A cache utilizes the RAM of your computer to store and serve frequently accessed data from backend systems like databases. As a result, a caching layer improves performance, minimizes costs, and reduces the load on a database.
The Redis in-memory database server is a suitable caching application because it requires minimal setup and scales well in a production environment. When you implement a cache, your application first queries Redis to get data. If the cache is empty, the application fetches data from the database. Then, it creates a cache for future queries. This in-memory caching solution is several times faster when compared to rotating or modern Solid State Disks (SSD).
This guide walks you through the process of implementing a caching layer with Redis to speed up MongoDB queries using the Go programming language.
Before you begin:
This guide uses a sample MongoDB database to store data permanently on a disk. Follow the steps below to create the database.
Log in to your MongoDB server. Replace mongo_db_admin with the correct user account.
$ mongosh -u mongo_db_admin -p --authenticationDatabase admin
Enter your password to proceed.
Run the statement below to create an e-commerce database.
test> use e-commerce
Output.
switched to db e-commerce
Insert five documents into a new products collection.
e-commerce> db.products.insertMany([
{"product_id" : 1,
"product_name" : "INSTANT WATER HEATER",
"retail_price" : 45.55
},
{"product_id" : 2,
"product_name" : "DOUBLE SOCKET WITH PATTRESS",
"retail_price" : 6.65
},
{"product_id" : 3,
"product_name" : "80MM USB PRINTER",
"retail_price" : 125.95
},
{"product_id" : 4,
"product_name" : "FITNESS SMARTWATCH",
"retail_price" : 39.85
},
{"product_id" : 5,
"product_name" : "3.1A FAST CHARGER",
"retail_price" : 23.90
}
]);
Output.
{
acknowledged: true,
insertedIds: {
'0': ObjectId("625d2694dd1d8bd34f469875"),
'1': ObjectId("625d2694dd1d8bd34f469876"),
'2': ObjectId("625d2694dd1d8bd34f469877"),
'3': ObjectId("625d2694dd1d8bd34f469878"),
'4': ObjectId("625d2694dd1d8bd34f469879")
}
}
Querying the products collection to verify the documents.
e-commerce> db.products.find()
Output.
[
{
_id: ObjectId("625d2694dd1d8bd34f469875"),
product_id: 1,
product_name: 'INSTANT WATER HEATER',
retail_price: 45.55
},
{
_id: ObjectId("625d2694dd1d8bd34f469876"),
product_id: 2,
product_name: 'DOUBLE SOCKET WITH PATTRESS',
retail_price: 6.65
},
{
_id: ObjectId("625d2694dd1d8bd34f469877"),
product_id: 3,
product_name: '80MM USB PRINTER',
retail_price: 125.95
},
{
_id: ObjectId("625d2694dd1d8bd34f469878"),
product_id: 4,
product_name: 'FITNESS SMARTWATCH',
retail_price: 39.85
},
{
_id: ObjectId("625d2694dd1d8bd34f469879"),
product_id: 5,
product_name: '3.1A FAST CHARGER',
retail_price: 23.9
}
]
Log out from the MongoDB database.
e-commerce> quit
The main.go file is the entry point in this sample application. This file contains the main(...){...} function that fires when your application starts. Execute the steps below to set up the file.
Create a new directory for your project. This approach keeps your project organized and avoids conflicting your source code with the rest of the Linux files.
$ mkdir project
Switch to the new project directory.
$ cd project
Open a new main.go file in an editor.
$ nano main.go
Enter the following information into the file. Replace mongo_db_admin and EXAMPLE_PASSWORD with the correct account details for the MongoDB server.
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
)
const (
MONGO_DB_USER = "mongo_db_admin"
MONGO_DB_PASS = "EXAMPLE_PASSWORD"
MONGO_DB_NAME = "e-commerce"
)
func main() {
http.HandleFunc("/products", requestHandler)
http.ListenAndServe(":8080", nil)
}
func requestHandler(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "application/json")
var respMessage map[string]interface{}
var respErr error
ctx := context.Background()
isCached, productsCache, err := getFromCache(ctx)
if err != nil {
respErr = err
} else {
if isCached == true {
respMessage = productsCache
respMessage["_source"] = "Redis Cache"
} else {
respMessage, err = getFromDb(ctx);
if err != nil {
respErr = err
}
err = addToCache(ctx, respMessage)
if err != nil {
respErr = err
}
respMessage["_source"] = "MongoDB database"
}
}
if respErr != nil {
fmt.Fprintf(w, respErr.Error())
} else {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if err := enc.Encode(respMessage); err != nil {
fmt.Fprintf(w, err.Error())
}
}
}
Save and close the file.
In the above file, you're creating an HTTP server that listens for incoming connections on port 8080.
In the main(...){...} function, you're redirecting incoming requests to the requestHandler(...){...} function. Under this function, you're using the isCached, productsCache, err := getFromCache(ctx) statement to retrieve products from a Redis Cache. In case the cache is empty, you're fetching the data from the MongoDB database using the statement respMessage, err = getFromDb(ctx);.
Then, you're outputting the products' data in a JSON format. You're also appending a _source attribute to mark the data source. That is, either from the cache or the database.
The Go programming language provides several packages for communicating with a MongoDB database. Execute the steps below to use the packages to connect to your database and retrieve documents from the products collection.
Open a new get_from_db.go file in an editor.
$ nano get_from_db.go
Enter the information below into the file.
package main
import (
"context"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func getFromDb(ctx context.Context) (map[string]interface{}, error) {
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://" + MONGO_DB_USER + ":" + MONGO_DB_PASS + "@localhost:27017"))
if err != nil {
return nil, err
}
collection := client.Database(MONGO_DB_NAME).Collection("products")
cur, err := collection.Find(ctx, bson.D{})
if err != nil {
return nil, err
}
defer cur.Close(ctx)
var records []bson.M
for cur.Next(ctx) {
var record bson.M
if err = cur.Decode(&record); err != nil {
return nil, err
}
records = append(records, record)
}
res := map[string]interface{}{}
res = map[string]interface{}{
"data" : records,
}
return res, nil
}
Save and close the file.
In the get_from_db.go file above, you're using the database constants declared in the main.go file to communicate to the MongoDB database. Then, you're assigning the documents to a bson.M slice and returning the documents as a map of [string]interface{}.
This step focuses on creating a function for populating a cache in the Redis server after retrieving the products from the MongoDB database.
Open a new add_to_cache.go file in a text editor.
$ nano add_to_cache.go
Enter the information below into the file.
package main
import (
"context"
"encoding/json"
"time"
"github.com/go-redis/redis"
)
func addToCache(ctx context.Context, data map[string]interface{}) error {
redisClient := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
jsonString, err := json.Marshal(data)
if err != nil {
return err
}
err = redisClient.Set(ctx, "products_cache", jsonString, 30 * time.Second).Err()
if err != nil {
return nil
}
return nil
}
Save and close the file.
In the above file, you're using the addToCache(...){...} function to connect to the Redis server. You're then converting data that you retrieved from the MongoDB database to a JSON string using the statement jsonString, err := json.Marshal(data).
You're then using the err = redisClient.Set(ctx, "products_cache", jsonString, 30 * time.Second).Err() statement to create a key into the Redis server with the JSON string as the value. You're using the '30 * time.Secondstatement to set the cache expiration period to30` seconds. Consider using a different timing in a production environment depending on your use case. Opt for a longer expiry period for data that rarely changes, like payment methods, product category names, and company address information.
The main.go file that you created earlier first checks if data is available from the Redis cache by calling a getFromCache(...){...} function. This function returns a boolean value (either true or false) depending on the availability of cached data. If the cache is not empty, the function returns the data as a map. Follow the steps below to create this logic.
Open a new get_from_cache.go file in an editor.
$ nano get_from_cache.go
Enter the information below into the file.
package main
import (
"context"
"encoding/json"
"github.com/go-redis/redis"
)
func getFromCache(ctx context.Context) (bool, map[string]interface{}, error) {
redisClient := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
productsCache, err := redisClient.Get(ctx, "products_cache").Bytes()
if err != nil {
return false, nil, nil
}
res := map[string]interface{}{}
err = json.Unmarshal(productsCache, &res)
if err != nil {
return false, nil, nil
}
return true, res, nil
}
Save and close the file.
In the above file, you're connecting to the Redis server and retrieving the cached data. Then, you're using the err = json.Unmarshal(productsCache, &res) statement to convert and return the cached data as a map to the calling function.
You've now created all the functions required by the application.
Your application is now ready for testing. After completing the coding, your project folder should contain the following files.
--project
-- main.go
-- get_from_db.go
-- add_to_cache.go
-- get_from_cache.go
Download all the packages used in the project.
$ go get github.com/go-redis/redis
$ go get go.mongodb.org/mongo-driver/mongo
Run the application. The command below has a blocking function. Therefore, don't enter other commands in your active terminal window. Your application now listens for incoming connections on port 8080.
$ go run ./
Establish another SSH connection to your server. Then, run the following curl command.
$ curl localhost:8080/products
You should get the following output. The application now lists MongoDB database as the value of the _source attribute. This value shows that the application fetches data from the database during the first request.
{
"_source": "MongoDB database",
"data": [
{
"_id": "625d2694dd1d8bd34f469875",
"product_id": 1,
"product_name": "INSTANT WATER HEATER",
"retail_price": 45.55
},
{
"_id": "625d2694dd1d8bd34f469876",
"product_id": 2,
"product_name": "DOUBLE SOCKET WITH PATTRESS",
"retail_price": 6.65
},
{
"_id": "625d2694dd1d8bd34f469877",
"product_id": 3,
"product_name": "80MM USB PRINTER",
"retail_price": 125.95
},
{
"_id": "625d2694dd1d8bd34f469878",
"product_id": 4,
"product_name": "FITNESS SMARTWATCH",
"retail_price": 39.85
},
{
"_id": "625d2694dd1d8bd34f469879",
"product_id": 5,
"product_name": "3.1A FAST CHARGER",
"retail_price": 23.9
}
]
}
Run the curl command again before the cache expires.
$ curl localhost:8080/products
This time around, the value of the _source attribute changes to Redis Cache. This value indicates the script is fetching data from the Redis server.
"_source": "Redis Cache",
"data": [
{
"_id": "625d2694dd1d8bd34f469875",
"product_id": 1,
"product_name": "INSTANT WATER HEATER",
"retail_price": 45.55
},
{
"_id": "625d2694dd1d8bd34f469876",
"product_id": 2,
"product_name": "DOUBLE SOCKET WITH PATTRESS",
"retail_price": 6.65
},
{
"_id": "625d2694dd1d8bd34f469877",
"product_id": 3,
"product_name": "80MM USB PRINTER",
"retail_price": 125.95
},
{
"_id": "625d2694dd1d8bd34f469878",
"product_id": 4,
"product_name": "FITNESS SMARTWATCH",
"retail_price": 39.85
},
{
"_id": "625d2694dd1d8bd34f469879",
"product_id": 5,
"product_name": "3.1A FAST CHARGER",
"retail_price": 23.9
}
]
}
Your application is now working as expected. If you hook up this script to an online e-commerce application with thousands of visitors, the response time is faster when compared to non-cache setups. A cache improves user experience and allows your customers to navigate your catalog with minimum delay.
The caching layer also reduces the number of round-trips your application makes to the database. In addition, this approach reduces bandwidth usage costs for managed MongoDB databases.
While a cache is a perfect mechanism for speeding up your application, be careful with your cache invalidation policy. Remember to flush your cache when data in the backend system changes to avoid serving stale data to the frontend users. For instance, if you add, delete, or update the MongoDB products collection, ensure your Golang script clears the cache in the Redis server.
Besides the products, consider caching user login tokens, customer profile information, lookup table values such as payment methods, and shopping cart items. The cached assets improve the entire performance of your e-commerce application.
In this guide, you've implemented a caching mechanism for your MongoDB database using the Redis server. Use the above caching technique to create a high-performance Golang application for your next MongoDB project.