Speed Up MongoDB Queries with Redis® on Golang

Published on May 25, 2022•Updated on June 22, 2024

Introduction

A cache utilizes the RAM of your computer to store and serve frequently accessed data from backend systems like databases. As a result, a caching layer improves performance, minimizes costs, and reduces the load on a database.

The Redis® in-memory database server is a suitable caching application because it requires minimal setup and scales well in a production environment. When you implement a cache, your application first queries Redis® to get data. If the cache is empty, the application fetches data from the database. Then, it creates a cache for future queries. This in-memory caching solution is several times faster when compared to rotating or modern Solid State Disks (SSD).

This guide walks you through the process of implementing a caching layer with Redis® to speed up MongoDB queries using the Go programming language.

Prerequisites

Before you begin:

1. Set Up a Sample Database

This guide uses a sample MongoDB database to store data permanently on a disk. Follow the steps below to create the database.

Log in to your MongoDB server. Replace mongo_db_admin with the correct user account.
```
 $ mongosh -u mongo_db_admin -p --authenticationDatabase admin
```
Enter your password to proceed.
Run the statement below to create an e-commerce database.
```
 test> use e-commerce
```
Output.
```
 switched to db e-commerce
```

Insert five documents into a new products collection.

 e-commerce> db.products.insertMany([
             {"product_id" : 1,
              "product_name" : "INSTANT WATER HEATER",
              "retail_price" : 45.55   
             },
             {"product_id" : 2,
              "product_name" : "DOUBLE SOCKET WITH PATTRESS",
              "retail_price" : 6.65    
             },
             {"product_id" : 3,
              "product_name" : "80MM USB PRINTER",
              "retail_price" : 125.95
             },
             {"product_id" : 4,
              "product_name" : "FITNESS SMARTWATCH",
              "retail_price" : 39.85
             },
             {"product_id" : 5,
              "product_name" : "3.1A FAST CHARGER",
              "retail_price" : 23.90
             }      
            ]);

Output.

 {
   acknowledged: true,
   insertedIds: {
     '0': ObjectId("625d2694dd1d8bd34f469875"),
     '1': ObjectId("625d2694dd1d8bd34f469876"),
     '2': ObjectId("625d2694dd1d8bd34f469877"),
     '3': ObjectId("625d2694dd1d8bd34f469878"),
     '4': ObjectId("625d2694dd1d8bd34f469879")
  }
 }

Querying the products collection to verify the documents.

 e-commerce> db.products.find()

Output.

 [
   {
     _id: ObjectId("625d2694dd1d8bd34f469875"),
     product_id: 1,
     product_name: 'INSTANT WATER HEATER',
     retail_price: 45.55
   },
   {
     _id: ObjectId("625d2694dd1d8bd34f469876"),
     product_id: 2,
     product_name: 'DOUBLE SOCKET WITH PATTRESS',
     retail_price: 6.65
   },
   {
     _id: ObjectId("625d2694dd1d8bd34f469877"),
     product_id: 3,
     product_name: '80MM USB PRINTER',
     retail_price: 125.95
   },
   {
     _id: ObjectId("625d2694dd1d8bd34f469878"),
     product_id: 4,
     product_name: 'FITNESS SMARTWATCH',
     retail_price: 39.85
   },
   {
     _id: ObjectId("625d2694dd1d8bd34f469879"),
     product_id: 5,
     product_name: '3.1A FAST CHARGER',
     retail_price: 23.9
   }
 ]

Log out from the MongoDB database.
```
 e-commerce> quit
```

2. Create the `main.go` File

The main.go file is the entry point in this sample application. This file contains the main(...){...} function that fires when your application starts. Execute the steps below to set up the file.

Create a new directory for your project. This approach keeps your project organized and avoids conflicting your source code with the rest of the Linux files.
```
 $ mkdir project
```
Switch to the new project directory.
```
 $ cd project 
```
Open a new main.go file in an editor.
```
 $ nano main.go
```

Enter the following information into the file. Replace mongo_db_admin and EXAMPLE_PASSWORD with the correct account details for the MongoDB server.

 package main

 import (
     "context"
     "encoding/json"
     "fmt"
     "net/http"
 )

 const (  
     MONGO_DB_USER = "mongo_db_admin"
     MONGO_DB_PASS = "EXAMPLE_PASSWORD"
     MONGO_DB_NAME = "e-commerce"    
 )

 func main() {
      http.HandleFunc("/products", requestHandler)
      http.ListenAndServe(":8080", nil)
 }

 func requestHandler(w http.ResponseWriter, req *http.Request) {    

     w.Header().Set("Content-Type", "application/json")

     var respMessage map[string]interface{}
     var respErr error

     ctx := context.Background()

     isCached, productsCache, err := getFromCache(ctx)

     if err != nil {  

         respErr = err

     } else {

         if isCached == true {

             respMessage = productsCache
             respMessage["_source"] = "Redis® Cache"

         } else {

             respMessage, err = getFromDb(ctx);

             if err != nil {       
                 respErr = err
             }   

             err = addToCache(ctx, respMessage)            

             if err != nil {       
                 respErr = err
             }

             respMessage["_source"] = "MongoDB database"
         }
     }

     if respErr != nil  {

         fmt.Fprintf(w, respErr.Error()) 

     } else {

         enc := json.NewEncoder(w)
         enc.SetIndent("", "  ")

         if err := enc.Encode(respMessage); err != nil {        
             fmt.Fprintf(w, err.Error()) 
         } 

     }    
 }

Save and close the file.

In the above file, you're creating an HTTP server that listens for incoming connections on port 8080.

In the main(...){...} function, you're redirecting incoming requests to the requestHandler(...){...} function. Under this function, you're using the isCached, productsCache, err := getFromCache(ctx) statement to retrieve products from a Redis® Cache. In case the cache is empty, you're fetching the data from the MongoDB database using the statement respMessage, err = getFromDb(ctx);.

Then, you're outputting the products' data in a JSON format. You're also appending a _source attribute to mark the data source. That is, either from the cache or the database.

3. Create the `get_from_db.go` File

The Go programming language provides several packages for communicating with a MongoDB database. Execute the steps below to use the packages to connect to your database and retrieve documents from the products collection.

Open a new get_from_db.go file in an editor.
```
 $ nano get_from_db.go
```

Enter the information below into the file.

 package main

 import (
     "context"
     "go.mongodb.org/mongo-driver/bson"
     "go.mongodb.org/mongo-driver/mongo"
     "go.mongodb.org/mongo-driver/mongo/options"
 )

 func getFromDb(ctx context.Context) (map[string]interface{}, error) {

     client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://" + MONGO_DB_USER + ":" + MONGO_DB_PASS + "@localhost:27017"))

     if err != nil { 
         return nil, err
     } 

     collection := client.Database(MONGO_DB_NAME).Collection("products")  

     cur, err := collection.Find(ctx, bson.D{})

     if err != nil { 
         return nil, err
     }

     defer cur.Close(ctx) 

     var records []bson.M           

     for cur.Next(ctx) {

         var record bson.M

         if err = cur.Decode(&record); err != nil {
             return nil, err
         }

         records = append(records, record)
     }

     res := map[string]interface{}{}

     res = map[string]interface{}{
               "data" : records,   
           }             

     return res, nil        
 }

Save and close the file.

In the get_from_db.go file above, you're using the database constants declared in the main.go file to communicate to the MongoDB database. Then, you're assigning the documents to a bson.M slice and returning the documents as a map of [string]interface{}.

4. Create the `add_to_cache.go` File

This step focuses on creating a function for populating a cache in the Redis® server after retrieving the products from the MongoDB database.

Open a new add_to_cache.go file in a text editor.
```
 $ nano add_to_cache.go
```

Enter the information below into the file.

 package main

 import (
     "context"
     "encoding/json"
     "time"
     "github.com/go-redis/redis"
 )

 func addToCache(ctx context.Context, data map[string]interface{}) error {

     redisClient := redis.NewClient(&redis.Options{
         Addr: "localhost:6379",
         Password: "",
         DB: 0,
     }) 

     jsonString, err := json.Marshal(data)

     if err != nil {
         return err
     }

     err = redisClient.Set(ctx, "products_cache", jsonString, 30 * time.Second).Err()

     if err != nil {
         return nil
     }  

     return nil
 }

Save and close the file.

In the above file, you're using the addToCache(...){...} function to connect to the Redis® server. You're then converting data that you retrieved from the MongoDB database to a JSON string using the statement jsonString, err := json.Marshal(data).

You're then using the err = redisClient.Set(ctx, "products_cache", jsonString, 30 * time.Second).Err() statement to create a key into the Redis® server with the JSON string as the value. You're using the '30 * time.Secondstatement to set the cache expiration period to30` seconds. Consider using a different timing in a production environment depending on your use case. Opt for a longer expiry period for data that rarely changes, like payment methods, product category names, and company address information.

5. Create the `get_from_cache.go` File

The main.go file that you created earlier first checks if data is available from the Redis® cache by calling a getFromCache(...){...} function. This function returns a boolean value (either true or false) depending on the availability of cached data. If the cache is not empty, the function returns the data as a map. Follow the steps below to create this logic.

Open a new get_from_cache.go file in an editor.
```
 $ nano get_from_cache.go
```

Enter the information below into the file.

 package main

 import (
     "context"
     "encoding/json"
     "github.com/go-redis/redis"
 )

 func getFromCache(ctx context.Context) (bool, map[string]interface{}, error) {

     redisClient := redis.NewClient(&redis.Options{
         Addr: "localhost:6379",
         Password: "",
         DB: 0,
     })

     productsCache, err := redisClient.Get(ctx, "products_cache").Bytes()

     if err != nil { 
         return false, nil, nil
     }

     res := map[string]interface{}{}

     err = json.Unmarshal(productsCache, &res)

     if err != nil {       
         return false, nil, nil
     }   

     return true, res, nil
 }

Save and close the file.

In the above file, you're connecting to the Redis® server and retrieving the cached data. Then, you're using the err = json.Unmarshal(productsCache, &res) statement to convert and return the cached data as a map to the calling function.

You've now created all the functions required by the application.

6. Test the Application

Your application is now ready for testing. After completing the coding, your project folder should contain the following files.

    --project
      -- main.go
      -- get_from_db.go
      -- add_to_cache.go  
      -- get_from_cache.go

Download all the packages used in the project.

    $ go get github.com/go-redis/redis
    $ go get go.mongodb.org/mongo-driver/mongo

Run the application. The command below has a blocking function. Therefore, don't enter other commands in your active terminal window. Your application now listens for incoming connections on port 8080.

    $ go run ./

Establish another SSH connection to your server. Then, run the following curl command.

    $ curl localhost:8080/products

You should get the following output. The application now lists MongoDB database as the value of the _source attribute. This value shows that the application fetches data from the database during the first request.

    {
      "_source": "MongoDB database",
      "data": [
        {
          "_id": "625d2694dd1d8bd34f469875",
          "product_id": 1,
          "product_name": "INSTANT WATER HEATER",
          "retail_price": 45.55
        },
        {
          "_id": "625d2694dd1d8bd34f469876",
          "product_id": 2,
          "product_name": "DOUBLE SOCKET WITH PATTRESS",
          "retail_price": 6.65
        },
        {
          "_id": "625d2694dd1d8bd34f469877",
          "product_id": 3,
          "product_name": "80MM USB PRINTER",
          "retail_price": 125.95
        },
        {
          "_id": "625d2694dd1d8bd34f469878",
          "product_id": 4,
          "product_name": "FITNESS SMARTWATCH",
          "retail_price": 39.85
        },
        {
          "_id": "625d2694dd1d8bd34f469879",
          "product_id": 5,
          "product_name": "3.1A FAST CHARGER",
          "retail_price": 23.9
        }
      ]
    }

Run the curl command again before the cache expires.

    $ curl localhost:8080/products

This time around, the value of the _source attribute changes to Redis® Cache. This value indicates the script is fetching data from the Redis® server.

      "_source": "Redis® Cache",
      "data": [
        {
          "_id": "625d2694dd1d8bd34f469875",
          "product_id": 1,
          "product_name": "INSTANT WATER HEATER",
          "retail_price": 45.55
        },
        {
          "_id": "625d2694dd1d8bd34f469876",
          "product_id": 2,
          "product_name": "DOUBLE SOCKET WITH PATTRESS",
          "retail_price": 6.65
        },
        {
          "_id": "625d2694dd1d8bd34f469877",
          "product_id": 3,
          "product_name": "80MM USB PRINTER",
          "retail_price": 125.95
        },
        {
          "_id": "625d2694dd1d8bd34f469878",
          "product_id": 4,
          "product_name": "FITNESS SMARTWATCH",
          "retail_price": 39.85
        },
        {
          "_id": "625d2694dd1d8bd34f469879",
          "product_id": 5,
          "product_name": "3.1A FAST CHARGER",
          "retail_price": 23.9
        }
      ]
    }

Your application is now working as expected. If you hook up this script to an online e-commerce application with thousands of visitors, the response time is faster when compared to non-cache setups. A cache improves user experience and allows your customers to navigate your catalog with minimum delay.

The caching layer also reduces the number of round-trips your application makes to the database. In addition, this approach reduces bandwidth usage costs for managed MongoDB databases.

While a cache is a perfect mechanism for speeding up your application, be careful with your cache invalidation policy. Remember to flush your cache when data in the backend system changes to avoid serving stale data to the frontend users. For instance, if you add, delete, or update the MongoDB products collection, ensure your Golang script clears the cache in the Redis® server.

Besides the products, consider caching user login tokens, customer profile information, lookup table values such as payment methods, and shopping cart items. The cached assets improve the entire performance of your e-commerce application.

Conclusion

In this guide, you've implemented a caching mechanism for your MongoDB database using the Redis® server. Use the above caching technique to create a high-performance Golang application for your next MongoDB project.

Speed Up MongoDB Queries with Redis® on Golang

Introduction

Prerequisites

1. Set Up a Sample Database

2. Create the `main.go` File

3. Create the `get_from_db.go` File

4. Create the `add_to_cache.go` File

5. Create the `get_from_cache.go` File

6. Test the Application

Conclusion

Products

Features

Marketplace

Resources

Company

Speed Up MongoDB Queries with Redis® on Golang

Introduction

Prerequisites

1. Set Up a Sample Database

2. Create the main.go File

3. Create the get_from_db.go File

4. Create the add_to_cache.go File

5. Create the get_from_cache.go File

6. Test the Application

Conclusion

2. Create the `main.go` File

3. Create the `get_from_db.go` File

4. Create the `add_to_cache.go` File

5. Create the `get_from_cache.go` File