How to Use a Redis® Cache with PostgreSQL in Golang

Introduction

A Redis® in-memory database cache allows you to access frequently used data from your server's RAM. This minimizes unnecessarily load and roundtrips to your PostgreSQL database server.

In circumstances where thousands or millions of users access your application, a cache can significantly improve user experience and reduce database costs if you're using a managed throughput-billed database.

To avoid serving stale data from the cache, ensure you're only caching data that is unlikely to change in a certain set duration. For instance, if your organization relies on different payment methods such as cash, check, and PayPal, they may never change for several months. Hence, caching the payment methods when loading them in a select box can improve your application responsiveness.

Other objects that you can cache for durations ranging from one second to several minutes include products in a shopping cart, bank account names in an accounting app, geographical names in drop-down lists, and more.

In this tutorial, you'll implement a caching strategy for your PostgreSQL database with Golang and Redis® on your Linux server.

Prerequisites

To complete this tutorial, you require:

• An Linux server.
• A non-root user with sudo privileges.
• A PostgreSQL database.
• A Redis® server.
• A Golang package.

1. Create a PostgreSQL Database

In this sample application, you'll set up a database for storing products permanently on a disk. Then, when end-users access your application for the first time in an URL that you'll define later, you'll retrieve products from the PostgreSQL database and immediately cache the data to the Redis® server. You'll then serve any subsequent products requests from the Redis® cache to improve performance.

Connect to your server through SSH and complete the following steps:

1. Log in to the PostgreSQL server as postgres.

    $ sudo -u postgres psql

2. Then, enter your postgres user account password and press Enter to proceed. Next, execute the following SQL command to create a sample_company database.

    postgres=# CREATE DATABASE sample_company;

3. Switch to the new sample_company database.

    postgres=# \c sample_company;

4. Create a products table. This table stores several products you intend to sell in your sample company.

    sample_company=# CREATE TABLE products (
                         product_id SERIAL PRIMARY KEY,
                         product_name VARCHAR (50),
                         retail_price  NUMERIC(5,2)  
                     );

5. Populate the products table with three entries. You might have hundreds or even thousands of records in a production environment.

    sample_company=# INSERT INTO products(product_name, retail_price) VALUES ('LEATHER BELT',  '12.25');
                     INSERT INTO products(product_name, retail_price) VALUES ('WINTER JACKET',  '89.65');
                     INSERT INTO products(product_name, retail_price) VALUES ('COTTON SOCKS',  '2.85');

6. Query the products table to ensure the items are in place.

    sample_company=# SELECT 
                         product_id,
                         product_name,
                         retail_price
                     FROM products;

7. Confirm the products list below.

     product_id | product_name  | retail_price
    ------------+---------------+--------------
              1 | LEATHER BELT  |        12.25
              2 | WINTER JACKET |        89.65
              3 | COTTON SOCKS  |         2.85
    (3 rows)

8. Log out from the PostgreSQL server.

    sample_company=# \q

2. Create a main.go Script

In this step, you'll create a main.go file that allows your application to run a web server that listens for incoming requests on port 8080.

1. To separate your project source code from the rest of the Linux files, create a new directory for your project.

    $ mkdir project

2. Next, switch to the directory.

    $ cd project

3. Use the nano text editor to create a main.go file under the project directory.

    $ nano main.go

4. Next, enter the following information into the main.go file.

    package main
   
    import (
    "net/http"
    "encoding/json"
    "fmt"
    )    
   
    func main() {
        http.HandleFunc("/products", httpHandler)        
        http.ListenAndServe(":8080", nil)
    }
   
    func httpHandler(w http.ResponseWriter, req *http.Request) { 
   
        response, err := getProducts()
   
        if err != nil {
   
             fmt.Fprintf(w, err.Error() + "\r\n") 
   
        } else {
   
            enc := json.NewEncoder(w)
            enc.SetIndent("", "  ")
   
            if err := enc.Encode(response); err != nil {
                fmt.Println(err.Error())
            }  
   
        } 
    }

5. Save and close the file when you're through with editing.

6. In the above file, you've imported the net/http package to implement HTTP functions in your application. Next, you've included the encoding/json package to format your application's response to JSON. You're then implementing the fmt package to display output in your application.

7. The main() function executes when you run the application and creates a HandleFunc to handle incoming requests directed to the /products URL. You're using the httpHandler function to retrieve products from a getProducts function, which you'll later create in a new products.go file. Finally, you're using the json functions to encode and output the response data.

2. Create a products.go Script

In this step, you'll create a products.go script that connects to the PostgreSQL database and the Redis® server. The main logic in this script is to only retrieve products from the PostgreSQL database server if they're not available in the Redis® cache.

1. Use nano to create the products.go file.

    $ nano products.go

2. Enter the following information into the products.go file.

    package main
   
    import (
    "context"
    "database/sql"
    "encoding/json"
    "fmt"
    "time"
   
    "github.com/go-redis/redis"
        _ "github.com/lib/pq" 
    )
   
    type Products struct {
        ProductId   int     `json:"product_id"`
        ProductName string  `json:"product_name"`
        RetailPrice float64 `json:"retail_price"`
    }
   
    type JsonResponse struct {
        Data   []Products `json:"data"`  
        Source string     `json:"source"`     
    }
   
    func getProducts() (*JsonResponse, error) {          
   
        redisClient := redis.NewClient(&redis.Options{
            Addr: "localhost:6379",
            Password: "",
            DB: 0,
        })
   
        ctx := context.Background()
   
        cachedProducts, err := redisClient.Get(ctx, "products").Bytes()
   
        response := JsonResponse{}
   
        if err != nil { 
   
            dbProducts, err := fetchFromDb()
   
            if err != nil { 
                return nil, err
            }
   
            cachedProducts, err = json.Marshal(dbProducts)
   
            if err != nil {
                return nil, err
            }
   
            err = redisClient.Set(ctx, "products", cachedProducts, 10 * time.Second).Err()
   
            if err != nil {
                return nil, err
            }
   
            response = JsonResponse{Data: dbProducts, Source: "PostgreSQL", }
   
            return &response, err
        }
   
        products := []Products{}
   
        err = json.Unmarshal(cachedProducts, &products)
   
        if err != nil {
            return nil, err
        }
   
        response = JsonResponse{Data: products, Source: "Redis Cache", }
   
        return &response, nil 
    }
   
    func fetchFromDb() ([]Products, error){
   
        dbUser     := "postgres"
        dbPassword := "EXAMPLE_PASSWORD"
        dbName     := "sample_company"
   
        conString := fmt.Sprintf("user=%s password=%s dbname=%s sslmode=disable", dbUser, dbPassword, dbName)
   
        db, err := sql.Open("postgres", conString)
   
        if err != nil { 
            return nil, err
        }
   
        queryString := `select
                         product_id,
                         product_name,
                         retail_price
                     from products`
   
        rows, err := db.Query(queryString)
   
        if err != nil { 
            return nil, err
        }
   
        var records []Products
   
        for rows.Next() {
   
            var p Products
   
            err = rows.Scan(&p.ProductId, &p.ProductName, &p.RetailPrice)
   
            records = append(records, p)
   
            if err != nil { 
                return nil, err
            }       
   
        }    
   
        return records, nil
    }

3. Save and close the products.go file when you're through with editing.

4. You've imported the context package to implement request-scoped values with Redis®. The database/sql package provides different methods for working with SQL databases, while the github.com/go-redis/redis allows you to connect to your Redis® server. The github.com/lib/pq provides a driver for your PostgreSQL database and the time library allows you to measure the time duration for evicting your cache.

5. In the above file, you've created a getProducts() function that connects to the Redis® server to check if a key named products exists using the statement cachedProducts, err := redisClient.Get(ctx, "products").Bytes(). In case the key exists, you're retrieving a string formatted response(cachedProducts) from the Redis® server that you're later converting to a Products struct using the products := []Products{} and err = json.Unmarshal(cachedProducts, &products) statements.

6. If the products key doesn't exist in the Redis® server, you're using the statement dbProducts, err := fetchFromDb() to fetch the products from the PostgreSQL database. That is, from disk.

7. When retrieving data from the PostgreSQL database, you're scanning the products' data into the Products struct and returning a slice of type []Products.

8. The getProducts() function returns a response of type JsonResponse with a data object and source string back to the main(...) function that you created in the main.go file. The source tag allows you to distinguish where your application retrieves the data. That is, either from the Redis® Cache or from the PostgreSQL database.

9. You've set a cache eviction policy of 10 seconds using the statement ...10 * time.Second... in this sample application. Depending on your business logic and how often you intend to update your product catalog, you may increase this value. You may code a function to delete the cache every time you update, delete, or insert a record into the products table in a more advanced application.

3. Test the Application

Your Redis® cache application is now ready for testing. In this step, you'll run the application twice to check whether the caching logic is working as expected.

1. Before doing this, use the go get command to download the packages you've used in your source code.

    $ go get github.com/go-redis/redis
    $ go get github.com/lib/pq

2. Next, execute the following go run command to start the application. The following command allows your application to listen on port 8080, which has a blocking function. Don't type any other command on this SSH session.

    $ go run ./

3. Next, open a new terminal window and execute the following curl command to test the application.

   $ curl localhost:8080/products

4. You should get the following output. As you can see from the source tag ("source": "PostgreSQL") towards the end of the JSON output, you've retrieved the data from the PostgreSQL database.

    {
      "data": [
        {
          "product_id": 1,
          "product_name": "LEATHER BELT",
          "retail_price": 12.25
        },
        {
          "product_id": 2,
          "product_name": "WINTER JACKET",
          "retail_price": 89.65
        },
        {
          "product_id": 3,
          "product_name": "COTTON SOCKS",
          "retail_price": 2.85
        }
      ],
      "source": "PostgreSQL"
    }

5. Next, run the curl command again before the cache expires. Remember, you've set the cache expiration policy to 10 seconds.

    $ curl localhost:8080/products

6. Your application fetches data from the Redis® cache this time around. Confirm this by examining the source tag ("source": "Redis Cache") towards the end of the JSON output.

    {
      "data": [
        {
          "product_id": 1,
          "product_name": "LEATHER BELT",
          "retail_price": 12.25
        },
        {
          "product_id": 2,
          "product_name": "WINTER JACKET",
          "retail_price": 89.65
        },
        {
          "product_id": 3,
          "product_name": "COTTON SOCKS",
          "retail_price": 2.85
        }
      ],
      "source": "Redis Cache"
    }

7. The above tests confirm that your Redis® cache is working as expected.

Conclusion

In this guide, you've implemented a Redis® cache for the PostgreSQL database server with Golang on your Linux server. You may extend the source code in this guide to cache other objects in your application. Remember to use the caching logic selectively to avoid serving stale data. As always, implement a fail-proof eviction policy to remove cached items when your backend data changes.

Visit the following resources to read more Golang tutorials:

• Designing a Modern API with Golang and MySQL 8 on Linux
• Secure a Golang Web Server with a Self-signed or Let's Encrypt SSL Certificate
• How to Create a Central Input Data Validator in Golang