Set Up High Availability Redis® using Sentinel in Ubuntu

Published on October 31, 2022•Updated on June 21, 2024

Introduction

Redis® is an open-source, in-memory data structure. Redis® stores data in memory and supports a mechanism to persist data to disk. Redis® is popular for caching, session management, real-time analytics, and pub/sub applications.

Redis® supports data replication using an asynchronous method between a master and multiple replicas. Redis® Sentinel is a solution to monitor, send notifications to system admins, and automatic failover if the master is not working as expected.

For a complex system, Redis® Cluster supports multiple masters and replicas.

This article shows how to set up a high-availability Redis® using Sentinel in Ubuntu.

At the end of this article, you'll know:

How to set up Redis® master-replica replication.
How to set up Redis® Sentinel.
How to test the failover of the high availability Redis® using Sentinel.
How to automate the setup using Ansible.

Set Up Master-Replica Configuration

This section shows how to set up Redis® master-replica replication. This setup contains a master server and two replica servers.

You can deploy three Ubuntu servers with 1GB of memory to follow this article. However, higher memory servers are better for production.

These servers should be independent of each other. For example, they can be different physical servers or virtual machines from different availability zones.

Create Three Vultr Server Instances

Navigate to your Customer Portal to deploy new servers.
Label the server as master, replica-1, and replica-2.
(Optional) Configure SSH Keys for better security. See SSH Keys in Customer Portal
(Optional) Enable Auto Backups

Here is an example result.

- master: 45.32.121.40
- replica-1: 139.180.217.55
- replica-2: 139.180.156.164

Install Redis®

Connect to each server over SSH.

Run the following commands to install Redis®.

 $ curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
 $ echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
 $ sudo apt-get update
 $ sudo apt-get install -y redis-server

Activate the redis-server service to start on boot.
```
 $ systemctl enable redis-server
```

Set Up the Master Node

This section shows how to set up the master node with password authentication.

Connect to your master server over SSH.
Open the configuration file at /etc/redis/redis.conf with your favorite editor.
Find the following configuration, uncomment and edit them. Change <YOUR_PASSWORD> to a secure password. The bind configuration allows the Redis® server to be accessible from other nodes.
```
 bind 0.0.0.0
 requirepass "<YOUR_PASSWORD>"
 masterauth "<YOUR_PASSWORD>"
```
Restart the redis-server service.
```
 $ sudo service redis-server restart
```

Set Up Replica Node

This section shows how to set up the replica nodes with password authentication.

Connect to each replica server over SSH.
Open the configuration file at /etc/redis/redis.conf with your favorite edit.
Find the following configuration, uncomment and edit them as follows. Change <YOUR_PASSWORD> to the same password on master node. Change <MASTER_NODE_IP> to the IP address of the master node.
```
 bind 0.0.0.0
 requirepass "<YOUR_PASSWORD>"
 masterauth "<YOUR_PASSWORD>"
 replicaof <MASTER_NODE_IP> 6379
```
Restart the redis-server service.
```
 $ sudo service redis-server restart
```

Set Up Firewall for Redis® Server

This section shows how to edit ufw firewall to allow connection to port 6379 so that Redis® Servers can talk with each other.

Connect to the master server and each replica server over SSH.

Run the following commands. Change <MASTER_NODE_IP> to the IP address of the master node. Change <REPLICA_1_NODE_IP> to the IP address of the replica-1 node. Change <REPLICA_2_NODE_IP> to the IP address of the replica-2 node.

 $ sudo ufw allow from <MASTER_NODE_IP> to any port 6379 comment 'MASTER'
 $ sudo ufw allow from <REPLICA_1_NODE_IP> to any port 6379 comment 'REPLICA-1'
 $ sudo ufw allow from <REPLICA_2_NODE_IP> to any port 6379 comment 'REPLICA-2'

Check the Master-Replica Setup

This section shows how to check if the master-replica setup is working as expected.

Connect to your master server.

Check the log of Redis® Sentinel.

 $ tail -f /var/log/redis/redis-server.log

Here is an example result:

 704:M 01 Oct 2022 09:20:58.524 * Loading RDB produced by version 7.0.5
 704:M 01 Oct 2022 09:20:58.524 * RDB age 15 seconds
 704:M 01 Oct 2022 09:20:58.524 * RDB memory usage when created 0.91 Mb
 704:M 01 Oct 2022 09:20:58.524 * Done loading RDB, keys loaded: 1, keys expired: 0.
 704:M 01 Oct 2022 09:20:58.525 * DB loaded from disk: 0.002 seconds
 704:M 01 Oct 2022 09:20:58.525 * Ready to accept connections
 704:M 01 Oct 2022 09:20:59.353 * Replica 139.180.217.55:6379 asks for synchronization
 704:M 01 Oct 2022 09:20:59.353 * Partial resynchronization request from 139.180.217.55:6379 accepted. Sending 0 bytes of backlog starting from offset 2873.
 704:M 01 Oct 2022 09:20:59.435 * Replica 139.180.156.164:6379 asks for synchronization
 704:M 01 Oct 2022 09:20:59.435 * Partial resynchronization request from 139.180.156.164:6379 accepted. Sending 0 bytes of backlog starting from offset 2873.

Run Redis® CLI to connect to the Redis® server.
```
 $ redis-cli
```
Run AUTH command to authenticate. Change <YOUR_PASSWORD> to the same password on master node.
```
 $ AUTH <YOUR_PASSWORD>
```
Run INFO command to check the Redis® replication information.
```
 $ INFO REPLICATION
```

Here is an example result. The output connected_slaves:2 shows that both replica nodes have successfully connected to the master node.

 # Replication
 role:master
 connected_slaves:2
 slave0:ip=139.180.217.55,port=6379,state=online,offset=14,lag=1
 slave1:ip=139.180.156.164,port=6379,state=online,offset=14,lag=1
 master_failover_state:no-failover
 master_replid:43855ff334c072ee5a3f4db39b1b27b31f55a3a4
 master_replid2:0000000000000000000000000000000000000000
 master_repl_offset:14
 second_repl_offset:-1
 repl_backlog_active:1
 repl_backlog_size:1048576
 repl_backlog_first_byte_offset:1
 repl_backlog_histlen:14

Run SET command to set a key hello to world on the master node.
```
 $ SET hello world
```
Connect to your Redis® replica server.
Run Redis® CLI to connect to the Redis® server.
```
$ redis-cli
```
Run AUTH command to authenticate. Change <YOUR_PASSWORD> to the same password on the master node.
```
$ AUTH <YOUR_PASSWORD>
```
Run GET command to get the value of key hello. The result is world shows that the replication from master to replica works successfully.
```
$ GET hello
```

Configure Redis® Sentinel

This section shows how to set up Redis® Sentinel to monitor the master-replica setup in the previous section.

Redis® Sentinel is a distributed system where there are multiple Sentinel processes running on different servers. This makes the system robust against failures. In a failure scenario, the majority of Sentinel processes monitor and vote to promote a replica to be the new master.

On all three servers, install Redis® Sentinel as follows:
```
 $ sudo apt install redis-sentinel
```
Activate the redis-sentinel service to start on boot.
```
 $ systemctl enable redis-sentinel
```
Open the configuration file at /etc/redis/sentinel.conf with your favorite editor.

Find the line sentinel monitor mymaster 127.0.0.1 6379 2, and edit them as follows. Change <YOUR_PASSWORD> to the same password on master node. Change <MASTER_NODE_IP> to the IP address of the master node.

 sentinel monitor mymaster <MASTER_NODE_IP> 6379 2
 sentinel auth-pass mymaster <YOUR_PASSWORD>
 sentinel down-after-milliseconds mymaster 5000
 sentinel failover-timeout mymaster 60000
 protected-mode no

Restart the Redis® Sentinel Service.
```
 $ sudo service redis-sentinel restart
```

Set Up Firewall for Sentinel

This section shows how to edit ufw firewall to allow connection to port 26379 so that Redis® Sentinel can talk with each other.

Connect to the master server and each replica server over SSH.

 $ sudo ufw allow from <MASTER_NODE_IP> to any port 26379 comment 'SENTINEL'
 $ sudo ufw allow from <REPLICA_1_NODE_IP> to any port 26379 comment 'SENTINEL'
 $ sudo ufw allow from <REPLICA_2_NODE_IP> to any port 26379 comment 'SENTINEL'

Check the Sentinel Setup

This section shows how to check if the Redis® Sentinel is working as expected.

Connect to your master server over SSH.

Check the log of Redis® Sentinel.

 $ tail -f /var/log/redis/redis-sentinel.log

Here is an example result:

 1049:X 01 Oct 2022 17:57:30.950 * Removing the pid file.
 1049:X 01 Oct 2022 17:57:30.950 # Sentinel is now ready to exit, bye bye...
 5020:X 01 Oct 2022 17:57:30.993 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
 5020:X 01 Oct 2022 17:57:30.993 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
 5020:X 01 Oct 2022 17:57:30.993 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=5020, just started
 5020:X 01 Oct 2022 17:57:30.993 # Configuration loaded
 5020:X 01 Oct 2022 17:57:30.994 * monotonic clock: POSIX clock_gettime
 5020:X 01 Oct 2022 17:57:30.994 * Running mode=sentinel, port=26379.
 5020:X 01 Oct 2022 17:57:30.995 # Sentinel ID is 8de12f567ebdfdfc8df229497e445667fd18f3e2
 5020:X 01 Oct 2022 17:57:30.995 # +monitor master mymaster 45.32.121.40 6379 quorum 2

Run Redis® CLI to connect to the Redis® Sentinel server.
```
 $ redis-cli -p 26379
```
Run INFO command to check the Redis® replication information.
```
 $ INFO SENTINEL
```

Here is an example result.

 # Sentinel
 sentinel_masters:1
 sentinel_tilt:0
 sentinel_tilt_since_seconds:-1
 sentinel_running_scripts:0
 sentinel_scripts_queue_length:0
 sentinel_simulate_failure_flags:0
 master0:name=mymaster,status=ok,address=45.32.121.40:6379,slaves=2,sentinels=3

Test the Failover

This section shows how to test the failover by stopping the Redis® master node manually.

Connect to your master server.

Stop the Redis® services on the master server.

 $ sudo service redis-sentinel stop
 $ sudo service redis-server stop

Open a new terminal and connect to a Redis® replica server.

Check the log of Redis® Sentinel.

 $ tail -f /var/log/redis/redis-sentinel.log

Here is an example result. The Sentinel starts a failover process and promotes a replica to be the new master.

 5222:X 01 Oct 2022 18:59:38.903 # +sdown sentinel e5b67f1e144dedf55cf71e60df52489debfdcb9d 45.32.121.40 26379 @ mymaster 45.32.121.40 6379
 5222:X 01 Oct 2022 18:59:39.836 # +sdown master mymaster 45.32.121.40 6379
 5222:X 01 Oct 2022 18:59:39.958 * Sentinel new configuration saved on disk
 5222:X 01 Oct 2022 18:59:39.958 # +new-epoch 3
 5222:X 01 Oct 2022 18:59:39.959 * Sentinel new configuration saved on disk
 5222:X 01 Oct 2022 18:59:39.960 # +vote-for-leader 852870198de5b3ced3abd82b8f85cd4866c7921c 3
 5222:X 01 Oct 2022 18:59:40.975 # +odown master mymaster 45.32.121.40 6379 #quorum 2/2
 5222:X 01 Oct 2022 18:59:40.976 # Next failover delay: I will not start a failover before Sat Oct  1 19:01:40 2022
 5222:X 01 Oct 2022 18:59:41.028 # +config-update-from sentinel 852870198de5b3ced3abd82b8f85cd4866c7921c 139.180.156.164 26379 @ mymaster 45.32.121.40 6379
 5222:X 01 Oct 2022 18:59:41.028 # +switch-master mymaster 45.32.121.40 6379 139.180.217.55 6379
 5222:X 01 Oct 2022 18:59:41.028 * +slave slave 139.180.156.164:6379 139.180.156.164 6379 @ mymaster 139.180.217.55 6379
 5222:X 01 Oct 2022 18:59:41.029 * +slave slave 45.32.121.40:6379 45.32.121.40 6379 @ mymaster 139.180.217.55 6379
 5222:X 01 Oct 2022 18:59:41.030 * Sentinel new configuration saved on disk
 5222:X 01 Oct 2022 18:59:46.128 # +sdown slave 45.32.121.40:6379 45.32.121.40 6379 @ mymaster 139.180.217.55 6379

Run Redis® CLI to connect to the Redis® Sentinel server.
```
 $ redis-cli -p 26379
```

Get the master address.

 $ SENTINEL get-master-addr-by-name mymaster

Here is an example result with the IP address and port of the master node.
```
 1) "139.180.217.55"
 2) "6379"
```

(Optional) Use Ansible to Automate the Setup

This section shows how to use Ansible to automate the Redis® master-replica setup with Redis® Sentinel. This is useful when you need to set up multiple environments and scale to multiple distributed servers.

Install Ansible

This section shows how to install Ansible from PyPI using pip. You can use your local PC with Python 3 installed. See the Ansible Installation Guide for more details.

Run the pip command to install Ansible.

$ pip install --user ansible

Setup Ansible Project

Clone the GitHub project of this article.

 $ git clone https://github.com/quanhua92/ansible-redis-sentinel

Create a file named vars.yml with the following content. Change <YOUR_PASSWORD> to a secure password.
```
 REDIS_PASSWORD: <YOUR_PASSWORD>
```

Edit the inventory file and replace <SERVER_01_IP>, <SERVER_02_IP>, and <SERVER_03_IP> with the corresponding IP address for each server.

 [redis]
 server-01 ansible_host=<SERVER_01_IP> role=master
 server-02 ansible_host=<SERVER_02_IP> role=replica
 server-03 ansible_host=<SERVER_03_IP> role=replica

Run Ansible playbook setup-redis-sentinel.yml.

 $ ansible-playbook setup-redis-sentinel.yml -i inventory

Here are the last few lines of the output. The failed=0 shows that there are no errors. Follow the previous section to test the failover.

 server-01                  : ok=14   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 server-02                  : ok=14   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 server-03                  : ok=14   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Set Up High Availability Redis® using Sentinel in Ubuntu

Introduction

Set Up Master-Replica Configuration

Create Three Vultr Server Instances

Install Redis®

Set Up the Master Node

Set Up Replica Node

Set Up Firewall for Redis® Server

Check the Master-Replica Setup

Configure Redis® Sentinel

Set Up Firewall for Sentinel

Check the Sentinel Setup

Test the Failover

(Optional) Use Ansible to Automate the Setup

Install Ansible

Setup Ansible Project

Products

Features

Marketplace

Resources

Company