Setup High Availability Redis using Sentinel in Ubuntu

Updated on October 31, 2022
Setup High Availability Redis using Sentinel in Ubuntu header image

Redis is an open-source, in-memory data structure. Redis stores data in memory and supports a mechanism to persist data to disk. Redis is popular for caching, session management, real-time analytics, and pub/sub applications.

Redis supports data replication using an asynchronous method between a master and multiple replicas. Redis Sentinel is a solution to monitor, send notifications to system admins, and automatic failover if the master is not working as expected.

For a complex system, Redis Cluster supports multiple masters and replicas.

This article shows how to set up a high-availability Redis using Sentinel in Ubuntu.

At the end of this article, you'll know:

  • How to set up Redis master-replica replication.
  • How to set up Redis Sentinel.
  • How to test the failover of the high availability Redis using Sentinel.
  • How to automate the setup using Ansible.

Setup Master-Replica Configuration

This section shows how to set up Redis master-replica replication. This setup contains a master server and two replica servers.

You can deploy three Ubuntu servers with 1GB of memory to follow this article. However, higher memory servers are better for production.

These servers should be independent of each other. For example, they can be different physical servers or virtual machines from different availability zones.

Create Three Vultr Server Instances

  1. Navigate to your Customer Portal to deploy new servers.
  2. Label the server as master, replica-1, and replica-2.
  3. (Optional) Configure SSH Keys for better security. See SSH Keys in Customer Portal
  4. (Optional) Enable Auto Backups

Here is an example result.

- master: 45.32.121.40
- replica-1: 139.180.217.55
- replica-2: 139.180.156.164

Install Redis

  1. Connect to each server over SSH.

  2. Run the following commands to install Redis.

     $ curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
     $ echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
     $ sudo apt-get update
     $ sudo apt-get install -y redis-server
  3. Activate the redis-server service to start on boot.

     $ systemctl enable redis-server

Setup the Master Node

This section shows how to set up the master node with password authentication.

  1. Connect to your master server over SSH.

  2. Open the configuration file at /etc/redis/redis.conf with your favorite editor.

  3. Find the following configuration, uncomment and edit them. Change <YOUR_PASSWORD> to a secure password. The bind configuration allows the Redis server to be accessible from other nodes.

     bind 0.0.0.0
     requirepass "<YOUR_PASSWORD>"
     masterauth "<YOUR_PASSWORD>"
  4. Restart the redis-server service.

     $ sudo service redis-server restart

Setup Replica Node

This section shows how to set up the replica nodes with password authentication.

  1. Connect to each replica server over SSH.

  2. Open the configuration file at /etc/redis/redis.conf with your favorite edit.

  3. Find the following configuration, uncomment and edit them as follows. Change <YOUR_PASSWORD> to the same password on master node. Change <MASTER_NODE_IP> to the IP address of the master node.

     bind 0.0.0.0
     requirepass "<YOUR_PASSWORD>"
     masterauth "<YOUR_PASSWORD>"
     replicaof <MASTER_NODE_IP> 6379
  4. Restart the redis-server service.

     $ sudo service redis-server restart

Setup Firewall for Redis Server

This section shows how to edit ufw firewall to allow connection to port 6379 so that Redis Servers can talk with each other.

  1. Connect to the master server and each replica server over SSH.

  2. Run the following commands. Change <MASTER_NODE_IP> to the IP address of the master node. Change <REPLICA_1_NODE_IP> to the IP address of the replica-1 node. Change <REPLICA_2_NODE_IP> to the IP address of the replica-2 node.

     $ sudo ufw allow from <MASTER_NODE_IP> to any port 6379 comment 'MASTER'
     $ sudo ufw allow from <REPLICA_1_NODE_IP> to any port 6379 comment 'REPLICA-1'
     $ sudo ufw allow from <REPLICA_2_NODE_IP> to any port 6379 comment 'REPLICA-2'

Check the Master-Replica Setup

This section shows how to check if the master-replica setup is working as expected.

  1. Connect to your master server.

  2. Check the log of Redis Sentinel.

     $ tail -f /var/log/redis/redis-server.log
  3. Here is an example result:

     704:M 01 Oct 2022 09:20:58.524 * Loading RDB produced by version 7.0.5
     704:M 01 Oct 2022 09:20:58.524 * RDB age 15 seconds
     704:M 01 Oct 2022 09:20:58.524 * RDB memory usage when created 0.91 Mb
     704:M 01 Oct 2022 09:20:58.524 * Done loading RDB, keys loaded: 1, keys expired: 0.
     704:M 01 Oct 2022 09:20:58.525 * DB loaded from disk: 0.002 seconds
     704:M 01 Oct 2022 09:20:58.525 * Ready to accept connections
     704:M 01 Oct 2022 09:20:59.353 * Replica 139.180.217.55:6379 asks for synchronization
     704:M 01 Oct 2022 09:20:59.353 * Partial resynchronization request from 139.180.217.55:6379 accepted. Sending 0 bytes of backlog starting from offset 2873.
     704:M 01 Oct 2022 09:20:59.435 * Replica 139.180.156.164:6379 asks for synchronization
     704:M 01 Oct 2022 09:20:59.435 * Partial resynchronization request from 139.180.156.164:6379 accepted. Sending 0 bytes of backlog starting from offset 2873.
  4. Run Redis CLI to connect to the Redis server.

     $ redis-cli
  5. Run AUTH command to authenticate. Change <YOUR_PASSWORD> to the same password on master node.

     $ AUTH <YOUR_PASSWORD>
  6. Run INFO command to check the Redis replication information.

     $ INFO REPLICATION
  7. Here is an example result. The output connected_slaves:2 shows that both replica nodes have successfully connected to the master node.

     # Replication
     role:master
     connected_slaves:2
     slave0:ip=139.180.217.55,port=6379,state=online,offset=14,lag=1
     slave1:ip=139.180.156.164,port=6379,state=online,offset=14,lag=1
     master_failover_state:no-failover
     master_replid:43855ff334c072ee5a3f4db39b1b27b31f55a3a4
     master_replid2:0000000000000000000000000000000000000000
     master_repl_offset:14
     second_repl_offset:-1
     repl_backlog_active:1
     repl_backlog_size:1048576
     repl_backlog_first_byte_offset:1
     repl_backlog_histlen:14
  8. Run SET command to set a key hello to world on the master node.

     $ SET hello world
  9. Connect to your redis replica server.

  10. Run Redis CLI to connect to the Redis server.

    $ redis-cli
  11. Run AUTH command to authenticate. Change <YOUR_PASSWORD> to the same password on the master node.

    $ AUTH <YOUR_PASSWORD>
  12. Run GET command to get the value of key hello. The result is world shows that the replication from master to replica works successfully.

    $ GET hello

Configure Redis Sentinel

This section shows how to set up Redis Sentinel to monitor the master-replica setup in the previous section.

Redis Sentinel is a distributed system where there are multiple Sentinel processes running on different servers. This makes the system robust against failures. In a failure scenario, the majority of Sentinel processes monitor and vote to promote a replica to be the new master.

  1. On all three servers, install Redis Sentinel as follows:

     $ sudo apt install redis-sentinel
  2. Activate the redis-sentinel service to start on boot.

     $ systemctl enable redis-sentinel
  3. Open the configuration file at /etc/redis/sentinel.conf with your favorite editor.

  4. Find the line sentinel monitor mymaster 127.0.0.1 6379 2, and edit them as follows. Change <YOUR_PASSWORD> to the same password on master node. Change <MASTER_NODE_IP> to the IP address of the master node.

     sentinel monitor mymaster <MASTER_NODE_IP> 6379 2
     sentinel auth-pass mymaster <YOUR_PASSWORD>
     sentinel down-after-milliseconds mymaster 5000
     sentinel failover-timeout mymaster 60000
     protected-mode no
  5. Restart the Redis Sentinel Service.

     $ sudo service redis-sentinel restart

Setup Firewall for Sentinel

This section shows how to edit ufw firewall to allow connection to port 26379 so that Redis Sentinel can talk with each other.

  1. Connect to the master server and each replica server over SSH.

  2. Run the following commands. Change <MASTER_NODE_IP> to the IP address of the master node. Change <REPLICA_1_NODE_IP> to the IP address of the replica-1 node. Change <REPLICA_2_NODE_IP> to the IP address of the replica-2 node.

     $ sudo ufw allow from <MASTER_NODE_IP> to any port 26379 comment 'SENTINEL'
     $ sudo ufw allow from <REPLICA_1_NODE_IP> to any port 26379 comment 'SENTINEL'
     $ sudo ufw allow from <REPLICA_2_NODE_IP> to any port 26379 comment 'SENTINEL'

Check the Sentinel Setup

This section shows how to check if the Redis Sentinel is working as expected.

  1. Connect to your master server over SSH.

  2. Check the log of Redis Sentinel.

     $ tail -f /var/log/redis/redis-sentinel.log
  3. Here is an example result:

     1049:X 01 Oct 2022 17:57:30.950 * Removing the pid file.
     1049:X 01 Oct 2022 17:57:30.950 # Sentinel is now ready to exit, bye bye...
     5020:X 01 Oct 2022 17:57:30.993 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
     5020:X 01 Oct 2022 17:57:30.993 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
     5020:X 01 Oct 2022 17:57:30.993 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=5020, just started
     5020:X 01 Oct 2022 17:57:30.993 # Configuration loaded
     5020:X 01 Oct 2022 17:57:30.994 * monotonic clock: POSIX clock_gettime
     5020:X 01 Oct 2022 17:57:30.994 * Running mode=sentinel, port=26379.
     5020:X 01 Oct 2022 17:57:30.995 # Sentinel ID is 8de12f567ebdfdfc8df229497e445667fd18f3e2
     5020:X 01 Oct 2022 17:57:30.995 # +monitor master mymaster 45.32.121.40 6379 quorum 2
  4. Run Redis CLI to connect to the Redis Sentinel server.

     $ redis-cli -p 26379
  5. Run INFO command to check the Redis replication information.

     $ INFO SENTINEL
  6. Here is an example result.

     # Sentinel
     sentinel_masters:1
     sentinel_tilt:0
     sentinel_tilt_since_seconds:-1
     sentinel_running_scripts:0
     sentinel_scripts_queue_length:0
     sentinel_simulate_failure_flags:0
     master0:name=mymaster,status=ok,address=45.32.121.40:6379,slaves=2,sentinels=3

Test the Failover

This section shows how to test the failover by stopping the Redis master node manually.

  1. Connect to your master server.

  2. Stop the Redis services on the master server.

     $ sudo service redis-sentinel stop
     $ sudo service redis-server stop
  3. Open a new terminal and connect to a Redis replica server.

  4. Check the log of Redis Sentinel.

     $ tail -f /var/log/redis/redis-sentinel.log
  5. Here is an example result. The Sentinel starts a failover process and promotes a replica to be the new master.

     5222:X 01 Oct 2022 18:59:38.903 # +sdown sentinel e5b67f1e144dedf55cf71e60df52489debfdcb9d 45.32.121.40 26379 @ mymaster 45.32.121.40 6379
     5222:X 01 Oct 2022 18:59:39.836 # +sdown master mymaster 45.32.121.40 6379
     5222:X 01 Oct 2022 18:59:39.958 * Sentinel new configuration saved on disk
     5222:X 01 Oct 2022 18:59:39.958 # +new-epoch 3
     5222:X 01 Oct 2022 18:59:39.959 * Sentinel new configuration saved on disk
     5222:X 01 Oct 2022 18:59:39.960 # +vote-for-leader 852870198de5b3ced3abd82b8f85cd4866c7921c 3
     5222:X 01 Oct 2022 18:59:40.975 # +odown master mymaster 45.32.121.40 6379 #quorum 2/2
     5222:X 01 Oct 2022 18:59:40.976 # Next failover delay: I will not start a failover before Sat Oct  1 19:01:40 2022
     5222:X 01 Oct 2022 18:59:41.028 # +config-update-from sentinel 852870198de5b3ced3abd82b8f85cd4866c7921c 139.180.156.164 26379 @ mymaster 45.32.121.40 6379
     5222:X 01 Oct 2022 18:59:41.028 # +switch-master mymaster 45.32.121.40 6379 139.180.217.55 6379
     5222:X 01 Oct 2022 18:59:41.028 * +slave slave 139.180.156.164:6379 139.180.156.164 6379 @ mymaster 139.180.217.55 6379
     5222:X 01 Oct 2022 18:59:41.029 * +slave slave 45.32.121.40:6379 45.32.121.40 6379 @ mymaster 139.180.217.55 6379
     5222:X 01 Oct 2022 18:59:41.030 * Sentinel new configuration saved on disk
     5222:X 01 Oct 2022 18:59:46.128 # +sdown slave 45.32.121.40:6379 45.32.121.40 6379 @ mymaster 139.180.217.55 6379
  6. Run Redis CLI to connect to the Redis Sentinel server.

     $ redis-cli -p 26379
  7. Get the master address.

     $ SENTINEL get-master-addr-by-name mymaster
  8. Here is an example result with the IP address and port of the master node.

     1) "139.180.217.55"
     2) "6379"

(Optional) Use Ansible to Automate the Setup

This section shows how to use Ansible to automate the Redis master-replica setup with Redis Sentinel. This is useful when you need to set up multiple environments and scale to multiple distributed servers.

Install Ansible

This section shows how to install Ansible from PyPI using pip. You can use your local PC with Python 3 installed. See the Ansible Installation Guide for more details.

Run the pip command to install Ansible.

$ pip install --user ansible

Setup Ansible Project

  1. Clone the GitHub project of this article.

     $ git clone https://github.com/quanhua92/ansible-redis-sentinel
  2. Create a file named vars.yml with the following content. Change <YOUR_PASSWORD> to a secure password.

     REDIS_PASSWORD: <YOUR_PASSWORD>
  3. Edit the inventory file and replace <SERVER_01_IP>, <SERVER_02_IP>, and <SERVER_03_IP> with the corresponding IP address for each server.

     [redis]
     server-01 ansible_host=<SERVER_01_IP> role=master
     server-02 ansible_host=<SERVER_02_IP> role=replica
     server-03 ansible_host=<SERVER_03_IP> role=replica
  4. Run Ansible playbook setup-redis-sentinel.yml.

     $ ansible-playbook setup-redis-sentinel.yml -i inventory
  5. Here are the last few lines of the output. The failed=0 shows that there are no errors. Follow the previous section to test the failover.

     server-01                  : ok=14   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
     server-02                  : ok=14   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
     server-03                  : ok=14   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0