File Mirroring with Rsync

Published on February 8, 2022•Updated on November 21, 2023

Introduction

Remote Sync (Rsync) is a file transfer program tool that allows you to copy, mirror data between servers, synchronize data locally and remotely across directories, disks, networks. The tool can quickly move large amounts of data between destinations with compression support, making it fast and secure to use.

In this guide, you will perform file mirroring with Rsync on specific files and directories, sync data between servers in a single Vultr Virtual Private Cloud (VPC), sync across to remote servers, and set up monitoring.

Prerequisites

Install Rsync

By default, Rsync is included in most Linux distributions. But in any case, install Rsync using the following commands:

On Ubuntu, Debian based distributions:

# apt install rsync

On RHEL based distributions:

# dnf install rsync

# yum install rsync

OpenBSD:

# pkg_add rsync

FreeBSD:

# pkg install rsync

Sync Specific Files and Directories

Rsync uses a straightforward syntax rsync on Debian, Ubuntu, OpenBSD, FreeBSD, and Redhat-based systems like CentOS, Rocky Linux, Alma Linux.

To sync specific files and directories using Rsync, create 3 test directories on your server.

# mkdir Dir1 
# mkdir Dir2 
# mkdir Dir3

Change to directory 1.

# cd Dir1

Create two simple text files and a simple script file in directory 1 using the following commands:

# echo "This is a text file" > file1.txt 
# echo "Just another text file" > file2.txt
# echo "#bin/sh echo"Hello World"" > script.sh

As well, create a test subdirectory in directory 1.

# mkdir Dir1/subdir1

To sync a single file to directory 2, use the following command:

# rsync Dir1/file2.txt  Dir2

To specifically sync only text files (.txt) in directory 1 to directory 2, use the following command.

# rsync Dir1/*.txt  Dir2

List all files in directory 2 to confirm that only .txt files are synced.

# ls

Output:

file1.txt   file2.txt

Next, use Rsync to mirror all files and subdirectories in directory 1 to directory 3, preserving symbolic links, time stamps, and ownership permissions.

# rsync -a Dir1/ Dir3

If you are hosting websites on your server, use Rsync to backup the entire /var/www/html/ directory to directory 2, view sync progress with v, and compress files during sync with z as command arguments.

# rsync -avz /var/www/html

The above command adds a new subdirectory html to directory 2 on your local server.

Sync within a single Vultr Virtual Private Cloud (VPC)

First, create a Vultr Virtual Private Cloud(VPC), assign addresses, then attach two or more servers to it. This guide uses the Rsync daemon to mirror files from a Ubuntu 20.04 server to a Rocky Linux server in a single VPC.

On server A (Ubuntu 20.04), create a dedicated backups user account.

# sudo adduser backups

Create a sample directory in the user home directory.

# mkdir /home/backups/backup

Then, create the Rsync daemon main configuration file in the /etc/ directory.

# nano /etc/rsyncd.conf

Paste the following contents:

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
port = 873
uid = backups
uid = backups

Save and exit the file.

Here is what each configuration line does:

pid file =: specifies the process id file Rsync uses.

lock file =: the Rsync daemon lock file.

log file =: specifies the log file location.

port =: Instructs the daemon to run on the specified port. By default, it runs on port 873.

uid =: specifies the user account Rsync should mirror files as.

gid =: specifies the user group.

Again, modify the configuration file to include a new [files] module with the backup directory path.

Open and edit the file.

# nano /etc/rsyncd.conf

Paste the following contents at the end of the file.

[files]
path = /home/backups/backup/
comment = VPC RSYNC Backup Files. 
read only = false
timeout = 300

Save and exit the file.

Here is what the new configuration lines do:

[files]: specifies the module name.

path =: specifies the root Rsync directory.

comment =: specifies the comment describing what the module is all about.

read only =: if set to true, clients will only be able to sync files from the directory. If set to false, clients will sync (pull) and write (put) to the root directory.

timeout =: the time in seconds that the Rsync daemon will stay active before terminating an inactive session.

You can create multiple modules pointing to different sync directories on the server. To tighten access to a specific directory, add another module with the path /var/www/ in the /etc/rsyncd.conf configuration file, and secure it with a password.

Add the following contents to the end of the /etc/rsyncd.conf file:

[confidential]
path = /var/www/html/
comment = Restricted Access, only Admins allowed here.
read only = true
timeout = 300 nano /etc/rsyncd.conf
auth users = admin,backups
secrets file = /etc/rsyncd.secrets

Save and close the file.

Here is what the new module parameters do:

auth users =: declares valid users authorized to sync files from the module directory.

secrets file =: specifies the file that contains the usernames and passwords for each of the authorized users.

read only = true: users are allowed to sync from the directory, but can't upload new files.

Next, create and edit the /etc/rsyncd.secrets file.

# nano /etc/rsyncd.secrets

Add the user account and password in the format: username:password.

admin:12345678
backups:123

Save and exit the file.

Change the file permissions to only allow the user root to read and edit the file.

# chmod 600 /etc/rsyncd.secrets

Your final daemon configuration file should now look like this:

$ cat /etc/rsyncd.conf


pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
port = 873
uid = backups
uid = backups

[files]
path = /home/backups/backup/
comment = VPC RSYNC Backup Files.
read only = false
timeout = 300

[confidential]
path = /var/www/html/
comment = Restricted Access, only Admins allowed here.
read only = true
timeout = 300 
auth users = admin,backups
secrets file = /etc/rsyncd.secrets

Setup the VPC Server Firewall

Now, add a new firewall rule to enable the Rsync daemon port.

# ufw allow 873/tcp

Restart firewall for changes to take effect.

# ufw reload

Then, start the Rsync daemon.

# rsync --daemon

Verify that the daemon is running:

# ps x | grep rsync

Alternatively, use systemd to enable the daemon to start at boot time.

# systemctl enable rsync

Start the Rsync daemon.

# systemctl start rsync

Sync VPC Server Files using the Rsync Daemon

Now, login to server B, and run the following command to read the available Rsync directory modules on server A.

# rsync rsync://10.9.96.3

10.9.96.3 refers to the server A IP address in the VPC. Your command output should be similar to the one below:

files           VPC RSYNC Backup Files.
confidential    Restricted Access, only Admins allowed here.

To, sync all files from the server module files, use the following command:

# rsync -av rsync://10.9.96.3/files/  /local/backups

To mirror a specific file, add the file name at the end of the module name. Then, specify a local directory to sync the file to.

# rsync -av rsync://10.9.96.3/files/helloworld  /local/backups

Next, sync files from the password-protected module confidential using the following command.

# rsync -av rsync://10.9.96.3/confidential/  /local/web-backup

Output:

Password:
receiving incremental file list
./
webfiles.tar.gz

sent 50 bytes  received 128 bytes  71.20 bytes/sec
total size is 0  speedup is 0.00

Now, try sending files to the server with the following command:

# rsync -av local/web-backup/ rsync://10.9.96.3/confidential/

Upon entering a valid user password, Rsync will throw an error message similar to:

Password:
sending incremental file list
rsync: read error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(785) [sender=3.1.3]

Setup CronJobs

To automate file mirroring, set up a new crontab using the following command:

# crontab -e

Then, paste the following command to sync files from the confidential directory once every week.

* * * * 6 rsync -az --password-file=/home/example/rsync_pass rsync://backups@10.9.96.3/confidential/ /home/example/local/web-backups

The above command automatically authenticates with the Rsync server using the password file /home/example/rsync_pass. Make sure it's created and only readable by the owner with permissions mode 600.

Also, paste the following command to sync files from server B to the server A files directory every day.

* *  * *  * rsync -az /home/backups/local  rsync://10.9.96.3/files/

Save and exit the file.

The above Cron jobs will sync files once every week and every day, respectively. To learn more about setting up Cronjobs, refer to Vultr's guide.

Sync to a Remote Server over the Internet

By default, Rsync uses Secure Shell (SSH) for secure data transfer over the Internet. This makes it possible to mirror files to any server in any location.

The following syntax is used to sync files to a remote server over SSH.

# rsync -a local-directory username@remoteserver:remote-directory

To sync the file backup.zip from your local home directory to a remote server, use the following command, replacing example with your actual server username.

# rsync -av  ~/backups/backup.zip  example@Vultr-Server-IP:Backup/files

Your output should be similar to:

building file list ... done
backup.zip

sent 15911 bytes  received 42 bytes  1679.26 bytes/sec
total size is 15782  speedup is 0.99

To sync files from the remote server to the local computer, use the following command.

# rsync -avz example@Vultr-Server-IP:Backup/files ~/backups

Your output should be similar to:

receiving file list ... done
./
backup.zip
helloworld.txt
webfiles.tar.gz

sent 88 bytes  received 15941 bytes  2137.20 bytes/sec
total size is 15782  speedup is 0.98

The above command mirrors all files from the ~/Backup/files directory, and saves them to the ~/backups directory on your local computer.

To strictly sync a file named website.tar.gz from the remote server, use the following command:

# rsync -avz example@Vultr-Server-IP:Backup/files/website.tar.gz ~/backups

A new file named website.tar.gz will be added to your local /home/user/backups directory.

Setup Rsync Logging and Monitoring

If you are running Rsync as a daemon similar to the VPC setup earlier, enable logging from the daemon configuration file. Else, if you are using Rsync on your local computer or mirroring files from a remote server, then add the --log-file= flag to every Rsync command.

The following command mirrors the file sample.txt to a VPC server and logs the output to /tmp/logs/rsync.log.

# rsync -avz sample.txt  backups@80.240.23.228:Hello/  --log-file=/tmp/logs/rsync.log

Contents of the log file will be similar to:

2022/01/21 02:48:51 [57728] receiving file list
2022/01/21 02:48:51 [57728] done
2022/01/21 02:48:51 [57747] .d..t.... ./
2022/01/21 02:48:52 [57747] sent 22 bytes  received 180 bytes  23.76 bytes/sec
2022/01/21 02:48:52 [57747] total size is 15782  speedup is 78.13

Next, to set up monitoring, and view the progress of Rsync directory transfers, add the --progress flag to every command.

The following command monitors and displays the progress of every transfer from the local directory /var/www/ to the remote server directory /var/www/html/.

# rsync -avz /var/www/ --progress root@Server-IP:/var/www/

Also, on the server side you can monitor active Rsync processes using the following command:

# ps -C rsync fw

Selective Sync with File and Directory Exclusion

Rsync allows you to include or exclude files you wish to transfer. Add the --exclude argument to the Rsync command to exclude specific files. For example, the following command mirrors / excluding the /mnt directory.

# rsync -avz --exclude '/mnt' example@Server-IP:/ ~/Systembackup --log-file=/tmp/logs/rsync.log

To strictly include files starting with h, and exclude all the others, use the following command:

# rsync -avz --include 'h*' --exclude '*' root@192.168.0.141:/ ~/Systembackup

The above command would sync the full system root directory but only include directories starting with h, meaning only the home directory will be mirrored.

Setup Firewall

By default, Rsync uses port 22 to transfer files using SSH and port 873 when running as a daemon.

Allow SSH Port 22 on your firewall.

On Debian bases systems:

# ufw allow 22/tcp

On RHEL based systems:

# firewall-cmd --zone=public --add-port=22/tcp --permanent

Allow your specified Rsync daemon port or default port 873.

On Debian based systems:

# ufw allow 873/tcp

On RHEL based systems:

# firewall-cmd --zone=public --add-port=873/tcp --permanent

Restart the Firewall.

# ufw reload

# firewall-cmd --reload

Conclusion

In this guide, you have performed file mirroring with Rsync on a local server, servers within a Vultr VPC, and performed sync operations on a remote location server. For more information on using the tool, read the Rsync manual by running the command man rsync on your server.

File Mirroring with Rsync

Introduction

Prerequisites

Install Rsync

Sync Specific Files and Directories

Sync within a single Vultr Virtual Private Cloud (VPC)

Setup the VPC Server Firewall

Sync VPC Server Files using the Rsync Daemon

Setup CronJobs

Sync to a Remote Server over the Internet

Setup Rsync Logging and Monitoring

Selective Sync with File and Directory Exclusion

Setup Firewall

Conclusion

Products

Features

Marketplace

Resources

Company