File Mirroring with Rsync
Introduction
Remote Sync (Rsync) is a file transfer program tool that allows you to copy, mirror data between servers, synchronize data locally and remotely across directories, disks, networks. The tool can quickly move large amounts of data between destinations with compression support, making it fast and secure to use.
In this guide, you will perform file mirroring with Rsync on specific files and directories, sync data between servers in a single Vultr Virtual Private Cloud (VPC), sync across to remote servers, and set up monitoring.
Prerequisites
- An active Virtual Private Server (VPS) on Vultr
- Access the server
- Create a Vultr Virtual Private cloud (VPC)
Install Rsync
By default, Rsync is included in most Linux distributions. But in any case, install Rsync using the following commands:
On Ubuntu, Debian based distributions:
On RHEL based distributions:
OR
OpenBSD:
FreeBSD:
Sync Specific Files and Directories
Rsync uses a straightforward syntax rsync on Debian, Ubuntu, OpenBSD, FreeBSD, and Redhat-based systems like CentOS, Rocky Linux, Alma Linux.
To sync specific files and directories using Rsync, create 3 test directories on your server.
Change to directory 1.
Create two simple text files and a simple script file in directory 1 using the following commands:
As well, create a test subdirectory in directory 1.
To sync a single file to directory 2, use the following command:
To specifically sync only text files (.txt) in directory 1 to directory 2, use the following command.
List all files in directory 2 to confirm that only .txt files are synced.
Output:
Next, use Rsync to mirror all files and subdirectories in directory 1 to directory 3, preserving symbolic links, time stamps, and ownership permissions.
If you are hosting websites on your server, use Rsync to backup the entire /var/www/html/ directory to directory 2, view sync progress with v, and compress files during sync with z as command arguments.
The above command adds a new subdirectory html to directory 2 on your local server.
Sync within a single Vultr Virtual Private Cloud (VPC)
First, create a Vultr Virtual Private Cloud(VPC), assign addresses, then attach two or more servers to it. This guide uses the Rsync daemon to mirror files from a Ubuntu 20.04 server to a Rocky Linux server in a single VPC.
On server A (Ubuntu 20.04), create a dedicated backups user account.
Create a sample directory in the user home directory.
Then, create the Rsync daemon main configuration file in the /etc/ directory.
Paste the following contents:
Save and exit the file.
Here is what each configuration line does:
pid file =: specifies the process id file Rsync uses.
lock file =: the Rsync daemon lock file.
log file =: specifies the log file location.
port =: Instructs the daemon to run on the specified port. By default, it runs on port 873.
uid =: specifies the user account Rsync should mirror files as.
gid =: specifies the user group.
Again, modify the configuration file to include a new [files] module with the backup directory path.
Open and edit the file.
Paste the following contents at the end of the file.
Save and exit the file.
Here is what the new configuration lines do:
[files]: specifies the module name.
path =: specifies the root Rsync directory.
comment =: specifies the comment describing what the module is all about.
read only =: if set to true, clients will only be able to sync files from the directory. If set to false, clients will sync (pull) and write (put) to the root directory.
timeout =: the time in seconds that the Rsync daemon will stay active before terminating an inactive session.
You can create multiple modules pointing to different sync directories on the server. To tighten access to a specific directory, add another module with the path /var/www/ in the /etc/rsyncd.conf configuration file, and secure it with a password.
Add the following contents to the end of the /etc/rsyncd.conf file:
Save and close the file.
Here is what the new module parameters do:
auth users =: declares valid users authorized to sync files from the module directory.
secrets file =: specifies the file that contains the usernames and passwords for each of the authorized users.
read only = true: users are allowed to sync from the directory, but can't upload new files.
Next, create and edit the /etc/rsyncd.secrets file.
Add the user account and password in the format: username:password.
Save and exit the file.
Change the file permissions to only allow the user root to read and edit the file.
Your final daemon configuration file should now look like this:
Setup the VPC Server Firewall
Now, add a new firewall rule to enable the Rsync daemon port.
Restart firewall for changes to take effect.
Then, start the Rsync daemon.
Verify that the daemon is running:
Alternatively, use systemd to enable the daemon to start at boot time.
Start the Rsync daemon.
Sync VPC Server Files using the Rsync Daemon
Now, login to server B, and run the following command to read the available Rsync directory modules on server A.
10.9.96.3 refers to the server A IP address in the VPC. Your command output should be similar to the one below:
To, sync all files from the server module files, use the following command:
To mirror a specific file, add the file name at the end of the module name. Then, specify a local directory to sync the file to.
Next, sync files from the password-protected module confidential using the following command.
Output:
Now, try sending files to the server with the following command:
Upon entering a valid user password, Rsync will throw an error message similar to:
Setup CronJobs
To automate file mirroring, set up a new crontab using the following command:
Then, paste the following command to sync files from the confidential directory once every week.
The above command automatically authenticates with the Rsync server using the password file
/home/example/rsync_pass. Make sure it's created and only readable by the owner with permissions mode 600.
Also, paste the following command to sync files from server B to the server A files directory every day.
Save and exit the file.
The above Cron jobs will sync files once every week and every day, respectively. To learn more about setting up Cronjobs, refer to Vultr's guide.
Sync to a Remote Server over the Internet
By default, Rsync uses Secure Shell (SSH) for secure data transfer over the Internet. This makes it possible to mirror files to any server in any location.
The following syntax is used to sync files to a remote server over SSH.
To sync the file backup.zip from your local home directory to a remote server, use the following command, replacing example with your actual server username.
Your output should be similar to:
To sync files from the remote server to the local computer, use the following command.
Your output should be similar to:
The above command mirrors all files from the ~/Backup/files directory, and saves them to the ~/backups directory on your local computer.
To strictly sync a file named website.tar.gz from the remote server, use the following command:
A new file named website.tar.gz will be added to your local /home/user/backups directory.
Setup Rsync Logging and Monitoring
If you are running Rsync as a daemon similar to the VPC setup earlier, enable logging from the daemon configuration file. Else, if you are using Rsync on your local computer or mirroring files from a remote server, then add the --log-file= flag to every Rsync command.
The following command mirrors the file sample.txt to a VPC server and logs the output to /tmp/logs/rsync.log.
Contents of the log file will be similar to:
Next, to set up monitoring, and view the progress of Rsync directory transfers, add the --progress flag to every command.
The following command monitors and displays the progress of every transfer from the local directory /var/www/ to the remote server directory /var/www/html/.
Also, on the server side you can monitor active Rsync processes using the following command:
Selective Sync with File and Directory Exclusion
Rsync allows you to include or exclude files you wish to transfer. Add the --exclude argument to the Rsync command to exclude specific files. For example, the following command mirrors / excluding the /mnt directory.
To strictly include files starting with h, and exclude all the others, use the following command:
The above command would sync the full system root directory but only include directories starting with h, meaning only the home directory will be mirrored.
Setup Firewall
By default, Rsync uses port 22 to transfer files using SSH and port 873 when running as a daemon.
Allow SSH Port 22 on your firewall.
On Debian bases systems:
On RHEL based systems:
Allow your specified Rsync daemon port or default port 873.
On Debian based systems:
On RHEL based systems:
Restart the Firewall.
Conclusion
In this guide, you have performed file mirroring with Rsync on a local server, servers within a Vultr VPC, and performed sync operations on a remote location server. For more information on using the tool, read the Rsync manual by running the command man rsync on your server.
0 Comments
Be the first to comment and share your perspective with the community.