How to Migrate Amazon S3 Storage to Vultr Object Storage

Introduction

Migrating Object Storage requires strategic planning to ensure minimal downtime for active applications. Vultr Object Storage is a managed cloud storage solution that securely stores data as objects. Migrating from Amazon S3 to Vultr Object Storage lets you store, optimize, and retrieve your data and applications with high availability.

Follow this guide to migrate from Amazon S3 to Vultr Object Storage and automate all data transfers to automatically synchronize buckets.

Prerequisites

Before you begin, you need to:

• Have an existing Amazon S3 subscription.
• Retrieve the Amazon S3 access credentials from the Amazon Identity and Access Management (IAM) dashboard.
• Provision a Vultr Object Storage subscription and retrieve the S3 Credentials in the Overview tab.
• Deploy an Ubuntu 24.04 instance on Vultr to use as the management workstation and enable the Limited User Login feature.
• Access the instance using SSH.

Set Up the Migration Environment

Rclone is an open-source application used to manage and migrate cloud storage solutions. It supports S3-compatible storage such as Vultr Object Storage to manage files as objects. Follow the steps below to install and configure Rclone to authenticate with Amazon S3 and Vultr Object Storage on your workstation.

1. Update the server's package index.

   CONSOLE

   Copy

   $ sudo apt update

   Explain Code

   Powered by Vultr AgentBeta

2. Download the latest Rclone installation script.

   CONSOLE

   Copy

   $ wget -O rclone.sh https://rclone.org/install.sh

   Explain Code

   Powered by Vultr AgentBeta

3. Run the script to install Rclone.

   CONSOLE

   Copy

   $ sudo bash rclone.sh 

   Explain Code

   Powered by Vultr AgentBeta

4. View the installed Rclone version.

   CONSOLE

   Copy

   $ rclone version

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below.

   rclone v1.68.2
   - os/version: ubuntu 24.04 (64 bit)
   - os/kernel: 6.8.0-51-generic (x86_64)
   - os/type: linux
   - os/arch: amd64
   - go/version: go1.23.3
   - go/linking: static
   - go/tags: none

Configure Rclone to Authenticate with Amazon S3 and Vultr Object Storage

The default Rclone configuration does not include any storage configurations. Follow the steps below to configure Rclone to securely authenticate with Amazon S3 and Vultr Object Storage to migrate existing buckets and objects.

1. Generate a new Rclone configuration in your user environment.

   CONSOLE

   Copy

   $ rclone config file

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below.

   Configuration file doesn't exist, but rclone will use this path:
   /home/linuxuser/.config/rclone/rclone.conf

2. Create the Rclone configuration using the generated path.

   CONSOLE

   Copy

   $ nano  /home/linuxuser/.config/rclone/rclone.conf

   Explain Code

   Powered by Vultr AgentBeta

3. Add the following Amazon S3 configuration block to the file. Replace YOUR_AWS_ACCESS_KEY and YOUR_AWS_SECRET_ACCESS_KEY with your actual credentials.

   INI

   Copy

   [aws]
   type = s3
   provider = AWS
   access_key_id = YOUR_AWS_ACCESS_KEY
   secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY
   env_auth = false

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following Vultr Object Storage configuration block to the file. Replace ewr1.vultrobjects.com with your actual Vultr Object Storage hostname and YOUR_VULTR_ACCESS_KEY,YOUR_VULTR_SECRET_ACCESS_KEY with your S3 credentials.

   INI

   Copy

   [vultr]
   type = s3
   provider = Other
   access_key_id = YOUR_VULTR_ACCESS_KEY
   secret_access_key = YOUR_VULTR_SECRET_ACCESS_KEY
   endpoint = ewr1.vultrobjects.com
   env_auth = false

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

5. Test the Rclone configuration and verify that both aws and vultr remotes are active.

   CONSOLE

   Copy

   $ rclone config

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   Current remotes:
   
   Name                 Type
   ====                 ====
   aws                   s3
   vultr                 s3

6. Enter Q to exit the Rclone configuration mode. Then, run the following command to verify your configuration values.

   CONSOLE

   Copy

   $ rclone config show

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below.

   [aws]
   type = s3
   provider = AWS
   access_key_id = example-access-key123
   secret_access_key = example-secret-key123
   env_auth = false
   
   [vultr]
   type = s3
   provider = Other
   access_key_id = example-access-key123
   secret_access_key = example-secret-key123
   endpoint = ewr1.vultrobjects.com
   env_auth = false

Migrate Amazon S3 Buckets to Vultr Object Storage

You can migrate specific Amazon S3 objects or entire storage buckets to Vultr Object Storage. Follow the steps below to migrate all Amazon S3 buckets to Vultr Object Storage including all existing objects.

1. Migrate all Amazon S3 buckets to Vultr Object Storage and monitor the transfer process.

   CONSOLE

   Copy

   $ rclone sync aws: vultr: --progress

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below when successful.

   Transferred:        1.191 MiB / 1.191 MiB, 100%, 287.950 KiB/s, ETA 0s
   Checks:                10 / 10, 100%
   Deleted:               10 (files), 0 (dirs), 725.997 KiB (freed)
   Transferred:           19 / 19, 100%
   Elapsed time:         5.8s

   The above transfer may take some time to complete depending on the size of your Amazon S3 storage. Monitor the process and verify that the transfer is successful with a 100% prompt in your output.

2. Run the following command to migrate a specific bucket to Vultr Object Storage. Replace vultrmigration with your actual bucket name.

   CONSOLE

   Copy

   $ rclone sync aws:awsvultrmigration vultr:awsvultrmigration --progress

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   Transferred:        4.424 MiB / 4.424 MiB, 100%, 0 B/s, ETA -
   Checks:                 1 / 1, 100%
   Transferred:            1 / 1, 100%
   Elapsed time:         1.8s

Set Up Automatic Object Storage Migration Using Cron

You can set up automatic migrations to Vultr Object Storage using Cron to run Rclone at specific intervals. Automating the migration process is important when the source Amazon S3 storage is actively in-use and requires real-time migration to the destination Vultr Object Storage subscription. Follow the steps below to create a new cron job that runs every 30 minutes to synchronize Amazon S3 buckets to Vultr Object Storage.

1. Open the Crontab file.

   CONSOLE

   Copy

   $ sudo nano /etc/crontab

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following cron entry at the end of the file.

   INI

   Copy

   */30 * * * * linuxuser /usr/bin/rclone sync aws: vultr: --progress  >> /var/log/awsvultrmigration.log 2>&1

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The above cron job runs the Rclone migration command every 30 minutes and redirects all logs to the /var/log/awsvultrmigration.log file.

Test Vultr Object Storage and Cut-over Applications

Follow the steps below to test the migrated Vultr Object Storage buckets and verify that the data matches with the source Amazon S3 storage.

1. Run the following command to verify that the source Amazon S3 data matches with the destination Vultr Object Storage.

   CONSOLE

   Copy

   $ rclone check aws: vultr:

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below.

   2025/01/09 18:04:39 NOTICE: S3 root: 0 differences found
   2025/01/09 18:04:39 NOTICE: S3 root: 19 matching files

2. Run the following command to verify that a specific Amazon S3 bucket matches the destination Vultr Object Storage bucket. Replace awsvultrmigration with your actual bucket name.

   CONSOLE

   Copy

   $ rclone check aws:awsvultrmigration vultr:awsvultrmigration

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below.

   2025/01/09 18:06:07 NOTICE: S3 bucket awsvultrmigration: 0 differences found
   2025/01/09 18:06:07 NOTICE: S3 bucket awsvultrmigration: 10 matching files

Conclusion

You have migrated from Amazon S3 to Vultr Object Storage using Rclone. You can migrate specific buckets to actively use both storage solutions to meet your application needs or move all existing buckets and objects to Vultr Object Storage. For more information on managing your Vultr Object Storage subscription, vist the product documentation page.