Example: Data migration with rclone
Rclone is a command line program to manage files on cloud storage. It supports a multitude of providers including OpenStack Swift and Amazon S3.
In this article, we will describe how to migrate data from one CityCloud region to another. In particular, we will use Karlskrona (Kna1) as SOURCE region and Frankfurt (Fra1) as TARGET region.
Level
INTERMEDIATE
Pre-requisites
- Openstack-client installed and CLI access to both SOURCE and TARGET regions through the openstack.rc file - How-To
In case you want to use S3 backend, EC2 credentials with both access_ and secret_ key - How-To
- A storage container and its objects available in SOURCE region - How-To
Step-by-step guide
1. Create S3 and/or Swift rclone configuration file (rclone.conf) for both regions:
|
Note
An alternative way to use your Swift or S3 config without exposing your credentials in rclone.conf is by using env_auth=true and then source your OS_* and AWS_* environment variables as needed.
2. View all objects in the container, e.g. test, in the SOURCE region.
|
3. Create a container, e.g. copy_from_kna1, in the TARGET region.
4. Use either S3 or Swift backend configuration to copy your files from the SOURCE to the TARGET region:
|
5. Check that all objects are copied to the new container in the TARGET region:
|