Example: Data migration with rclone

Rclone is a command line program to manage files on cloud storage. It supports a multitude of providers including OpenStack Swift and Amazon S3.

In this article, we will describe how to migrate data from one CityCloud region to another. In particular, we will use Karlskrona (Kna1) as SOURCE region and Frankfurt (Fra1) as TARGET region.

Level

INTERMEDIATE

Pre-requisites

  • Openstack-client installed and CLI access to both SOURCE and TARGET regions through the openstack.rc file - How-To
  • In case you want to use S3 backend, EC2 credentials with both access_ and secret_ key - How-To

  • A storage container and its objects available in SOURCE region  - How-To

Step-by-step guide

1. Create S3 and/or Swift rclone configuration file (rclone.conf) for both regions:

[kna1-s3]
type = s3
provider = Other
env_auth = false
access_key_id = <access key>
secret_access_key = <secret>
region = s3-kna1.citycloud.com:8080
endpoint = s3-kna1.citycloud.com:8080
acl = private
 
[kna1-swift]
type = swift
user = <username>
key = <password>
auth = https://kna1.citycloud.com:5000/v3
tenant = <Project name>
region = Kna1
domain = <Domain name>
tenant_domain = <Domain name>
auth_version = 3
 
[fra1-s3]
type = s3
provider = Other
env_auth = false
access_key_id = <access key>
secret_access_key = <secret>
region = s3-fra1.citycloud.com:8080
endpoint = s3-fra1.citycloud.com:8080
acl = private
 
[fra1-swift]
type = swift
user = <username>
key = <password>
auth = https://fra1.citycloud.com:5000/v3
tenant = <Project name>
region = Fra1
domain = <Domain name>
tenant_domain = <Domain name>
auth_version = 3

Note

An alternative way to use your Swift or S3 config without exposing your credentials in rclone.conf is by using env_auth=true and then source your OS_* and AWS_*  environment variables as needed.

2. View all objects in the container, e.g. test, in the SOURCE region.

$ openstack object list test
+---------+
| Name    |
+---------+
| file.ex |
+---------+

3. Create a container, e.g. copy_from_kna1, in the TARGET region.

4. Use either S3 or Swift backend configuration to copy your files from the SOURCE to the TARGET region:

$ rclone sync --config rclone.conf kna1-s3:test fra1-s3:copy_from_kna1 -P
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s

5. Check that all objects are copied to the new container in the TARGET region:

$ openstack object list copy_from_kna1
+---------+
| Name    |
+---------+
| file.ex |
+---------+