How to use ECS-sync for S3 to S3 migration

ecs-syncECS-sync is an external tool developed by the Object EMC Development team. It supports Centera, Atmos, S3, file to CAS, Atmos, S3, file migration. In this post I’m focused on S3 to S3 migration.

Features

ECS-sync supports the following functionality:

  • Migration Pause, Resume, Stop
  • Thread control
  • Clip list only
  • Logging including the SDK error and timestamp.
  • Retry of a failed clip
  • Mysql Database to log all migration transactions
  • Email home
  • ECS-sync will resume where it left off if the tool stops as long as the DB table it was populating is still present.
  • Multiple sync servers can be deployed; rule of thumb is 1 sync server for every 2 Centera access nodes. Best practice is try not go over 4 servers per site
  • ECS-sync copies and validates the MD5 after it is on a target
Note: Validation prevents having to perform a long reconciliation at the end of the migration. After the migration, we only have work on the errors.

Documentation and CLI

Some ECS-sync description and CLI syntax have to be checked here

Sample configuration XML files for ECS-sync 2.1 are available here

Plugins and filters

ecs-sync-is-java-applicationECS-sync is a Java application that joins 3rd-party storage clients with an abstract “sync” framework. When we migrate data, we “sync” it.
The framework consists of the abstract notion of a sync object and a set of sync plugins.

For any given sync operation, source plugin, target plugin, and optional object filters have to be defined. A source plugin iterates with objects on the source storage. A target is responsible for replicating object to a target storage. Sometimes source attributes cannot be transferred or represented on target. However, target plugin is trying to do its best to create objects as close as possible to a source.
Each plugin is configured using a set of common and custom properties. Before the sync operation is started, all of the plugins configure themselves and verify any compatibility issues with other plugins in a source-target chain.
Plugins are loaded dynamically based on command line parameters. Each plugin is registered and knows what kind of URL it can process (i.e. “s3://” is processed by S3 plugin).

Object filters transform the objects or collect information about the objects as they are processed. Filters are activated by listing the filter name in the comma-delimited –filters parameter.

Migration server

To use ECS-sync you need to provide a migration server. The migration server is the machine on which ECS-sync will run. All migration traffic goes through this server. Therefore you should have high bandwidth network between ECS-sync and both the source and target systems .

There are two option to install ECS-sync:

  • manual
  • vApp OVA bundle

Manual

If the customer cannot support the vApp deployment, a manual installation can be performed on any OS that supports Java 7+. Only one instance of the sync tool can run on a single server.

The link to get the latest version of ECS-sync for manual installation is here

vApp installation

OVA requirements.

  • CentOS7
  • 16B RAM
  • 200 GB disk
  • Single NIC at 1GB or 10GB
  • Quad core
  • Up to 4 VM’s per site can be deployed for the migration

Please use this link to download the latest version of OVA bundle.

The OVA for ECS-sync 2.1 contains the following:

  • Scripts to clean a clip list
  • Script to create database tables
  • JCAScript
  • CCV verify
  • SDK binaries
  • ecs_sync 2.0 jar file
  • ecs-sync-cli-1.1.jar
  • xml
  • MariaDB
  • ECS-sync run script

Deploy and Configure the OVA:

  • Upload the OVA to a datastore
  • From a vSphere client connected to either an ESX host or VCenterselect select File > Deploy OVF Template

ecssync-1

  • Power on the VM
  • Open a Virtual Machine Console to the VM

ecssync-2

  • Login is root:ECSSync

ecssync-3

  • Change the hostname to ecs_sync1, or whatever you want
[root@localhost ~]# vi /etc/hostname
  • Add hostname to /etc/hosts
[root@localhost ~]# vi /etc/hosts
127.0.0.1 ecs_syncX localhost localhost.localdomain
  • Configure network
[root@ecs_sync1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
DEVICE=ens33
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.76.246.122
NETMASK=255.255.255.0
GATEWAY=10.76.246.1
DNS1=10.76.246.101
  • Reboot
  • Bring up the interface
[root@localhost ~]# ifup /etc/sysconfig/network-scripts/ifcfg-ens33

Source S3 bucket preparation

In our document ECS test drive is used as a source for data migration. ECS test drive is available for everybody registered here: https://portal.ecstestdrive.com

  • Login to ECS test drive portal
  • Navigate to Manage Secret Keys
  • Get your credentials

ecssync-4

  • Configure source S3 account
  • Specify:
    • Storage Type = S3 Compatible Storage
    • REST Endpoint = ecstestdrive.com
    • Access Key = 130790262785219969@ecstestdrive.emc.com
    • Secret Key1 = wIe2HzvgnrEGhassL+iTPv6wggi+5xxxD
    • Use secure transfer = checked

ecssync-5

  • Check existing buckets
  • Check objects in the bucket you need to migrate

ecssync-6

  • We are going to migrate the sourcebucket bucket
  • Several objects (ECS documentation files) are inside

ecssync-7

Target ECS Preparation

  • Login into ECS using a browser: Default is root:ChangeMe
  • Create the VDC’s, replication group and storage pool if it hasn’t been done so already

ecssync-8Note: All configurations made on ECS will automatically get pushed to the other VDC(s)

  • Create a namespace

ecssync-9

  • Create new object user targetuser
  • Generate S3 password

ecssync-10

  • Create target S3 bucket targetbucket
  • Assign target user as the owner of the bucket

 

Migration with ECSsync

Execute migration

  • Login to ECSsync server
  • Execute ecs-sync utility with the parameters:
    • –source s3:https://130790262785219969@ecstestdrive.emc.com: wIe2HzvgnrEGhassL+iTPv6wggi+5xxxD@ecstestdrive.com

Use format s3:[http[s]://]access_key:secret_key@[host[:port]]

  • –source-legacy-signatures

Note: –source-legacy-signatures is used for S3 v2 authentication used by ECS test drive. That option can be skipped if v2 auth is not required by the specific S3 source.

  • –source-bucket sourcebucket
  • –target ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVxxxDY@76.246.143

Use format ecs-s3:[http[s]://]access_key:secret_key@host-spec

  • –target-bucket targetbucket
[root@ecs_sync1 ecs-sync-2.0]# java -jar ecs-sync-2.0.jar --verbose --source s3:https://130790262785219969@ecstestdrive.emc.com:wIe2HzvgnrEGhassL+iTPv6wggi+5G86YS+5wz3D@object.ecstestdrive.com --source-bucket sourcebucket --source-legacy-signatures --target ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVki9TmxC16JlSWDY@10.76.246.143 --target-bucket targetbucket --source-legacy-signatures
EcsSync v2.0
2016-10-24 01:42:40 WARN [main           ] RestServer: REST server listening at http://localhost:9200/
2016-10-24 01:42:40 INFO [main          ] EcsSync: Sync started at Mon Oct 24 13:42:40 CDT 2016
2016-10-24 01:42:40 INFO [main           ] EcsSync: Configuration Summary:
EcsSync:
- queryThreadCount: 2
- syncThreadCount: 2
- recursive: true
- timingsEnabled: false
- timingWindow: 10000
- rememberFailed: true
- verify: false
- verifyOnly: false
- deleteSource: false- logLevel: verbose
- retryAttempts: 2
- dbService: null
- dbFile: null
- dbConnectString: null
- dbTable: null
- reprocessObjects: false
- splitPoolsThreshold: 0

Source: S3Source:
- metadataOnly: false
- ignoreMetadata: false
- includeAcl: false
- ignoreInvalidAcls: false
- includeRetentionExpiration: false
- force: false
- bufferSize: 262144
- monitorPerformance: false
- sourceUri: s3:https://130790262785219969@ecstestdrive.emc.com:wIe2HzvgnrEGhassL+iTPv6wggi+5G86YS+5wz3D@object.ecstestdrive.com

Target: EcsS3Target:
- metadataOnly: false
- ignoreMetadata: false
- includeAcl: false
- ignoreInvalidAcls: false
- includeRetentionExpiration: false
- force: false
- bufferSize: 262144
- monitorPerformance: false
- targetUri: ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVki9TmxC16JlSWDY@10.76.246.143
- enableVHosts: false
- bucketName: targetbucket
- createBucket: false
- includeVersions: false
- apacheClientEnabled: false
- mpuDisabled: false
- mpuThresholdMB: 512
- mpuPartSizeMB: 128
- mpuThreadCount: 4

2016-10-24 01:42:48 INFO [main           ] EcsSync: syncing from s3:https://130790262785219969@ecstestdrive.emc.com:wIe2HzvgnrEGhassL+iTPv6wggi+5G86YS+5wz3D@object.ecstestdrive.com to ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVki9TmxC16JlSWDY@10.76.246.143
2016-10-24 01:42:48 INFO [Thread-3       ] EcsHostListProvider: adding host 10.76.246.144 to VDC 10.76.246.143
2016-10-24 01:42:48 INFO [Thread-3       ] EcsHostListProvider: adding host 10.76.246.145 to VDC 10.76.246.143
2016-10-24 01:42:48 INFO [Thread-3       ] EcsHostListProvider: adding host 10.76.246.146 to VDC 10.76.246.143
…
S3SyncObject(docu79377_ECS-3.0-Release-Notes.pdf) (178306 bytes transferred)
2016-10-24 01:43:10 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing object S3SyncObject(docu79370_ECS-3.0-Security-Configuration-Guide.pdf) (487555 bytes transferred)
2016-10-24 01:43:11 INFO [main           ] EcsSync: all tasks complete
2016-10-24 01:43:11 INFO [main           ] RestServer: stopping REST server...
Transferred 17,152,723 bytes in 30 seconds (571,757 bytes/s)

Successful files: 8 (0.27/s) Failed Files: 0
Failed files: []

 

Check if migration is successful

Before migration targetbucket on ECS was empty

  • Configure target S3 account
  • Specify:
    • Storage Type = S3 Compatible Storage
    • REST Endpoint = 76.246.143:9020
    • Access Key = targetuser
    • Secret Key1 = 4+V1P1Pr7c2VHtOCPRUpyfgFVxxxDY
    • Use secure transfer = unhecked

 

ecssync-12

  • Check if targetbucket is emty

ecssync-13

After migration targetbucket contains objects

  • Refresh bucket
  • It contains new objects now

ecssync-14

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s