ECS-sync is an external tool developed by the Object EMC Development team. It supports Centera, Atmos, S3, file to CAS, Atmos, S3, file migration. In this post I’m focused on S3 to S3 migration.
ECS-sync supports the following functionality:
- Migration Pause, Resume, Stop
- Thread control
- Clip list only
- Logging including the SDK error and timestamp.
- Retry of a failed clip
- Mysql Database to log all migration transactions
- Email home
- ECS-sync will resume where it left off if the tool stops as long as the DB table it was populating is still present.
- Multiple sync servers can be deployed; rule of thumb is 1 sync server for every 2 Centera access nodes. Best practice is try not go over 4 servers per site
- ECS-sync copies and validates the MD5 after it is on a target
Note: Validation prevents having to perform a long reconciliation at the end of the migration. After the migration, we only have work on the errors.
Documentation and CLI
Some ECS-sync description and CLI syntax have to be checked here
Sample configuration XML files for ECS-sync 2.1 are available here
Plugins and filters
ECS-sync is a Java application that joins 3rd-party storage clients with an abstract “sync” framework. When we migrate data, we “sync” it.
The framework consists of the abstract notion of a sync object and a set of sync plugins.
For any given sync operation, source plugin, target plugin, and optional object filters have to be defined. A source plugin iterates with objects on the source storage. A target is responsible for replicating object to a target storage. Sometimes source attributes cannot be transferred or represented on target. However, target plugin is trying to do its best to create objects as close as possible to a source.
Each plugin is configured using a set of common and custom properties. Before the sync operation is started, all of the plugins configure themselves and verify any compatibility issues with other plugins in a source-target chain.
Plugins are loaded dynamically based on command line parameters. Each plugin is registered and knows what kind of URL it can process (i.e. “s3://” is processed by S3 plugin).
Object filters transform the objects or collect information about the objects as they are processed. Filters are activated by listing the filter name in the comma-delimited –filters parameter.
To use ECS-sync you need to provide a migration server. The migration server is the machine on which ECS-sync will run. All migration traffic goes through this server. Therefore you should have high bandwidth network between ECS-sync and both the source and target systems .
There are two option to install ECS-sync:
- vApp OVA bundle
If the customer cannot support the vApp deployment, a manual installation can be performed on any OS that supports Java 7+. Only one instance of the sync tool can run on a single server.
The link to get the latest version of ECS-sync for manual installation is here
- 16B RAM
- 200 GB disk
- Single NIC at 1GB or 10GB
- Quad core
- Up to 4 VM’s per site can be deployed for the migration
Please use this link to download the latest version of OVA bundle.
The OVA for ECS-sync 2.1 contains the following:
- Scripts to clean a clip list
- Script to create database tables
- CCV verify
- SDK binaries
- ecs_sync 2.0 jar file
- ECS-sync run script
Deploy and Configure the OVA:
- Upload the OVA to a datastore
- From a vSphere client connected to either an ESX host or VCenterselect select File > Deploy OVF Template
- Power on the VM
- Open a Virtual Machine Console to the VM
- Login is root:ECSSync
- Change the hostname to ecs_sync1, or whatever you want
[root@localhost ~]# vi /etc/hostname
- Add hostname to /etc/hosts
[root@localhost ~]# vi /etc/hosts 127.0.0.1 ecs_syncX localhost localhost.localdomain
- Configure network
[root@ecs_sync1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33 DEVICE=ens33 ONBOOT=yes BOOTPROTO=static IPADDR=10.76.246.122 NETMASK=255.255.255.0 GATEWAY=10.76.246.1 DNS1=10.76.246.101
- Bring up the interface
[root@localhost ~]# ifup /etc/sysconfig/network-scripts/ifcfg-ens33
Source S3 bucket preparation
In our document ECS test drive is used as a source for data migration. ECS test drive is available for everybody registered here: https://portal.ecstestdrive.com
- Login to ECS test drive portal
- Navigate to Manage Secret Keys
- Get your credentials
- Configure source S3 account
- Storage Type = S3 Compatible Storage
- REST Endpoint = ecstestdrive.com
- Access Key = email@example.com
- Secret Key1 = wIe2HzvgnrEGhassL+iTPv6wggi+5xxxD
- Use secure transfer = checked
- Check existing buckets
- Check objects in the bucket you need to migrate
- We are going to migrate the sourcebucket bucket
- Several objects (ECS documentation files) are inside
Target ECS Preparation
- Login into ECS using a browser: Default is root:ChangeMe
- Create the VDC’s, replication group and storage pool if it hasn’t been done so already
Note: All configurations made on ECS will automatically get pushed to the other VDC(s)
- Create a namespace
- Create new object user targetuser
- Generate S3 password
- Create target S3 bucket targetbucket
- Assign target user as the owner of the bucket
Migration with ECSsync
- Login to ECSsync server
- Execute ecs-sync utility with the parameters:
- –source s3:https://firstname.lastname@example.org: wIe2HzvgnrEGhassL+iTPv6wggi+5xxxD@ecstestdrive.com
Use format s3:[http[s]://]access_key:secret_key@[host[:port]]
Note: –source-legacy-signatures is used for S3 v2 authentication used by ECS test drive. That option can be skipped if v2 auth is not required by the specific S3 source.
- –source-bucket sourcebucket
- –target ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVxxxDY@76.246.143
Use format ecs-s3:[http[s]://]access_key:secret_key@host-spec
- –target-bucket targetbucket
[root@ecs_sync1 ecs-sync-2.0]# java -jar ecs-sync-2.0.jar --verbose --source s3:https://email@example.com:wIe2HzvgnrEGhassL+iTPv6wggi+5G86YS+5wz3D@object.ecstestdrive.com --source-bucket sourcebucket --source-legacy-signatures --target ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVki9TmxC16JlSWDY@10.76.246.143 --target-bucket targetbucket --source-legacy-signatures EcsSync v2.0 2016-10-24 01:42:40 WARN [main ] RestServer: REST server listening at http://localhost:9200/ 2016-10-24 01:42:40 INFO [main ] EcsSync: Sync started at Mon Oct 24 13:42:40 CDT 2016 2016-10-24 01:42:40 INFO [main ] EcsSync: Configuration Summary: EcsSync: - queryThreadCount: 2 - syncThreadCount: 2 - recursive: true - timingsEnabled: false - timingWindow: 10000 - rememberFailed: true - verify: false - verifyOnly: false - deleteSource: false- logLevel: verbose - retryAttempts: 2 - dbService: null - dbFile: null - dbConnectString: null - dbTable: null - reprocessObjects: false - splitPoolsThreshold: 0 Source: S3Source: - metadataOnly: false - ignoreMetadata: false - includeAcl: false - ignoreInvalidAcls: false - includeRetentionExpiration: false - force: false - bufferSize: 262144 - monitorPerformance: false - sourceUri: s3:https://firstname.lastname@example.org:wIe2HzvgnrEGhassL+iTPv6wggi+5G86YS+5wz3D@object.ecstestdrive.com Target: EcsS3Target: - metadataOnly: false - ignoreMetadata: false - includeAcl: false - ignoreInvalidAcls: false - includeRetentionExpiration: false - force: false - bufferSize: 262144 - monitorPerformance: false - targetUri: ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVki9TmxC16JlSWDY@10.76.246.143 - enableVHosts: false - bucketName: targetbucket - createBucket: false - includeVersions: false - apacheClientEnabled: false - mpuDisabled: false - mpuThresholdMB: 512 - mpuPartSizeMB: 128 - mpuThreadCount: 4 2016-10-24 01:42:48 INFO [main ] EcsSync: syncing from s3:https://email@example.com:wIe2HzvgnrEGhassL+iTPv6wggi+5G86YS+5wz3D@object.ecstestdrive.com to ecs-s3:http://targetuser:4+V1P1Pr7c2VHtOCPRUpyfgFVki9TmxC16JlSWDY@10.76.246.143 2016-10-24 01:42:48 INFO [Thread-3 ] EcsHostListProvider: adding host 10.76.246.144 to VDC 10.76.246.143 2016-10-24 01:42:48 INFO [Thread-3 ] EcsHostListProvider: adding host 10.76.246.145 to VDC 10.76.246.143 2016-10-24 01:42:48 INFO [Thread-3 ] EcsHostListProvider: adding host 10.76.246.146 to VDC 10.76.246.143 … S3SyncObject(docu79377_ECS-3.0-Release-Notes.pdf) (178306 bytes transferred) 2016-10-24 01:43:10 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing object S3SyncObject(docu79370_ECS-3.0-Security-Configuration-Guide.pdf) (487555 bytes transferred) 2016-10-24 01:43:11 INFO [main ] EcsSync: all tasks complete 2016-10-24 01:43:11 INFO [main ] RestServer: stopping REST server... Transferred 17,152,723 bytes in 30 seconds (571,757 bytes/s) Successful files: 8 (0.27/s) Failed Files: 0 Failed files: 
Check if migration is successful
Before migration targetbucket on ECS was empty
- Configure target S3 account
- Storage Type = S3 Compatible Storage
- REST Endpoint = 76.246.143:9020
- Access Key = targetuser
- Secret Key1 = 4+V1P1Pr7c2VHtOCPRUpyfgFVxxxDY
- Use secure transfer = unhecked
- Check if targetbucket is emty
After migration targetbucket contains objects
- Refresh bucket
- It contains new objects now