Testing ecs-sync

ECS-sync AtmosBasic tests of very powerful tool ECS-sync.

Distribution and docs

2.1.2 version was used in my tests

 

Copy files from directory to S3 bucket

  • Lets create a target object userecs-sync
  • Copy files from the local directory to S3 bucket
  • Create the TargetBucket automatically
  • Define the number of threads = 10
host:ecs-sync-2.1.2 pantyv$ java -jar ecs-sync-2.1.2.jar --verbose --source file:///Users/pantyv/Work/TestData --target ecs-s3:http://targetuser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143 --target-create-bucket --target-bucket TargetBucket --sync-threads 10
EcsSync v2.1.2
2016-05-19 06:42:37 WARN [main ] RestServer: REST server listening at http://localhost:9200/
2016-05-19 06:42:37 INFO [main ] EcsSync: Sync started at Thu May 19 18:42:37 MSK 2016
2016-05-19 06:42:37 INFO [main ] EcsSync: Configuration Summary:
EcsSync:
- queryThreadCount: 2
- syncThreadCount: 10
- recursive: true
- timingsEnabled: false
- timingWindow: 10000
- rememberFailed: true
- verify: false
- verifyOnly: false
- deleteSource: false
- logLevel: verbose
- retryAttempts: 2
- dbService: null
- dbFile: null
- dbConnectString: null
- dbTable: null
- reprocessObjects: false
- splitPoolsThreshold: 0
Source: FilesystemSource:
- metadataOnly: false
- ignoreMetadata: false
- includeAcl: false
- ignoreInvalidAcls: false
- includeRetentionExpiration: false
- force: false
- bufferSize: 262144
- monitorPerformance: false
- sourceUri: file:///Users/pantyv/Work/TestData
Target: EcsS3Target:
- metadataOnly: false
- ignoreMetadata: false
- includeAcl: false
- ignoreInvalidAcls: false
- includeRetentionExpiration: false
- force: false
- bufferSize: 262144
- monitorPerformance: false
- targetUri: ecs-s3:http://targetuser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143
- enableVHosts: false
- smartClientEnabled: true
- bucketName: TargetBucket
- createBucket: true
- includeVersions: false
- apacheClientEnabled: false
- mpuDisabled: false
- mpuThresholdMB: 512
- mpuPartSizeMB: 128
- mpuThreadCount: 4
 
2016-05-19 06:42:39 INFO [main ] EcsSync: syncing from file:///Users/pantyv/Work/TestData to ecs-s3:http://targetuser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143
2016-05-19 06:42:39 INFO [Thread-3 ] EcsHostListProvider: adding host 10.76.246.144 to VDC 10.76.246.143
2016-05-19 06:42:39 INFO [Thread-3 ] EcsHostListProvider: adding host 10.76.246.145 to VDC 10.76.246.143
2016-05-19 06:42:39 INFO [Thread-3 ] EcsHostListProvider: adding host 10.76.246.146 to VDC 10.76.246.143
2016-05-19 06:42:39 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing directory FileSyncObject()
2016-05-19 06:42:40 INFO [sync-pool-t-3 ] EcsSync: O--O finished syncing object FileSyncObject(file5.pl) (11849 bytes transferred)
2016-05-19 06:42:40 INFO [sync-pool-t-5 ] EcsSync: O--O finished syncing directory FileSyncObject(Folder2_1)
2016-05-19 06:42:40 INFO [sync-pool-t-8 ] EcsSync: O--O finished syncing object FileSyncObject(Folder2_1/file3.txt) (3941 bytes transferred)
2016-05-19 06:42:40 INFO [sync-pool-t-6 ] EcsSync: O--O finished syncing object FileSyncObject(Folder1_1/file1.txt) (9140 bytes transferred)
2016-05-19 06:42:40 INFO [sync-pool-t-2 ] EcsSync: O--O finished syncing object FileSyncObject(.DS_Store) (6148 bytes transferred)
2016-05-19 06:42:41 INFO [sync-pool-t-4 ] EcsSync: O--O finished syncing directory FileSyncObject(Folder1_1)
2016-05-19 06:42:41 INFO [sync-pool-t-7 ] EcsSync: O--O finished syncing object FileSyncObject(Folder1_1/file2.txt) (117 bytes transferred)
2016-05-19 06:42:42 INFO [sync-pool-t-9 ] EcsSync: O--O finished syncing object FileSyncObject(Folder2_1/file4.zip) (9758 bytes transferred)
2016-05-19 06:42:43 INFO [main ] EcsSync: all tasks complete
2016-05-19 06:42:43 INFO [main ] RestServer: stopping REST server...
Transferred 40,953 bytes in 5 seconds (8,190 bytes/s)
Successful files: 9 (1.8/s) Failed Files: 0
Failed files: []
  • Check the created TargetBucket and uploaded files with Cyberduck
  • Folders’ structure is keptw2
  • Interesting that despite of the fact that in the command I defined only the 1st node, ecs-sync recognized all 4 existing ECS nodes.
  • Lets check what happen underneath using with wireshark
  • The first request asked for the list of all endpointsw3
  • The list of all nodes is reported backw4 
  • All requests are load balanced noww5
  • Option –source-no-smart-client disables the client-side load balancing. That is necessary when using a proxy or external load balancer without DNS configuration.

Note: the phrase “external load balancer without DNS configuration” is from the ecs-sync help documentation. Honestly I don’t understand it. Please explain if you know why DNS config of LB is important here?

w6

 w7

  • If I re-execute the same replication command it will not update already uploaded files cause ecs-sync recognizes that the target files are newer than source files.
2016-05-19 08:03:23 INFO  [sync-pool-t-2  ] EcsS3Target: Target newer than source.  Skipping .DS_Store
2016-05-19 08:03:23 INFO  [sync-pool-t-2  ] EcsSync: O--O finished syncing object FileSyncObject(.DS_Store) (0 bytes transferred)
2016-05-19 08:03:23 INFO  [sync-pool-t-6  ] EcsS3Target: Target newer than source.  Skipping Folder1_1/file1.txt

Transferred 0 bytes in 3 seconds (0 bytes/s)
Successful files: 9 (3/s) Failed Files: 0
  • –force option to force overwriting target objects even if they are the same or newer than the source.
2016-05-19 08:54:40 INFO  [sync-pool-t-1  ] EcsSync: O--O finished syncing directory FileSyncObject()
2016-05-19 08:54:41 INFO  [sync-pool-t-4  ] EcsSync: O--O finished syncing directory FileSyncObject(Folder1_1)
2016-05-19 08:54:41 INFO  [sync-pool-t-8  ] EcsSync: O--O finished syncing object

 

Copy from S3 bucket to another S3 bucket

  • Create the SourceUserw8
  • Create the SourceBucketw9 
  • Upload a file to the SourceBucketw10
  • Lets copy from one to another bucket
  • Different object users are used to get access to the source and target buckets
host:ecs-sync-2.1.2 pantyv$ java -jar ecs-sync-2.1.2.jar --verbose --source ecs-s3:http://sourceuser:LKYyjpEXBYBqyYbWpx/Xc+tn1PKOG24T+DzwgQdz@10.76.246.143 --source-bucket SourceBucket --target ecs-s3:http://targetuser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143 --target-bucket TargetBucket
EcsSync v2.1.2

2016-05-19 09:55:14 INFO  [main           ] EcsSync: syncing from ecs-s3:http://sourceuser:LKYyjpEXBYBqyYbWpx/Xc+tn1PKOG24T+DzwgQdz@10.76.246.143 to ecs-s3:http://targetuser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143
2016-05-19 09:55:56 INFO  [sync-pool-t-1  ] EcsSync: O--O finished syncing object file10.jar (1163150 bytes transferred)
2016-05-19 09:55:56 INFO  [main           ] EcsSync: all tasks complete
2016-05-19 09:55:56 INFO  [main           ] RestServer: stopping REST server...
Transferred 1,163,150 bytes in 43 seconds (27,050 bytes/s)
Successful files: 1 (0.02/s) Failed Files: 0
Failed files: []
  • Upload is successful

w11

Copy files to Atmos (namespace API)

  • Create the Atmos subtenant
host:ecs-sync-2.1.2 pantyv$ ./create_subtenant.sh
+++++++++++++++++++++++++++++++++++++++++++++++CREATE SUBTENANT++++++++++++++++++++++++++++++++++++++++++++++++++++
* Trying 10.76.246.143...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Connected to 10.76.246.143 (10.76.246.143) port 9022 (#0)
> PUT /rest/subtenant HTTP/1.1
> Host: 10.76.246.143:9022
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type:text/plain
> x-emc-date:Thu, 19 May 2016 19:16:05 GMT
> x-emc-uid:TargetUser
> x-emc-signature:d95OusW1BFFqmYBSejzDR++aHTI=
>
< HTTP/1.1 204 No Content
< subtenantID: 5cd21ab046d34dbebfa5fa004691e3a8
< x-emc-request-id: 0a4cf68f:154c2433ca2:11a1:3
< Server: Jetty(7.6.4.v20120524)
<
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host 10.76.246.143 left intact
Sub tenant id is...
5cd21ab046d34dbebfa5fa004691e3a8
UID string is:
5cd21ab046d34dbebfa5fa004691e3a8/TargetUser
  • Subtenant is createdw12 
  • Lets configure access in AtmosChromew13
  • Namespace view
  • Atmos is empty noww14
  • Lets copy files from the folder to Atmos
  • The URI for the synchronisation has to be in the format atmos:http[s]://uid:secret@host[,host..][:port][/namespace-path]
  • The uid should be the full token ID including the subtenant ID and the user name concatenated by a slash
  • If you want to load balance across multiple hosts, you can provide a comma-delimited list of hostnames or IPs in the host part of the URI.
host:ecs-sync-2.1.2 pantyv$ java -jar ecs-sync-2.1.2.jar --verbose --source file:///Users/pantyv/Work/TestData --target atmos:http://5cd21ab046d34dbebfa5fa004691e3a8/TargetUser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143:9022/Folder1/
EcsSync v2.1.2
…
2016-05-19 11:15:37 INFO [main ] AtmosTarget: Connected to Atmos 2.0.1 on [http://10.76.246.143:9022]
2016-05-19 11:15:37 INFO [main ] EcsSync: syncing from file:///Users/pantyv/Work/TestData to atmos:http://5cd21ab046d34dbebfa5fa004691e3a8/TargetUser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143:9022/Folder1/
2016-05-19 11:15:38 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing directory FileSyncObject()
2016-05-19 11:15:39 INFO [sync-pool-t-2 ] EcsSync: O--O finished syncing object FileSyncObject(.DS_Store) (6148 bytes transferred)
2016-05-19 11:15:39 INFO [sync-pool-t-2 ] EcsSync: O--O finished syncing directory FileSyncObject(Folder1_1)
2016-05-19 11:15:39 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing object FileSyncObject(file5.pl) (11849 bytes transferred)
2016-05-19 11:15:40 INFO [sync-pool-t-2 ] EcsSync: O--O finished syncing directory FileSyncObject(Folder2_1)
2016-05-19 11:15:40 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing object FileSyncObject(Folder2_1/file3.txt) (3941 bytes transferred)
2016-05-19 11:15:40 INFO [sync-pool-t-2 ] EcsSync: O--O finished syncing object FileSyncObject(Folder1_1/file1.txt) (9140 bytes transferred)
2016-05-19 11:15:41 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing object FileSyncObject(Folder2_1/file4.zip) (9758 bytes transferred)
2016-05-19 11:15:41 INFO [sync-pool-t-2 ] EcsSync: O--O finished syncing object FileSyncObject(Folder1_1/file2.txt) (117 bytes transferred)
2016-05-19 11:15:41 INFO [main ] EcsSync: all tasks complete
2016-05-19 11:15:41 INFO [main ] RestServer: stopping REST server...
Transferred 40,953 bytes in 5 seconds (8,190 bytes/s)
Successful files: 9 (1.8/s) Failed Files: 0
Failed files: []
  • Files successfully uploadedw15

 

Copy from S3 bucket to Atmos (object API)

  • Object view is empty noww16
  • By default, objects will be written to Atmos using the object API unless namespace-path is specified.
  • Use –debug option to see the Atmos object ID
host:ecs-sync-2.1.2 pantyv$ java -jar ecs-sync-2.1.2.jar --debug --source ecs-s3:http://sourceuser:LKYyjpEXBYBqyYbWpx/Xc+tn1PKOG24T+DzwgQdz@10.76.246.143 --source-bucket SourceBucket --target atmos:http://5cd21ab046d34dbebfa5fa004691e3a8/TargetUser:hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA@10.76.246.143:9022
EcsSync v2.1.2
…
2016-05-20 12:49:36 DEBUG [main ] headers: >> GET /rest/service HTTP/1.1
2016-05-20 12:49:36 DEBUG [main ] headers: >> Date: Thu, 19 May 2016 21:49:36 UTC
2016-05-20 12:49:36 DEBUG [main ] headers: >> x-emc-date: Thu, 19 May 2016 21:49:36 UTC
2016-05-20 12:49:36 DEBUG [main ] headers: >> x-emc-uid: 5cd21ab046d34dbebfa5fa004691e3a8/TargetUser
2016-05-20 12:49:36 DEBUG [main ] headers: >> x-emc-signature: 81PFRaNNnUgOBWeMbqzmBSbYL3o=
2016-05-20 12:49:36 DEBUG [main ] headers: >> Host: 10.76.246.143:9022
2016-05-20 12:49:36 DEBUG [main ] headers: >> Connection: Keep-Alive
2016-05-20 12:49:36 DEBUG [main ] headers: >> User-Agent: Apache-HttpClient/4.4.1 (Java/1.8.0_92)
…
2016-05-20 12:50:09 DEBUG [sync-pool-t-1 ] AtmosTarget: Wrote source SourceBucket/file10.jar to dest a55bd0b621f02421f7f882ca6655cd1026bd7a366137c9fd140076dfd0af1fd3
2016-05-20 12:50:09 INFO [sync-pool-t-1 ] EcsSync: O--O finished syncing object file10.jar (1163150 bytes transferred)
2016-05-20 12:50:10 INFO [main ] EcsSync: all tasks complete
2016-05-20 12:50:10 DEBUG [main ] SmartClientFactory: terminating polling daemon
2016-05-20 12:50:10 DEBUG [main ] SmartClientFactory: destroying host list provider
2016-05-20 12:50:10 DEBUG [main ] SmartClientFactory: destroying Jersey client
2016-05-20 12:50:10 INFO [main ] RestServer: stopping REST server...
Transferred 1,163,150 bytes in 35 seconds (33,232 bytes/s)
Successful files: 1 (0.03/s) Failed Files: 0
Failed files: []
  • Find the object by specifying its ID

w17

  • Object is foundw18

 

Execute a run.sh script with .xml configuration

  • Prepare config file
  • Use the .xml files located in the samples directory as templates
  • In that example I configured copying of files from directory to S3 bucke
host:ecs-sync-2.1.2 pantyv$ vi directory-to-ecs-s3.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
 
<!-- This is the required 'root' bean that will bootstrap the operation -->
<bean name="sync" class="com.emc.ecs.sync.EcsSync">
<property name="source" ref="source" />
<property name="target" ref="target" />
 
<!-- database file -->
<property name="dbService">
<bean class="com.emc.ecs.sync.service.SqliteDbService">
<constructor-arg value="files-to-ecs.db" />
</bean>
</property>
 
<!-- adjust threading to increase/throttle performance -->
<property name="queryThreadCount" value="16" />
<property name="syncThreadCount" value="32" />
<property name="rememberFailed" value="false" />
 
<!-- adjust logging output (debug, verbose, quiet, silent) - default is quiet -->
 <property name="logLevel" value="verbose" />
 
<!-- uncomment if the source files are still changing and you want to run incremental syncs -->
<!--property name="reprocessObjects" value="true" /-->
</bean>
 
 <!-- The source -->
<bean name="source" class="com.emc.ecs.sync.source.FilesystemSource">
 <property name="rootFile" value="/Users/pantyv/Work/TestData/" />
<property name="monitorPerformance" value="true" />
</bean>
 
 <!-- The target -->
<bean name="target" class="com.emc.ecs.sync.target.EcsS3Target">
<property name="protocol" value="http"/>
<property name="vdcs"><list>
<bean class="com.emc.rest.smart.ecs.Vdc"><constructor-arg><array value-type="java.lang.String">
<value>10.76.246.143</value>
<value>10.76.246.144</value>
<value>10.76.246.145</value>
<value>10.76.246.146</value>
</array></constructor-arg></bean>
</list></property>
 <property name="port" value="9020" />
 <property name="accessKey" value="targetuser" />
 <property name="secretKey" value="hECPQbdJylWH6dMzGy3FEcw82B7+dim8nr4uOyFA" />
 <property name="bucketName" value="TargetBucket" />
<property name="createBucket" value="false" />
<property name="monitorPerformance" value="true" />
</bean>
</beans>
  • Simple execute the run.sh script
host:ecs-sync-2.1.2 pantyv$ ./run.sh directory-to-ecs-s3.xml
EcsSync v2.1.2

Successful files: 9 (9/s) Failed Files: 0
Failed files: []

 

Execute and schedule in UI (didn’t work)

  • UI (v2.1.2) starts fine, configuration is stored in S3 bucket
  • BUT scheduled synchronisation just does NOT work
  • Debug mode doesn’t provide any detail. Why?
  • Please let me know if it works for youw20

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s