ECS bucket S3 versioning via API

ECS bucket S3 versioning via APIIn that post I play with S3 versioning using s3curl. Major thing to do are: enable versioning, create several versions of an object, delete and restore the latest version, restore one of the non-latest version, enable and check version expiration policy, suspend versioning and clean a bucket.

  • Create a bucket
# ./s3curl.pl --id=ecs_profile -- -X PUT -ks https://10.0.0.1:9021/newbucket10
  • Create a file
# echo "version 1" > file
  • Write the file as a new object
# ./s3curl.pl --id=ecs_profile -- -X PUT -T file -ks https://10.0.0.1:9021/newbucket10/file
  • Check the file
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10/file
version 1
  • Create a config file to enable versioning
# vi EnableVersioning.txt
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Status>Enabled</Status>
</VersioningConfiguration>
  • Enable versioning
# ./s3curl.pl --id=ecs_profile -- -X PUT -d @EnableVersioning.txt -ks https://10.0.0.1:9021/newbucket10?versioning
  • Check if versioning is enabled
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versioning | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Status>Enabled</Status>
</VersioningConfiguration>
  • The file written before versioning was enabled has versionId=null
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Version>
   <Key>file</Key>
   <VersionId>null</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T19:43:16Z</LastModified>
   <ETag>"81127ad129dd2249f5ab0667ca0aeb84"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
</ListVersionsResult>
  • Create and write another version of the object
# echo "version 2" > file

# ./s3curl.pl --id=ecs_profile -- -X PUT -T file -ks https://10.0.0.1:9021/newbucket10/file

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10/file
version 2
  • New version is written with its own versionID
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Version>
   <Key>file</Key>
   <VersionId>1492804701587</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T19:58:21Z</LastModified>
   <ETag>"b04d0b29d4bac0781fe844e328c5e2ac"</ETag>
   <Size>10</Size>
   <Owner>
   <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>null</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:43:16Z</LastModified>
   <ETag>"81127ad129dd2249f5ab0667ca0aeb84"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
</ListVersionsResult>
  • Write one more version of the object
  • It is the latest version of the object now.
# echo "version 3" > file

# ./s3curl.pl --id=ecs_profile -- -X PUT -T file -ks https://10.0.0.1:9021/newbucket10/file

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10/file
version 3

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Version>
   <Key>file</Key>
   <VersionId>1492804816605</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T20:00:16Z</LastModified>
   <ETag>"c0e8cda1c7b12c94d017e634208b828e"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>1492804701587</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:58:21Z</LastModified>
   <ETag>"b04d0b29d4bac0781fe844e328c5e2ac"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>null</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:43:16Z</LastModified>
   <ETag>"81127ad129dd2249f5ab0667ca0aeb84"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
</ListVersionsResult>
  • Let’s delete the latest version of the object
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10/file

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10/file
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>file</Resource><RequestId>0a91c721:15a3bd62ce4:495e6:97</RequestId></Error>#
  • Delete Market is created
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<DeleteMarker>
   <Key>file</Key>
   <VersionId>1492804881806</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T20:01:21Z</LastModified>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
</DeleteMarker>

<Version>
   <Key>file</Key>
   <VersionId>1492804816605</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T20:00:16Z</LastModified>
   <ETag>"c0e8cda1c7b12c94d017e634208b828e"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>1492804701587</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:58:21Z</LastModified>
   <ETag>"b04d0b29d4bac0781fe844e328c5e2ac"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>null</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:43:16Z</LastModified>
   <ETag>"81127ad129dd2249f5ab0667ca0aeb84"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
</ListVersionsResult>

 

  • If we delete the object again, one more Delete Market is created.
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<DeleteMarker>
   <Key>file</Key>
   <VersionId>1492805233454</VersionId>
 <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T20:07:13Z</LastModified>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
</DeleteMarker>

<DeleteMarker>
   <Key>file</Key>
   <VersionId>1492804881806</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T20:01:21Z</LastModified>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
</DeleteMarker>

<Version>
   <Key>file</Key>
   <VersionId>1492804816605</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T20:00:16Z</LastModified>
   <ETag>"c0e8cda1c7b12c94d017e634208b828e"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
...
  • To recover the latest version of the object we have to delete all corresponding Delete Markers.
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10/file?versionId=1492805233454
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10/file?versionId=1492804881806

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10/file
version 3
  • Let’s recover version 2 of the object.
  • We copy the object with specified versionId and make it the latest.
# ./s3curl.pl --id=ecs_profile -- -X PUT -H 'x-amz-copy-source: /newbucket10/file?versionId=1492804701587' -ks https://10.0.0.1:9021/newbucket10/file
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LastModified>2017-04-21T20:12:34.332Z</LastModified><ETag>"b04d0b29d4bac0781fe844e328c5e2ac"</ETag></CopyObjectResult>

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10/file
version 2
  • Delete the object and bucket.
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10/file

# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10 | xmllint --format -
<?xml version="1.0"?>
<Error>
<Code>BucketNotEmpty</Code>
<Message>The bucket you tried to delete is not empty.</Message>
<Resource>newbucket10</Resource>
<RequestId>0a91c715:15a3baf365e:49a0b:11</RequestId>
</Error>
  • Bucket is not empty because all versions are still there.
  • Lets assign 1 day versioning expiration
# vi VersioningExpiration_1day.txt
<LifecycleConfiguration>
   <Rule>
       <ID>expires in a day</ID>
       <Prefix></Prefix>
       <Status>Enabled</Status>
       <Expiration>
           <Days>1000000</Days>
       </Expiration>
       <NoncurrentVersionExpiration>
           <NoncurrentDays>1</NoncurrentDays>
       </NoncurrentVersionExpiration>
   </Rule>

# ./s3curl.pl --id=ecs_profile -- -X PUT -d @VersioningExpiration_1day.txt -ks https://10.0.0.1:9021/newbucket10?lifecycle

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?lifecycle | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
   <ID>expires in a day</ID>
   <Prefix/>
   <Status>Enabled</Status>
   <Expiration>
     <Days>1000000</Days>
   </Expiration>
   <NoncurrentVersionExpiration>
     <NoncurrentDays>1</NoncurrentDays>
   </NoncurrentVersionExpiration>
</Rule>
</LifecycleConfiguration>
  • Before expiration
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<DeleteMarker>
   <Key>file</Key>
   <VersionId>1492805675844</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T20:14:35Z</LastModified>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
</DeleteMarker>

<Version>
   <Key>file</Key>
   <VersionId>1492805554332</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T20:12:34Z</LastModified>
   <ETag>"b04d0b29d4bac0781fe844e328c5e2ac"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>1492804816605</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T20:07:13Z</LastModified>
   <ETag>"c0e8cda1c7b12c94d017e634208b828e"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
<Version>

   <Key>file</Key>
   <VersionId>1492804701587</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:58:21Z</LastModified>
   <ETag>"b04d0b29d4bac0781fe844e328c5e2ac"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>

<Version>
   <Key>file</Key>
   <VersionId>null</VersionId>
   <IsLatest>false</IsLatest>
   <LastModified>2017-04-21T19:43:16Z</LastModified>
   <ETag>"81127ad129dd2249f5ab0667ca0aeb84"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
</ListVersionsResult>
  • 1 day later
  • Only Delete Marker remains.
# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<DeleteMarker>
   <Key>file</Key>
   <VersionId>1492805675844</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T20:14:35Z</LastModified>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
</DeleteMarker>
</ListVersionsResult>
  • Delete the Delete Marker
  • The latest version is still there. It was not influenced by a versions’ lifecycle policy.
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10/file?versionId=1492805675844

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versions | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>newbucket10</Name>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Version>
   <Key>file</Key>
   <VersionId>1492804816605</VersionId>
   <IsLatest>true</IsLatest>
   <LastModified>2017-04-21T20:07:13Z</LastModified>
   <ETag>"c0e8cda1c7b12c94d017e634208b828e"</ETag>
   <Size>10</Size>
   <Owner>
     <ID>newuser</ID>
     <DisplayName>newuser</DisplayName>
   </Owner>
   <StorageClass>STANDARD</StorageClass>
</Version>
</ListVersionsResult>
  • Suspend versioning.
# vi SuspendVersioning.txt
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Status>Suspended</Status>
</VersioningConfiguration>

# ./s3curl.pl --id=ecs_profile -- -X PUT -d @SuspendVersioning.txt -ks https://10.0.0.1:9021/newbucket10?versioning

# ./s3curl.pl --id=ecs_profile -- -ks https://10.0.0.1:9021/newbucket10?versioning |xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Status>Suspended</Status>
</VersioningConfiguration>
  • We can delete the file and the bucket now.
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10/file
# ./s3curl.pl --id=ecs_profile -- -X DELETE -ks https://10.0.0.1:9021/newbucket10

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s