Examples of ECS API usage

S3 API testingIn this post several typical cases how to use API with curl and Advanced REST API Client are presented.

Authenticate with the ECS management REST API and check existing namespaces using curl

ECS uses a token-based authentication system for all its REST API calls. Once a user is authenticated against ECS, an authentication token is returned and can be used to authenticate the user in subsequent calls.

  • An HTTP 401 code is returned if the client is automatically following redirects, indicating that you need to login and authenticate to obtain a new token.
  • An HTTP 302 code is returned if the client is not automatically following redirects.
  • The 302 code directs the client to where to get re-authenticated.

You can retrieve and use authentication tokens by:

  • Saving the X-SDS-AUTH-TOKEN cookie from a successful authentication request and sending that cookie along in subsequent requests.
  • Reading the X-SDS-AUTH-TOKEN HTTP header from a successful authentication request and copying that header into any subsequent request.

The REST API is available on port :4443 and clients access ECS by issuing a login request in the form: https://:4443/login

Authenticate with auth-token

This example shows how to use authentication tokens by reading the X-SDS-AUTH-TOKEN http header from a successful authentication request and copying that header into a subsequent request. This example does not use cookies.

  • The examples here are written in curl and formatted for readability. Curl is a common command-line tool for getting or sending files using URL syntax. It supports both HTTP and HTTPS protocols.
  • Authentication command executes a GET on the /login resource.
  • The -u option indicates the user of basic authentication header. The user designation must be included in the request. Upon successful authentication, a HTTP 200 code is returned as well as the X-SDS-AUTH-TOKEN header containing the encoded token.

Note: Special characters in URL have to be substituted by their codes. So the P@ssw0rd has to be specified as P%40ssw0rd.

testhost$ curl -v -L -k https://root:P%40ssw0rd@10.76.246.143/login
*   Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 443 (#0)
* TLS 1.0 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: localhost
* Server auth using Basic with user 'root'
> GET /login HTTP/1.1
> Host: 10.76.246.143
> Authorization: Basic cm9vdDpQQHNzdzByZA==
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 29 Aug 2016 11:55:39 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 240
< Connection: keep-alive
< Set-Cookie: ECSUI_ERRORS=;Expires=Mon, 29-Aug-2016 11:55:39 GMT;Path=/;Secure;HTTPOnly
< Set-Cookie: ECSUI_SESSION=;Expires=Mon, 29-Aug-2016 11:55:39 GMT;Path=/;Secure;HTTPOnly
< Set-Cookie: XSRF-TOKEN=fa39760b75d47a48d4ab77e2584e1671;Path=/
< Set-Cookie: ECSUI_FLASH=;Expires=Mon, 29-Aug-2016 11:55:39 GMT;Path=/;Secure;HTTPOnly
< Cache-Control: no-cache
<
* Connection #0 to host 10.76.246.143 left intact
BAAcdktiVTRkM1BNUk1lZ2pnK3ZudUJNL2w2MFFzPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46Yjk0N2UzMTItNjAzZS00YTJlLThjYTctMzcyODE3OTc2YTJmAgAC0A8=

The token can then be passed back in the next API call. You can copy the X-SDS-AUTH-TOKEN contents and pass it to the next request through curl’s -H switch. XML output is formatted with xmllint tool.

  • API to list all namespaces is used in the example.
testhost$ curl https://10.76.246.143:4443/object/namespaces -k -H "X-SDS-AUTH-TOKEN: BAAcZVFkT0dsZVo5R2NzYWZ1blk0c2MvWEZPTnlzPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46M2Q0ODMxNWUtYmUxNi00ZDkyLThjZDQtMTJkOTljMDA2ZDIzAgAC0A8=" |xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<namespaces>
 <Filter>name=*</Filter>
 <namespace>
 <id>ns1</id>
 <link rel="self" href="/object/namespaces/namespace/ns1"/>
 <name>ns1</name>
 </namespace>
</namespaces>

Authenticate with cookies

This example shows how to use authentication tokens by saving the cookie from a successful authentication request, then passing the cookie in a subsequent request.

  • In this example, you specify the ?using-cookies=true parameter to indicate that you want to receive cookies in addition to the normal HTTP header. This curl command saves the authentication token to a file named cookiefile in the current directory.

 

testhost$ curl -L -v -k https://10.76.246.143:4443/login?using-cookies=true -u "root:P@ssw0rd" -c cookiefile
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 4443 (#0)
* TLS 1.0 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: localhost
* Server auth using Basic with user 'root'
> GET /login?using-cookies=true HTTP/1.1
> Host: 10.76.246.143:4443
> Authorization: Basic cm9vdDpQQHNzdzByZA==
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 29 Aug 2016 12:00:51 GMT
< Content-Type: application/xml
< Content-Length: 93
< Connection: keep-alive
< X-SDS-AUTH-TOKEN: BAAcd0ZEd0VNVmZ3d3V0YUtadzN2YWhDS0krUlhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46MDYxNDJjMTktYjVjMC00MmIxLWI4MjgtOTdhNzFjMTRlZGNhAgAC0A8=
* Added cookie X-SDS-AUTH-TOKEN="BAAcd0ZEd0VNVmZ3d3V0YUtadzN2YWhDS0krUlhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46MDYxNDJjMTktYjVjMC00MmIxLWI4MjgtOTdhNzFjMTRlZGNhAgAC0A8=" for domain 10.76.246.143, path /, expire 1472500854
< Set-Cookie: X-SDS-AUTH-TOKEN=BAAcd0ZEd0VNVmZ3d3V0YUtadzN2YWhDS0krUlhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46MDYxNDJjMTktYjVjMC00MmIxLWI4MjgtOTdhNzFjMTRlZGNhAgAC0A8=;Version=1;Max-Age=28800;Secure
<
* Connection #0 to host 10.76.246.143 left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><loggedIn><user>root</user></loggedIn>

testhost$ cat cookiefile
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

10.76.246.143 FALSE / TRUE 1472500854 X-SDS-AUTH-TOKEN BAAcd0ZEd0VNVmZ3d3V0YUtadzN2YWhDS0krUlhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46MDYxNDJjMTktYjVjMC00MmIxLWI4MjgtOTdhNzFjMTRlZGNhAgAC0A8= 
  •  The next command passes the cookie with the authentication token through the -b switch, and returns the user’s tenant information.
testhost$ curl -v -k -b cookiefile https://10.76.246.143:4443/object/namespaces | xmllint --format -
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 4443 (#0)
* TLS 1.0 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: localhost
> GET /object/namespaces HTTP/1.1
> Host: 10.76.246.143:4443
> User-Agent: curl/7.43.0
> Accept: */*
> Cookie: X-SDS-AUTH-TOKEN= BAAcd0ZEd0VNVmZ3d3V0YUtadzN2YWhDS0krUlhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmIzYzljYjEwLTVjNmItNGExOC1hYzZmLWE4NWFkZWEwYmQxOQIADTE0NzIzODczNDIzMDADAC51cm46VG9rZW46MDYxNDJjMTktYjVjMC00MmIxLWI4MjgtOTdhNzFjMTRlZGNhAgAC0A8=
>
< HTTP/1.1 200 OK
< Date: Mon, 29 Aug 2016 12:06:19 GMT
< Content-Type: application/xml
< Content-Length: 212
< Connection: keep-alive
<
{ [212 bytes data]
* Connection #0 to host 10.76.246.143 left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<namespaces>
 <Filter>name=*</Filter>
 <namespace>
 <id>ns1</id>
 <link rel="self" href="/object/namespaces/namespace/ns1"/>
 <name>ns1</name>
 </namespace>
</namespaces>

 

Authenticate with the ECS management REST API and check existing namespaces using Postman

The authentication REST API is available on port :4443 and clients access ECS by issuing a login request in the form: https://:4443/login

Free Postman GUI tool is used in the example below to test REST API (getpostman.com).

  • Create authentication request specifying Basic Auth (root/P@ssw0rd)S2 API testing

 

  • The management root is logged in as shown in the response body screenshot.ecs-api-2

 

  • X-SDS-AUTH-TOKEN response header contains the auth token.ecs-api-3

 

  • Specify object/namespaces request and X_SDS_AUTH_TOKEN header.ecs-api-4

 

  • Response header contains the existing namespace.ecs-api-5

 

Create an object user and S3 bucket using using Postman

  • Add Content-type: application/xml headerecs-api-6
  • Request /object/users API call specifying the user name in a body.ecs-api-7
  • Response confirms the successful object user creation.ecs-api-8
  • Request the /object/user-secret-keys/objuser1 API call
  • Specify secret key (object user password) in a body.
    ecs-api-9
  • Response confirms object user secret key creation.

ecs-api-10

  • ECS portal confirms the object user existence and the correct secret key.

ecs-api-11

 

S3 access via s3curl

S3 REST API authentication is dynamic because it uses a custom HTTP scheme based on a keyed-HMAC (Hash Message Authentication Code). For more details, please check the resource: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html.

s3curl is a wrapper around curl, a popular command line http client, that will help us calculating dynamic authentication parameters for the ECS REST API requests.

  • Modify the .s3curl in your home directory (dot in the beginning of the file name means that it is hidden). 600 permissions should be assigned to .s3curl file.
  • Specify object user and password for “ecsid” profile. Specify all ECS nodes as endpoints.
testhost$ vi .s3curl
%awsSecretAccessKeys = (
   # ECS account
   ecsid => {
     id => 'objuser1',
     key => 'sMUE724b3yX8xFSceBiFNBl+7sCL6SFfYRIvmZN+',
   },
   @endpoints = ('10.76.246.146','10.76.246.144','10.76.246.145','10.76.246.143')
);
  • Check authentication using ECS account nickname you specified by ecsid profile in .s3curl file.
  • Bucket1 found.

 

testhost$ ./s3curl.pl --id=ecsid --debug -- -vs http://10.76.246.143:9020/ | xmllint --format -
s3curl: Found the url: host=10.76.246.143; port=9020; uri=/; query=;
s3curl: ordinary endpoint signing case
s3curl: StringToSign='GET\n\n\nMon, 05 Sep 2016 12:14:11 +0000\n/'
s3curl: exec curl -H Date: Mon, 05 Sep 2016 12:14:11 +0000 -H Authorization: AWS objuser1:F5Ai5yB/gTV8eGHb3Ko7uQrldF0= -L -H content-type: -v -s http://10.76.246.143:9020/
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 9020 (#0)
> GET / HTTP/1.1
> Host: 10.76.246.143:9020
> User-Agent: curl/7.43.0
> Accept: */*
> Date: Mon, 05 Sep 2016 12:14:11 +0000
> Authorization: AWS objuser1:F5Ai5yB/gTV8eGHb3Ko7uQrldF0=
>
< HTTP/1.1 200 OK
< Date: Mon, 05 Sep 2016 12:13:58 GMT
< Server: ViPR/1.0
< x-amz-request-id: 0a4cf68f:156f9999cac:54:7
< x-amz-id-2:
< Content-Type: application/xml
< Content-Length: 424
<
{ [424 bytes data]
* Connection #0 to host 10.76.246.143 left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>objuser1</ID>
 <DisplayName>objuser1</DisplayName>
 </Owner>
 <Buckets>
 <Bucket>
 <Name>bucket1</Name>
 <CreationDate>2016-09-05T12:13:54.069Z</CreationDate>
 <ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>
 </Bucket>
 </Buckets>
 <IsTruncated>false</IsTruncated>
</ListAllMyBucketsResult>
  • Upload a new file to the bucket1.
testhost$ ./s3curl.pl --id=ecsid --put=README -- -vs http://10.76.246.143:9020/bucket1/README
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 9020 (#0)
> PUT /bucket1/README HTTP/1.1
> Host: 10.76.246.143:9020
> User-Agent: curl/7.43.0
> Accept: */*
> Date: Mon, 05 Sep 2016 12:19:57 +0000
> Authorization: AWS objuser1:E/S+iFAN2V+oOd0z82ofJDTmuFM=
> Content-Length: 3941
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 05 Sep 2016 12:19:43 GMT
< Server: ViPR/1.0
< x-amz-request-id: 0a4cf68f:156f9999cac:5d:1
< x-amz-id-2: 77ee43c70cc786ab81acd0b570740805a6c759f3e79df2ebf06ea69ff4b0f75b
< ETag: "5e714348185ffe355a76b754f79176d6"
< Last-Modified: Mon, 05 Sep 2016 12:19:43 GMT
< x-emc-mtime: 1473077983868
< x-emc-previous-object-size: 0
< x-amz-copy-source-version-id:
< Content-Length: 0
<
* Connection #0 to host 10.76.246.143 left intact
  • The file README is uploaded in to the bucket.
testhost$ ./s3curl.pl --id=ecsid -- -v -s http://10.76.246.143:9020/bucket1/ |xmllint --format -
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 9020 (#0)
> GET /bucket1/ HTTP/1.1
> Host: 10.76.246.143:9020
> User-Agent: curl/7.43.0
> Accept: */*
> Date: Mon, 05 Sep 2016 12:20:43 +0000
> Authorization: AWS objuser1:4ARYmmqAeNYuxbUQfxtPjCids+I=
>
< HTTP/1.1 200 OK
< Date: Mon, 05 Sep 2016 12:20:30 GMT
< Server: ViPR/1.0
< x-amz-request-id: 0a4cf68f:156f9999cac:a9:3
< x-amz-id-2: 6ff76792d8e34a321dc27cdff7f5293172f358f338cc8e1c805ff923ba76385f
< x-emc-retention-period: 0
< Content-Type: application/xml
< Content-Length: 568
<
{ [568 bytes data]
* Connection #0 to host 10.76.246.143 left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Name>bucket1</Name>
 <Prefix/>
 <Marker/>
 <MaxKeys>1000</MaxKeys>
 <IsTruncated>false</IsTruncated>
 <ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>
 <Contents>
 <Key>README</Key>
 <LastModified>2016-09-05T12:19:43Z</LastModified>
 <ETag>"5e714348185ffe355a76b754f79176d6"</ETag>
 <Size>3941</Size>
 <StorageClass>STANDARD</StorageClass>
 <Owner>
 <ID>objuser1</ID>
 <DisplayName>objuser1</DisplayName>
 </Owner>
 </Contents>
</ListBucketResult>
  • Prepare .xml file to describe all files to be deleted.
testhost$ vi delete.xml
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
 
 README
 
</Delete>

Note: Multiply <Object> sections can be specified.

  • Delete the file README.
  • We have to calculate base64 encoded MD5 sum of POST body.
testhost$ ./s3curl.pl --id=ecsid --post delete.xml --contentMd5 `cat delete.xml | openssl dgst -md5 -binary | base64` -- -v -s http://10.76.246.143:9020/bucket1?delete |xmllint --format -
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 9020 (#0)
> POST /bucket1?delete HTTP/1.1
> Host: 10.76.246.143:9020
> User-Agent: curl/7.43.0
> Accept: */*
> Date: Mon, 05 Sep 2016 12:43:09 +0000
> Authorization: AWS objuser1:CvciVEgcLHDn1R5f0xWS1M3+rHE=
> Content-MD5: ynD9DKJIcR6AphBmHc6jlw==
> Content-Length: 112
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
} [112 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Mon, 05 Sep 2016 12:42:56 GMT
< Server: ViPR/1.0
< x-amz-request-id: 0a4cf68f:156f9999cac:ba:2
< x-amz-id-2: 6ff76792d8e34a321dc27cdff7f5293172f358f338cc8e1c805ff923ba76385f
< Content-Type: application/xml
< Content-Length: 202
<
{ [202 bytes data]
* Connection #0 to host 10.76.246.143 left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Deleted>
 <Key>README</Key>
 <DeleteMarker>false</DeleteMarker>
 </Deleted>
</DeleteResult>
  • File README is deleted from the bucket1.

 

testhost$ ./s3curl.pl --id=ecsid -- -v -s http://10.76.246.143:9020/bucket1/ |xmllint --format -
* Trying 10.76.246.143...
* Connected to 10.76.246.143 (10.76.246.143) port 9020 (#0)
> GET /bucket1/ HTTP/1.1
> Host: 10.76.246.143:9020
> User-Agent: curl/7.43.0
> Accept: */*
> Date: Mon, 05 Sep 2016 12:43:52 +0000
> Authorization: AWS objuser1:eaDGJ2f/zcmDmEu/BZ/ySoW1Rso=
>
< HTTP/1.1 200 OK
< Date: Mon, 05 Sep 2016 12:43:39 GMT
< Server: ViPR/1.0
< x-amz-request-id: 0a4cf68f:156f9999cac:bb:3
< x-amz-id-2: 6ff76792d8e34a321dc27cdff7f5293172f358f338cc8e1c805ff923ba76385f
< x-emc-retention-period: 0
< Content-Type: application/xml
< Content-Length: 313
<
{ [313 bytes data]
* Connection #0 to host 10.76.246.143 left intact
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Name>bucket1</Name>
 <Prefix/>
 <Marker/>
 <MaxKeys>1000</MaxKeys>
 <IsTruncated>false</IsTruncated>
 <ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>
</ListBucketResult>

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: