Simple empty bucket scripts

empty-bucket-scriptsSometimes we need to delete all data in a S3 bucket or Swift container. Below are the examples of scripts to do that.

Make empty S3 bucket script

#! /bin/bash
# That is a script to test S3
echo "--- Clean S3 bucket ---"

# Specify s3curl directory here
export S3curl_DIR=~/Work/s3curl/
cd $S3curl_DIR

# S3 credentials are in the .s3curl file
# Specify s3curl profile id here
export ID=ecsid

# Specify ECS or Load Balancer IP here
export IP=10.10.10.10

# Specify Namespace and Bucket here
export BUCKET=bucket1

# Specify Protocol and Port here
# Use http/9020 or https/9021 for protocol/port
export PROTOCOL=http
export PORT=9020

# URL is defined here
STORAGE_URL=$PROTOCOL://$IP:9020/$BUCKET
echo "URL is: " $STORAGE_URL

echo "Generation list of S3 objects... it takes a while..."
./s3curl.pl --id=ecsid -- -s $STORAGE_URL/|xmllint --format -| grep Key | grep -v MaxKeys |awk -F "[<>]" '{print $3}' > s3_files.lst

file_number=`cat s3_files.lst|wc -l`
let file_number_all=file_number
echo "We have $file_number_all S3 objects stored"

if [ "$file_number" -eq 999 ]
then echo "S3curl returns list of objects in groups of 999 objects. If you bucket have more objects, several executions of the script would be needed !!!"
fi

echo "Deleting object..."

let file_number=file_number+1

while read line;
do
let file_number=file_number-1
echo "Deleting the object # $file_number $line"
nospaceline=${line// /"%20"}
./s3curl.pl --id=ecsid -- -ks -X DELETE $STORAGE_URL/$nospaceline
done < s3_files.lst

echo "--- $file_number_all S3 objects were deleted ---"

Note: the script can be optimised using Delete Multi-object API call.

.

.

Make empty Swift container script

#! /bin/bash
# That is a script to test Swift
echo "--- Clean Swift container ---"

# Scpecify ECS or Load Balancer IP
export IP=10.10.10.10

# Specify Swift credentials here
export USER=objuser1
export PASSWORD=123456

# Specify testing Namespace and Bucket here
export NAMESPACE=ns1
export BUCKET=container1

# Specify Protocol and Port here
# Use http/9024 or https/9025 for protocol/port
export PROTOCOL=https
export PORT=9025

# Receive Authentication Token
export TOKEN=`curl -ski -H "X-Auth-User: $USER" -H "X-Auth-Key:$PASSWORD" $PROTOCOL://$IP:$PORT/auth/v1.0 | grep -i X-Auth-Token | awk '{ print $2 }' | tr -d '\015'`
echo "Authenticated with: " $TOKEN

# URL is defined here
export STORAGE_URL=$PROTOCOL://$IP:$PORT/v1/$NAMESPACE/$BUCKET
echo "URL is: " $STORAGE_URL

echo "Generation list of Swift objects... it takes a while..."
curl -ks --connect-timeout 600 -H "X-Auth-Token:$TOKEN" $STORAGE_URL/ > swift_files.lst

file_number=`cat swift_files.lst|wc -l`
let file_number_all=file_number
echo "We have" $file_number_all "Swift objects stored"
echo "Deleting all of them..."

let file_number=file_number+1
while read line;
do
let file_number=file_number-1
echo "Deleting the object # $file_number $line"
nospaceline=${line// /"%20"}
curl -ks -X DELETE -H "X-Auth-Token:$TOKEN" $STORAGE_URL/$nospaceline
done < swift_files.lst

echo "--- $file_number_all Swift objects were deleted ---"

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s