Tag: linux

Description

A bash script will be used to copy a file from a Linux server to an S3 bucket. Next, it will run a checksum on the results to verify the upload. Finally, it will output the local file size, the local etag , the aws file size, and the aws etag value for easy comparison. This should give the end user enough confidence that the uploaded file has maintained it’s integrity.

The script assumes you have an account in AWS with a login credentials. You have the cli AWS tools and credentials downloaded to /home/user/.aws/config and /home/user/.aws/credentials. These two files are needed to successfully authenticate to the s3 bucket.

Amazon Web Service S3 Bucket

AWS is a flat file system. There are no folders or directories. The “full” name of a file includes all the subdirectories as well. i.e. “/file1/file2/file3.txt” is the file name and not “file3.txt”. AWS will show all subdirectories as folders in the console, for ease of human navigate.

Begin

Start the script by defining that it will run as bash and add any notes to the head.

#!/bin/bash
# 04/18/23, backup.sh
# created by mbb

Send any log output to a custom log file and code to exit the script if any commands in a pipeline fails.

exec 2>> /var/log/backups/aws.log
set -euo pipefail

Get the number of processing units available and add it to a variable.

NUM_PARALLEL=$(nproc)

Define the remaining local variables.

HOSTNAME=`hostname`
DATE=`date "+%F %T"`

DAY=$(date -d "1 day ago" +%d)
MONTH=$(date -d "1 day ago" +%b)
YEAR=$(date -d "1 day ago" +%Y)

LOG="/var/log/backups/aws.log"

src_dir="/mnt/logs/$YEAR/$MONTH"
src_file="log-$DAY.log.gz"

Define the AWS variables.

aws_dir="s3://bucket01/folder1/$YEAR/$MONTH" 

s3_bucket="bucket01"
s3_dir="folder1/$YEAR/$MONTH"
s3_key="$s3_dir/$src_file"     # name of the file in s3 
s3_body="$src_dir/$src_file"   # the location of the data to be uploaded

When a file is uploaded to AWS, it will calculate what is called an ETAG value. This is the checksum value of the upload file. To verify file integrity, we will compare the uploaded aws calculated ETAG against the local file’s calculated ETAG.

The ETAG will match a true md5 hash value if the file size is < 5 GB. If the file is > 5 GB, the aws ‘cp’ command will automatically break the file into 8 MB chunks and upload 4 threads of data simultaneously, until the upload is complete. Each uploaded thread will have an md5 calculated. The resulting ETAG will be a sum of all the uploaded data chunks, rather than a true md5 hash against the completed file.

In order to compare the ETAG’s and verify they match, we must calculate the local file’s ETAG value. Then compare that value to the value calculated by AWS. The script contains two methods to calculate the ETAG value, you will need to review and consider what is needed. In my case, I always know the value I will upload will be > 5 GB.

To calculate the local files ETAG value, for files < 5GB. use:

# calculate ETAG value of local file, if < 5GB.
md5_hash="$(md5sum "$src_dir/$src_file" | awk '{ print $1 }')"

For files > 5 GB, we can use the code from https://gist.github.com/rajivnarayan/1a8e5f2b6783701e0b3717dbcfd324ba.

# ---------- Begin Code ---------------
MULTIPART_MINSIZE=$((8*1024*1024))

file="$src_dir/$src_file"
partSizeInMb=8

if [[ ! -f "$file" ]]; then
   echo "$DATE Error: $file not found to compute hash." >> $LOG
   exit 1;
fi

# Calculate checksum for a specified file chunk
# inputs: file, partSizeInMb, chunk
# output: chunk md5sum
hash_chunk(){
   file="$1"
   partSizeInMb="$2"
   chunk="$3"
   skip=$((partSizeInMb * chunk))
   # output chunk + md5 (to allow sorting later)
   dd bs=1M count="$partSizeInMb" skip="$skip" if="$file" 2> /dev/null | echo -e "$chunk $(md5sum)"
}

# Integer quotient a/b after rounding up 
div_round_up(){
   echo $((($1 + $2 - 1)/$2))
}

partSizeInB=$((partSizeInMb * 1024 * 1024))
fileSizeInB=$(du -b "$file" | cut -f1 )
parts=$(div_round_up fileSizeInB partSizeInB)

if [[ $fileSizeInB -gt $MULTIPART_MINSIZE ]]; then
   export -f hash_chunk
   etag=$(seq 0 $((parts-1)) | \
       xargs -P ${NUM_PARALLEL} -I{} bash -c 'hash_chunk "$@"' -- "$file" "$partSizeInMb" {} | \
       sort -n -k1,1 |tr -s ' '|cut -f2,3 -d' '|xxd -r -p|md5sum|cut -f1 -d' ')"-$parts"
else
   etag=$(md5sum $file|cut -f1 -d' ')
fi
# ---------------- end of code ----------------------
# Calculate ETAG Value of local file, if > 5GB
md5_hash=$etag

Next, we will copy the files to the s3 bucket using the ‘cp’ command. We will be using the CLI copy command, rather than the s3api command, as the api can not handle file’s large then 5 GB. Copy the content to S3 and tell AWS that the data is just a plain text file.

aws s3 cp $src_dir/$src_file s3://$s3_bucket/$s3_dir/$src_file --content-type=text/plain

Get the ETAG value that AWS calculated during the upload.

s3_md5_hash=$(aws s3api head-object --bucket "$s3_bucket" --key "$s3_key" --query ETag --output text | sed 's/"//'g)

Next, we will get both the local file size and the uploaded file sizes.

aws_list=`aws s3 ls "$aws_dst_dir/$src_file" --human-readable --output text`
local_list=`ls -alhF "$src_dir/$src_file"`

Finally, display the file sizes and the ETAG values of both the uploaded file and the local file side by side for comparison.

echo "File size check:"
echo "$HOSTNAME: $local_list"
echo "s3.amazon: $aws_list"
echo ""
echo "ETAG check (md5 sum):"
echo "$HOSTNAME: $src_file  ETAG: $md5_hash"
echo "s3.amazon: $src_file  ETAG: $s3_md5_hash"