Tag: bash

Backup to a Remote Server using Bash
Introduction

A great way to backup your Linux files is through automation. Linux contains many built in commands that can be used to automate this process. In this article, we will write a simple bash script to backup critical files to a remote host.

For this to work, a user account is needed on both hosts to transfer the files. The sending server will need the private SSH key and the receiving server will need the public SSH key. These keys are used to automatically authenticate to the remote server. Creating and deploying a SSH key pair for authentication is beyond the scope of this article.

The script will use the scp command as it uses SSH underneath the hood. This means it will natively automatically check for an identity file stored under the user’s account at ~/.ssh/config. When creating the SSH credentials, do not add a passphrase to the private key.

Secure copy (scp) command is good if you just want to copy a single file. You can use the -R switch to copy all files in a directory. If you need to backup multiple files, you can modify the script and just add additional variables, update checks, and then add more scp commands.

If you need to transfer entire directories consider using the rsync command. Rsync like scp should automatically use the ~/.ssh/conf file by default. However, rsync sometime has issues using an identity file, so do proper testing.

Finally, Consider setting up a CRON job to run the script nightly.

Procedure

Lets build a script on server1 and we will be connecting and coping our file to server2. First, set up the head of the script and call bash and add any comments.
```
#!/bin/bash
# backup.sh
```
Next, let’s set up the script variables.
```
DATE=`date`
DAY=$(date -d "1 day ago" +%d)
MONTH=$(date -d "1 day ago" +%b)
YEAR=$(date -d "1 day ago" +%Y)
SRC_DIR="/var/logs/$YEAR/$MONTH"
DST_DIR="/user1/backup/$YEAR/$MONTH"
FILE="important-file-$DAY.sql"
```
Next, lets run a check on the remote server and test if the file path exists!
```
# test if path exist on remote server.
# -d=test for file & if file exists is it a dir?
TEST=`ssh server2.company.com "test -d $DST_DIR && echo 1 || echo 0"`
if [ $TEST = 0 ] ; then
     echo "$DATE : $DST_DIR was not found. Making ..."
     # -p=make all missing directories in the file path.
     ssh server2.company.com "mkdir -p $DST_DIR"
else
     echo "$DATE : $DST_DIR was found."
fi
```
Finally, copy the file to the remote server.
```
# Copy file to remote server.
#-p=preserve permissions
scp -p $SRC_DIR/$FILE user1@server1:$DST_DIR
```
Lastly

Create a cronjob to run nightly at 9:00 PM.

References

https://unix.stackexchange.com/questions/127352/specify-identity-file-id-rsa-with-rsync
January 20, 2025
Backup Files to S3 using Bash
Description

A bash script will be used to copy a file from a Linux server to an S3 bucket. Next, it will run a checksum on the results to verify the upload. Finally, it will output the local file size, the local etag , the aws file size, and the aws etag value for easy comparison. This should give the end user enough confidence that the uploaded file has maintained it’s integrity.

The script assumes you have an account in AWS with a login credentials. You have the cli AWS tools and credentials downloaded to /home/user/.aws/config and /home/user/.aws/credentials. These two files are needed to successfully authenticate to the s3 bucket.

Amazon Web Service S3 Bucket

AWS is a flat file system. There are no folders or directories. The “full” name of a file includes all the subdirectories as well. i.e. “/file1/file2/file3.txt” is the file name and not “file3.txt”. AWS will show all subdirectories as folders in the console, for ease of human navigate.

Begin

Start the script by defining that it will run as bash and add any notes to the head.
```
#!/bin/bash
# 04/18/23, backup.sh
# created by mbb
```
Send any log output to a custom log file and code to exit the script if any commands in a pipeline fails.
```
exec 2>> /var/log/backups/aws.log
set -euo pipefail
```
Get the number of processing units available and add it to a variable.
```
NUM_PARALLEL=$(nproc)
```
Define the remaining local variables.
```
HOSTNAME=`hostname`
DATE=`date "+%F %T"`

DAY=$(date -d "1 day ago" +%d)
MONTH=$(date -d "1 day ago" +%b)
YEAR=$(date -d "1 day ago" +%Y)

LOG="/var/log/backups/aws.log"

src_dir="/mnt/logs/$YEAR/$MONTH"
src_file="log-$DAY.log.gz"
```
Define the AWS variables.
```
aws_dir="s3://bucket01/folder1/$YEAR/$MONTH" 

s3_bucket="bucket01"
s3_dir="folder1/$YEAR/$MONTH"
s3_key="$s3_dir/$src_file"     # name of the file in s3 
s3_body="$src_dir/$src_file"   # the location of the data to be uploaded
```
When a file is uploaded to AWS, it will calculate what is called an ETAG value. This is the checksum value of the upload file. To verify file integrity, we will compare the uploaded aws calculated ETAG against the local file’s calculated ETAG.

The ETAG will match a true md5 hash value if the file size is < 5 GB. If the file is > 5 GB, the aws ‘cp’ command will automatically break the file into 8 MB chunks and upload 4 threads of data simultaneously, until the upload is complete. Each uploaded thread will have an md5 calculated. The resulting ETAG will be a sum of all the uploaded data chunks, rather than a true md5 hash against the completed file.

In order to compare the ETAG’s and verify they match, we must calculate the local file’s ETAG value. Then compare that value to the value calculated by AWS. The script contains two methods to calculate the ETAG value, you will need to review and consider what is needed. In my case, I always know the value I will upload will be > 5 GB.

To calculate the local files ETAG value, for files < 5GB. use:
```
# calculate ETAG value of local file, if < 5GB.
md5_hash="$(md5sum "$src_dir/$src_file" | awk '{ print $1 }')"
```
For files > 5 GB, we can use the code from https://gist.github.com/rajivnarayan/1a8e5f2b6783701e0b3717dbcfd324ba.
```
# ---------- Begin Code ---------------
MULTIPART_MINSIZE=$((8*1024*1024))

file="$src_dir/$src_file"
partSizeInMb=8

if [[ ! -f "$file" ]]; then
   echo "$DATE Error: $file not found to compute hash." >> $LOG
   exit 1;
fi

# Calculate checksum for a specified file chunk
# inputs: file, partSizeInMb, chunk
# output: chunk md5sum
hash_chunk(){
   file="$1"
   partSizeInMb="$2"
   chunk="$3"
   skip=$((partSizeInMb * chunk))
   # output chunk + md5 (to allow sorting later)
   dd bs=1M count="$partSizeInMb" skip="$skip" if="$file" 2> /dev/null | echo -e "$chunk $(md5sum)"
}

# Integer quotient a/b after rounding up 
div_round_up(){
   echo $((($1 + $2 - 1)/$2))
}

partSizeInB=$((partSizeInMb * 1024 * 1024))
fileSizeInB=$(du -b "$file" | cut -f1 )
parts=$(div_round_up fileSizeInB partSizeInB)

if [[ $fileSizeInB -gt $MULTIPART_MINSIZE ]]; then
   export -f hash_chunk
   etag=$(seq 0 $((parts-1)) | \
       xargs -P ${NUM_PARALLEL} -I{} bash -c 'hash_chunk "$@"' -- "$file" "$partSizeInMb" {} | \
       sort -n -k1,1 |tr -s ' '|cut -f2,3 -d' '|xxd -r -p|md5sum|cut -f1 -d' ')"-$parts"
else
   etag=$(md5sum $file|cut -f1 -d' ')
fi
# ---------------- end of code ----------------------
# Calculate ETAG Value of local file, if > 5GB
md5_hash=$etag
```
Next, we will copy the files to the s3 bucket using the ‘cp’ command. We will be using the CLI copy command, rather than the s3api command, as the api can not handle file’s large then 5 GB. Copy the content to S3 and tell AWS that the data is just a plain text file.
```
aws s3 cp $src_dir/$src_file s3://$s3_bucket/$s3_dir/$src_file --content-type=text/plain
```
Get the ETAG value that AWS calculated during the upload.
```
s3_md5_hash=$(aws s3api head-object --bucket "$s3_bucket" --key "$s3_key" --query ETag --output text | sed 's/"//'g)
```
Next, we will get both the local file size and the uploaded file sizes.
```
aws_list=`aws s3 ls "$aws_dst_dir/$src_file" --human-readable --output text`
local_list=`ls -alhF "$src_dir/$src_file"`
```
Finally, display the file sizes and the ETAG values of both the uploaded file and the local file side by side for comparison.
```
echo "File size check:"
echo "$HOSTNAME: $local_list"
echo "s3.amazon: $aws_list"
echo ""
echo "ETAG check (md5 sum):"
echo "$HOSTNAME: $src_file  ETAG: $md5_hash"
echo "s3.amazon: $src_file  ETAG: $s3_md5_hash"
```
April 17, 2024

Tag: bash

Backup to a Remote Server using Bash

Introduction

Procedure

Lastly

References

Backup Files to S3 using Bash

Description

Amazon Web Service S3 Bucket

Begin