Category: Linux

Add a Personal Package Archive to Ubuntu
Introduction

PPA stands for Personal Package Archive. The primary purpose of a PPA is to provide a way for developers to distribute their own software. PPA are stored on Launchpad.net, which is the official website for PPA’s and is managed by Canonical. PPA’s can be created by anyone and you should only install a PPA from a trusted developer. Use at your own risk.

A secondary purpose of PPA’s is for updating well known software. When Ubuntu reaches out to the official archive repository to get updates, these packages are often outdated and do not contain the latest available software. To get the most recent version, you can set up and install an appropriate PPA. Then, when you run the update command, your host will reach out to both the official archive and the PPA. Whom ever has the most recent software available, will get download and installed on your host.

Typically, each PPA has a GPG key that needs to be installed. Each time an update is run, the keys are exchanged, to verify that you have connected to the correct repository, prior to any downloads. The key should be automatically installed with the initial PPA installation setup.

After a PPA is initially setup. It should survive any system reboots. If you need to disable the PPA for any reason, you can open the PPA file under /etc/apt/sources.list.d/ and comment out the line calling the PPA. Then run an ‘apt update’ again, before running ‘apt upgrade’.

Example PPA Install & Setup

We will be using the popular PPA Apache2 for installation. This is a well known PPA and is maintained by ondrej, a Debian developer. So, it should be safe.
1. apache2 -v (Note the current installed version).
2. sudo apt -y install software-properties-common (to install a PPA, the add-apt-repository command is needed).
3. sudo ls /etc/apt/trusted.gpg.d/ (Check the key was installed).
4. sudo add-apt-repository ppa:ondrej/apache2 (The GPG keys are also installed w this command).
5. sudo ls /etc/apt/sources.list.d/ (verify the package was installed).
6. sudo apt update
7. sudo apt install apache2
8. apache2 -v (verify that the new version is installed).
9. service apache2 restart (restart the service).
References

https://launchpad.net/~ondrej/+archive/ubuntu/apache2

https://www.digitalocean.com/community/tutorials/how-to-handle-apt-key-and-add-apt-repository-deprecation-using-gpg-to-add-external-repositories-on-ubuntu-22-04
April 25, 2025
Backup to a Remote Server using Bash
Introduction

A great way to backup your Linux files is through automation. Linux contains many built in commands that can be used to automate this process. In this article, we will write a simple bash script to backup critical files to a remote host.

For this to work, a user account is needed on both hosts to transfer the files. The sending server will need the private SSH key and the receiving server will need the public SSH key. These keys are used to automatically authenticate to the remote server. Creating and deploying a SSH key pair for authentication is beyond the scope of this article.

The script will use the scp command as it uses SSH underneath the hood. This means it will natively automatically check for an identity file stored under the user’s account at ~/.ssh/config. When creating the SSH credentials, do not add a passphrase to the private key.

Secure copy (scp) command is good if you just want to copy a single file. You can use the -R switch to copy all files in a directory. If you need to backup multiple files, you can modify the script and just add additional variables, update checks, and then add more scp commands.

If you need to transfer entire directories consider using the rsync command. Rsync like scp should automatically use the ~/.ssh/conf file by default. However, rsync sometime has issues using an identity file, so do proper testing.

Finally, Consider setting up a CRON job to run the script nightly.

Procedure

Lets build a script on server1 and we will be connecting and coping our file to server2. First, set up the head of the script and call bash and add any comments.
```
#!/bin/bash
# backup.sh
```
Next, let’s set up the script variables.
```
DATE=`date`
DAY=$(date -d "1 day ago" +%d)
MONTH=$(date -d "1 day ago" +%b)
YEAR=$(date -d "1 day ago" +%Y)
SRC_DIR="/var/logs/$YEAR/$MONTH"
DST_DIR="/user1/backup/$YEAR/$MONTH"
FILE="important-file-$DAY.sql"
```
Next, lets run a check on the remote server and test if the file path exists!
```
# test if path exist on remote server.
# -d=test for file & if file exists is it a dir?
TEST=`ssh server2.company.com "test -d $DST_DIR && echo 1 || echo 0"`
if [ $TEST = 0 ] ; then
     echo "$DATE : $DST_DIR was not found. Making ..."
     # -p=make all missing directories in the file path.
     ssh server2.company.com "mkdir -p $DST_DIR"
else
     echo "$DATE : $DST_DIR was found."
fi
```
Finally, copy the file to the remote server.
```
# Copy file to remote server.
#-p=preserve permissions
scp -p $SRC_DIR/$FILE user1@server1:$DST_DIR
```
Lastly

Create a cronjob to run nightly at 9:00 PM.

References

https://unix.stackexchange.com/questions/127352/specify-identity-file-id-rsa-with-rsync
January 20, 2025
Manage Users & Groups
Introduction

Managing user and groups in Linux is an essential administrative task. We will cover the manual method to create a new user and add that user to a group. Also, remove that user in a multi step process. Finally, we will cover selective tasks concerning Groups.

There is a separate article concerning adding the same user to multiple servers using Ansible. We will not duplicate that information here.

Create a User Account

First, get a list of all users on the host.
```
sudo cat /etc/passwd
```
Now, create a new user called mark.
```
sudo adduser mark
```
Remove a User Account

If we want to fully remove a user we will need to follow several steps. Also, the user’s files on other remote systems will need to be manually searched for and removed or have the ownership changed.

You can not remove an account if the user is currently logged in. This is because the user will have existing running processes.

Lock the user’s account, so they can not login.
```
sudo password -l mark
```
Backup the user’s account data.
```
sudo tar -zcvf /nas/backups/accounts/removed/mark_011225.tar.gz /home/mark/
```
See if the user has any running processes and kill them.
```
sudo ps -u username
```
```
sudo killall -9 -u username
```
Remove the user’s crontab jobs.
```
sudo crontab -r -u username
```
If necessary, cancel any running print jobs. (Linux print remove).
```
sudo lprm -U username
```
Assign Mark’s files to another user named Tom.
```
sudo find / -user mark -exec chown tom:tom {} \;
```
-exec = execute script.
chown tom:tom = Change ownership to tom.
{} = for each file that is found
\ = Terminate script when done.

Since we will be deleting the home and mail spool directories, and we have already made a backup, we do not need or want to search those directories by changing the file permissions right now. This will prevent us from deleting them and lead to orphan files. So we will modify the above command to exclude those. We only want to change ownership for files outside of those two directories.
```
sudo find / -user mark ( -path /home -o -path /var/spool/mail/mark ) -prune -o -exec chown tom:tom {} \;
```
-p = path to exclude
-prune = Do not search specified path.
-o = OR

Finally, remove the user’s account. Some users like to use the deluser command and some like to use the older userdel command. They do essentially the same thing. I am using deluser, as it is a higher level command and also deletes the user’s /home directory and mail spool.
```
sudo deluser --remove-home mark
```
–remove-home = removes /home and /var/spool/mail.
–remove-all-files = removes /home,/var/spool/mail, and attempts removal of all other files.

Lastly, check to verify there are no remaining files assigned to the user.
```
sudo find / -user mark -group mark
```
Create a Group

Lets create a group called analyst.
```
sudo addgroup analyst
```
Add a User to a Group

Let’s add the new user mark to the analyst group. NOTE: You must be a member of a group before you can add others to the same group.
```
sudo usermod -aG analyst mark
```
-a = Append
-G = Group

NOTE: For centOS systems, need to run the command ‘$ usermod -aG wheel <user>’

Review Group Memberships

Lets see who is in the group analyst and see what groups the user ‘mark’ is in.
```
sudo cat /etc/group | grep analyst
```
```
sudo groups mark
```
Remove a User from Group

We can remove the user mark from the analyst group. The command is not as clean as adding a user. It is not obvious that these are group commands. You can use either of the two below commands.
```
sudo deluser mark analyst 
```
```
sudo gpasswd -d mark analyst
```
-d = delete user from group.

Change a File’s Group Permissions

Next, lets change ownership of a file to the group ‘analyst’. Although there are other methods, I prefer the one shown below, as it is more granular. After you change a file’s group permissions, users will not be able to access the file until they log off and back on again.
```
chown sudo:analyst /home/mark/sample.txt
```
References

https://linuxize.com/post/how-to-add-and-delete-users-on-ubuntu-18-04/

https://www.digitalocean.com/community/tutorials/how-to-add-and-delete-users-on-an-ubuntu-14-04-vps
January 19, 2025
Updating the Linux OS & Installed Software
Introduction

The apt (aptitude) command is just a a shortened version of the apt-get command. They are synonymous terms. Use the ‘apt’ command to update and manage your installed software packages. Use with Ubuntu or Debian Linux servers.

Update and upgrade your system

# apt update (date your local repository list)
# apt upgrade (update all installed packages)
# apt autoremove (remove packages that were installed as dependencies)
# apt autoclean (clean the /var/cache/apt/archive folder).

List all installed packages
```
dpkg --list
```
```
apt list --installed
```
Get a list of all packages that can be upgraded
```
apt list --upgradable
```
Remove packages with out uninstalling config files
```
sudo apt remove {package name}
```
Remove packages including config Files
```
sudo apt remove --purge {package name}
```
Repository location
```
sudo cat /etc/apt/sources.list
```
```
sudo ls /etc/apt/sources.list.d
```
References

https://askubuntu.com/questions/668582/false-disk-full-error-apt-get-unable-to-install-or-remove
October 21, 2024
Copy Files to S3 Using AWS CLI Tools
Introduction to the AWS CLI

There are three methods to upload and download data to Amazon Web Services. You can use the command line (CLI), AWS SDK, or the S3 REST API. In this article, we will explore the command Line interface, and the most common commands to manage an S3 bucket.

The maximum size of a file that you can upload by using the Amazon S3 console is 160 GB. The maximum bucket size is 5TB. You can not use s3api on files uploads larger than 5GB. Command line tools can achieve upload speeds greater than 7 MB’s. But, you can go even faster if you turn on acceleration. It is not recommended because an additional cost will be incurred.

Common switches
- –dryrun = test what files would be uploaded, prior to running command.
- — summarize = include a total at the bottom of the output.
- — human-readable = show files sizes in Gb and not Bytes.
- –output text = format the output on separate lines
- –content-type=text/plain = Tell aws the upload data is text data (not video or other).
- –recursive = show full file path
- –exclude – leave out certain files.
- –include = include certain files.
- –delete = this flag is needed to remove any files.
- –meta-data = Use this flag to upload custom data like the true MD5 hash
List contents of a bucket
```
aws s3 ls s3://bucket1/directory1/directory2/ --summarize --human-readable
```
Copy a single file

If the file is large, the cp command will automatically handle a multi-part upload dynamically. If the full path is not present, it will create it automatically in the s3 bucket.
```
aws s3 cp ~/test.txt s3://bucket1/dir1/dir2/ --human-readable 
```
Copy multiple files from a local directory

There are two commands that can be used to copy multiple files. Use sync or cp with the –recursive switch.
```
aws s3 sync ./test/May s3://bucket1/May/ --human-readable --summarize
```
OR
```
aws s3 cp ./test/May s3://bucket1/May/ --recursive --content-type=text/plain
```
Copy only files with .sum extension
```
aws s3 sync ~/dir1/ s3://bucket1/dir1/ --exclude '*' --include '*.sum' --dryrun 
```
Copy a directory and exclude two files
```
aws s3 sync ~/dir1/ s3://bucket1/dir1/ --exclude 'file1.txt' --exclude 'file2.txt'
```
October 11, 2024
Synchronize the Time & Date in Ubuntu
Understanding Time

Time is a critical component of every computer. When to run updates, launch scheduled tasks, or just to keep user’s informed, are all dependent on time.

Every computer has a built in clock on the motherboard, usually powered by a battery, to keep track of the time. This hardware based clock is called the Real Time Clock (RTC) and is used to power the human readable ‘system time’, based on time zones.
- System Time = Uses Time Zones
- RTC = Real Time Zone = Uses Hardware clock
Unfortunately, the hardware clock will always eventually get out of sync with real time and need to be adjusted. This occurs because of power outages, battery on the MB dies, or other reasons.

Ubuntu Settings

Two commands are used to control the date and time. First, the ‘timedatectl‘ command is used to set the time zone. Next, the ‘timesyncd‘ command is used start or stop the sync service and to turn on or off the network time protocol (NTP).

An older process to manage these functions is called ntpd. While this process is still supported, it is recommended to use the newer methods.

Check Your Time Zone

In the United States, there are three primary time zones.
- UTC (Universal Time Coordinated)
- EDT (Eastern Daylight Time) = 2nd Sunday March to 1st Sunday in Nov. = 4 hr behind UTC
- EST (Eastern Standard Time) = 1st Sunday Nov to 2nd Sunday March. = 5 hr behind UTC
```
date
```
Get a list of available time zones
```
timedatectl list-timezones
```
Change the Time Zone
```
sudo timedatectl set-timezone America/New_York
```
Check the Sync & NTP Services

Note: Typically, the Real Time Clock or ‘RTC in local TZ’ should always be set to ‘no’. This is due to that most of the US uses spring/fall time changes and the RTC does not know anything about that.

Turn on the ‘System clock synchronized’

Check that there is a name server or two listed in the configuration file. The entries should be space separated.
```
sudo vim /etc/systemd/timesyncd.conf
```
Add the following lines under [Time]:
NTP=ntp.myserver.com time.nist.gov

Next, run the below command.
```
sudo systemctl start systemd-timesyncd
```
Turn on the ‘NTP Service’
```
sudo timedatectl set-ntp on
```
Verify the Clock is Synced
```
systemctl status systemd-timesyncd
```
Ref: https://www.linuxfordevices.com/tutorials/ubuntu/set-up-time-synchronization-ubuntu

Ref: https://opensource.com/article/20/6/time-date-systemd

Ref: https://ubuntu.com/server/docs/about-time-synchronisation
October 10, 2024