Perform backups with Systemd

 May 31, 2024

Many strategies can be employed to build resilience in IT systems. Personally, I think one of the most critical yet overlooked ones - both in personal and corporate settings - is backups.

I recently had to back up a folder containing the state of a service running on a Fedora machine. As often happens, an interesting aspect of this service is that the backups are consistent and, therefore, restorable only if the service is stopped while the configuration folder is backed up. Due to the design of this host, I wanted to use Systemd as the backup driver and keep it as simple and obvious as possible.

To start, we need to describe the process using the exact tasks we will need to implement. In this case, it is probably better to create two simple and distinct processes:

1. Backup
   1. Stop the service
   2. Backup the configuration folder
   3. Start the service
2. Clean backups
   1. Keep the last seven backups and remove the backups in excess

It is now possible to structure the first process in a Systemd service. Systemd is designed to execute a single command per service, so it might not be obvious how to run three steps without creating an external script, which would make it more complex. One way to do this is to leverage ExecStartPre and ExecStartPost to execute some parts of the logic. Therefore, we can use the following lines, assuming the service we need to backup is called SERVICE.

ExecStartPre=/usr/bin/systemctl stop SERVICE
ExecStartPost=/usr/bin/systemctl start SERVICE

Another important aspect is to ensure that the backup file has a filename containing the backup’s date and time. Using nested commands in the ExecStart is possible, but it requires a little care since we will need to leverage bash (/bin/bash -c 'COMMAND') and double the $ and % symbols to escape them. If we assume that the folder to be backed up is /opt/SERVICE/ and we want to have the backups in /opt/bkp/ with names such as SERVICE_2024-05-31_05-00-09.tar.gz, we can write the following line:

ExecStart=/bin/bash -c 'tar -czf /opt/bkp/SERVICE_$$(date +%%Y-%%m-%%d_%%H-%%M-%%S).tar.gz /opt/SERVICE/'

So, putting it all together, we will end up with the following systemd service unit in the /etc/systemd/system/SERVICE-bkp.service file:

[Unit]
Description=SERVICE backup
After=local-fs.target

[Service]
ExecStartPre=/usr/bin/systemctl stop SERVICE
ExecStart=/bin/bash -c 'tar -czf /opt/bkp/SERVICE_$$(date +%%Y-%%m-%%d_%%H-%%M-%%S).tar.gz /opt/SERVICE/'
ExecStartPost=/usr/bin/systemctl start SERVICE
Type=oneshot

We can now proceed to create the systemd time by creating the file /etc/systemd/system/SERVICE-bkp.timer with the following content:

[Unit]
Description=SERVICE backup

[Timer]
OnCalendar=*-*-* 05:00:00

[Install]
WantedBy=timers.target

This will ensure that the SERVICE-bkp.service service is executed shortly after 05:00:00 every day.

It is now possible to create the files required to prune the old backups in a very similar way. In this case, we will not need any ExecStartPre or ExecStartPost, but we will still need to leverage bash.

To ensure that only the last seven backups are kept, we can list the backups on the system, exclude the last seven, and then remove the ones left on the list. The way of doing this in bash is: ExecStart=/bin/bash -c 'ls -tr /opt/bkp/SERVICE_* | head -n -7 | xargs rm'.

So, the resulting systemd unit located in /etc/systemd/system/SERVICE-bkp-clean.service would be:

[Unit]
Description=SERVICE backup cleaner
After=local-fs.target

[Service]
ExecStart=/bin/bash -c 'ls -tr /opt/bkp/SERVICE_* | head -n -7 | xargs rm'
Type=oneshot

In a way similar to the previous timer, we can create /etc/systemd/system/SERVICE-bkp-clean.timer with the following content:

[Unit]
Description=SERVICE backup cleaner

[Timer]
OnCalendar=*-*-* 06:00:00

[Install]
WantedBy=timers.target

This timer will start one hour after the previous one. In my case, that’s okay since the backup task lasts less than 5 minutes, but your mileage might vary based on the backup time.

I hope this is as useful for others as it is for me. I heavily use this pattern because it is very simple and straightforward, as I like it to be.