
Perform backups with Systemd
May 31, 2024
Many strategies can be employed to build resilience in IT systems. Personally, I think one of the most critical yet overlooked ones - both in personal and corporate settings - is backups.
I recently had to back up a folder containing the state of a service running on a Fedora machine. As often happens, an interesting aspect of this service is that the backups are consistent and, therefore, restorable only if the service is stopped while the configuration folder is backed up. Due to the design of this host, I wanted to use Systemd as the backup driver and keep it as simple and obvious as possible.
To start, we need to describe the process using the exact tasks we will need to implement. In this case, it is probably better to create two simple and distinct processes:
- Backup
- Stop the service
- Backup the configuration folder
- Start the service
- Clean backups
- Keep the last seven backups and remove the backups in excess
It is now possible to structure the first process in a Systemd service.
Systemd is designed to execute a single command per service, so it might not be obvious how to run three steps without creating an external script, which would make it more complex.
One way to do this is to leverage ExecStartPre
and ExecStartPost
to execute some parts of the logic.
Therefore, we can use the following lines, assuming the service we need to backup is called SERVICE
.
ExecStartPre=/usr/bin/systemctl stop SERVICE
ExecStartPost=/usr/bin/systemctl start SERVICE
Another important aspect is to ensure that the backup file has a filename containing the backup’s date and time.
Using nested commands in the ExecStart
is possible, but it requires a little care since we will need to leverage bash (/bin/bash -c 'COMMAND'
) and double the $
and %
symbols to escape them.
If we assume that the folder to be backed up is /opt/SERVICE/
and we want to have the backups in /opt/bkp/
with names such as SERVICE_2024-05-31_05-00-09.tar.gz
, we can write the following line:
ExecStart=/bin/bash -c 'tar -czf /opt/bkp/SERVICE_$$(date +%%Y-%%m-%%d_%%H-%%M-%%S).tar.gz /opt/SERVICE/'
So, putting it all together, we will end up with the following systemd service unit in the /etc/systemd/system/SERVICE-bkp.service
file:
[Unit]
Description=SERVICE backup
After=local-fs.target
[Service]
ExecStartPre=/usr/bin/systemctl stop SERVICE
ExecStart=/bin/bash -c 'tar -czf /opt/bkp/SERVICE_$$(date +%%Y-%%m-%%d_%%H-%%M-%%S).tar.gz /opt/SERVICE/'
ExecStartPost=/usr/bin/systemctl start SERVICE
Type=oneshot
We can now proceed to create the systemd time by creating the file /etc/systemd/system/SERVICE-bkp.timer
with the following content:
[Unit]
Description=SERVICE backup
[Timer]
OnCalendar=*-*-* 05:00:00
[Install]
WantedBy=timers.target
This will ensure that the SERVICE-bkp.service
service is executed shortly after 05:00:00 every day.
It is now possible to create the files required to prune the old backups in a very similar way.
In this case, we will not need any ExecStartPre
or ExecStartPost
, but we will still need to leverage bash
.
To ensure that only the last seven backups are kept, we can list the backups on the system, exclude the last seven, and then remove the ones left on the list.
The way of doing this in bash is: ExecStart=/bin/bash -c 'ls -tr /opt/bkp/SERVICE_* | head -n -7 | xargs rm'
.
So, the resulting systemd unit located in /etc/systemd/system/SERVICE-bkp-clean.service
would be:
[Unit]
Description=SERVICE backup cleaner
After=local-fs.target
[Service]
ExecStart=/bin/bash -c 'ls -tr /opt/bkp/SERVICE_* | head -n -7 | xargs rm'
Type=oneshot
In a way similar to the previous timer, we can create /etc/systemd/system/SERVICE-bkp-clean.timer
with the following content:
[Unit]
Description=SERVICE backup cleaner
[Timer]
OnCalendar=*-*-* 06:00:00
[Install]
WantedBy=timers.target
This timer will start one hour after the previous one. In my case, that’s okay since the backup task lasts less than 5 minutes, but your mileage might vary based on the backup time.
I hope this is as useful for others as it is for me. I heavily use this pattern because it is very simple and straightforward, as I like it to be.