Google Cloud Snapshot Cleaner v0.1.0

 September 16, 2019

I’ve just tagged the first version (0.1.0) of gcsc (Google Cloud Snapshot Cleaner). The idea behind this small software is to create a more flexible way to keep the Google Cloud Disks Snapshots tidy.

Google Cloud does provide a very nice way to automatically snapshot your disks, leveraging the resource policies. This is very nice, since it allows you to be sure that the Google Cloud always and reliably snapshots your disks. The tool also allows you to auto-delete the snapshots after a certain period, but I found this feature a little bit too limited. The main reason is that it does not allow you to have complex retention policies. In fact, the tool only allows to set a single expiration date for all snapshots.

Google is very smart in the management of the snapshots, so you end up paying only for the deltas from one snapshot to the next one, so it could seem like if you have two snapshots 24 hours apart with 1Gb of difference, no matter how many snapshots you have in between them, you end up paying the same. This is not always true, because it really depends how the disks in your infrastructure is used. In the case I was following, there were a lot of cases of disks where hourly snapshots had a 20% delta (each one against the previous one), while daily snapshots only had 25% delta. This could happen for many reasons, one of those are cases (like in some databases use cases), where some data are changing more than once every hour.

In addition to this problem, there is the quota of 25'000 snapshots per project. Now, this limit can be raised, but I think it still makes sense to stay below this number, if you don’t have a really strange use case.

What gcsc can do for you, is to clean-up the snapshots you have based on the age of the snapshots. In it’s current version, the configuration file needs to be present and be placed in $HOME/.gcsc/config.yaml. An example valid file is the following:

project-id: my-project
retention-policies:
  - begin: 0h
    end: 168h # 7 days
    cadence: 1h
  - begin: 168h # 7 days
    end: 336h # 14 days
    cadence: 24h
  - begin: 336h # 14 days
    end: 1512h # 63 days
    cadence: 168h # 7 days

In this case, we have declared the name of the project (my-project), and declared a retention schedule. For the first 7 days, we will have hourly backups, then daily backups for the next 7 days, and then weekly backups for the next 49 days.

This system (at least with the presented configuration) is not meant to replace completely the Google provided retention-policy, in fact in this particular instance is set to delete all snapshots after 70 days. This guarantees that no matter what (ie: gcsc did not completed for some reason), in maximum 70 days the snapshots are gone.

To lunch the script, you also need to export the GOOGLE_APPLICATION_CREDENTIALS environment variable, unless gcsc is running on a machine that has a service account with the right permissions.

To be sure to have the right permissions you can create a custom role like the following one:

title: Snapshot Cleaner (v0)
description: Snapshot Cleaner custom role
stage: ALPHA
includedPermissions:
  - compute.snapshots.delete
  - compute.snapshots.get
  - compute.snapshots.list

Executing gcsc is very easy, as it can be notice from running it naked:

$ gcsc 
Usage:
  snapshot-cleaner [command]

Available Commands:
  clean       execute a cleaning
  help        Help about any command

Flags:
      --automatic           Include automatic backups (default true)
      --dry-run             Dry run mode
  -h, --help                help for snapshot-cleaner
      --manual              Include manual backups
  -p, --project-id string   Google Cloud Project ID

Use "snapshot-cleaner [command] --help" for more information about a command.
subcommand is required

In the future I hope to add new features to make it more flexible, since at the moment it is very basic and limited to the specific user-case I needed to solve.

UPDATE: I’ve then published a new version and another article about gcsc.