GitHub Actions and containers

 April 17, 2023

GitHub Actions allows the use of containers with different Operating Systems. Although, it does not mean that everything is seamless when you are using them. I’ve discovered this the hard way! Below are my findings and the process I followed to make the GitHub Action pipeline work properly with containers.

It all started with the addition of a new tool in the pipeline, which was not installable on Ubuntu (the GitHub Actions default operating system), due to a packaging issue. Therefore, I decided to use a Fedora container since I was sure that our toolchain was present and well-maintained in Fedora. After adding the container definition and changing the toolchain installation command from apt to dnf, the pipeline seemed to be working properly. After a few days, it was noticed that the pipeline did not fail but was not working.

In debugging the pipeline, I first noticed that the Git repository seemed not to be a Git repository at all! The pipeline used actions/checkout@v3 to checkout the code, did some compilation to create HTML pages, and then pushed the result to a branch called docs using stefanzweifel/git-auto-commit-action@v4. The issue was that the stefanzweifel/git-auto-commit-action@v4 module was returning errors such as:

fatal: not a git repository (or any parent up to mount point /)

It took a few days before anyone noticed because even if this was clearly a fatal error, the action did not return a failure to the Actions engine, making the pipeline apparently successful. Aside from the weirdness of handling the error, I found the error even more puzzling itself since I was expecting actions/checkout@v3 to create a Git repository as it does outside the container. After looking intensely at actions/checkout@v3 logs, I’ve noticed the following lines:

The repository will be downloaded using the GitHub REST API
To create a local Git repository instead, add Git 2.18 or higher to the PATH

This got quickly solved by adding an action to install Git before the invocation of actions/checkout@v3. Trying to rerun the pipeline, the result was still no error, but a new log line in stefanzweifel/git-auto-commit-action@v4:

fatal: detected dubious ownership in repository at '/github/workspace'

This got quickly solved with the addition of a new step between the checkout and the commit phases:

- run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

After this issue was solved as well, I noticed that stefanzweifel/git-auto-commit-action@v4 was not committing anything leaving in the log:

Working tree clean. Nothing to commit.

With a bit of tweaking, I figured out that the reason was its inability to “notice” that some files were changed. As of now, I’ve not yet figured out the exact dynamic of the issue, but I managed to work around it by adding a step before calling the stefanzweifel/git-auto-commit-action@v4 action:

- run: git add -f .

In this way, the files are already staged to be committed before stefanzweifel/git-auto-commit-action@v4 is called, so it can finalize and push the commit properly.

I wanted to share this story because it highlights many problems with GitHub actions (as well as many other similar systems). What I took out of this experience, and hopefully you will as well, before going in loops as I did is:

1. The fact that a GitHub Action was successful does not guarantee that it had no fatal issues.
2. Some actions might behave very differently based on the environment with no clear way of notifying it
3. Reading all logs is critical when operating or changing GitHub Actions