Haskell Monorepo GitHub Actions (Part 2)

I now have a plan for using GitHub Actions to run automated tests for multiple Haskell packages in a single repository (aka “monorepo”).

As discussed in the previous blog entry, I would like to keep the number of jobs low to better utilize resources. I would also like to keep providing meaningful jobs and steps, to make it easy to navigate test results (including timing) within the GitHub Actions UI. If you implement everything in a script (or Nix), the usability is terrible. At the same time, I would like to minimize the cost of maintenance, and maintenance of complicated YAML configuration is error prone.

Given these constraints, I have decided to move forward with an old idea that I have not needed until now: use software to generate the configuration. Note that this software is run locally by a developer, not as part of a GitHub Actions workflow. The software creates/updates the configuration, which the developer can review before committing.

Here are my current plans:

1. Get everything working in TTC using manual configuration, including the release of the new packages
2. Develop a Haskell script that generates the CI configuration, still within the TTC project
3. Migrate my other public projects to generate CI configuration in the same way, updating the general functions of the script in preparation for putting them in a separate library
4. Develop and release a library, updating all of the scripts to use it

This software is not limited to generating CI configuration, of course. It can also generate Makefiles and other scripts. My current Makefiles require GNU Make and Bash because those are easy for me to develop and test locally, but this change will provide a good chance to generate more portable Makefiles as well as scripts for other platforms. With configuration in Haskell, regular maintenance will not require messing with Makefiles or Bash at all.

I started on the first step this morning by refactoring the TTC CI configuration with the current package organization. I will move the ttc package into a subdirectory next, and then I will add the new packages.

I plan on writing better documentation as part of the final step, but here is a concise overview of the changes that I made so far:

• As I wrote in the previous blog entry, TTC had 32 test jobs per push. The new configuration significantly decreases the number of jobs by running Cabal, Stack, dependency lower bounds tests, and dependency upper bounds tests within the same job for a given GHC version. In addition, I went ahead and removed most of the Cabal version tests, so that only the oldest and latest supported Cabal versions are tested. There are now only 12 test jobs per push. While I will get more data over time, it looks like fewer jobs results in faster completion because all of them are able to run in parallel without hitting parallel job restrictions.
• The haskell-actions/setup step, which installs the Haskell software, takes a significant amount of time. Reducing the number of jobs significantly decreases the amount of times this step is run, resulting in faster completion.
• Each package (currently just ttc) uses separate Cabal and Stack caches. In addition, ~/.stack is cached separately and includes artifacts for all packages. As usual, cache keys include the month, so all caches are reset each month. This prevents a cache from growing without bound as well as limits any non-optimal caching that may arise.
• Dependency lower and upper bounds tests are implemented as conditional steps. They show up in the UI even in jobs where they are not run, but icons make it clear that they are not run in that case. I think that the output is pretty user-friendly even though the number of jobs is decreased.

While working on this, I found an action-validator project that validates GitHub Actions workflow YAML files. I love static analysis, but this tool was unfortunately not useful for me. It did not find any issues in my initial configuration. I tried purposefully misspelling matrix, but it did not catch that. After pushing to GitHub, there were a number of issues that this tool does not check for. I am not going to continue to use this tool.

Here are three issues that I wish static analysis could have found:

• I mistakenly had two if conditionals in the same step. Perhaps this was missed because many JSON/YAML parsers silently ignore duplicate keys in objects. I am not a fan of this behavior.
• I wrote conditionals like if: matrix.ghc == "8.2.2", using double quotes. Single quotes are required, like if: matrix.ghc == '8.2.2'.
• I accidentally put an && operator outside of ${{ ... }} syntax. Correct usage is documented in the expressions documentation.

In addition, I had two other issues:

• The haskell-actions/setup@v2 command updates Cabal by default only if Stack is not enabled (source). To resolve this issue, I explicitly configured haskell-actions/setup to not update Cabal (in case this behavior ever changes) and update Cabal in a separate step.
• I had removed the v2 prefix from Cabal commands, but this broke the test for Cabal 2.4.1.0, which needs the v2 prefix to run the newer commands. Restoring the v2 prefix for the steps in that job resolved the issue.

The refactored CI is now all green!