GitHub Actions Scripted Matrix

In the Dependency version bounds are a lie discussion, Joachim Breitner let me know that GitHub Actions is able to configure the test matrix based on the output of commands. Researching, I found that a fromJSON expression can be used to decode JSON output. I have since experimented with using this functionality.

My Haskell projects test using both Cabal and Stack. The CI tests, as well as the test-all command, run tests using specific versions of GHC, which may vary across projects. I have been maintaining the list of tested versions in three files (per project):

1. .cabal file
2. Makefile
3. .github/workflows/ci.yml

Note that some projects have cabal.project files for each tested GHC version, but most do not. All Stack tests are configured using stack.yaml files, but not all of these are tested in CI or using test-all.

The definitive list of tested GHC versions is in the .cabal file, so I experimented with updating the Makefile and CI configuration to load the list from that single location. I did not do this in the Makefile before because Make makes this difficult to do correctly. (I do not want a macro to load the test versions on every command, just the commands that require the information.)

After experimenting with various possibilities, I ended up implementing a test-all.sh script that manages this functionality. This script simply takes a command to determine the behavior:

Usage: test-all.sh COMMAND

Commands:
  cabal   test all GHC versions using Cabal
  github  output GitHub actions matrix configuration
  stack   test all GHC versions using Stack

The github command is used to output the JSON that is used to configure the GitHub Actions test matrix. The CI configuration has a new config job that executes the script to get the tested GHC versions. The cabal and stack jobs are then able to use this information to configure each job matrix.

$ ./test-all.sh github
ghcvers=["8.2.2","8.4.4","8.6.5","8.8.4","8.10.7","9.0.2","9.2.5","9.4.4"]

Note that the cabal-version job matrix is still configured in the YAML file. All versions of Cabal that are supported by both the project and the GitHub Actions environment are tested using the oldest supported GHC version. I considered configuring this elsewhere, but doing so is pointless. This still needs to be maintained separately.

The test-all command is updated to simply call the new test-all.sh script with the selected MODE (cabal or stack). I considered calling make from the script to run the actual tests, but I decided against it and instead run the tests from the script. This unfortunately results in some duplicate logic. Why did I do this? The issue is that Make does not provide a clean interface. A lot of logic is shared across all commands even when it does not make sense. The script behavior is determined by the argument and the configured GHC versions, while the Make behavior can be affected by environment variables that I do not want to affect the test-all command.

I am once again quite unhappy with using Make. I tend to feel this way whenever I make a non-trivial change to a Makefile, and I have written about it before in The Make Dilemma. Now that I have one Bash script to cleanly implement test-all functionality, I am thinking about implementing all of functionality in my current Makefile using scripts and using a simple Makefile that just calls the scripts. As I wrote about in Rewrite When More Than N Lines, shell scripts are a poor solution as well, but at least they are more sound than Make. I have also considered using a Python script, using just the standard library so that only Python 3 is required to run it, but I am not keen on adding a bunch of Python to all of my projects.

To summarize my experiments so far, I now only configure the tested GHC versions in the .cabal file, and a new Bash script parses that information. The Makefile and CI configuration no longer need to configure the tested GHC versions since they call the new script. This works well, but I feel unsatisfied with the Make/script situation and will likely experiment further.

Joachim has since created a new cabal-plan-bounds tool that can be used to automatically maintain dependency version bounds, as discussed in Don’t edit dependency bounds manually, with this CI setup. It looks quite nice and is probably a great solution for some people! I do not plan on using it myself, however, because it is more automation than I am comfortable with.

Technically, changing dependency bounds should result in a new release. The Hackage policy is to create package revisions for active releases, however. Fewer releases helps keep Hackage sustainable, at the cost of decreased safety. I follow this policy, and I prefer to maintain dependency bounds myself so that I can judge if a package revision is sufficiently safe or if a new release is required.

On a side note, I avoid the Cabal “Caret” operator (^>=) in my .cabal files. The reason for this is just that I strive to support as wide of a range of dependency versions as possible, not just a single major version. My article on Software Extent discusses the motivation for this.