Testing GHC Versions

I think that it is important for libraries on Hackage to support a wide range of versions of GHC, Cabal, and dependencies. (See my article on software extent). In some environments, people are unable to use bleeding edge releases, so supporting older versions allows them to make use of the package. Many of these people are working in industry, so I think that backwards compatibility is quite important for making Haskell a viable choice in industry. Supporting new versions is also important, particularly when the package is a dependency of other packages, as not supporting new releases can block those other packages from doing so. Also, it can be very frustrating for new Haskell users when they install the latest Haskell development tools and find that they are unable to use a package.

Maintaining such compatibility has a cost, of course. It takes time and effort to maintain packages. This time is particularly valuable to open source maintainers, as spending “personal” time can take away from time spent with loved ones and time maintaining personal health, not to mention the pressures of other life obligations such as employment. Time spent on maintenance also takes away from time spent developing new software, which tends to be more interesting. There can also be a complexity cost in the code, when the code needs to provide compatibility with different APIs exposed in different versions of dependencies. On the tooling side, one has to refrain from using new features in order to maintain compatibility.

Personally, my goal is to maintain Haskell software to work with five years of GHC/base and Cabal versions. In practice, compatibility in my projects varies according to the compatibility provided by the dependencies. Currently, most of my projects are compatible with the following versions of GHC and Cabal.

Stack

I think that it is very important to be able to test changes against all supported GHC versions before committing to the repository. I use Stack because it makes this very easy. My projects contain a stack-$VERSION.yaml configuration file for each supported GHC version, and a stack.yaml soft link configures the default for the project. When I need to test something special, such as a new release of a package, I can create a stack-$TESTNAME.yaml configuration file for that specific test.

Most commands in the project Makefile use the default configuration by default. For example, the following command runs tests using the default configuration.

$ make test

These commands can be passed an argument in order to use a specific configuration.

$ make test CONFIG=stack-9.0.1.yaml

An exception is the make test-all command, which runs tests with all of the configured GHC versions.

$ make test-all

Nix

I use Nix for testing my projects with Cabal-install. I try to configure Nix to test the same versions that I test with Stack, but it is very difficult to do, and the cost of doing so is extremely high.

The nixpkgs repository does not support many versions of software in a given commit. To test specific versions of software, one therefore needs to search for a commit that has that version. Finding a revision where everything works can be very challenging as well as extremely time consuming! Sometimes a specific version of software is not available in any revision. For example, GHC 8.10.6 is not in nixpkgs yet. While it is possible to write a derivation for it oneself, this incurs an even greater cost.

I wish that nixpkgs would provide (working) versions of every GHC and Cabal release. For example, providing the latest point release for every minor version of GHC since version 8 would be very helpful. I suspect that this is not done because of the way that Nix treats every Hackage package as a Nix package. IMHO, this is a huge problem with the design.

ghcup

ghcup now supports a wide range of GHC versions! I really like it!

I use ghcup in containers sometimes, but I have not yet tried implementing a make test-all command. In general, ghcup uses a set command, which activates a specific version of GHC/Cabal by creating/updating a soft link. This method of implementation is unfortunate because it is global, not limited to the current project/shell. With Stack, I am able to run tests without global side effects, which allows you to work on multiple projects without worrying about interference. For example, it is common to continue development of a project while doing maintenance and running tests for different projects in the background. That said, I am pretty sure that I can use cabal --with-PROG options to specify tool versions. In this case, the soft links can merely specify the defaults, and different versions can be used without global side effects. I will try this out when I get a chance.

GitHub

I use GitHub Actions to implement continuous integration (CI) tests that run when commits are pushed to GitHub. It is a free service, and I am very happy with it!

GitHub Actions uses software on the image when it is available, and it installs the required software otherwise. The Haskell actions use ghcup to install Haskell development software. Since ghcup only supports the following cabal-install versions, my CI tests do not test older versions.

• cabal-install 2.4.1.0
• cabal-install 3.0.0.0
• cabal-install 3.2.0.0
• cabal-install 3.4.0.0

This is not a big concern, but I mention it because this blog entry is about testing across a wide range of versions. The version number 2.4.1.0 looks pretty old, but note that it came out in 2019. That is not very long ago…

I would really like to cache versions of GHC that are installed, so that they do not have to be installed every time. This should speed up the tests, which is always nice, but my primary motivation is to reduce the cost of the tests. The service is free to me, but installing such software over and over again has a cost to GitHub/Microsoft as well as the environment. I would be unhappy to see GitHub Actions go the way of Travis CI.

So far, I have not found a good example of how to cache the software that is installed by ghcup. Perhaps nobody is doing it yet? Caching a .ghcup directory may be sufficient… Unless I find a solution, I plan on experimenting with this in a test project. If you know of a project that already does this, please let me know!