Reflex Package Versions

I was hesitant to write the Nix Reproducibility blog entry yesterday, but I am glad that I did. Thanks to writing that blog entry, I noticed that the version of reflex-dom-core used was quite old. It makes sense that a project developed with Nix uses versions of packages that are determined by a fixed nixpkgs (or other) repository revision, and an old project likely uses old revisions. Obelisk is installed from the master branch, however. In this blog entry, I investigate which package versions are used when using Reflex FRP.

First, I prepared a clean Nix environment. See the Nix Cleanup Everything blog entry for details.

$ nix-env --query
nix-2.7.0
$ du -sh /nix/
434M    /nix/

For my first test, I experimented with the tutorial on the Reflex homepage. I installed Obelisk by following the Installing Obelisk instructions in the project README. With Nix already installed and caches configured, I just needed to run the following command.

$ nix-env -f https://github.com/obsidiansystems/obelisk/archive/master.tar.gz -iA command

It does not take too long to install.

$ nix-env --query
nix-2.7.0
obelisk-command-0.9.0.1
$ du -sh /nix/
1.8G    /nix/

Yesterday, I noticed that I had many versions of Obelisk packages in my Nix store and wondered which version I was using. My profile pointed to version 0.9.0.1 like above, but I assume that the older versions were installed and used by the project repositories that I was experimenting with. The changelog and releases indicate that the latest version of Obelisk is 1.0.0.0, but I did not see that version. The ob command unfortunately does not provide a --version option or version command, so commands like ls -l $(which ob) need to be used to investigate which version is actually being used.

I now realize that the 1.0.0.0 version is for the master branch as a whole and is unrelated to the version of the packages. Indeed, lib/command/obelisk-command.cabal specifies version 0.9.0.1.

At this point, even running ob --help causes Nix to build the command. This downloads lots of stuff, including many obsolete Python 2.7 libraries, which is puzzling. (Why is Python 2.7 still being used!?!?) Nix 2.2.2 is also downloaded! I guess this explains why I saw old versions of Nix in my store. I found that running ob --help increased the size of my store by a gigabyte!

$ du -sh /nix/
2.8G    /nix/

The tutorial repository includes an .obelisk/impl directory that points to a specific commit in the Obelisk repository. The specified branch is a few years old, and the specified revision is not even the latest on that branch.

{
  "owner": "obsidiansystems",
  "repo": "obelisk",
  "branch": "ls-reflex-frp",
  "rev": "50a5b9e0da7986de0f885575ab302ca05dc793ac",
  "sha256": "0qx74sl710zd50zgccyn8y4c37d5lj3xhh7ckicjvhy5kbk12bps"
}

Checking Obelisk revision 50a5b9e0da, I found that obelisk-command is at version 0.1, which is something that had puzzled me yesterday. I guess that this revision is also what determines the versions of the libraries used to build the project. Indeed, dep/reflex-platform points to a specific commit in the Reflex Platform repository.

{
  "owner": "reflex-frp",
  "repo": "reflex-platform",
  "branch": "je-reflex-frp-website-temp",
  "rev": "a3acd23d3242f7cd80baff38b44d73c41d527c5f",
  "sha256": "1cv4i0wxgyhalx04vqilrw8ybni0ka51zgx4g9yx4vljk82qdbjd"
}

Checking Reflex Platform revision a3acd23d32, I found that nixpkgs points to a specific commit in the nixpkgs repository. The specified revision is a pretty old revision in the nixos-19.03 branch.

{
  "owner": "nixos",
  "repo": "nixpkgs-channels",
  "branch": "nixos-19.03",
  "rev": "9d55c1430af72ace3a479d5e0a90451108e774b4",
  "sha256": "05625fwgsa15i2jlsf2ymv3jx68362nf3zqbpnrwq6d3sn89liny"
}

Checking nixpkgs revision 9d55c1430a, I found that reflex-dom-core is set at version 0.4 in pkgs/development/haskell-modules/hackage-packages.nix.

Fixing the revisions/versions of the whole development ecosystem is important for providing reproducibility. When experimenting with numerous Reflex projects, however, I ended up building many similar yet slightly different development ecosystems. This consumes a lot of storage space, takes a lot of time, and uses a lot of bandwidth. I imagine/hope that this is not an issue when focusing on a single project.

I am curious if the tutorial works with the latest release of Obelisk. It should not take too long to try! I first created a new directory and ran ob init to initialize a new project. The new .obelisk/impl directory points to the latest revision on the master branch! Wow!

{
  "owner": "obsidiansystems",
  "repo": "obelisk",
  "branch": "master",
  "private": false,
  "rev": "e3ec75f6988eed47189dd20f8c8514748b33ea81",
  "sha256": "0sdlcyzdx9nyql5vmpjydhzsakixc8y050v8m2rz87l2rmhkqnw9"
}

Obelisk revision e3ec75f698 uses Reflex Platform revision 123a6f487c, the latest revision of the release/0.9.2.0 branch. Interestingly, a fork of nixpkgs is used! Revision a78062dc78 on this fork is the second newest revision of the reflex-platform-20.09 branch. GitHub reports that the branch is 4,444 commits ahead and 122,868 commits behind the nixpkgs master branch. There are newer reflex-platform-* branches, but perhaps they are not stable yet. I found that reflex-dom-core version 0.6.0.0 is used in this revision.

{
  "owner": "obsidiansystems",
  "repo": "nixpkgs",
  "branch": "reflex-platform-20.09",
  "private": false,
  "rev": "a78062dc78a85f5d553443ea636a3242c0e6de04",
  "sha256": "1kda0hdmvscma86c13z01mm29ka43djd9fskfa9g6f170sd9f4qc"
}

I manually merged the tutorial with the new project template.

The cabal.project configuration now configures write-ghc-environment-files to never, which is the default anyway. I kept this setting.

There are a few changes to the default.nix file. A system parameter is added so that builtins.currentSystem can be overridden. A newer iOS SDK version is used. There is new terms.security.acme.acceptTerms configuration that is required to use Let’s Encrypt. I merged in all of these new changes.

The new .cabal files have updated ghc-options. Libraries use the following options, while executables use the same options with the additions of -threaded.

-O
-Wall
-Wincomplete-record-updates
-Wincomplete-uni-patterns
-Wredundant-constraints
-fno-show-valid-hole-fits

The .gitignore file is significantly updated. I kept the new version. I also kept the new .obelisk directory, of course.

The ob run command takes a long time to run. A log is displayed indicating that the command is built, and then there is long time without any output at all! Perhaps I should run the command using the --verbose flag… In this case, I monitored the system using htop (sorting by CPU percentage and using the full width of my monitor in order to see the full paths of programs under /nix/store) and Nethogs.

I of course did something else while waiting for the build. After returning from lunch, I found that the build had failed. The following error was displayed, reformatted to multiple lines.

Process exited with code 100;
/nix/store/4z4336r8yhyznz3fiscvz53wgdj43kjb-nix-2.3.11/bin/nix-shell -E
$'{root, pkgs, shell}: ((import root {}).passthru.__unstable__.self.extend
(_: _: {shellPackages = builtins.fromJSON pkgs;})).project.shells.${shell}'
--arg root ./. --argstr pkgs $'{"backend":"/tmp/calculator-tutorial/backend",
"common":"/tmp/calculator-tutorial/common",
"frontend":"/tmp/calculator-tutorial/frontend",
"obelisk-generated-static":
"/nix/store/lsaqdxf2vhw6ipnj5jb1d9ylfw0aisr0-asset-manifest-haskellManifest"}'
--argstr shell ghc --run $'export
$\'NIX_PATH=nixpkgs=/nix/store/yxa12wh8fc6j8w61y3jn27vw8dhydby7-source\' ;
bash -c \'type -p ghc\''

I ran ob run again, and (after more waiting and near 100% CPU usage) it failed with the same error, with no additional output. I tried again with the --verbose option and got different output.

$ ob run --verbose
Starting Obelisk </nix/store/ndgqr350z5rnr58wd70nwkzjg9847nxq-obelisk-command-0.9.0.1/bin/.ob-wrapped> args=["run","--verbose"] logging-level=Debug
Thunk specification git-v5 did not match "./.obelisk/impl": ReadThunkError_UnrecognizedPaths ("./.obelisk/impl/github.json" :| [])
Thunk specification github-v5 matched "./.obelisk/impl"
./.obelisk/impl: command not cached, building ...
Creating process: nix-build ./.obelisk/impl -A command --out-link ./.obelisk/impl/.attr-cache/command.out
DONE Built on ./.obelisk/impl [command]
Handing off to ./.obelisk/impl/.attr-cache/command.out/bin/ob
Starting Obelisk </nix/store/ndgqr350z5rnr58wd70nwkzjg9847nxq-obelisk-command-0.9.0.1/bin/.ob-wrapped> args=["--no-handoff","run","--verbose"] logging-level=Debug
Finding packages with root "." and interpret paths:

ob: <stdout>: commitAndReleaseBuffer: invalid argument (invalid character)

The thunk error is intriguing. After researching it a bit, I found a thread on /r/haskell that indicates that the error is not an issue.

That final error is strange. It is an error that is commonly seen when a Haskell program is run in a non-UTF-8 locale. My shell of course has the locale configured correctly, but perhaps something is wrong within the Nix environment…

I tried building a few more times, and I discovered that the “invalid character” error now occurs very quickly as soon as I run ob run --verbose, while running just ob run takes a lot longer and uses a lot of CPU before eventually failing with the above “Process exited with code 100” error message. This leads me to believe that the “invalid character” error is an issue with the Obelisk command, not the tutorial project. Unfortunately, it keeps me from seeing what is really making the build fail.

I tried running ob --verbose run, on the off chance that the option order is significant, but I get the same “invalid character” error.

I created a new project, initialized using ob init, and tried running ob --verbose run in that project without any changes. It fails with the same “invalid character” error.

I tried searching the Obelisk issues and found an old/closed issue about a similar problem that used to happen with ob init (issue 335). I once again confirmed that my LANG environment variable is set. I also searched the currently open pull requests but did not see anything relevant. I went ahead and submitted a new issue.

This is an unsatisfactory way to fail at making the tutorial use the latest release of Obelisk. I do not even know the real error! Alas, at least I am getting more familiar with Obelisk projects.

Author

Travis Cardwell

Published

April 1, 2022