Random Reproducibility

My PhatSort project supports random ordering, so it is a good project for testing use of HMock with MonadRandom. It was quite straightforward to refactor the code to a tagless-final representation, but writing the tests did not go as well as I had hoped.

I abstracted the effects to the following four monads:

• MonadFileSystem is used for filesystem I/O.
• MonadProcess is used for process execution, to call the sync command. Running a separate process just to call the sync system call is quite undesirable, but the System.Posix.Unistd module of the unix package does not provide a sync function. I will likely change the implementation to use FFI in the future.
• MonadRandom is used for randomness.
• MonadStdio is used for writing to STDOUT. Normally, it is used to display progress when the verbose option is used. When the script option is used, the script it written to STDOUT. If the library were to support other types of user interfaces, it would be a good idea to handle these two cases separately. PhatSort is specifically a CLI utility, however, so I decided to just use MonadStdio for simplicity.

When I wrote the initial tests, I decided to only mock MonadFileSystem and MonadProcess. I cannot mock MonadRandom because of the reasons described in the HMock with MonadRandom blog entry, and I accumulated the MonadStdio output in an attempt to make it less tightly coupled with the filesystem I/O. While I am not a fan of stacking many monad transformers in the actual program, I do not mind doing so in tests.

For MonadRandom, I simply tested with the RandT monad transformer. Since I use MockT as the outermost transformer, the following instance is required to lift the MonadRandom methods.

instance MonadRandom m => MonadRandom (MockT m) where
  getRandomR  = lift . Rand.getRandomR
  getRandom   = lift   Rand.getRandom
  getRandomRs = lift . Rand.getRandomRs
  getRandoms  = lift   Rand.getRandoms

For MonadStdio, I created an instance that accumulates the output into a DList String using a WriterT monad transformer.

instance Monad m => MonadStdio (MockT (WriterT (DList String) m)) where
  putStrLn = lift . tell . DList.singleton

All tests are run in a concretely-specified monad using the runTest function. A test may terminate with an error message or success (unit), and there may be output to STDOUT in either case, but my tests only check the error messages in error cases.

runTest
  :: MockT (WriterT (DList String) (RandT StdGen IO)) (Either String ())
  -> IO (Either String [String])
runTest
    = fmap getResult
    . flip evalRandT (Rand.mkStdGen 42)
    . runWriterT
    . runMockT
  where
    getResult :: (Either String (), DList String) -> Either String [String]
    getResult (Left err, _outLines) = Left err
    getResult (Right (), outLines)  = Right $ DList.toList outLines

The tests were pretty easy to write, and I really like the HMock design! The ExpectContext combinators are very useful.

Unfortunately, I ran into trouble when I started to test across many GHC versions. The StdGen pseudo-random number generator is not consistent across different versions of the random library. Searching the documentation, I found the following note about reproducibility:

If you have two builds of a particular piece of code against this library, any deterministic function call should give the same result in the two builds if the builds are

• compiled against the same major version of this library
• on the same architecture (32-bit or 64-bit)

I considered adding CPP pragmas to specify test results for different versions of the library, but I do not like that idea, especially since results may also differ on different architectures. Perhaps I could use a different pseudo-random number generator that provides consistent results, but my searches did not come up with a suitable one. (I have a different project in my queue that requires reproducible randomness, so I will likely revisit this problem again someday.)

For PhatSort, I plan on refactoring the tests to just check that all items are processed without checking the order when using random ordering. To do that correctly, I will need to mock MonadStdio after all. Since there is no more need for deterministic randomness in this case, I will remove RandT and simply use the IO instance of MonadRandom.