Recipe

Fix a test that fails only sometimes

Use when: a test fails now and then in the full suite, or passes only when another test has run first

Goal: tests that each set up their own files, and pass alone, together and in any order

Taught in Fixtures and organization

1. 1

   Run the whole suite a few times with cargo test -- --test-threads=1. Then run the failing test alone, by name.

   Example

   In the ordered project, cargo test shares_load_back fails alone with No such file or directory. It only passed before because saves_the_shares had just made the file.

   • it passes with one thread and fails with many — two tests are racing each other.
   • it passes in the suite and fails alone — it needs another test to run first.

   Why: cargo test runs tests at the same time, in no fixed order, so a shared file is the usual cause.

2. 2

   Search the tests for a file name, folder or path that more than one test uses.

   Example

   Both tests in ordered use shares.txt in the package folder. One writes it. The other reads it and deletes it.

   Why: two tests that touch one path can run in either order, or both at once.

3. 3

   Add tempfile under [dev-dependencies]. At the top of each test, make a fresh folder and keep it in a variable. Build every path inside that folder.

   Example

   let dir = tempfile::tempdir()?; and then let path = dir.path().join("shares.txt");. Two tests can both use the name shares.txt now, each in its own folder.

   Why: each tempdir() call makes a new folder with a unique name, and dropping the TempDir deletes it when the test ends.

4. 4

   A test that reads a file writes that file first, itself. Put setup that many tests share in a helper function, such as scratch().

   Example

   saved_shares_load_back calls save_shares(&path, &[334, 333, 333])? itself, and only then load_shares(&path)?. It no longer relies on another test.

   Why: a test that builds its own world gives the same result alone, in any order, and in parallel.

5. 5

   Run plain cargo test a few times, and the fixed test alone. Take --test-threads=1 back out of any script that used it as a cure.

   Example

   cargo test and cargo test saved_shares_load_back both pass, run after run.

   Why: one thread only hid the problem; the fix has to hold with many threads.