tvix-glue: add test_resources style tests

#357
Opened by flokli at 2023-12-13T18·23+00

Some of our tests require impure builtins, like those using builtins.derivation or piece of code that triggers imports into the store (or not).

We can't test this from tvix/eval/src/tests, as it doesn't know about the derivation builtin, and doesn't have a "real" EvalIO interface available.

tvix-glue knows about both of these things, and tvix/glue/src/tvix_store_io.rs and tvix/glue/src/builtins/mod.rs already started getting some small tests (defined at unit tests).

Dealing with these tests in there gets a bit annoying (escaping {} in format strings is very unreadable, especially if you use {} to put a variable inside).

We also want to add multiline tests, like

[
  ("foo" + "bar")
  ("hello " + ./directory/default.nix)
  (/foo/bar + /foo/bar)
  (/deep + "/thought")
  (builtins.concatStringsSep "/" [ ./directory/default.nix "deep" ])
]

We should replicate something like the test_resources infrastructure in tvix/eval over to there, then possibly see which of these tests can be move to become file-based (use that infrastructure).

At least the file-specific ones would also need to have a way to specify how to set up the files that we want to import.

  1. flokli updated the body of this issue at 2023-12-13T18·24+00

  2. It is probably wise to move the place where tvix_tests and nix_tests (“lang tests”) are executed to a place where a proper EvalIO interface is available. The long term idea would be to have a common test suite for C++ Nix and Tvix, so spreading it over more places on our side may be unwise?

    sterni at 2023-12-13T18·49+00

  3. crate2nix builds every crate inside our workspace in isolation, so we can't really reach outside each crate either.

    Maybe the tests should be moved into their own crate, and use the necessary bits and pieces from tvix-eval and glue.

    flokli at 2024-01-14T17·11+00

  4. There's now tvix_tests in both eval and glue, with the ones requiring more builtins moved to the latter.

    This has also been migrated from test_resources to rstest in the meantime.

    I think properly categorizing various tests in the test suite, and defining a set of fixtures to deal with is out of scope for this issue.

    flokli at 2024-05-23T15·52+00

  5. flokli closed this issue at 2024-05-23T15·52+00