tvix: add criterion benchmark for store path ingestion
While we don't want to do any too fine-grained performance tuning in the evaluator yet, it'd still be helpful to be able to easily reason about the impact of a change for evaluation.
https://github.com/bheisler/criterion.rs provides a nice way to create some small benchmarks.
We should write a little bench for eval.
- Seed a pinned nixpkgs checkout into a simple implementation of EvalIO (to make sure we don't benchmark the filesystem or more complicated EvalIO implementations)
- Evaluate a bunch of attribute sets (like hello, stdenv etc. from our output path regression tests)
For now, this can be run on-demand by developers, invoking cargo bench
before and after a certain change locally, similar to the existing benches.
An example for ATerm parsing is in https://cl.tvl.fyi/c/depot/+/9733
I noticed that Gerrit adds extra git footers to commits, noting things like who approved the commit.
Looping it into CI somehow could be a later step.
It would be nifty if we logged these values in a git footer, like
X-Benchmark-Milliseconds: 1058
instead of doing what OfBorg does with eval reports and just chucking it on some random webpage (that probably gets garbage-collected periodically).
amjoseph at 2023-09-13T10·22+00
I'm not sure pulling this into the critical path is a good idea.
flokli at 2023-10-16T08·37+00
Let's start with some criterion benchmarks first, once we have them we can still see how to expose them somewhere.
flokli at 2023-10-16T08·38+00
- flokli updated the body of this issue at 2023-10-16T08·39+00
- flokli changed the subject of this issue from "tvix: add criterion benchmark for some store paths" to "tvix: add criterion benchmark for store path ingestion" at 2023-10-16T08·54+00
@amjoseph Exposing it in CI should be discussed in another issue (please open one if you want to), we already have other criterion benches and it's independent of this specific testbench.
flokli at 2023-10-16T08·58+00
- flokli updated the body of this issue at 2023-10-16T08·58+00