tvix: evaluating any depot attribute seems to force the entire readTree hierarchy

#275
Opened by sterni at 2023-05-30T22·02+00

When evaluating any attribute belonging to depot, tvix crashes due to it not having proper bytestrings (this is besides the point for now, b/189 describes the problem):

tvix-repl> (import ./. {}).nix.stateMonad.pure
thread 'main' panicked at 'byte index 1 is not a char boundary; it is inside '🕰' (bytes 0..4) of `🕰️`', eval/src/builtins/mod.rs:879:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is caused by this line in readTree:

45:      isVisible = f: f == ".skip-subtree" || f == ".skip-tree" || (substring 0 1 f) != ".";

However, it should never be necessary to check isVisible on //fun/🕰️ for, say nix.stateMonad.pure. We can verify this in C++ Nix:

let
  overrides = {
    builtins = builtins // {
      readDir = dir: builtins.trace dir (builtins.readDir dir);
      scopedImport = o: builtins.scopedImport (o // overrides);
      import = builtins.scopedImport overrides;
    };
    import = builtins.scopedImport overrides;
  };
in
(builtins.scopedImport overrides ./. { }).nix.stateMonad.pure

# yields:
#
# trace: /home/lukas/src/depot
# trace: /home/lukas/src/depot/corp
# trace: /home/lukas/src/depot/docs
# trace: /home/lukas/src/depot/fun
# trace: /home/lukas/src/depot/garbage
# trace: /home/lukas/src/depot/lisp
# trace: /home/lukas/src/depot/net
# trace: /home/lukas/src/depot/nix
# trace: /home/lukas/src/depot/ops
# trace: /home/lukas/src/depot/third_party
# trace: /home/lukas/src/depot/tools
# trace: /home/lukas/src/depot/tvix
# trace: /home/lukas/src/depot/users
# trace: /home/lukas/src/depot/views
# trace: /home/lukas/src/depot/web
# trace: /home/lukas/src/depot/nix/binify
# trace: /home/lukas/src/depot/nix/bufCheck
# trace: /home/lukas/src/depot/nix/buildGo
# trace: /home/lukas/src/depot/nix/buildLisp
# trace: /home/lukas/src/depot/nix/buildManPages
# trace: /home/lukas/src/depot/nix/buildkite
# trace: /home/lukas/src/depot/nix/dependency-analyzer
# trace: /home/lukas/src/depot/nix/drvSeqL
# trace: /home/lukas/src/depot/nix/emptyDerivation
# trace: /home/lukas/src/depot/nix/escapeExecline
# trace: /home/lukas/src/depot/nix/getBins
# trace: /home/lukas/src/depot/nix/lazy-deps
# trace: /home/lukas/src/depot/nix/mergePatch
# trace: /home/lukas/src/depot/nix/netstring
# trace: /home/lukas/src/depot/nix/nint
# trace: /home/lukas/src/depot/nix/nix-1p
# trace: /home/lukas/src/depot/nix/readTree
# trace: /home/lukas/src/depot/nix/renderMarkdown
# trace: /home/lukas/src/depot/nix/runExecline
# trace: /home/lukas/src/depot/nix/runTestsuite
# trace: /home/lukas/src/depot/nix/sparseTree
# trace: /home/lukas/src/depot/nix/stateMonad
# trace: /home/lukas/src/depot/nix/tag
# trace: /home/lukas/src/depot/nix/tailscale
# trace: /home/lukas/src/depot/nix/utils
# trace: /home/lukas/src/depot/nix/writeElispBin
# trace: /home/lukas/src/depot/nix/writeExecline
# trace: /home/lukas/src/depot/nix/writeScript
# trace: /home/lukas/src/depot/nix/writeScriptBin
# trace: /home/lukas/src/depot/nix/writers
# trace: /home/lukas/src/depot/nix/yants
# trace: /home/lukas/src/depot/nix/stateMonad/tests
# <LAMBDA>

The tvix runtime trace shows that we apparently walk the tree deep enough to encounter the dreaded mantelpiece clock—so likely a(nother) case of overeager evaluation.

  1. Nevermind — it's better to trace substring here. The readDir of //fun also happens in C++ Nix which incurs a isVisible check (including the substring) on //fun/🕰️.

    sterni at 2023-05-30T22·05+00

  2. sterni closed this issue at 2023-05-30T22·05+00