tvix: stack overflow when calculating list of nixpkgs attributes using derivation-attr-paths.nix

#265
Opened by sterni at 2023-04-07T11·17+00

To reproduce:

> wget https://raw.githubusercontent.com/NixOS/cabal2nix/master/distribution-nixpkgs/derivation-attr-paths.nix
> tvix --expr '(import ./derivation-attr-paths.nix) { } # uses <nixpkgs>
thread 'main' has overflowed its stack
fatal runtime error: stack overflow

What's interesting here is that very few outPaths etc. should be calculated. This Nix expression was written to replace nix-env -qaP to obtain a list of attribute paths visible in nixpkgs — without the need to calculate every derivation's output path, as nix-env does.

  1. I think the underlying problem is a duplicate of b/261, which is our weird eagerness bug where we're traversing some path before it's ready.

    In your case this stackoverflows because the panic handler is trying to print a context value that is too large, and it's recursing into some infinitely large data structure or something like that. We probably don't actually want to print that ... I'll have to figure out what to do with that.

    For reference, here's a snippet from the backtrace of running this under rust-gdb:

    # ... and so on
#177172 0x00005634818a747b in core::fmt::builders::{impl#4}::field::{closure#0} () at library/core/src/fmt/builders.rs:322
#177173 core::result::Result::and_then<(), core::fmt::Error, (), core::fmt::builders::{impl#4}::field::{closure_env#0}> ()
    at library/core/src/result.rs:1372
#177174 core::fmt::builders::DebugTuple::field () at library/core/src/fmt/builders.rs:309
#177175 0x00005634818a9e39 in core::fmt::Formatter::debug_tuple_field1_finish () at library/core/src/fmt/mod.rs:2139
#177176 0x00005634816a1bcd in <&T as core::fmt::Debug>::fmt ()
#177177 0x00005634818a747b in core::fmt::builders::{impl#4}::field::{closure#0} () at library/core/src/fmt/builders.rs:322
#177178 core::result::Result::and_then<(), core::fmt::Error, (), core::fmt::builders::{impl#4}::field::{closure_env#0}> ()
    at library/core/src/result.rs:1372
#177179 core::fmt::builders::DebugTuple::field () at library/core/src/fmt/builders.rs:309
#177180 0x00005634818a9e39 in core::fmt::Formatter::debug_tuple_field1_finish () at library/core/src/fmt/mod.rs:2139
#177181 0x00005634816adcc0 in <tvix_eval::value::Value as core::fmt::Debug>::fmt ()
#177182 0x00005634818a747b in core::fmt::builders::{impl#4}::field::{closure#0} () at library/core/src/fmt/builders.rs:322
#177183 core::result::Result::and_then<(), core::fmt::Error, (), core::fmt::builders::{impl#4}::field::{closure_env#0}> ()
    at library/core/src/result.rs:1372
#177184 core::fmt::builders::DebugTuple::field () at library/core/src/fmt/builders.rs:309
#177185 0x00005634818a9e39 in core::fmt::Formatter::debug_tuple_field1_finish () at library/core/src/fmt/mod.rs:2139
#177186 0x0000563481722742 in <tvix_eval::value::thunk::ThunkRepr as core::fmt::Debug>::fmt ()
#177187 0x00005634818a88ce in core::fmt::write () at library/core/src/fmt/mod.rs:1208
#177188 0x0000563481884fa6 in core::fmt::Write::write_fmt<alloc::string::String> () at library/core/src/fmt/mod.rs:192
#177189 std::panicking::begin_panic_handler::{impl#0}::fill::{closure#0} () at library/std/src/panicking.rs:541
#177190 core::option::Option::get_or_insert_with<alloc::string::String, std::panicking::begin_panic_handler::{impl#0}::fill::{closure_env#0}> () at library/core/src/option.rs:1546
#177191 std::panicking::begin_panic_handler::PanicPayload::fill () at library/std/src/panicking.rs:539
#177192 std::panicking::begin_panic_handler::{impl#1}::get () at library/std/src/panicking.rs:557
#177193 0x0000563481885357 in std::panicking::rust_panic_with_hook () at library/std/src/panicking.rs:687
#177194 0x0000563481885109 in std::panicking::begin_panic_handler::{closure#0} () at library/std/src/panicking.rs:579
#177195 0x0000563481883b9c in std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::{closure_env#0}, !> () at library/std/src/sys_common/backtrace.rs:137
#177196 0x0000563481884e12 in std::panicking::begin_panic_handler () at library/std/src/panicking.rs:575
#177197 0x00005634815abde3 in core::panicking::panic_fmt () at library/core/src/panicking.rs:64
#177198 0x0000563481721c66 in tvix_eval::value::thunk::Thunk::upvalues_mut ()
#177199 0x00005634817218fa in tvix_eval::value::thunk::Thunk::finalise ()
#177200 0x00005634816bacd2 in tvix_eval::vm::VM::execute_bytecode ()
#177201 0x00005634816b54fd in tvix_eval::vm::VM::execute ()
#177202 0x00005634816c02e2 in tvix_eval::vm::run_lambda ()
#177203 0x00005634816cb620 in tvix_eval::Evaluation::evaluate ()
#177204 0x00005634815bbacb in tvix::interpret ()
#177205 0x00005634815bc93e in tvix::main ()

    tazjin at 2023-04-07T15·41+00

  2. I overflowed also the stack with the changes in https://cl.tvl.fyi/c/depot/+/8572/3.

    This didn't include any changes in eagerness, and is with the same attribute paths as before, so it might not be related to b/261.

    flokli at 2023-05-14T18·40+00

  3. Ignore the previous comment - this was me being stupid.

    flokli at 2023-05-14T19·06+00

  4. This is no longer an issue after r/6336, but the fix reveals a new issue: b/281

    sterni at 2023-06-20T10·53+00

  5. sterni closed this issue at 2023-06-20T10·53+00