nix-build -A tvix.store fails to build (test failures) on MacOS

#403
Opened by flokli at 2024-05-25T20·16+00

I invoked nix-build -A tvix.store on a current depot checkout, and noticed it currently fails to build on a MacOS with Nix installed (and sandbox = true in /etc/nix/nix.conf), as some of the from_addr PathInfoService tests fail:

failures:

---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http' panicked at src/dynamic_store.rs:154:1:
Attempted to create a NULL object.
stack backtrace:
   0: std::panicking::begin_panic
   1: system_configuration::dynamic_store::SCDynamicStore::create
   2: core::ops::function::FnOnce::call_once
   3: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   4: once_cell::imp::initialize_or_wait
   5: once_cell::imp::OnceCell<T>::initialize
   6: reqwest::async_impl::client::ClientBuilder::build
   7: reqwest::async_impl::client::Client::new
   8: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   9: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  11: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http::{{closure}}
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  16: tokio::runtime::scheduler::current_thread::Context::enter
  17: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  18: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  19: tokio::runtime::context::scoped::Scoped<T>::set
  20: tokio::runtime::context::set_scheduler::{{closure}}
  21: std::thread::local::LocalKey<T>::try_with
  22: tokio::runtime::context::set_scheduler
  23: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  24: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  25: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  26: tokio::runtime::context::runtime::enter_runtime
  27: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  28: tokio::runtime::runtime::Runtime::block_on
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http
  30: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http::{{closure}}
  31: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys' panicked at /once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: reqwest::async_impl::client::ClientBuilder::build
   6: reqwest::async_impl::client::Client::new
   7: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   8: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
   9: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::Context::enter
  16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::set_scheduler::{{closure}}
  20: std::thread::local::LocalKey<T>::try_with
  21: tokio::runtime::context::set_scheduler
  22: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  23: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  27: tokio::runtime::runtime::Runtime::block_on
  28: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys::{{closure}}
  30: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https' panicked at /once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: reqwest::async_impl::client::ClientBuilder::build
   6: reqwest::async_impl::client::Client::new
   7: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   8: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
   9: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::Context::enter
  16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::set_scheduler::{{closure}}
  20: std::thread::local::LocalKey<T>::try_with
  21: tokio::runtime::context::set_scheduler
  22: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  23: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  27: tokio::runtime::runtime::Runtime::block_on
  28: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https::{{closure}}
  30: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key' panicked at /once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: reqwest::async_impl::client::ClientBuilder::build
   6: reqwest::async_impl::client::Client::new
   7: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   8: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
   9: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::Context::enter
  16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::set_scheduler::{{closure}}
  20: std::thread::local::LocalKey<T>::try_with
  21: tokio::runtime::context::set_scheduler
  22: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  23: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  27: tokio::runtime::runtime::Runtime::block_on
  28: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key::{{closure}}
  30: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath' panicked at /once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: reqwest::async_impl::client::ClientBuilder::build
   6: reqwest::async_impl::client::Client::new
   7: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   8: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
   9: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::Context::enter
  16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::set_scheduler::{{closure}}
  20: std::thread::local::LocalKey<T>::try_with
  21: tokio::runtime::context::set_scheduler
  22: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  23: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  27: tokio::runtime::runtime::Runtime::block_on
  28: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath::{{closure}}
  30: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port' panicked at /once_cell-1.19.0/src/lib.rs:1311:25:
Lazy instance has previously been poisoned
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   3: once_cell::imp::initialize_or_wait
   4: once_cell::imp::OnceCell<T>::initialize
   5: reqwest::async_impl::client::ClientBuilder::build
   6: reqwest::async_impl::client::Client::new
   7: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   8: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
   9: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port::{{closure}}
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::Context::enter
  16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  18: tokio::runtime::context::scoped::Scoped<T>::set
  19: tokio::runtime::context::set_scheduler::{{closure}}
  20: std::thread::local::LocalKey<T>::try_with
  21: tokio::runtime::context::set_scheduler
  22: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  23: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  25: tokio::runtime::context::runtime::enter_runtime
  26: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  27: tokio::runtime::runtime::Runtime::block_on
  28: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port::{{closure}}
  30: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https
    pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http
    pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath
    pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port
    pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key
    pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys

test result: FAILED. 62 passed; 6 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s

error: builder for '/nix/store/mdzjj05if7d34sjlc7cxywa74qrzgjgy-run-tests-rust_tvix-store-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > failures:
       >     pathinfoservice::from_addr::tests::test_from_addr_tokio::case_11_correct_nix_https
       >     pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http
       >     pathinfoservice::from_addr::tests::test_from_addr_tokio::case_13_correct_nix_http_with_subpath
       >     pathinfoservice::from_addr::tests::test_from_addr_tokio::case_14_correct_nix_http_with_subpath_and_port
       >     pathinfoservice::from_addr::tests::test_from_addr_tokio::case_15_correct_nix_https_with_trusted_public_key
       >     pathinfoservice::from_addr::tests::test_from_addr_tokio::case_16_correct_nix_https_with_two_trusted_public_keys
       >
       > test result: FAILED. 62 passed; 6 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
       >
       For full logs, run 'nix-store -l /nix/store/mdzjj05if7d34sjlc7cxywa74qrzgjgy-run-tests-rust_tvix-store-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/w86g7i2xwbdcrf101z4gpk8hgnpsrlk5-rust_tvix-store-0.1.0-linked.drv' failed to build

I recall these tests needing SSL_CERT_FILE to be set (on Linux) to succeed, as the TLS stack wants to load some certificates, even before doing any actual connection:

https://cs.tvl.fyi/depot@edd93b19628e655c5903dd79393e6387b23b8168/-/blob/tvix/store/default.nix?L43

Maybe this is a similar problem, and we need to poke a darwin-specific hole into the sandbox, as it also tries to access another MacOS-specific path?

  1. Could it be as simple as setting __darwinAllowLocalNetworking = true on the derivation? To test it I need to enable sandboxing on my own MacOS nix daemon but I did some research and SDDynamicStore is among other things used to get network status and nix has a __darwinAllowLocalNetworking flag for allowing local networking in the sandbox.

    griff at 2024-05-25T22·21+00

  2. I went down the rabbit hole of setting __darwinAllowLocalNetworking for the run-tests- derivation, but it's not sufficient. Above stacktrace has a lot of "Lazy instance has previously been poisoned", but this one is interesting:

    
---- pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http stdout ----
thread 'pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http' panicked at src/dynamic_store.rs:154:1:
Attempted to create a NULL object.
stack backtrace:
   0: std::panicking::begin_panic
   1: system_configuration::dynamic_store::SCDynamicStore::create
   2: core::ops::function::FnOnce::call_once
   3: once_cell::imp::OnceCell<T>::initialize::{{closure}}
   4: once_cell::imp::initialize_or_wait
   5: once_cell::imp::OnceCell<T>::initialize
   6: reqwest::async_impl::client::ClientBuilder::build
   7: reqwest::async_impl::client::Client::new
   8: tvix_store::pathinfoservice::nix_http::NixHTTPPathInfoService<BS,DS>::new
   9: tvix_store::pathinfoservice::from_addr::from_addr::{{closure}}
  10: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::{{closure}}
  11: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http::{{closure}}
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
  14: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
  15: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
  16: tokio::runtime::scheduler::current_thread::Context::enter
  17: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
  18: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
  19: tokio::runtime::context::scoped::Scoped<T>::set
  20: tokio::runtime::context::set_scheduler::{{closure}}
  21: std::thread::local::LocalKey<T>::try_with
  22: tokio::runtime::context::set_scheduler
  23: tokio::runtime::scheduler::current_thread::CoreGuard::enter
  24: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  25: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  26: tokio::runtime::context::runtime::enter_runtime
  27: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  28: tokio::runtime::runtime::Runtime::block_on
  29: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http
  30: tvix_store::pathinfoservice::from_addr::tests::test_from_addr_tokio::case_12_correct_nix_http::{{closure}}
  31: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

    flokli at 2024-06-17T18·11+00

  3. It is a sandboxing issue, it fails as I set sandbox = true in my nix.conf (and restart the nix-daemon).

    If I uncomment that, which then uses the "relaxed" mode, the tests successfully run inside the nix derivation.

    I think what's going on is that the test (or reqwest) tries to load SystemConfiguration, but gets blocked as no hole for it is punched through the sandbox.

    I had no success in finding a way to propagate impure host deps into the runcommand via the crate2nix machinery.

    flokli at 2024-06-17T19·28+00