tvix: Multiple simultaneous builds cause the daemon to abort

#58
Opened by aspen at 2020-09-01T00·01+00

systemd log:

Aug 31 23:58:39 whitby systemd[1]: nix-daemon.service: Main process exited, code=killed, status=6/ABRT
Aug 31 23:58:39 whitby systemd[1]: nix-daemon.service: Failed with result 'signal'.
Aug 31 23:58:39 whitby systemd[1]: nix-daemon.service: Consumed 10.965s CPU time, no IP traffic.
Aug 31 23:58:40 whitby systemd-coredump[5963]: Process 57884 (nix-daemon) of user 0 dumped core.

core dumps available at

/var/lib/systemd/coredump/
 core.nix-daemon.0.375c4aa8660341d293722a7e7dc8192e.3001.1598918246000000000000.lz4
 core.nix-daemon.0.375c4aa8660341d293722a7e7dc8192e.45259.1598918300000000000000.lz4
 core.nix-daemon.0.375c4aa8660341d293722a7e7dc8192e.57884.1598918319000000000000.lz4

  1. more info: this appears to be triggered by building certain nixos systems (frog, camden, chupacabra) - see https://buildkite.com/tvl/depot/builds/3191

    aspen at 2020-09-01T00·05+00

  2. More information - this appears to only happen when multiple nixos systems are building simultaneously. Maybe this is in fact a resource exhaustion?

    aspen at 2020-09-01T02·15+00

  3. Disregard that last comment, the behavior was observed after switching back to upstream nix :/

    aspen at 2020-09-01T02·33+00

  4. It appears that the last log line before the abort is the following, when building frog:

    Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man...

    aspen at 2020-09-01T02·36+00

  5. grfn changed the subject of this issue from "tvix: Abort when running full CI build" to "tvix: Abort when building frog" at 2020-09-01T02·47+00

  6. full build log:

    [grfn@whitby:/depot]$ nix build -f . users.tazjin.nixos.frogSystem
building '/nix/store/a08zmr6narx7gknkb3anq9d3r2kihvqd-man-cache.drv'[/nix/store/a08zmr6narx7gknkb3anq9d3r2kihvqd-man-cache.drv]
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/man7'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/zh_TW...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/zh_TW/man5'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/fi...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/fi/man1'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/sv...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/sv/man8'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/ko...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/ko/man1'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/ru...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/ru/man1'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/id...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/id/man8'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/de...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/de/man5'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/zh_CN...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/zh_CN/man1'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/ja...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/ja/man1'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/tr...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/tr/man1'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/da...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/da/man8'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/it...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/it/man8'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/pl...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/pl/man3'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/hu...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/hu/man5'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/pt_BR...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/pt_BR/man5'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/fr...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/fr/man3'. Wait...done.
Processing manual pages under /nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/cs...
Updating index cache for path `/nix/store/nv6ppp48581rk8fgpvzfiqm0j2f6q723-man-paths/share/man/cs/man5'. Wait...done.
Processing manual pages under /run/current-system/sw/share/man...
/nix/store/mz4nbxb75b7nh02np6qh5gk1a8xcdf5z-man-db-2.9.3/bin/mandb: warning: cannot create catdir /var/cache/man/nixos
/nix/store/mz4nbxb75b7nh02np6qh5gk1a8xcdf5z-man-db-2.9.3/bin/mandb: can't create index cache /var/cache/man/nixos/51565: No such file or directory
/nix/store/mz4nbxb75b7nh02np6qh5gk1a8xcdf5z-man-db-2.9.3/bin/mandb: can't update index cache /var/cache/man/nixos/51565: No such file or directory
done.
building '/nix/store/kv7lzr7nyxfxmj8z4dck8ry53pb0xx56-nixos-manual.desktop_fish-completions.drv'[/nix/store/kv7lzr7nyxfxmj8z4dck8ry53pb0xx56-nixos-manual.desktop_fish-completions.drv]
E20200901 02:45:50.691630 51302 shared.cc:306] error: Operation failed: INTERNAL: build of '/nix/store/5nly6c1j523d7js2k2a9aq1k6d7i670l-nixos-system-frog-20.09pre-git.drv' failed (exit code 100)

    aspen at 2020-09-01T02·47+00

  7. The issue described in the description here appears to have just been that frog's man-db doesn't build outside of a sandbox (see https://b.tvl.fyi/issues/58#comment-80). The original issue - wherre building multiple nixos'es simultaneously causes the daemon to abort - does in fact appear to be a real issue.

    aspen at 2020-09-02T03·28+00

  8. grfn changed the subject of this issue from "tvix: Abort when building frog" to "tvix: Multiple simultaneous nixos builds cause the daemon to abort" at 2020-09-02T03·29+00

  9. here's a backtrace of the current situation:

    (gdb) bt
#0  0x00007f7bfedcf08a in raise () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libc.so.6
#1  0x00007f7bfedb9528 in abort () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libc.so.6
#2  0x00007f7c00359373 in nix::LocalStore::buildPaths(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, nix::BuildMode) ()
   from /nix/store/pycasfpda4i12fsli754qy63k0xqjh40-tvix-2.3.4/lib64/libnixstore.so
#3  0x00007f7c00359653 in virtual thunk to nix::LocalStore::buildPaths(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, nix::BuildMode) ()
   from /nix/store/pycasfpda4i12fsli754qy63k0xqjh40-tvix-2.3.4/lib64/libnixstore.so
#4  0x0000000000420a5b in nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}::operator()() const ()
#5  0x00000000004208bd in std::__1::__function::__func<nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, grpc::Status ()>::operator()() ()
#6  0x000000000041ad7c in nix::daemon::WorkerServiceImpl::HandleExceptions(std::__1::function<grpc::Status ()>, std::__1::basic_string_view<char, std::__1::char_traits<char> >) ()
#7  0x0000000000419877 in nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*) ()
#8  0x00007f7bfff07006 in std::__1::__function::__func<nix::proto::WorkerService::Service::Service()::$_5, std::__1::allocator<nix::proto::WorkerService::Service::Service()::$_5>, grpc::Status (nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)>::operator()(nix::proto::WorkerService::Service*&&, grpc_impl::ServerContext*&&, nix::proto::BuildPathsRequest const*&&, grpc_impl::ServerWriter<nix::proto::BuildEvent>*&&) () from /nix/store/pycasfpda4i12fsli754qy63k0xqjh40-tvix-2.3.4/lib64/libnixproto.so
#9  0x00007f7bfff70387 in grpc::Status grpc_impl::internal::CatchingFunctionHandler<grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&) () from /nix/store/pycasfpda4i12fsli754qy63k0xqjh40-tvix-2.3.4/lib64/libnixproto.so
#10 0x00007f7bfff6fb04 in grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) () from /nix/store/pycasfpda4i12fsli754qy63k0xqjh40-tvix-2.3.4/lib64/libnixproto.so
#11 0x00007f7bffda7ea7 in grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception() ()
   from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#12 0x00007f7bffda7c8d in grpc_impl::Server::SyncRequest::CallData::Run(std::__1::shared_ptr<grpc_impl::Server::GlobalCallbacks> const&, bool) ()
   from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#13 0x00007f7bffdb1f85 in grpc::ThreadManager::MainWorkLoop() () from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#14 0x00007f7bffdb2b7c in grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) ()
   from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#15 0x00007f7bff749c74 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::__invoke(void*) () from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgpr.so.10
#16 0x00007f7bff5b9ead in start_thread () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libpthread.so.0
#17 0x00007f7bfee8ed2f in clone () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libc.so.6

    aspen at 2020-09-02T03·33+00

  10. disassembly of the coredump appears to indicate we're hitting a stack overflow(?):

    => 0x00007f7c00359373 <+1891>:	callq  0x7f7c0030c330 <__stack_chk_fail@plt

    aspen at 2020-09-02T03·37+00

  11. a little more context, including the instruction that's jumping to +1891:

       0x00007f7c003592d6 <+1734>:	je     0x7f7c003592e4 <_ZN3nix10LocalStore10buildPathsERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEERKNS1_3setINS1_12basic_stringIcS4_NS1_9allocatorIcEEEENS1_4lessISB_EENS9_ISB_EEEENS_9BuildModeE+1748>
   0x00007f7c003592d8 <+1736>:	mov    0x38(%rsp),%rdx
   0x00007f7c003592dd <+1741>:	mov    0x30(%rsp),%rcx
   0x00007f7c003592e2 <+1746>:	jmp    0x7f7c003592ec <_ZN3nix10LocalStore10buildPathsERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEERKNS1_3setINS1_12basic_stringIcS4_NS1_9allocatorIcEEEENS1_4lessISB_EENS9_ISB_EEEENS_9BuildModeE+1756>
   0x00007f7c003592e4 <+1748>:	shr    %rcx
   0x00007f7c003592e7 <+1751>:	lea    0x29(%rsp),%rdx
   0x00007f7c003592ec <+1756>:	mov    %rbp,%rdi
   0x00007f7c003592ef <+1759>:	mov    $0xd,%esi
   0x00007f7c003592f4 <+1764>:	callq  0x7f7c0030f030 <_ZN4absl6StatusC1ENS_10StatusCodeENSt3__117basic_string_viewIcNS2_11char_traitsIcEEEE@plt>
   0x00007f7c003592f9 <+1769>:	testb  $0x1,0x28(%rsp)
   0x00007f7c003592fe <+1774>:	je     0x7f7c0035930a <_ZN3nix10LocalStore10buildPathsERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEERKNS1_3setINS1_12basic_stringIcS4_NS1_9allocatorIcEEEENS1_4lessISB_EENS9_ISB_EEEENS_9BuildModeE+1786>
   0x00007f7c00359300 <+1776>:	mov    0x38(%rsp),%rdi
   0x00007f7c00359305 <+1781>:	callq  0x7f7c0030e770 <_ZdlPv@plt>
   0x00007f7c0035930a <+1786>:	testb  $0x1,0x10(%rsp)
   0x00007f7c0035930f <+1791>:	je     0x7f7c0035931b <_ZN3nix10LocalStore10buildPathsERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEERKNS1_3setINS1_12basic_stringIcS4_NS1_9allocatorIcEEEENS1_4lessISB_EENS9_ISB_EEEENS_9BuildModeE+1803>
   0x00007f7c00359311 <+1793>:	mov    0x20(%rsp),%rdi
   0x00007f7c00359316 <+1798>:	callq  0x7f7c0030e770 <_ZdlPv@plt>
   0x00007f7c0035931b <+1803>:	mov    0x48(%rsp),%rsi
   0x00007f7c00359320 <+1808>:	lea    0x40(%rsp),%rdi
   0x00007f7c00359325 <+1813>:	callq  0x7f7c003111a0 <_ZNSt3__16__treeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS_4lessIS6_EENS4_IS6_EEE7destroyEPNS_11__tree_nodeIS6_PvEE@plt>
   0x00007f7c0035932a <+1818>:	mov    0x60(%rsp),%rsi
   0x00007f7c0035932f <+1823>:	lea    0x58(%rsp),%rdi
   0x00007f7c00359334 <+1828>:	callq  0x7f7c0030cab0 <_ZNSt3__16__treeINS_10shared_ptrIN3nix4GoalEEENS2_15CompareGoalPtrsENS_9allocatorIS4_EEE7destroyEPNS_11__tree_nodeIS4_PvEE@plt>
   0x00007f7c00359339 <+1833>:	lea    0xa0(%rsp),%rdi
   0x00007f7c00359341 <+1841>:	callq  0x7f7c0030ddf0 <_ZN3nix6WorkerD2Ev@plt>
   0x00007f7c00359346 <+1846>:	mov    %fs:0x28,%rax
   0x00007f7c0035934f <+1855>:	cmp    0x210(%rsp),%rax
   0x00007f7c00359357 <+1863>:	jne    0x7f7c00359373 <_ZN3nix10LocalStore10buildPathsERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEERKNS1_3setINS1_12basic_stringIcS4_NS1_9allocatorIcEEEENS1_4lessISB_EENS9_ISB_EEEENS_9BuildModeE+1891>
   0x00007f7c00359359 <+1865>:	mov    %rbp,%rax
   0x00007f7c0035935c <+1868>:	add    $0x218,%rsp
   0x00007f7c00359363 <+1875>:	pop    %rbx
   0x00007f7c00359364 <+1876>:	pop    %r12
   0x00007f7c00359366 <+1878>:	pop    %r13
   0x00007f7c00359368 <+1880>:	pop    %r14
   0x00007f7c0035936a <+1882>:	pop    %r15
   0x00007f7c0035936c <+1884>:	pop    %rbp
   0x00007f7c0035936d <+1885>:	retq
   0x00007f7c0035936e <+1886>:	callq  0x7f7c0030e780 <abort@plt>
=> 0x00007f7c00359373 <+1891>:	callq  0x7f7c0030c330 <__stack_chk_fail@plt>

    aspen at 2020-09-02T03·40+00

  12. with dontStrip:

    #0  0x00007f8cb5efb08a in raise () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libc.so.6
#1  0x00007f8cb5ee5528 in abort () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libc.so.6
#2  0x00007f8cb7485383 in nix::Worker::Worker (this=0x7f8c8f7f5a40, store=..., log_sink=...) at /build/nix/src/libstore/build.cc:4250
#3  nix::LocalStore::buildPaths (this=0x7f8c8f7f5760, log_sink=..., drvPaths=..., build_mode=nix::bmNormal) at /build/nix/src/libstore/build.cc:4713
#4  0x00007f8cb7485663 in virtual thunk to nix::LocalStore::buildPaths(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, nix::BuildMode) () at /build/nix/src/libstore/derivations.hh:83
#5  0x0000000000420bfb in nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}::operator()() const (this=0x7f8c8f7f5e58) at /build/nix/src/nix-daemon/nix-daemon-proto.cc:334
#6  0x0000000000420a5d in std::__1::__invoke<nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}&> (__f=...) at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/type_traits:3539
#7  std::__1::__invoke_void_return_wrapper<grpc::Status>::__call<nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}&>(nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}&) (__args=...)
    at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/__functional_base:317
#8  std::__1::__function::__alloc_func<nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, grpc::Status ()>::operator()() (this=<optimized out>) at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:1540
#9  std::__1::__function::__func<nix::daemon::WorkerServiceImpl::BuildPaths(grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, grpc::Status ()>::operator()() (this=<optimized out>) at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:1714
#10 0x000000000041af1c in std::__1::__function::__value_func<grpc::Status ()>::operator()() const (this=0x0) at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:1867
#11 std::__1::function<grpc::Status ()>::operator()() const (this=0x0) at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:2473
#12 nix::daemon::WorkerServiceImpl::HandleExceptions(std::__1::function<grpc::Status ()>, std::__1::basic_string_view<char, std::__1::char_traits<char> >) (this=<optimized out>,
    fn=<error reading variable: Cannot access memory at address 0x0>, methodName=...) at /build/nix/src/nix-daemon/nix-daemon-proto.cc:771
#13 0x0000000000419877 in nix::daemon::WorkerServiceImpl::BuildPaths (this=0x7f8c8f7f5760, request=<optimized out>, writer=<optimized out>) at /build/nix/src/nix-daemon/nix-daemon-proto.cc:315
#14 0x00007f8cb7033006 in nix::proto::WorkerService::Service::Service()::$_5::operator()(nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*) const (this=<optimized out>, service=0x7f8c8f7f5760, ctx=0x0, req=0x7f8cb5efb08a <raise+202>, writer=0x0)
    at /nix/store/y3i4zc7n9fzbjbx0zsniph5spjvii94j-nix-proto-srcs/libproto/worker.grpc.pb.cc:936
#15 std::__1::__invoke<nix::proto::WorkerService::Service::Service()::$_5&, nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*> (__f=..., __args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0)
    at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/type_traits:3539
#16 std::__1::__invoke_void_return_wrapper<grpc::Status>::__call<nix::proto::WorkerService::Service::Service()::$_5&, nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*>(nix::proto::WorkerService::Service::Service()::$_5&, nix::proto::WorkerService::Service*&&, grpc_impl::ServerContext*&&, nix::proto::BuildPathsRequest const*&&, grpc_impl::ServerWriter<nix::proto::BuildEvent>*&&) (__args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0, __args=@0x7f8c8f7f5760: 0x0)
    at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/__functional_base:317
#17 std::__1::__function::__alloc_func<nix::proto::WorkerService::Service::Service()::$_5, std::__1::allocator<nix::proto::WorkerService::Service::Service()::$_5>, grpc::Status (nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)>::operator()(nix::proto::WorkerService::Service*&&, grpc_impl::ServerContext*&&, nix::proto::BuildPathsRequest const*&&, grpc_impl::ServerWriter<nix::proto::BuildEvent>*&&) (this=<optimized out>, __arg=@0x7f8c8f7f5760: 0x0, __arg=@0x7f8c8f7f5760: 0x0, __arg=@0x7f8c8f7f5760: 0x0, __arg=@0x7f8c8f7f5760: 0x0)
    at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:1540
#18 std::__1::__function::__func<nix::proto::WorkerService::Service::Service()::$_5, std::__1::allocator<nix::proto::WorkerService::Service::Service()::$_5>, grpc::Status (nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)>::operator()(nix::proto::WorkerService::Service*&&, grpc_impl::ServerContext*&&, nix::proto::BuildPathsRequest const*&&, grpc_impl::ServerWriter<nix::proto::BuildEvent>*&&) (this=<optimized out>, __arg=@0x7f8c8f7f5760: 0x0, __arg=@0x7f8c8f7f5760: 0x0, __arg=@0x7f8c8f7f5760: 0x0, __arg=@0x7f8c8f7f5760: 0x0)
    at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:1714
#19 0x00007f8cb709c387 in std::__1::__function::__value_func<grpc::Status (nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)>::operator()(nix::proto::WorkerService::Service*&&, grpc_impl::ServerContext*&&, nix::proto::BuildPathsRequest const*&&, grpc_impl::ServerWriter<nix::proto::BuildEvent>*&&) const (this=<optimized out>,
    __args=@0x7f8c8f7f5ec8: 0x7f8c8f7f5f38, __args=@0x7f8c8f7f5ec8: 0x7f8c8f7f5f38, __args=@0x7f8c8f7f5ec8: 0x7f8c8f7f5f38, __args=@0x7f8c8f7f5ec8: 0x7f8c8f7f5f38)
    at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:1867
#20 std::__1::function<grpc::Status (nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*)>::operator()(nix::proto::WorkerService::Service*, grpc_impl::ServerContext*, nix::proto::BuildPathsRequest const*, grpc_impl::ServerWriter<nix::proto::BuildEvent>*) const (this=<optimized out>, __arg=0x7f8c8f7f5f38, __arg=0x7f8c8f7f5f38,
    __arg=0x7f8c8f7f5f38, __arg=0x7f8c8f7f5f38) at /nix/store/jksylk0cs111zvgfni5mhzxphkp43g6z-libc++-10.0.1/include/c++/v1/functional:2473
#21 grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}::operator()() const (this=<optimized out>) at /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/include/grpcpp/impl/codegen/method_handler_impl.h:191
#22 grpc_impl::internal::CatchingFunctionHandler<grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&) (handler=...) at /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/include/grpcpp/impl/codegen/method_handler_impl.h:42
#23 0x00007f8cb709bb04 in grpc_impl::internal::ServerStreamingHandler<nix::proto::WorkerService::Service, nix::proto::BuildPathsRequest, nix::proto::BuildEvent>::RunHandler (this=0x1f06510, param=...)
    at /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/include/grpcpp/impl/codegen/method_handler_impl.h:190
#24 0x00007f8cb6ed3ea7 in grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception() () from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#25 0x00007f8cb6ed3c8d in grpc_impl::Server::SyncRequest::CallData::Run(std::__1::shared_ptr<grpc_impl::Server::GlobalCallbacks> const&, bool) ()
   from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#26 0x00007f8cb6eddf85 in grpc::ThreadManager::MainWorkLoop() () from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#27 0x00007f8cb6edeb7c in grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) () from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgrpc++.so.1
#28 0x00007f8cb6875c74 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::__invoke(void*) () from /nix/store/dqbirffbhq8psssndwjnxwbrf2nb0sl8-grpc-1.30.0/lib/libgpr.so.10
#29 0x00007f8cb66e5ead in start_thread () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libpthread.so.0
#30 0x00007f8cb5fbad2f in clone () from /nix/store/mh78fk3x12q2a77srgkzv16h0irl8r61-glibc-2.31/lib/libc.so.6

    aspen at 2020-09-06T22·02+00

  13. Here's the abort: https://cs.tvl.fyi/depot/-/blob/third_party/nix/src/libstore/build.cc#L4248-4251

    aspen at 2020-09-06T22·05+00

  14. upstream nix forks a child process per connection, so that flag is isolated to each request: https://cs.tvl.fyi/depot/-/blob/third_party/nix/src/nix-daemon/nix-daemon-legacy.cc#L1056-1061

    aspen at 2020-09-06T22·18+00

  15. grfn changed the subject of this issue from "tvix: Multiple simultaneous nixos builds cause the daemon to abort" to "tvix: Multiple simultaneous builds cause the daemon to abort" at 2020-09-06T22·18+00

  16. https://cl.tvl.fyi/c/depot/+/1938/2

    aspen at 2020-09-06T22·23+00

  17. ya I think riking's original suggestion of keeping an absl::Mutex there might be better

    aspen at 2020-09-06T22·30+00

  18. https://cl.tvl.fyi/c/depot/+/1940/3 fixes this

    aspen at 2020-09-07T17·39+00

  19. grfn closed this issue at 2020-09-12T16·21+00