tvix nix+https pathinfoservice is unreliable

#405
Opened by yuka at 2024-06-04T16·10+00

mount a nix+https:// remote

target/release/tvix-store mount --path-info-service-addr "nix+https://cache.nixos.org?trusted-public-keys=cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" --directory-service-addr sled://$PWD/test/dirs.sled --blob-service-addr objectstore+file://$PWD/test test2

attempt to list the directories in a bash store path

~/proj/depot/tvix $ ls test2/n2p10bjfnsc43hcbd9dv1qph65p2aph8-bash-interactive-5.2p26/ "test2/n2p10bjfnsc43hcbd9dv1qph65p2aph8-bash-interactive-5.2p26/": Input/output error (os error 5)

~/proj/depot/tvix $

Meanwhile in the logs of the mount process:

2024-06-04T15:31:47.039212Z ERROR get:get:ingest_entries: tvix_castore::import: error=error from producer: unable to read blob contents for root/bin/bash: invalid trailer path.digest="n2p10bjfnsc43hcbd9dv1qph65p2aph8" 2024-06-04T15:31:47.039285Z ERROR get:get: tvix_store::pathinfoservice::nix_http: error=internal storage error: error from producer: unable to read blob contents for root/bin/bash: invalid trailer path.digest="n2p10bjfnsc43hcbd9dv1qph65p2aph8"

This is not consistent, after ~1-5 tries it usually works

  1. This can be reproduced much more reliably with the following setup (3 Terminals)

    Checkout https://cl.tvl.fyi/c/depot/+/11903

    cargo run --profile=release-with-debug -p nar-bridge -- --otlp=false

    RUST_LOG=debug cargo run -p tvix-store --bin=tvix-store -- --otlp=false daemon --remote-path-info-service-addr 'nix+https://cache.nixos.org?trusted-public-keys=cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY='

    curl -v http://localhost:3000/cvcl2fygqr6h1cfnrka7n2p0b893nr08.narinfo

    It reliably fails at

    unable to read blob contents for root/demo/jfc/Font2DTest/Font2DTest.jar: invalid trailer

    flokli at 2024-06-30T20·10+00

  2. cl/11898 needs to be cherrypicked, not cl/11903. cl/11903 was a parent, rebased out and landed separately.

    flokli at 2024-07-18T16·44+00

  3. Fixed by https://cl.tvl.fyi/c/depot/+/11978.

    flokli at 2024-07-18T22·00+00

  4. flokli closed this issue at 2024-07-19T08·44+00