fuse-backend-rs doesn't mount inside buildkite agents

#309
Opened by flokli at 2023-09-22T12·58+00

We noticed the `nix-shell --run "cargo build && cargo test" pipeline for tvlfyi/tvix was broken since cl/9341.

All fileystem-related tests were failing with the following output:

failures:
---- fs::tests::root stdout ----
thread 'fs::tests::root' panicked at 'must succeed: Custom { kind: Other, error: "descriptor I/O error: No such file or directory (os error 2)" }', store/src/fs/tests.rs:360:6
---- fs::tests::read_stat_directory stdout ----
thread 'fs::tests::read_stat_directory' panicked at 'must succeed: Custom { kind: Other, error: "descriptor I/O error: No such file or directory (os error 2)" }', store/src/fs/tests.rs:634:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- fs::tests::stat_file_at_root stdout ----
thread 'fs::tests::stat_file_at_root' panicked at 'must succeed: Custom { kind: Other, error: "descriptor I/O error: No such file or directory (os error 2)" }', store/src/fs/tests.rs:438:6
---- fs::tests::readdir_deep stdout ----
thread 'fs::tests::readdir_deep' panicked at 'must succeed: Custom { kind: Other, error: "descriptor I/O error: No such file or directory (os error 2)" }', store/src/fs/tests.rs:799:6
---- fs::tests::compare_inodes_symlinks stdout ----
thread 'fs::tests::compare_inodes_symlinks' panicked at 'must succeed: Custom { kind: Other, error: "descriptor I/O error: No such file or directory (os error 2)" }', store/src/fs/tests.rs:996:6
---- fs::tests::symlink_readlink stdout ----
thread 'fs::tests::symlink_readlink' panicked at 'must succeed: Custom { kind: Other, error: "descriptor I/O error: No such file or directory (os error 2)" }', store/src/fs/tests.rs:548:6

After some investigation, we guessed it might be due to the fusermount binary not being available in $PATH, and hacked together cl/9386 to add it to $PATH.

After that, the same build succeeded.

While we could just keep this workaround, it's probably worthwhile to compare the behaviour with the fuser crate, and see why it's able to accomplish this without having the binary avaialble.

  1. For now we applied the workaround, but ideally we should get the fuse-backend-rs to behave the same way (and work) in the buildkite environment.

    flokli at 2023-09-25T19·34+00

  2. A comment in src/transport/fusedev/linux_session.rs suggested this is due to fuse-backend-rs using multiple threads:

        // mount in another mntns requires mounting with fusermount, which is a new process, as
    // multithreaded program is not allowed to join to another mntns, and the process running fuse
    // session might be multithreaded.

    flokli at 2024-06-18T06·08+00