tvix/store: Add *Service Registry Based on URI Scheme

#308
Opened by cbrewster at 2023-09-21T23·20+00

The registry would store a mapping of URI schemes -> constructors for a given *Service backend. This would make it easy for downstream users to add their own backends with their own CLI.

Additionally it makes testing a bit easier since we can have some custom schemes that setup up testing backends, like one that spins up a test gRPC server under the hood. The test can then use a list of URIs as different test cases so we can get more coverage of all the backends.

  1. Another I had with tazjin: In case the URI scheme is not known, some fallback case is triggered, in which everything talking to stores is invoking a binary with a specific name derived from the protocol, speaking gRPC over stdin/stdout, similar to how restic uses rclone.

    This would allow plugging in non-native store implementations without having to even link them to tvix codebase at all.

    flokli at 2023-09-26T10·43+00

  2. //tvix:nar-bridge, specifically the golang implementation we have now would be something that'd be nice to be able to easily plug into Tvix (as one layer), so let's try to come up with some calling convention for these external store providers.

    flokli at 2023-10-04T21·13+00

  3. cbrewster changed the subject of this issue from "tvix/store: Add `*Service` Regsitry Based on URI Scheme" to "tvix/store: Add `*Service` Registry Based on URI Scheme" at 2023-10-12T17·36+00

  4. cl/10030 (and parents) centralize most of the URL parsing, especially for the gRPC-based processes.

    I assume it should be fairly easy to now add support for a grpc+stdio://$executableName URL, that'll invoke the binary from $PATH.

    We should probably still discuss how the calling conventions will look like:

    • how to enable stdio mode in the called binary (env var?)
    • how to pass url parameters in the addr to the called process

    flokli at 2023-11-13T12·47+00

  5. https://cl.tvl.fyi/c/depot/+/10043 introduces a grpc+inetd: scheme, allowing to call external processes and speaking grpc over stdin and stdout

    flokli at 2023-11-15T17·18+00

  6. It's not clear to me that we should be building a process manager into tvix. It encourages adding more things to our process tree, and more ambient state that we pass along to other programs.

    Even if not intentionally used, the fact that it can spawn arbitrary processes becomes part of any tvix user's security considerations.

    I would much prefer to leave socket activation to the system service manager (eg, systemd), which is where that piece of responsibility belongs.

    edef at 2023-11-15T21·51+00

  7. It's not clear to me that we should be building a process manager into tvix. It encourages adding more things to our process tree, and more ambient state that we pass along to other programs.

    Even if not intentionally used, the fact that it can spawn arbitrary processes becomes part of any tvix user's security considerations.

    I would much prefer to leave socket activation to the system service manager (eg, systemd), which is where that piece of responsibility belongs.

    edef at 2023-11-15T21·51+00

  8. Alright, let's keep the spinning up processes part to socket activation only, no problem :-)

    flokli at 2023-11-16T21·10+00

  9. flokli closed this issue at 2023-11-16T21·10+00