ci: allow failures for certain packages

#41
Opened by tazjin at 2020-08-05T11·41+00

It might be useful to have a way of marking packages as "known broken" in ci-builds.nix, in cases where we know that they will not build (e.g. Haskell packages after a channel update, as we are facing at the moment).

The primary benefit would be that we could let the rest of the repository advance, without building up CLs on top of one that is blocked by a specific failure. The main drawback would be that we would rely on people to actually move those packages back into the "proper" list.

Implementation could be relatively simple by updating our Buildkite pipeline generator to generate a separate build step which allows failure, so that we still get the information in CI but aren't blocked from submitting.

We could also introduce a convention that each build marked broken needs an issue link attached.