Skip to content

Running Remote Builds

The Getting Started guide shows you how to setup Nix and run a remote build. Many users can just set and forget this, and go on using Nix as they always have done, with nixbuild.net transparently scaling out their builds behind the curtains.

However, there a number of twists and turns to remote building in Nix. For some use cases, you can make your builds significantly more performant by not following the standard path. This chapter aims to summarize the differences between various remote build setups.

When it comes to figuring out and setting up build processes and CI pipelines for your Nix builds, you should conceptualize nixbuild.net as just a single Nix machine, which happens to have an infinite number of CPUs and extra clever caching of Nix closures. You can run builds on it, push and pull closures to and from it, and use it as a binary cache. All with the standard Nix tools. It also means that this chapter is actually not that specific to nixbuild.net, you can apply much of it to Nix in general.

This chapter describes Nix building in detail, and highlights possible issues and bottlenecks along the way. If you want, you can skip directly to the section that recommends how to set up Nix for best performance based on your use case. However, we would recommend you to go back and read the other sections too, in order to fully understand the possibilities and limitations with Nix remote builds.

Remote Builders or Stores

Nix has two main ways of distributing builds; using remote builders or remote stores. When using remote builders, Nix may offload builds to one or more Nix machines, with the end goal of fetching the build result back into your local Nix store. When using a remote store, Nix instead tries to get a single remote Nix machine to perform the build, without involving the local Nix store. When the build is done, the remote Nix machine should have the result in its store, but it will not be in the local Nix store. The remote Nix machine might have used remote builders to help out with the build, but that is outside the control of your local Nix invocation.

For certain setups, using a remote store can greatly improve performance due to the fact that so much less closure copying is involved. For other setups, remote builders is the more convenient choice.

The following sections will describe these two remote build methods in detail.

Using Remote Builders

Remote builders is usually what people mean when they talk about remote Nix builds. It is also what is described in the official Nix documentation. I suggest you read through that section, everything in there is applicable to nixbuild.net.

The Build Process

The general process Nix follows when building is outlined below.

  1. Nix evaluates the expression it has been requested to build. When doing this, one or more derivation files (with the extension .drv) will be created and stored in the local Nix store. The set of .drv files represent the build tree of the Nix expression. Each .drv file contains information about which Nix paths will be produced by the derivation, and what derivations must be built before the given derivation can be built.

  2. Now Nix looks at the output paths for the derivation that represents the expression it is supposed to build. If those paths already exist and are valid in the local Nix store, Nix stops the process here. No build is needed.

  3. For any missing output path, Nix will query the substituters that have been set up in nix.conf, or specified on the command line. If all missing paths can be fetched from a substituter, Nix will do it and unpack the paths into the local store. The process is then stopped. It is worth to note that remote builders are not considered substituters, so Nix will not query them for the output paths at this point. This means that Nix might now proceed building or fetching inputs even though a remote builder already has the build result in place. We'll talk more about this later.

  4. If there are still missing paths after the substitution phase, Nix has to build the derivation. Before doing that, all required input paths must exist in the local store. So Nix will simply walk the derivation tree and build any missing inputs until the inputs that are needed to build the "top-level" derivation is in place in the local store. It will follow the same build process (check for existing paths, query substituters, build) in a recursive fashion. Nix processes multiple inputs concurrently, whenever possible.

  5. When a derivation has all its inputs available in the local Nix store, the actual build can start. Nix will now check the builders configuration option (setup in nix.conf or provided on the command line). If builders has remote Nix machines, Nix tries to decide if the derivation can be built on any of those machines. The matching process is decribed in the official docs. In short, Nix looks at the platform (x86_64-linux, aarch64-linux, etc) of the derivation, and the special derivation attribute requiredSystemFeatures to find a suitable remote builder. Builders can also be configured to have different priorities. If Nix can't find a suitable remote builder, it will try to build the derivation locally instead.

  6. Once a remote builder has been selected for building the derivation, Nix will check if the builder has all necessary inputs in place. While doing this, Nix can ask the builder itself to fetch any missing inputs from the substituters that are configured on the builder machine. Any inputs that still are missing on the remote are uploaded to the builder from the local store. Remember, Nix has already made sure that all inputs are valid in the local store, before even selecting a remote builder. This means that Nix might fetch or build inputs in the local store that never are needed locally, since the build that needs the inputs is built remotely. This is just the way the Nix build process works.

  7. Once Nix has made sure that the remote builder has all build inputs in place, it sends the derivation to the builder and asks for it to be built.

  8. If the build finishes successfully, Nix will fetch the build result and its transitive run-time closure from the remote builder to the local store. The derivation has now been built.

Drawbacks of Remote Builders

Remote builders are not treated as substituters

As mentioned above, Nix will only query the local store and substituters when checking if a build is needed. If no existing output is found, the whole build process will be started, meaning all build inputs will be fetched or built and then upload to the remote builder before asking the remote builder to run the build. At that point, it it possible that the remote machine has the build in place and can serve it back without running the build. All the time and network bandwidth spent on getting build inputs in place might have been wasted.

The workaround for this is to configure your remote builders as substituters, then Nix will be able to fetch build outputs from them in a much earlier phase. If you have a small number of static remote builders it is fairly simple to do this configuration, but it gets much trickier if you have a large number of builders, or if you use auto-scaling for your builder instances.

If you use nixbuild.net, you can configure Nix to use it as a substituter. The nixbuild.net action for GitHub Actions automatically sets up nixbuild.net as a substituter.

Build dependencies are fetched locally

Build dependencies (inputs) are always fetched (possibly after first being built) to the local store before Nix starts interacting with a remote builder (as described above). If you run all builds remotely, you can end up with closures in your local store that were never needed there. For some setups, this is not a big issue, but for other cases it can affect performance dramatically. We will talk more about these different scenarios.

The workaround to the build input handling is to switch from using remote builders to using a remote store.

Build outputs are fetched to the local store

When using remote builders, the final build output (together with its complete transitive closure) is copied from the remote builder to the local Nix store. This might not sound like a drawback at all, but for some cases it actually is. Consider CI setups: many times your are primarly interested in seeing that builds and tests pass, but you are less interested in actually using any build artifacts. So, if you use remote builders in your CI setup, fetching build outputs can be a waste of time, storage and network bandwidth. This is especially true when you are using CI runners that are underpowered or "cold" (starting out with empty Nix stores).

To avoid having to fetch build outputs, you must switch to using a remote store, instead of using remote builders.

Tricky to let multiple clients use the same builders

The build scheduling on remote builders is simplistic and entirely directed from the Nix client. This means that if you'd like use the same remote builder from multiple clients at the same time, you can run into problems. The different Nix clients will not be aware of each other and they might therefore submit builds to the same remote builder, causing congestion.

If you use nixbuild.net as your remote builder, this is a non-issue. nixbuild.net acts as a normal Nix machine externally, but internally it has knowledge about all builds that all clients submits and can direct builds to different backend builders in an optimal fashion.

Using Remote Stores

Nix has a command line option named --store that can be used to select which Nix store to operate on. If you don't specify any --store argument, Nix will use the default store, called auto. Auto means that if you run as the same user that owns /nix, Nix will use the Nix database file directly, and write to the nix store directly. If you run as any other user, Nix will connect to nix-daemon running on the local host. nix-daemon will then handle database usage and store updating. This way, nix-daemon can use different trust-levels for different local users.

It is possible to redirect Nix to connect to a nix-daemon running on another machine by specifying the --store option. This way, all Nix operations will take place entirely in the other store, bypassing the local store. It also means that if you build Nix expressions, they will only end up in the remote store, not in the local store.

Nix also has an option called --eval-store, which instructs Nix to use a specific Nix store during the evaluation phase (before starting the build). It should be possible to set the evaluation phase to a remote store too, but in practice it makes most sense to always set it to auto. There currently seems to be a few bugs in Nix which make building in a remote store not possible if the evaluation store is also remote. Besides, the evaluation store is used for storing Nix expressions and source trees. These things are usually only available locally so it doesn't make any sense to try running evaluation remotely.

The recommended way of building in a remote Nix store (your nixbuild.net store, in this case) is to launch your Nix build like this:

nix build --eval-store auto --store ssh-ng://eu.nixbuild.net ...

You need to use ssh-ng:// instead of ssh://. ssh-ng is a newer Nix build protocol that will replace ssh, and only it has support for remote store builds.

The Build Process

If you run nix build --eval-store auto --store ssh-ng://remote-host, Nix will follow this process:

  1. Evaluate your Nix expression as usual. Derivation (.drv) files will be created and stored in your local Nix store. This step is identical to when you use a remote builder, or build locally.

  2. Upload all .drv files of the build to the remote Nix machine.

  3. Ask the remote Nix machine to build the .drv file corresponding to the build you requested.

  4. Now, the remote Nix machine will run the build. Depending on how the remote machine is configured, it might use remote builders to run the builds, or it might run them itself. Any build logs will be forwarded to your local console.

  5. When the build is done (or failed), your local Nix client will not do anything more. The build output will now only be on the remote Nix machine.

IFD Builds (Import From Derivation)

IFD requires Nix to perform one or more builds during the evaluation of a Nix derivation. This is useful in some cases but also comes with drawbacks. The general advice is to avoid IFD if possible. However, if you need IFD there is one thing you should watch out for when using remote store builds: If you have configured remote builders, Nix will try to use them for the builds that happen during evaluation, even if you are running a remote store build. This, in turn, causes Nix to not copy the evaluated derivations to the remote store, leading to error messages mentioning missing paths. To workaround this you need to make sure that no remote builders are used during IFD evaluation. The simplest way to do that is using the --builders option:

nix build --builders "" --eval-store auto --store ssh-ng://eu.nixbuild.net ...

Depending on how you have configured Nix, you might also need to specify --max-jobs 1 to allow local builds to run during evaluation.

Retrieving Build Output from Remote Stores

Often, you actually do want to get the build results into your local Nix store. This will not happen by itself when using a remote store. Instead you need to manually run nix copy or nix copy-closure to retrieve the build output. To simplify this process you can run nix build with the --json option so that you get the output paths of the build results:

$ nix build --json ...
[{"drvPath":"/nix/store/7grz4ylh94a0cfffk3m1cwh3511la71k-test.drv","outputs":{"out":"/nix/store/pzzxqxjfwidlqdvwkgy4fndx6axbpm7d-test"}}]

Now you can fetch the built path like this:

nix copy --from ssh-ng://eu.nixbuild.net /nix/store/pzzxqxjfwidlqdvwkgy4fndx6axbpm7d-test

Drawbacks of Remote Stores

Retrieving build outputs requires extra steps

This is not really a drawback, but simply the intended mode of operation. If you require the build output locally even when you've built in a remote store, you need to explicitly fetch the build results with nix copy or nix-copy-closure.

Build dependencies are not copied from the local store

When using a remote store, the local store is minimally involved (only used during evaluation). This also means that even if you have build dependencies (inputs) available locally, they will not be sent to the remote store. Instead the remote machine will have to fetch or build all inputs itself.

If this is an issue in your setup, you could explictly copy the build inputs to the remote machine before starting the build.

The remote store might suffer from the drawbacks of remote builders

When using a remote store you are really just delegating the Nix operations to another machine. You might also just be delegating your issues, because if the remote machine is configured to use remote builders it will of course suffer from all its drawbacks. However, if you use nixbuild.net as your remote store, that is not an issue. Internally, nixbuild.net uses a much more efficient way of handling build inputs, outputs and builder allocation than standard Nix.

Not all Nix options are propagated to the remote Nix store

If you provide Nix with build-time command line options like --keep-going and similar, those options will not be propagated to the remote store. This is something that has not been fully worked out and implemented in Nix yet.

For some cases, this is mitigated in nixbuild.net since it has its own setting system in addition to the settings available in Nix.

Limitations of remote store builds in nixbuild.net

Remote store builds is a beta feature in nixbuild.net. The intention is for nixbuild.net to fully support remote store builds, but currently there are some limitations. Except for these limitations, the feature is fully usable by all nixbuild.net users.

Your Ideal Setup

Remote building in Nix is not one-size-fits-all. To get the best performance, you need to think a little bit about how you make use of it in your particular setup, and where your bottlenecks are. The good thing is that a few typical use cases cover most peoples needs - and nixbuild.net supports all of those cases.

Here follows a list of typical setups:

Developer Machine

If you regularly run Nix builds on your local development machine, running builds remotely can help you with the following:

  • Running builds for other platforms than your machine's platform. For example, running ARM builds from an x86_64 machine, or runnning Linux builds from MacOS.

  • Running builds with larger resource requirements (in terms of CPU and/or memory) than your machine can handle.

  • Running lots of builds in parallel by scaling out to multiple remote builders.

  • Reducing fan noise.

In this case, using one or more remote builders is the way to go. Unless you have specific needs, it will not be a problem that build inputs are fetched to the local machine, since you likely will have use for those inputs when running local builds. You can setup your remote builders as substituters if you notice your remote builders often have builds in place when you're running builds.

Short-lived CI Runners

Short-lived CI runners includes services like GitHub Actions. When you run builds in such environments, every build will run inside a completely fresh machine or container. This means the Nix store will be empty each time a build starts, and the transitive closure of all build inputs will have to be fetched (or in worst case built) on each build, if you run builds locally or use remote builders. You can mitigate this somewhat by making use of a binary cache that is updated with new build outputs, to make sure that you at least avoid building the same inputs multiple times. But the time it takes to download and upload all those Nix closure can be really prohibitive.

Instead, you should run your builds in a remote Nix store. It completely removes the need of copying closures back and forth, saving a possibly tremendous amount of time in your CI.

In nixbuild.net's own CI, we managed to cut down the best-case build time of our slowest build from 20 minutes to 20 seconds by switching from using nixbuild.net as a remote builder to using it as a remote store. You can read about this on our blog.

Underpowered Nix machines

You might have machines where you want to use Nix/NixOS, but for whatever reasons they can't handle building well enough. There might be limitations in network bandwidth, storage or CPU making Nix builds slow or unfeasible. In this case, using a remote store guarantees that no building happens on the local machine, only evaluation. And you can still fetch the build outputs that you need locally.

Long-lived CI Runners

A long-lived CI runner is a machine that will keep its Nix store between builds, reducing the need to fetch inputs for each build. However, if you have many CI runner machines, you can still run into issues of missing inputs that need to be fetched, if builds tend to "jump around" between different CI runners.

In this case, it's harder to say which build mode is best since it depends both on the CI machines and the builds themselves. You can simply try it out and see what works best in your setup.

Additional Resources