Getting Started¶

This guide describes how you create a nixbuild.net account and set up your system to use nixbuild.net for running remote Nix builds.

The guide is written to be followed from start to end. If you run into any troubles when going through the guide, or during your future use of nixbuild.net, just ask [email protected] for help. You are also welcome to submit public issue reports, questions or any other feedback here.

Register¶

When registering for nixbuild.net you will receive 25 free CPU hours of build time, which means your can try out the service on your own terms without committing to anything. No credit card is needed for registration.

To register, simply enter your email in the registration form and check your inbox for confirmation.

Add SSH Key¶

To run builds or use the the admin shell you need to add a public SSH key to your nixbuild.net. You can then use the corresponding private SSH key to connect to nixbuild.net.

The SSH key must be of type Ed25519. With OpenSSH, you can generate the key in the following way:

ssh-keygen -t ed25519 -C my-key-comment -f my-nixbuild-key

If you're using nix-daemon to run nix builds (which is usually the case), you shouldn't set a password on the SSH key. The key comment provided with the -C option will help you distinguish keys within nixbuild.net, if you add multiple SSH keys to your account, but is not used in any other way.

The public key generated by the above command should look something like this:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDzLYWrTuAlSWsdTHkTwLGoIaXOq6vrifbBt/X060KwL my-key-comment

Now go to the SSH keys settings page and add your public SSH key. You can add more keys if you like. You could for example have different keys for your nix-daemon user and your normal user.

Quick NixOS Configuration¶

If you use a Linux distribution that isn't NixOS, or if you use MacOS, follow the instructions in SSH Configuration and Nix Configuration to get up and running.

If you use NixOS, you can simply add the following to your configuration.nix in order to start using nixbuild.net:

programs.ssh.extraConfig = ''
  Host eu.nixbuild.net
    PubkeyAcceptedKeyTypes ssh-ed25519
    ServerAliveInterval 60
    IPQoS throughput
    IdentityFile /path/to/your/private/key
'';

programs.ssh.knownHosts = {
  nixbuild = {
    hostNames = [ "eu.nixbuild.net" ];
    publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPIQCZc54poJ8vqawd8TraNryQeJnvH1eLpIDgbiqymM";
  };
};

nix = {
  distributedBuilds = true;
  buildMachines = [
    { hostName = "eu.nixbuild.net";
      system = "x86_64-linux";
      maxJobs = 100;
      supportedFeatures = [ "benchmark" "big-parallel" ];
    }
  ];
};

Above, the remote builder is setup for the x86_64-linux system. nixbuild.net also supports i686-linux, aarch64-linux and armv7l-linux builds. If you want to use nixbuild.net for builds of different systems you need to add one entry per system to buildMachines, all pointing to nixbuild.net. Nix will then be able to send builds of different systems to nixbuild.net.

Make sure that the private key file (IdentityFile) is readable by the root user (that runs nix-daemon).

If you use the above configuration you can skip the SSH Configuration and Nix Configuration sections below and go directly to Verify Shell Access and then Your First Build. It doesn't hurt skimming through all sections, though, to get a better feeling for how things work.

SSH Configuration¶

If you use nix-daemon to run builds on your local machine you need to make the private SSH key available (with permissions 0600) to the user that nix-daemon runs as (root). The easiest way to do this is to add the following entry to your system wide (or root user) SSH client config:

Host eu.nixbuild.net
  PubkeyAcceptedKeyTypes ssh-ed25519
  ServerAliveInterval 60
  IPQoS throughput
  IdentityFile /path/to/your/private/key

The PubkeyAcceptedKeyTypes ssh-ed25519 entry above is important, since it will keep SSH from using key types that nixbuild.net can't handle.

Then, you need to add the public host key of eu.nixbuild.net to the known hosts file of the root user (/root/.ssh/known_hosts). Alternatively, you can add it to the global known hosts file (/etc/ssh/ssh_known_hosts). The reason you have to add the public host key manually is that when Nix sets up an SSH connection, it can't ask you to confirm the host key.

You should add the following line to the known hosts file:

eu.nixbuild.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPIQCZc54poJ8vqawd8TraNryQeJnvH1eLpIDgbiqymM

The ServerAliveInterval and IPQoS settings have been found to make remote builds more reliable, especially if there are long silent periods with no logs emitted by a build.

Verify Shell Access¶

You should now verify that you can connect to the nixbuild.net shell, which is used to administer your account.

Run the following command (using the root user via sudo, to emulate how nix-daemon works):

sudo ssh eu.nixbuild.net shell

You should be greeted by a screen looking similar to this:

••••••••••••••••••••••••••••
••••••• nixbuild.net •••••••
••••••••••••••••••••••••••••

Welcome to nixbuild.net!

This shell allows you to administer your account
and retrieve information about your nix builds.

Account id: IIERI7
Unbilled CPU hours: 0
Free build time left: 49:59:59

Available shell commands:

  help         Show help
  settings     Configure account or SSH key
               settings
  usage        Show resource usage
  ssh-keys     Manage your public SSH keys
  billing      Billing activation and info
  signing-keys Manage Nix signing keys
  tokens       Manage auth tokens
  builds       Manage builds

Run 'help COMMAND' for help on individual shell commands.
For help on subcommands run 'COMMAND SUBCOMMAND --help'.
For more documentation, see:
https://docs.nixbuild.net/configuration

Adding an SSH Key¶

For convenience, you probably want to generate another SSH key and add it to your nixbuild.net account now. This way, you can have one key owned by the root user, used by nix-daemon, and one key owned by your normal user, for accessing the nixbuild.net shell.

Your user key can have a password or be managed by an SSH agent, since it will be used interactively. It must be of type Ed25519, and you need to have the PubkeyAcceptedKeyTypes ssh-ed25519 configured as above.

Each key associated with your account must have a unique key name (sometimes referred to as key comment) for identifying the key.

To add a new SSH key, use the ssh-keys add command, like this:

nixbuild.net> ssh-keys add --help
ssh-keys add - Add a public SSH key to your account

Usage: ssh-keys add KEY_TYPE KEY KEY_NAME
  Add a key

Available options:
  KEY_TYPE                 Type of SSH key (must be 'ssh-ed25519')
  KEY                      The base64-encoded key byte sequence
  KEY_NAME                 A key name (comment), for identifying keys
  -h,--help                Show this help text

Example usage:
  ssh-keys add ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDrqgdPd6EB9YOUPy5CRNc+3dDvkc5jc2ZXfBNusi3LF laptop

The options match the format of OpenSSH public keys.
Only SSH keys of type 'ssh-ed25519' are supported.
Each key in your account must have a unique key name.
The same key cannot be associated with more than one account.


WARNING! When adding a new public key, any person with access to the
corresponding private key will gain immediate access to this account, and will
be able to run builds and make changes to the account. Look into the setting
'default-permissions' to restrict permissions of new and existing SSH keys.

As mentioned in the warning printed above, new SSH keys will get full access to your account by default. You can use the setting default-permissions to restrict access for the key.

You can add as many SSH keys as you like to your account, to set up for example team users or CI builders. To support complex access control, it is possible to use auth tokens instead of SSH keys. Read more about this in reference documentation.

Nix Configuration¶

You can now go ahead and configure nixbuild.net as a Nix remote build machine. As such, you can add it as an entry in your /etc/nix/machines file:

eu.nixbuild.net x86_64-linux - 100 1 big-parallel,benchmark

or simply use the --builders option of nix-build:

--builders "ssh://eu.nixbuild.net x86_64-linux - 100 1 big-parallel,benchmark"

You can also use nixbuild.net for i686-linux, aarch64-linux, and armv7l-linux builds. If you want to use nixbuild.net for builds of different systems you need to specify one builder per system, all pointing to nixbuild.net. Nix will then be able to send builds of different systems to nixbuild.net.

The big-parallel,benchmark assignment is something that is called system features in Nix. You can use that as a primitive scheduling strategy if you have multiple remote machines. Nix will only submit builds that have been marked as requiring a specific system feature to machines that are assigned that feature.

The number 100 in the file above tells Nix that it is allowed to submit up to 100 simultaneous builds to eu.nixbuild.net. Usually, you use this property to balance builds between remote machines, and to make sure that a machine doesn't run too many builds at the same time. This works OK when you have rather homogeneous builds, and only one single Nix client is using a set of build servers. If multiple Nix clients use the same set of build servers, this simplistic scheduling breaks down, since a given Nix client loses track on how many builds are really running on a server.

However, when you're using nixbuild.net, you can set this number to anything really, since nixbuild.net will take care of the scheduling and scaling on its own, and it will not let multiple Nix clients step on each other's toes.

You probably want to activate the builders-use-substitutes Nix option. This option allows nixbuild.net to download dependencies directly from cache.nixos.org. You can also add additional substituters using the substituters setting.

There is an alternative way of performing distributed builds with Nix. Instead of sending builds off to remote builders and then fetching the result back, Nix can run the entire build in a remote store, entirely by-passing the local Nix store. This can have a tremendously positive performance impact in certain cases, such as in CI setups. This mode of operation is described in detail in the Running Remote Builds chapter.

Your First Build¶

Run the following build to verify that your configuration works as expected:

nix-build \
  --max-jobs 0 \
  --builders "ssh://eu.nixbuild.net x86_64-linux - 100 1 big-parallel,benchmark" \
  -I nixpkgs=channel:nixos-20.03 \
  --expr '((import <nixpkgs> {}).runCommand "test${toString builtins.currentTime}" {} "echo Hello nixbuild.net; touch $out")'

Note, if you use MacOS, you need to add --system x86_64-linux (or any supported system) to the above invocation. nixbuild.net does not yet support MacOS as build target, but you can use the service to build Linux builds from your MacOS machine.

The output should look something like this:

these derivations will be built:
  /nix/store/0alhvbxdqq4hakna5sp6l51180q2l1l9-test1598214851.drv
building '/nix/store/0alhvbxdqq4hakna5sp6l51180q2l1l9-test1598214851.drv' on 'ssh://eu.nixbuild.net'...
Hello nixbuild.net
copying 1 paths...
copying path '/nix/store/8zj14ysp2jd3whldi99hk501628a9gaf-test1598214851' from 'ssh://eu.nixbuild.net'...
/nix/store/8zj14ysp2jd3whldi99hk501628a9gaf-test1598214851

That's it, now you can use nixbuild.net!

It is recommended to continue from here by reading the chapters about configuring your nixbuild.net account, and about running remote builds. You should also take a look at how access control is implemented in nixbuild.net.