System Configuration¶

nixbuild.net deployments are configured by placing one or more configuration files in /etc/nixbuild.net/conf.d on the nxb-server instance. The configuration files should be written in the libucl configuration language, or any of languages that libucl can read, like JSON.

Merging of configuration files uses the same semantics that multiplication of objects in jq uses. This is also called recursive merge.

The configuration files are merged in this order:

The file /etc/nixbuild.net/base_config.json is read. This file is created when the nixbuild.net distribution is built and can't be changed by system administrators.
Files found directly under /etc/nixbuild.net/conf.d are sorted alphabetically, converted to JSON by libucl and merged into the final configuration using the recursive merge approach described above.

On nxb-server you can run the command nixbuild-config get to show the complete nixbuild.net configuration.

Note that the recursive merge approach used does not allow for an overriding configuration file to append to a list, only override it. For example, base_conf.json usually includes the standard Nix substituter in the default list of substituters to use. So if you have no extra config you'll get this:

$ nixbuild-config get nixbuild default-user-settings substituters
[
  "https://cache.nixos.org"
]

Adding a file /etc/nixbuild.net/conf.d/substituters.conf that looks like this:

nixbuild {
  default-user-settings {
    substituters = [ "https://my-cache.example.com" ]
  }
}

will effectively remove https://cache.nixos.org:

$ nixbuild-config get nixbuild default-user-settings substituters
[
  "https://my-cache.example.com"
]

Configuration example¶

slurmctld {
  host = nxb-server-1
  ip = 172.31.32.20
}
nixbuild {
  predefined-accounts = [
    { account-id = 1
    , email = rickard@nixbuild.net
    , ssh-keys = [
        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPo2vH/gNTctZy2R4Xg9BuExLwqc/xFnqBGLx7WndgQG rickard"
      ]
    }
  ]
}
ec2 {
  build-node-templates = [
    { ami = ami-0917406788c1b79e5
    , region = eu-north-1
    , security-group = sg-030abe5e2f9ec5e74
    , subnet = subnet-07dc948c6500abf4b
    , ssh-key-name = nxb-ssh-key-1
    , instance-type = c5.large
    , count = 2
    , weight = 0
    }
  ]
}

Configuration Options¶

`nard.profiling.enable`¶

Type: bool

Default: false

If set to true, profiling will be activated for nard.service. Profiling output can be found in /var/log/nard/profiling. Most of the profiling output will only be available after nard.service has been stopped or restarted. Each time nard.service is restarted, a backup of the profiling directory will be created in /var/log/nard.

The config option nard.profiling.rts-options controls what is included in the profiling outputs, see below.

`nard.profiling.rts-options`¶

Type: list of strings

Default: [ "-p" ]

Defines a list of GHC RTS options to be used when running the profiled nard binary. While you can set any GHC RTS options here, you should probably limit yourself to the ones that concern profiling.

For the options to take effect, you must also set nard.profiling.enable to true.

`nard.s3.allow-ambient-credentials`¶

Type: bool

Default: false

If false (the default), nard will not pick up any ambient credentials (like EC2 IAM instance profiles). Instead you have to configure an access-token for each S3 bucket your NixBuild account wants to use for substitution or push store paths to.

`nard.push.s3.default-compression-level`¶

Type: integer

Default: 3

The zstd compression level used when uploading store paths to S3 buckets.

`nard.push.s3.compression-chunk-bytes`¶

Type: integer

Default: 1048576

When compressing and uploading store paths nard will compress individual chunks (in parallel if possible) and then concatenate the compressed chunks into a buffer that will be uploaded to S3 once it is big enough. The compression-chunk-bytes specifies the preferred chunking size for compression.

`nard.push.s3.concurrency`¶

Type: integer

Default: 4

This setting specifies the number of threads nard is allowed to have running compression for S3 uploads at the same time. It should be pretty safe to set this to the number of available CPU threads, or even slightly more. If you see the CPU consumption of nard cannibalize other services on the server, you can turn this setting down.

`nard.push.s3.buffer-size`¶

Type: integer

Default: 12582912

This is the preferred size of the individual multi-part S3 uploads. Regardless what size you configure, nard will make sure that the buffer size is within S3's allowed limits (minimum 5MB, and maximum 10000 parts per multi-part upload). Due to the way concurrent compression works in nard, a bigger buffer size will improve compression somewhat, since it will avoid having to pad compressed data to fulfill the minimum limit. A bigger buffer size will increase memory consumption.

`nard.push.s3.max-buffers-per-nar`¶

Type: integer

Default: 6

This setting multiplied with the buffer-size setting specifies the maximum amount of (compressed) data that is allowed to be kept in memory per processed nar file. Only "large" nar files have this limit. Small nar files are loaded into memory, compressed and uploaded to S3 as a single HTTP request.

`nard.push.s3.max-concurrent-nars`¶

Type: integer

Default: 4

The number of large nar files that can be processed concurrently. It is recommended to increase this value if you have enough memory on your server.

`nard.push.s3.max-concurrent-requests`¶

Type: integer

Default: 32

The maximum number of in-flight HTTP requests to S3.

`nixbuild.ssh-auto-login-account-id`¶

Type: integer

Default: Not set

If set, SSH authentication will be skipped and the SSH session will be automatically tied to the account id specified. You should only enable this if you have other means of access control, like network boundaries etc.

`nixbuild.default-user-settings`¶

Type: attribute set

An attribute set that defines the initial configuration of accounts that are added to your NixBuild Enterprise deployment. The configuration here is a superset of the settings available through the settings command in the NixBuild shell, because not all account settings are configurable by users.

For the settings that are available through the NixBuild shell, please see the documentation for the public nixbuild.net service. For other settings, see below.

`nixbuild.default-user-settings.storage-limit-days`¶

Type: integer

Default: 1000

Any store path older than the specified number of days will be automatically deleted. The age is based on when the path was last accessed, queried or stored.

This setting puts an absolute upper limit on the time a path can exist in the NixBuild database and storage. However, the related settings input-closure-ttl-seconds, output-closure-ttl-seconds, uploaded-path-ttl-seconds and queried-path-ttl-seconds let you define more fine-grained lifetimes for paths.

Also see the related setting storage-limit-gigabytes.

`nixbuild.default-user-settings.storage-limit-gigabytes`¶

Type: float

Default: 1000.0

Limits the total storage space that is allowed for an account. If an account exceeds the specified storage usage, the oldest (in terms of when they were last accessed, queried or stored) paths will be removed until storage-limit-gigabytes is no longer exceeded.

`nixbuild.default-user-settings.input-closure-ttl-seconds`¶

Type: integer

Default: 90 * 24 * 3600

Anytime a path closure is used as input for a build, its time to live (TTL) will be set to specified number of seconds. If a path in the closure already has a longer TTL, that TTL will not be changed. This means the complete path closure will remain in the NixBuild store for at least the specified time, as long as the settings storage-limit-days or storage-limit-gigabytes are not violated.

`nixbuild.default-user-settings.output-closure-ttl-seconds`¶

Type: integer

Default: 90 * 24 * 3600

Anytime a build output path closure is produced, its time to live (TTL) will be set to specified number of seconds. If a path in the closure already has a longer TTL, that TTL will not be changed. This means the complete path closure will remain in the NixBuild store for at least the specified time, as long as the settings storage-limit-days or storage-limit-gigabytes are not violated.

`nixbuild.default-user-settings.uploaded-path-ttl-seconds`¶

Type: integer

Default: 7 * 24 * 3600

Anytime a path is uploaded to the NixBuild store, its time to live (TTL) will be set to specified number of seconds. If the path in the already has a longer TTL, that TTL will not be changed. This means the complete path closure will remain in the NixBuild store for at least the specified time, as long as the settings storage-limit-days or storage-limit-gigabytes are not violated.

`nixbuild.default-user-settings.queried-path-ttl-seconds`¶

Type: integer

Default: 7 * 24 * 3600

Anytime a path is queried by a Nix client, its time to live (TTL) will be set to specified number of seconds. If the path in the already has a longer TTL, that TTL will not be changed. This means the complete path closure will remain in the NixBuild store for at least the specified time, as long as the settings storage-limit-days or storage-limit-gigabytes are not violated.

`nixbuild.gc-build-logs-max-age-days`¶

Type: integer

If set, build logs will be automatically deleted once they reach the specified age (number of days) specified by this setting. By default, this setting is unset, and no log deletion happens.

`nixbuild.oidc-audience`¶

A list of strings that nixbuild.net should accept as values for the aud field in incoming OIDC JWTs.

Type: list of strings

Default: [ "nixbuild.net" ]

`nixbuild.predefined-accounts`¶

A list of accounts that should be added automatically to the database when NixBuild starts. You can add new accounts to an existing NixBuild deployment by adding new entries here, but if you remove entries here they will not be removed from the NixBuild database. Updating entries here will only be reflected in the database if the overwrite flag is set for the account entry, see below.

Type: list of attribute sets

Default: []

Example:

[
  { account-id = 1
  , email = rickard@nixbuild.net
  , ssh-keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM/mBylFblBYkAoESGu3dPAYulFOqEmykhCgHhNb2taP rickard"
    ]
  }
]

`nixbuild.predefined-accounts.*.account-id`¶

Type: integer

Required

The unique integer id of the account. This id is used for looking up existing accounts.

`nixbuild.predefined-accounts.*.overwrite`¶

Type: bool

Default: true

If set to true (the default), any manual changes to this user account will be overwritten by the configuration defined here. If set to false, the configuration defined here will only be used for the initial creation of the user account.

`nixbuild.predefined-accounts.*.email`¶

Type: string

Required

The account's email address.

`nixbuild.predefined-accounts.*.ssh-keys`¶

Type: list of strings

Default: []

A list of public SSH keys of type ed25519 that users can use to access the account. The SSH keys must be unique between accounts.

You can add and remove SSH keys later using the NixBuild shell.

Example:

ssh-keys = [
  "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM/mBylFblBYkAoESGu3dPAYulFOqEmykhCgHhNb2taP rickard"
]

`nixbuild.predefined-accounts.*.user-settings`¶

Type: attribute set of user settings

Default: {}

Initial settings that should be set for the account. You can change these account settings later through the NixBuild shell.

Example:

{
  substituters = []
}

`nixbuild.profiling.enable`¶

Type: bool

Default: false

If set to true, profiling will be activated for nixbuild-frontend.service. Profiling output can be found in /var/log/nixbuild-frontend/profiling. Most of the profiling output will only be available after nixbuild-frontend.service has been stopped or restarted. Each time nixbuild-frontend.service is restarted, a backup of the profiling directory will be created in /var/log/nixbuild-frontend.

The config option nixbuild.profiling.rts-options controls what is included in the profiling outputs, see below.

`nixbuild.profiling.rts-options`¶

Type: list of strings

Default: [ "-p" ]

Defines a list of GHC RTS options to be used when running the profiled nixbuild-frontend binary. While you can set any GHC RTS options here, you should probably limit yourself to the ones that concern profiling.

For the options to take effect, you must also set nixbuild.profiling.enable to true.

`nixbuild.require-state-volume`¶

Type: bool

Default: false

If set to true, a filesystem separate from the boot disk is required to store all state (database, nar contents etc) that NixBuild produces. This filesystem must have the label nixbuild. If no such filesystem is found, and this setting is true, the NixBuild services will not start. If you start the server without the required filesystem and then make the filesystem available after boot, the NixBuild services will start automatically once the filesystem is detected. For more details see NixBuild State Management.

`nixbuild.state-volume-device`¶

Type: string

If you set this option to the device name of the volume you are using for NixBuild state, this device will be automatically initialised (a filesystem labeled nixbuild will be created if needed) and activated. You also need to set the nixbuild.require-state-volume option to true for this automatic initialisation to happen. For more details see NixBuild State Management.

`nixbuild.upload-to-caches-from-builders`¶

Type: bool

Default: true

If true, uploads to caches will be performed from the builder hosts. If false, uploads will be performed by nard. Any uploads that fails on the builder will be re-tried by nard.

`nixbuild.verify-oidc-jwts`¶

Type: bool

Default: true

If set to true, any OIDC ID token provided in the NIXBUILDNET_OIDC_ID_TOKEN SSH env var or NIXBUILDNET-OIDC-ID-TOKEN HTTP header will automatically be verified. If the token is valid the relevant Biscuit facts will be populated.

`ec2.instance-creation-retries`¶

Type: `integer

Default: 10

The number of times EC2 instance creation should be retried when provisioning new builder instances. When creating EC2 instances the "retry mode" will be set to standard and the "max attempts" will be set to the configured value. For more info see https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html.

`ec2.instance-termination-retries`¶

Type: `integer

Default: 20

The number of times EC2 instance termination should be retried when terminating idle builder instances. When terminating EC2 instances the "retry mode" will be set to standard and the "max attempts" will be set to the configured value. For more info see https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html.

`ec2.sandbox-paths`¶

Type: list of strings

Default: []

Example:

[ "/var/lib/sandbox-files/etc/ssl:/etc/ssl"
, "/var/lib/sandbox-files/tmp/krb5cc_30000:/tmp/krb5cc_30000"
, "/usr/share/zoneinfo:/usr/share/zoneinfo"
]

A list of paths bind-mounted read-only into the builder sandbox environments. Each entry should have the form SRC:DST, where SRC is a file or directory on your nxb-server-ec2 instance, and DST is the target path inside the builder sandbox. Both SRC and DST must be absolute paths. SRC can be either a regular file or a directory. You must take care not to set DST to any existing file or directory inside the sandbox that is vital for the sandbox functioning.

When a builder instance is provisioned, any files or directories defined in ec2.sandbox-paths will be recursively copied over to the new builder instance. When a build is launched the corresponding local files on the builder instance will bind-mounted into the build sandbox.

This is setting is similar to using the sandbox-paths setting in Nix.

`ec2.build-node-templates`¶

A list of "templates" that define the EC2 instances that should be available for running builds. Each entry in this list defines a group of instances sharing the same AMI, instance type etc. Each entry also specifies a count attribute specifying the number of instances that can be created. In total number of instances that can be used is the sum of all count attributes in this list.

The instances are provisioned when they are needed for running builds, and will be terminated automatically when no longer needed. slurm handles the selection of which and how many instances to provision depending how many CPU cores and how much memory is required by the active builds. If the resources needed by the builds exceeds what has been defined in this list, builds will be queued. This list of node templates therefore put an upper limit on how many EC2 instance can be active at the same time.

`ec2.build-node-templates.*.ami`¶

Type: string

Required

Specifies the AMI ID to be used for the nodes. You should make sure that the CPU architecture of the AMI ID configured is compatible with the configured EC2 instance type.

`ec2.build-node-templates.*.boot-disk-extra-gigabytes`¶

Type: int

Default: 8

Specifies the extra size that will be added to the boot disk, in addition to the space required for the AMI. This extra size will be used for the /var mount, which is used for storing nar caches.

`ec2.build-node-templates.*.count`¶

Type: int

Required

The maximal number of nodes that can be created for this node group.

`ec2.build-node-templates.*.instance-type`¶

Type: string

Example:

c6i.xlarge

The name of the EC2 instance type used for the instances. You can specify any valid EC2 instance type here, but your AWS account might have limitations on what instance types you can use. You should also make sure that the instance type you select is available in the region you've specified.

If you select an Arm instance type, make sure that the AMI you specify is built for aarch64-linux.

`ec2.build-node-templates.*.keep-alive`¶

Type: bool

Default: false

If keep-alive is set to true, the instances in the template are not terminated after a period of idleness as is usually the case. The instances will still start in a turned-off state, only when they first are needed for a build they will be turned on and then stay on after that. The instances will still be auto-terminated if the nxb-server instance is turned off or becomes unreponsive for a longer time.

This can be used to keep one or more instances available with less latency, for interactive builds. For this to work the instances must be large enough to handle the interactive builds, otherwise you still get the latency when NixBuild starts any larger instances needed.

`ec2.build-node-templates.*.max-uptime`¶

Type: integer

Default: 0

If max-uptime is set to a non-zero integer, the instances in the template will be terminated after they have been up for the number of seconds defined by max-uptime. This can be useful together with the keep-alive setting to avoid instances from being kept up for too long. You can, however, also use this setting on instances where keep-alive is not set.

Note, the instances will be terminated without any consideration on them being idle or not. Any builds that happen to be running on the instances will therefore be cancelled and restarted.

`ec2.build-node-templates.*.memory.zram.enable`¶

Type: bool

Default: false

If enabled, part of the memory on the EC2 builder instance will be reserved for zram swap usage. This increases the usable memory for builds beyond what is offered by the EC2 instance type.

The total memory available to builds is calculated like this:

m + ((m * p / 100) * (r - 1))

where:

m = EC2 instance memory
p = memory.zram.max-memory-percentage
r = memory.zram.compression-ratio

The memory.zram.compression-ratio parameter configures the expected compression rate. If the actual compression rate is lower, it is possible that the EC2 instance will run out of memory. Currently, nixbuild.net has no ability to detect such an event. Instead, nixbuild.net will simply detect that the builder instance stops working and will restart the build. However, it is then likely that the same situation will occur again. Once the number of allowed build restarts has been exhausted, nixbuild.net will fail the build. So, you should configure the memory.zram.compression-ratio conservatively, and monitor your builds for excessive failures.

`ec2.build-node-templates.*.memory.zram.compression-ratio`¶

Type: float

Default: 2.0

The expected zram compression ratio (see explanation above).

`ec2.build-node-templates.*.memory.zram.max-memory-percentage`¶

Type: integer

Default: 50

The percentage of memory set aside for zram. See above for explanation on how zram usage is calculated.

`ec2.build-node-templates.*.public-ip`¶

Type: bool

Default: true

Example:

false

If set to true (the default) a public IPv4 address will be added to the EC2 instance. If set to false, no public IP will be setup for the instance, in which case you should make sure that you have configured a NAT Gateway or similar for the instance's subnet.

`ec2.build-node-templates.*.region`¶

Type: string

Required

EC2 region.

`ec2.build-node-templates.*.security-group`¶

Type: string

Required

EC2 security group.

`ec2.build-node-templates.*.ssh-key-name`¶

Type: string

Required

EC2 SSH key name.

`ec2.build-node-templates.*.subnet`¶

Type: string

Required

EC2 subnet.

`ec2.build-node-templates.*.tags`¶

Type: attribute set of strings

Default: {}

A key-value mapping of EC2 instance tags that should be assigned to the instances.

Example:

{ env = "qa"
, deployment = "A"
}

`ec2.build-node-templates.*.weight`¶

Type: int

Required

The priority of this node group. A lower number means nodes from this group will have a higher priority when deciding which node to provision if there is no running node that can run a build request.

System Configuration¶

Configuration example¶

Configuration Options¶

nard.profiling.enable¶

nard.profiling.rts-options¶

nard.s3.allow-ambient-credentials¶

nard.push.s3.default-compression-level¶

nard.push.s3.compression-chunk-bytes¶

nard.push.s3.concurrency¶

nard.push.s3.buffer-size¶

nard.push.s3.max-buffers-per-nar¶

nard.push.s3.max-concurrent-nars¶

nard.push.s3.max-concurrent-requests¶

nixbuild.ssh-auto-login-account-id¶

nixbuild.default-user-settings¶

nixbuild.default-user-settings.storage-limit-days¶

nixbuild.default-user-settings.storage-limit-gigabytes¶

nixbuild.default-user-settings.input-closure-ttl-seconds¶

nixbuild.default-user-settings.output-closure-ttl-seconds¶

nixbuild.default-user-settings.uploaded-path-ttl-seconds¶

nixbuild.default-user-settings.queried-path-ttl-seconds¶

nixbuild.gc-build-logs-max-age-days¶

nixbuild.oidc-audience¶

nixbuild.predefined-accounts¶

nixbuild.predefined-accounts.*.account-id¶

nixbuild.predefined-accounts.*.overwrite¶

nixbuild.predefined-accounts.*.email¶

nixbuild.predefined-accounts.*.ssh-keys¶

nixbuild.predefined-accounts.*.user-settings¶

nixbuild.profiling.enable¶

nixbuild.profiling.rts-options¶

nixbuild.require-state-volume¶

nixbuild.state-volume-device¶

nixbuild.upload-to-caches-from-builders¶

nixbuild.verify-oidc-jwts¶

ec2.instance-creation-retries¶

ec2.instance-termination-retries¶

ec2.sandbox-paths¶

ec2.build-node-templates¶

ec2.build-node-templates.*.ami¶

ec2.build-node-templates.*.boot-disk-extra-gigabytes¶

ec2.build-node-templates.*.count¶

ec2.build-node-templates.*.instance-type¶

ec2.build-node-templates.*.keep-alive¶

ec2.build-node-templates.*.max-uptime¶

ec2.build-node-templates.*.memory.zram.enable¶

ec2.build-node-templates.*.memory.zram.compression-ratio¶

ec2.build-node-templates.*.memory.zram.max-memory-percentage¶

ec2.build-node-templates.*.public-ip¶

ec2.build-node-templates.*.region¶

ec2.build-node-templates.*.security-group¶

ec2.build-node-templates.*.ssh-key-name¶

ec2.build-node-templates.*.subnet¶

ec2.build-node-templates.*.tags¶

ec2.build-node-templates.*.weight¶

`nard.profiling.enable`¶

`nard.profiling.rts-options`¶

`nard.s3.allow-ambient-credentials`¶

`nard.push.s3.default-compression-level`¶

`nard.push.s3.compression-chunk-bytes`¶

`nard.push.s3.concurrency`¶

`nard.push.s3.buffer-size`¶

`nard.push.s3.max-buffers-per-nar`¶

`nard.push.s3.max-concurrent-nars`¶

`nard.push.s3.max-concurrent-requests`¶

`nixbuild.ssh-auto-login-account-id`¶

`nixbuild.default-user-settings`¶

`nixbuild.default-user-settings.storage-limit-days`¶

`nixbuild.default-user-settings.storage-limit-gigabytes`¶

`nixbuild.default-user-settings.input-closure-ttl-seconds`¶

`nixbuild.default-user-settings.output-closure-ttl-seconds`¶

`nixbuild.default-user-settings.uploaded-path-ttl-seconds`¶

`nixbuild.default-user-settings.queried-path-ttl-seconds`¶

`nixbuild.gc-build-logs-max-age-days`¶

`nixbuild.oidc-audience`¶

`nixbuild.predefined-accounts`¶

`nixbuild.predefined-accounts.*.account-id`¶

`nixbuild.predefined-accounts.*.overwrite`¶

`nixbuild.predefined-accounts.*.email`¶

`nixbuild.predefined-accounts.*.ssh-keys`¶

`nixbuild.predefined-accounts.*.user-settings`¶

`nixbuild.profiling.enable`¶

`nixbuild.profiling.rts-options`¶

`nixbuild.require-state-volume`¶

`nixbuild.state-volume-device`¶

`nixbuild.upload-to-caches-from-builders`¶

`nixbuild.verify-oidc-jwts`¶

`ec2.instance-creation-retries`¶

`ec2.instance-termination-retries`¶

`ec2.sandbox-paths`¶

`ec2.build-node-templates`¶

`ec2.build-node-templates.*.ami`¶

`ec2.build-node-templates.*.boot-disk-extra-gigabytes`¶

`ec2.build-node-templates.*.count`¶

`ec2.build-node-templates.*.instance-type`¶

`ec2.build-node-templates.*.keep-alive`¶

`ec2.build-node-templates.*.max-uptime`¶

`ec2.build-node-templates.*.memory.zram.enable`¶

`ec2.build-node-templates.*.memory.zram.compression-ratio`¶

`ec2.build-node-templates.*.memory.zram.max-memory-percentage`¶

`ec2.build-node-templates.*.public-ip`¶

`ec2.build-node-templates.*.region`¶

`ec2.build-node-templates.*.security-group`¶

`ec2.build-node-templates.*.ssh-key-name`¶

`ec2.build-node-templates.*.subnet`¶

`ec2.build-node-templates.*.tags`¶

`ec2.build-node-templates.*.weight`¶