Skip to content

Changelog

nixbuild.net uses semantic versioning (MAJOR.MINOR.PATCH) with the following meaning:

  • MAJOR version changes are used to signal large feature additions and milestones. There are no compatibility guarantees between different MAJOR versions.

  • MINOR version changes are used to signal changes in internal and external compatibility. Internal compatibilty means database schema and builder protocol. You need to use the same MINOR version on the frontend server as you do on your build hosts. Additionally, after updating to a new MINOR version it is not guaranteed that the database (and other state) can be used together with an older MINOR version anymore. External compatibility means the APIs exposed to nixbuild.net clients, essentially the Nix protocol and the HTTP API. It is likely that a separate versioning scheme will be introduced for representing external compatibility.

  • PATCH version can introduce fixes and features in a backwards-compatible way.

2.9.1

Release date: 2026-05-18

Fixed

  • Fixed reading the special NIXBUILDNET_* derivation attributes also when structured attributes (__structuredAttrs) are enabled.

  • Predefined accounts had their manually added SSH keys deleted on service restarts, even though overwrite = false was set for the account. This was supposed to have been fixed in 2.9.0, but it was only working if you had a single predefined account.

2.9.0

Release date: 2026-05-18

Fixed

  • Fixed inefficient db queries used when searching for builds having specific tags, using the HTTP API. This fix required bumping the minor version number due to database schema changes.

  • Fixed a problem when registering build outputs with identical nar hash as an already stored nar. Due to an oversight in the implementation, such build outputs were still fully parsed and written to storage, before finally being deleted in favour of the already existing content. This could waste a lot of IO if you had builds repeatedly create the same large outputs.

  • Predefined accounts had their manually added SSH keys deleted on service restarts, even though overwrite = false was set for the account.

Added

  • New Prometheus metric nixbuild_nard_stored_bytes_total exported by the nixbuild-frontend service, tracking nar ingestion in more detail than the current nard_uploaded_nar_bytes metric exported by the nard service.

  • New user setting build-summary-log-format that lets you select one of none, pretty and logfmt to control the look of the build summary log.

  • New user setting log-build-url that lets you turn off displaying of the build url printed in the build summary.

Changed

  • Worker protocol (ssh-ng://) updated from 1.32 to 1.34, adding support for some newer more efficient Nix operations.

  • The build summary log is now printed only after the final build (not after build restarts caused by transient errors), after any post-build activities finished (like uploading to external caches).

2.8.7

Release date: 2026-05-06

Fixed

  • Optimize db queries that registers GC roots.

  • Remove an unused db index.

Changed

  • Re-introduce the nard changes that was rolled back in 2.8.1, with further fixes and optimizations that makes sure CPU and memory usage stays under control during cache uploads.

2.8.6

Release date: 2026-04-30

Fixed

  • Updated nixpkgs from 25.11 3f6aea94bebc142cff1f7a9dd531c91013ef1129 to 5a9c58fc6ac2ec48bf9cf4c07de27f912b1ed1cc. This includes kernel patches for CVE-2026-31431 (copy.fail).

  • Fix a lock starvation that could occur when building several large derivation graphs using remote store building.

  • Make the substitution process more efficient by chunking work more.

  • Vaccuum garbage collected db tables less frequently.

Added

  • Add metrics for counting fetched narinfos and 404 narinfos.

  • Add metrics for tracking queued db connections.

  • Add metrics for tracking session received/transmitted bytes.

2.8.5

Release date: 2026-04-27

Fixed

  • AWS deployments: Fix support for KVM builds on aarch64 instances.

Added

  • Enabled the postgresql prometheus exporter on port 9187.

2.8.4

Release date: 2026-04-26

Fixed

  • Add two missing indexes to the database, speeding up the path deletion GC job.

Added

  • Add new config options system.services.*.rts-options for setting GHC RTS options for the nixbuild services. Run nixbuild-config get system services to see the default values.

2.8.3

Release date: 2026-04-24

Fixed

  • Switched to using PostgreSQL's statement timeout instead of killing connections, which could leave statements running on the PostgreSQL side.

  • Fixed collection of the nixbuild_gc_pending_deletion_* metrics to better reflect reality.

  • Removed the 10% margin added to account storage limit in the GC.

  • Reduced CPU usage caused by the GHC garbage collector during idle periods by GHC GC settings. This fixes the increase of CPU usage that was introduced in 2.8.2.

Changed

  • Renamed the metric nixbuild_gc_invalidated_paths_total to nixbuild_gc_deleted_paths_total.

  • Chunk the database transactions used by the nar invalidation and path deletion jobs.

  • Increased Biscuit policy execution timeout to 1s.

  • Randomize start of recurring jobs a little to avoid thundering herd effects.

  • Changed the default maximum size of the derivation cache introduced in 2.8.2 from 65536 to 100000 (the setting derivation-cache-size).

  • Changed the default interval of the nar claim deletion deletion from six hours to one hour (the setting gc-nar-claim-deletion-interval-seconds).

  • Replaced the setting gc-nar-invalidation-daily-batch-divisor with gc-nar-invalidation-sample-percentage, controlling many of the rows in the nar table that should be randomly sampled on each run of the nar invalidation job. Defaults to 1 percent.

Added

  • Added TTL to the derivation cache, configurable with the new derivation-cache-ttl-seconds setting. Defaults to one hour.

  • Added the setting gc-nar-invalidation-interval-seconds controlling how often the nar invalidation job should run. This was previously hard-coded to 12 hours. Defaults to one hour.

  • Added the setting gc-nar-invalidation-chunk-size to control how many of the nar rows will be checked in a single database transaction when the nar invalidation job works its way through the sampled nars.

  • Added the setting gc-path-deletion-sample-percentage, controlling many of the paths should be randomly sampled on each run of the path deletion job. Defaults to 1 percent.

  • Added the setting gc-path-deletion-interval-seconds controlling how often the path deletion job should run. This was previously hard-coded to 12 hours. Defaults to one hour.

  • Added the setting gc-path-deletion-chunk-size to control how many of the paths will be checked in a single database transaction when the path deletion job works its way through the sampled paths.

  • New Prometheus metric nixbuild_db_connections for tracking active database connections per nixbuild-* service. The maximum number of connections can be configured with the (existing) setting db-config.connection-pool-size.

  • New Prometheus metric nixbuild_gc_deleted_nar_claims_total, tracking how many rows have been removed from the nar_claim table. This makes it easier to track GC progress.

2.8.2

Release date: 2026-04-17

Fixed

  • Fixed excessive memory when building many identical or similar large derivations concurrently using remote store builds.

  • Fix slow database queries updating process status

  • In the container-based build environment, make the build directory /build instead of a randomly named subdirectory of that directory. This fixes cases where non-determinism is introduced in the build through the path name.

2.8.1

Release date: 2026-04-15

Fixed

  • Fixed excessive memory usage when uploading large path closures via ssh-ng://. Also cut down on the number of database transactions during the upload, to improve ingestion time.

Changed

  • Rollback the change in how nard uploads nars to external caches. Go back to the previous independent handling of caches. The rollback is done since CPU and memory usage jumped up in the new implementation. More research is needed to improve the new implementation.

  • Use jumboframes on EC2 instances.

Added

  • Make database connection pool size and session timeout configurable.

2.8.0

Release date: 2026-04-13

Changed

  • When nard uploads build inputs to external S3 caches, it will now compress one part at a time and then upload that part to all configured caches. Previously, each cache was handled independently, which meant the work of compressing parts was duplicated per cache.

Note that the new implementation currently has a performance regression when just one or a few very large inputs are uploaded. The current implementation use no concurrency within individual nar files, while the old implementation was able to upload different parts of the same nar file concurrently. There is a plan to improve the new implementation to do the same. Overall, the change in the way multiple caches are handled and the fact that concurrency between multiple inputs still is supported, should outweigh the (temporary) performance regression.

This fix bumps the MINOR version since it introduces changes in the internal builder protocol.

Fixed

  • A bug in the parser of cache URIs could cause Cachix auth tokens not to be picked up as expected. The problematic case was a cache URI on the format cachix://xyz.cachix.org together with an access token on the format cachix://xyz, even though these two formats works interchangeably in other parts of nixbuild.net.

  • Several improvements in database queries, reducing the risk of timeouts.

  • Increased reliability of recurring background jobs.

Added

  • Add a new setting called nard-read-connections-per-account-limit that limits the number of concurrent reads from nard (triggered by nix copy --from etc) each account is allowed. By default the per-account limit is set to the same value as the global limit (nard-read-connections-limit).

2.7.5

Release date: 2026-04-04

Fixed

  • Update the Prometheus client library used to fix edge case exception.

Added

  • Add histogram versions of db transaction duration summary Prometheus metrics.

2.7.4

Release date: 2026-03-30

Fixed

  • Fix nar GC job so that it can't run concurrently, causing db transaction restarts. Also make the nar GC job faster by splitting it up into smaller pieces.

  • Fix a potential database deadlock by sorting inserts to one of the derivation tables.

  • Fix a type cast exception in one of the GC background jobs, causing it to fail. This issue was a regression introduced in 2.7.3.

Changed

  • Change defaults for nard-connection-limit settings.

Added

  • Add histogram versions of build summary Prometheus metrics.

  • Add a new setting called nard-write-connections-per-account-limit that limits the number of concurrent writes to nard (uploads, substitutions) each account is allowed. By default the per-account limit is set to the same value as the global limit (nard-write-connections-limit).

  • Cache parsed Biscuit tokens for 30s and the revocation list for 5m.

2.7.3

Release date: 2026-03-23

Fixed

  • Improve performance and efficiency of the garbage collector that removes NAR content claims that have expired or exceed allowed storage limit.

  • Remove excessive debug logging in process interpreter.

  • Remove problematic udev rule that caused trouble when running multiple simultaneous kvm builds on the same builder host.

2.7.2

Release date: 2026-03-19

Added

  • Changelog integrated into enterprise docs.

2.7.1

Release date: 2026-03-16

Fixed

  • Fix an issue in the nixbuild-* services where the Prometheus exporter could stop working after running into an exception. Care is now taken to restart the exporter on internal errors.

  • Fix an issue in the S3 substituter, where the region parameter wasn't used when fetching nar contents, only when fetching narinfo.

  • Fixed a regression introduced with the support for nested virtualization on EC2 that made provisioning of metal instances fail.

  • Support Biscuit secret keys that include the key type prefix.

Changed

  • Tweak the memory allocator to be a little more conservative

Added

  • When nard uploads to S3 it will now retry individual S3 requests that failed in a transient way with exponential backoff and jitter.

  • Add a new configuration setting ec2.build-node-templates.*.iam-instance-profile. If set to a non-empty string, it will define the IAM instance profile that should be used for EC2 builders within that group. This is useful for allowing S3 cache uploads from build nodes to use ambient AWS credentials, avoiding using an access token.

2.7.0

Release date: 2026-03-02

Added

  • New process-based build orchestrator, currently enabled only for build requests made through the HTTP API. This changes the database schema and the HTTP API.

2.6.6

Release date: 2026-02-22

Added

  • KVM builds are now supported on EC2 builders that uses an instance type that supports nested virtualization (like C8i, M8i and R8i). If you add any such instances (or bare-metal instances, like before) to your cluster configuration it will automatically gain support for running KVM builds.

2.6.5

Release date: 2026-02-09

Fixed

  • The garbage collector background job that looks for unused nar references and store paths has been split into smaller parts to avoid timeouts.

  • Fix an issue in the periodic background job that deletes unused nar data that could lead to the job not recurring until nixbuild-jobs.service was restarted.

  • The slurm configuration has been tweaked for lower latency when new builds are submitted. The build cluster has been split into one partition per Nix system type (x86_64-linux, aarch64-linux etc). This improves latency since slurm doesn't check system constraints in its initial scheduling attempt, only in periodic backfill loop. Submitting builds to separate partitions avoids a potential delay if all nodes of a specific architecture was busy. Additionally, the backfill loop has been tweaked for lower latency.

Changed

  • Update nixpkgs from nixpkgs-25.11 a35a20f6d3d51b6b42b68a6879701f7d87381896 to nixpkgs-25.11 3f6aea94bebc142cff1f7a9dd531c91013ef1129.

  • The narinfo JSON sent to exec:// caches now also includes the ca, deriver, signatures and nar_hash fields.

  • For EC2 builders with an instance type that supports it, Nitro Enclave support is now enabled. This has no practical use for the moment, but in the future enclave support could potentially be added to the build sandbox.

Added

  • New user setting optional-caches which is just like caches except that failed uploads do not interrupt the build process.

2.6.4

Release date: 2026-01-28

Fixed

  • Improve performance of the derivation graph walker used during remote store building. For derivations with large number of dependencies this can significantly cut down the time until the first build starts.

  • Run billing processing jobs in smaller chunks to avoid database transaction timeouts.

  • Cache slurm cluster state to avoid querying it on each build to find out if there are builders matching the derivation requirements.

  • Avoid excessive database transaction retries when concurrent builds of the same derivation is started at the same time.

2.6.3

Release date: 2026-01-26

Fixed

  • When uploading build inputs to nard and/or external caches, memory usage on the builder host could balloon if the upload slowed down due to a missing buffer limit when reading nar data from the build sandbox.

  • Switch to using the Cachix multipart API, fixing errors that occurred when uploading large paths to Cachix and improving overall Cachix upload performance.

2.6.2

Release date: 2026-01-19

Fixed

  • Fixed caching of narinfo when uploading to external caches. Previously our cache was only updated querying for missing narinfos, not when uploading narinfos. This caused an extra check for missing narinfos.

  • Fixed an issue with the nixbuild-jobs service that made it fail to start on machines with more than 256 CPU threads.

  • Fixed nard so it queries external caches with multiple store paths at once, instead of one at a time which kills performance.

  • Fixed nard so it runs exec:// cache uploads concurrently.

  • Use a separate nard connection pool for requests that upload to external caches from requests that uploads nar files to nard, to avoid cache uploads to starve other operations.

  • Fix exception handling during substitution to avoid killing Nix sessions on certain connection errors.

Changed

  • For the exec:// caches, narinfo is now provided on a file descriptor defined by the NARINFO_JSON_FD environment variable, instead of being directly available in the NARINFO_JSON environment variable. This avoids a 128 KiB limit on the narinfo.

Added

  • Add metrics for tracking nard connections (nixbuild_nard_connections).

2.6.1

Release date: 2026-01-12

Fixed

  • Fixed a regression introduced in version 2.6.0 causing many builds to hang or run very slowly.

2.5.6

Release date: 2026-01-19

Fixed

  • Fixed nard so it runs exec:// cache uploads concurrently.

  • Use a separate nard connection pool for requests that upload to external caches from requests that uploads nar files to nard, to avoid cache uploads to starve other operations.

  • Fix exception handling during substitution to avoid killing Nix sessions on certain connection errors.

Changed

  • For the exec:// caches, narinfo is now provided on a file descriptor defined by the NARINFO_JSON_FD environment variable, instead of being directly available in the NARINFO_JSON environment variable. This avoids a 128 KiB limit on the narinfo.

Added

  • Add metrics for tracking nard connections (nixbuild_nard_connections).

2.5.5

Release date: 2026-01-14

Fixed

  • Fixed another narinfo caching issue when uploading to external caches.

  • Fixed an issue with the nixbuild-jobs service that made it fail to start on machines with more than 256 CPU threads.

  • Fixed nard so it queries external caches with multiple store paths at once, instead of one at a time which kills performance.

2.5.4

Release date: 2026-01-14

Fixed

  • Fixed caching of narinfo when uploading to external caches. Previously our cache was only updated querying for missing narinfos, not when uploading narinfos. This caused an extra check for missing narinfos.

Changed

  • For the exec:// caches, narinfo is now provided on a file descriptor defined by the NARINFO_JSON_FD environment variable, instead of being directly available in the NARINFO_JSON environment variable. This avoids a 128 KiB limit on the narinfo.

2.5.3

Release date: 2026-01-11

Fixed

  • Fix a bug causing cache uploads to hang if more than one exec:// cache was configured. (Back-ported from 2.6.0)

2.6.0

Release date: 2026-01-09

Changed

  • Update nixpkgs from nixpkgs-25.05 7b936b6c804867b0eb87378eefe9b93ced546d8f to nixpkgs-25.11 a35a20f6d3d51b6b42b68a6879701f7d87381896.

Fixed

  • Fix a bug causing cache uploads to hang if more than one exec:// cache was configured.

2.5.2

Release date: 2025-12-17

Added

  • Added a configuration option called nixbuild.ssh-auto-login-account-id that allows bypassing normal SSH or Biscuit authentication and directly login users to the specified account.

  • Add boto3 and requests to the system python3 on nxb-server

  • Access tokens (as defined in the access-tokens user setting) will now be propagated to matching exec:// caches in the ACCESS_TOKEN environment variable.

  • Output authentication logs directly to the SSH session's stderr if the setting nixbuild.debug-logs is activated.

Fixed

  • Update the AWS EC2 instance type database so that builders using newer instance types like c8a can be provisioned.

  • Fix the User Guides section in enterprise docs.

2.5.1

Release date: 2025-12-13

Fixed

  • Fixed broken links in enterprise docs.

2.5.0

Release date: 2025-12-13

Added

  • Add jless and python3 to the system path of nxb-server.

  • Log OIDC JWKS errors. These are printed to the journal for now.

  • You can now set IMDSv2 to required for nxb-server-ec2 instances.

Fixed

  • Avoid excessive database transaction retries when many builds finish at the same time.

  • Fix a regression introduced in 2.4.3 that caused provisioning of large sandbox files to builders sometimes result in incomplete or missing files.

  • Update the libucl version used, fixing a JSON parser issue. It is now possible to use either JSON or ucl configuration files to configure nixbuild.net.

  • Swap out the cloud-init implementation used for EC2 deployments, since the previous one could corrupt the nixbuild configuration in certain situations.

  • Correctly specify build node region when provisioning new EC2 build nodes.

Changed

  • Change the way configuration files in /etc/nixbuild.net/conf.d are merged. See the System Configuration documentation for details. In short, configurations are now recursively merged, but any "leaf" settings will be overwritten. This means that it is not possible to append to lists like it was before, but on the other hand it is now possible to override list settings which was not possible previously (system-defined list items couldn't be removed).

  • Use (require) IMDSv2 for build nodes provisioned on EC2.

2.4.5

Release date: 2025-12-03

Added

  • Expanded support for uploading to exec:// caches (upload from builder hosts implemented).

2.4.4

Release date: 2025-11-24

Fixed

  • Fix an edge case where build outputs were not uploaded to Cachix caches if we already had an identical nar file in nard.

Added

  • Configuration option for accepted values for the aud field in OIDC JWTs

  • aarch64 builder image for Azure

  • Initial support for uploading to exec:// caches

2.4.3

Release date: 2025-11-11

Fixed

  • The iptables configuration script for slurmd is now more idempotent.

Added

  • Enforce CPU usage limit through Biscuit policy.

  • Tweaks to how free CPU hours are configured and re-filled.

  • New configuration option ec2.build-node-files for provisioning files to EC2 builders.

  • Smarter provisioning of EC2 sandbox files.

  • Tweaks to slurm scheduler settings.

  • New process attribute NIXBUILDNET_GITHUB_CHECK_RUN_SHA for passing a commit SHA to be associated with GitHub Check Runs created by nixbuild.net.

  • Improved OIDC token exchange. The new token is the original token, but extended with a Biscuit "third-party block", containing the JWT claims from the OIDC token. This way, all content of the original token is preserved.

2.4.2

Release date: 2025-11-02

Fixed

  • Handle a case where a slurm node could get out of sync with its cloud instance, leading to lingering cloud instances.

Added

  • Enable support for running builds requiring KVM on metal EC2 instances. nixbuild.net will automatically route KVM builds to the correct instance if you add one or more metal instances to your build cluster definition.

  • Experimental new HTTP endpoint for exchanging an OIDC token for a nixbuild.net biscuit token

  • Experimental support for running installation processes in parallel

2.4.1

Release date: 2025-10-21

Fixed

  • Fixed incorrect sleep time in database transaction retry logic.

  • Fixed backoff time calculation in database transaction retry logic.

  • Improved registration of completed builds (avoid some db serialization issues).

Changed

  • Run monthly CPU usage update in periodic background job instead of on every build.

Added

  • Automatically fix biscuit and ssh-host key premissions at startup, if necessary.

  • First versions of NixOS modules for Azure server and builder.

  • First version of nodectl implementation for Azure.

2.4.0

Release date: 2025-09-23

Fixed

  • Fixed issue that made it impossible to set the settings max-cpu and max-mem in the admin shell.

  • Fixed regression that caused the nixbuild_active_builds metric go negative.

  • Add missing back-off delay when new builds couldn't be created due to database serialization errors.

  • Make HTTP header parsing for the NIXBUILDNET* headers case insensitive.

Changed

  • Make HTTP header parsing for the NIXBUILDNET* headers accept hyphens instead of underscores. Using hyphens is the correct approach, since underscores in header names can cause headers to be removed by HTTP proxies.

Added

  • Add experimental support for submitting build requests through the HTTP API. This is work in progress and currently not documented. This change bumps the MINOR version. When updating to this version the db schema is no longer compatible with older versions.

2.3.1

Release date: 2025-09-03

Changed

  • Update nixpkgs from nixpkgs-25.05 a5e9291e97f5ba0b4ba7d657ddedd5f86d11acfd to nixpkgs-25.05 7b936b6c804867b0eb87378eefe9b93ced546d8f.

  • We now support at most one OIDC ID Token to be provided during auth, passed in through the SSH environment variable or HTTP header NIXBUILDNET_OIDC_ID_TOKEN.

Added

  • The HTTP API now accepts HTTP request headers prefixed NIXBUILDNET_ and parses them to extract settings, just like for the SSH environment.

Fixed

  • The Nix version used internally in the nixbuild.net build sandbox was updated from 2.29.0 to 2.30.2. Nix 2.29.0 had a bug that caused some builds using the __json or __structuredAttrs derivation attribute to fail.

2.3.0

Release date: 2025-08-20

Added

  • Add the configuration setting ec2.build-node-creation-hook that allows specifying a script that should run each time a new EC2 builder has been provisioned. This can be used to copy files (like SSL certs) to the instance.

Changed

  • When build outputs are uploaded to any S3 caches specified in the account settings, the upload will now be performed directly from the builder host that ran the build. When running many concurrent builds with large outputs, this change increases performance radically, since the work of compressing and uploading outputs will be spread out on many builder servers instead of doing all work in nard. In some benchmarks, speedups of x2.7 has been seen. This change bumps the MINOR version, and you need to make sure you are using compatible builder and frontend versions.

2.2.0

Release date: 2025-07-22

Changed

  • Change the way build outputs are uploaded to nard, by performing the upload directly from the host that ran the build. This avoids tunneling uploads through nixbuild-frontend, saving on resources and improving performance. For builds with large outputs, this can cut down the post-build time with 20-30%. This change bumps the MINOR version, and you need to make sure you are using compatible builder and frontend versions.

  • Update nixpkgs from nixpkgs-24.11 f44bd8ca21e026135061a0a57dcf3d0775b67a49 to nixpkgs-25.05 a5e9291e97f5ba0b4ba7d657ddedd5f86d11acfd.

  • Remove the status tag from the nixbuild_build_queue_time_seconds Prometheus metric.

Fixed

  • Fix broken recording of active builds Prometheus metric.

  • Improve backoff logic when uploads to external caches fail.

2.1.1

Release date: 2025-06-05

Added

  • If enabled (default disabled), nixbuild.net will verify any OIDC ID Tokens passed in through the SSH environment as variables prefixed NIXBUILDNET_OIDC_ID_TOKEN_. The iss and sub claims from verified tokens will be added as the Biscuit facts jwt_claim("iss", $iss) and jwt_claim("sub", $sub). This allows you to create attenuated auth tokens that can only be used from requests that originated in a specific OIDC provider, like GitHub Actions or similar. This feature can be turned on by enabling the boolean config setting nixbuild.verify-oidc-jwts.

NOTE: This feature required new tables to be added to the database, and the migration scripts can take a long time (10 minutes or so) if there are many entries in the ssh_session table. The db changes are still backwards compatible (you can run older versions of nixbuild.net on the same database), since no modifications or deletions are performed, only additions.

Fixed

  • A regression in 2.1.0 caused the 'ssh-keys list' command to fail in the nixbuild.net shell. This has been fixed now.

  • Improve database transaction reliability on errors. Retry transactions on more kind of errors like temporary OOM failures etc. Increase maximal backoff time between transaction retry from 10s to 30s.

  • Fix regressions introduced in 2.1.0 that broke the settings page in the Web UI.

  • Fix failures in the sandbox-files-daemon service that could be caused by misbehaving EC2 builder nodes.

Changed

  • When checking for already running builds before starting a new build, don't look further back than 2h.

2.1.0

Release date: 2025-05-21

Fixed

  • Some Nix operations could cause resource exhaustion if the Nix client didn't exit the SSH session cleanly (for example due to network issues). This in turn caused future Nix operations to hang. This was successfully reproduced and fixed (the fix in 2.0.21 was not enough to resolve the issue).

  • Changed a database entity id type from integer to bigint, working around id exhaustion. This needed a backwards incompatible db schema change.

Changed

  • With this version we change the meaning of the MAJOR.MINOR.PATCH version number schema and departs from the "official" Semantic Versioning schema.

From now on, the MAJOR version represents external compatibility. This number will be bumped when we change our API or integration interfaces in a way that needs changes in external systems.

The MINOR version represents internal compatibility. This number will be bumped when the nixbuild.net database schema or internal builder protocol changes. This means that if you update to a new MINOR version, it is not guaranteed that you can go back to an older MINOR version using the same database contents. You also need to keep your builder AMIs on the same MINOR version as the main nixbuild.net server.

The PATCH version represents changes, fixes or additions that are backwards compatible both internally (db schema) and externally.

  • Tweak the memory allocator to be a little more conservative since statistics still show that we tend to over-provision memory and a very low percentage of builds run out of memory. Previously, new builds was allocated 35% more memory than earlier successful builds or 85% more than unsuccessful builds. These percentages have been tweaked to 25% and 80% now.

  • If we get spurios build errors and also registers memory usage at 85% or more, we assume the build ran out of memory.

  • If a running build exceeds 95% memory we consider it out of memory and restart it. Previously, the limit was 90%.

2.0.21

Release date: 2025-05-14

Added

  • Added support for specifying a name when creating an access token in the Web UI or in the shell.

  • Added support for revoking access tokens in the Web UI.

Fixed

  • Fixed a suspected resource-leak caused by Nix clients not shutting down the SSH session cleanly (due to network issues or similar).

  • Fix performance issue when looking up historic resource usage for builds. For certain derivations, a considerable delay was introduced during this process.

  • Fix performance issues in nar garbage collector.

  • Enable mmap of large files in aarch64-linux builds when KVM-based isolation is used. Previously mmap was limited to 512 GiB in this situation.

2.0.20

Release date: 2025-04-23

Added

  • When configuring predefined-accounts in your nixbuild.net configuration, you can now set overwrite = false for accounts that you want to manage manually. Such accounts will not have their settings overwritten by changes made to the configuration, they will only be created with their initial settings from the configuration. Note, however, that if you remove the accounts entirely from predefined-accounts in your configuration they will also be removed from your nixbuild.net database. A special case is if you set predefined-accounts to an empty list, in which case the nixbuild.net database will not be modified at all (this is the same behavior as before).

Fixed

  • Use the default-permissions setting only for SSH key logins, not for the Web UI. This fixes an issue where you could lock yourself out of the Web UI if you've used default-permissions to make SSH keys have restricted permissions by default.

  • Introduce an HTTP connection pool between nixbuild-frontend and nard that can improve performance and reliability when storing build outputs.

  • Increase database transaction isolation level when creating new builds to be sure we don't create "orphaned" builds.

  • Make sure Superset uses the same state volume the other nixbuild.net services uses on EC2.

2.0.19

Release date: 2025-04-10

Added

  • Apache Superset (a data exploration and visualization platform) is now available on the AWS server images, connected to the nixbuild database and served under the /analytics URL, by nginx. The service is pre-loaded with a set of basic dashboards that we have developed.

Fixed

  • Refactored the way nar files are uploaded to S3 by nard. Previously, there was a chance for run-away memory usage when uploading multiple large nar files at the same time, since there were no limit on the number of compressed chunks of data kept in memory while waiting for being uploaded. Now there is a configurable limit on the amount of compressed data is allowed per (large) nar file, and a configurable limit on how many (large) nar files can be processed concurrently. Together with the configurable buffer size, this sets an approximate upper limit to the amount of memory nard can keep occupied during S3 uploads. The following new settings has been introduced to control the memory and CPU usage of nard: max-buffers-per-nar, max-concurrent-nars. The default values of the other related settings have also been adjusted.

Changed

  • Various minor tweaks to the Web UI.

  • Tweak the memory allocator to be a little more conservative since statistics show that we tend to over-provision memory and a very low percentage of builds run out of memory. Previously, new builds were allocated 50% more memory than earlier successful builds or 100% more than unsuccessful builds. These percentages have been tweaked to 35% and 85% now.

2.0.18

Release date: 2025-03-28

  • Serve the NixBuild API under /api/v1/ from nginx on nxb-server in the AWS deployments.

2.0.17

Release date: 2025-03-28

Added

  • Enable the NixBuild Web UI for AWS deployments, running as nixbuild-web.service on nxb-server. A small nginx instance is running in front of it, listening on HTTP port 80.

To access the Web UI, you need to set an Authorization: Bearer <TOKEN> header containing a NixBuild access token that you can generate using the tokens create command in the NixBuild shell.

  • Add a configuration option for setting the number of concurrent HTTP requests nard is allowed to perform against S3 when pushing to caches. The option is named nard.push.s3.max-concurrent-requests and defaults to 32. Before introducing this configuration option the (hard-coded) limit was set to 512. In tests, it seems like performance of HEAD requests can improve for higher values of this setting, but at around 32 to 64 the improvements diminish.

Fixed

  • Minor fixes and tweaks to the way nard issues HEAD requests to query caches about existing narinfo files.

2.0.16

Release date: 2025-02-28

Added

  • Make it possible to profile the nixbuild-http-api service just like the other nixbuild services can be profiled.

Changed

  • Update nixpkgs from nixpkgs-24.11 394571358ce82dff7411395829aa6a3aad45b907 to nixpkgs-24.11 f44bd8ca21e026135061a0a57dcf3d0775b67a49.

Fixed

  • Improved robustness during service startup in AWS deployments, where the nixbuild configuration could end up corrupted.

  • Register the number of (compressed) bytes that has been pushed to S3 caches (the nard_pushed_nar_bytes Prometheus metric). Previously this was only done for Cachix caches.

2.0.15

Release date: 2025-02-09

Added

  • Add support for the nix-daemon ops 'wopAddTempRoot', 'wopAddToStoreNar' and 'wopEnsurePath'. This makes it possible to do remote store builds (--store) without setting --eval-store auto. This means nixbuild.net can be used as "eval store". However, this makes evaluation slower since it adds latency, so it is still recommend to use the local machine as eval store.

Fixed

  • When running settings --reset in the nixbuild.net shell, the signing keys for the account was incorrectly removed, causing new build outputs to not be signed and therefore not possible to use as build inputs.

  • Fix a parser bug that could cause errors if the build cluster contains nodes with some specific memory amounts.

2.0.14

Release date: 2025-01-27

Added

  • Add support for using zram to expand the memory usable to builds on EC2 builder instances. To use this feature, builder AMIs must also be updated to 2.0.14.

2.0.13

Release date: 2025-01-21

Fixed

  • Fixed a regression introduced in 2.0.12 causing some similar build lookups to take a long time.

2.0.12

Release date: 2025-01-17

Fixed

  • Fixed bad performance of a database query that is used during substitution.

  • Fixed bad performance of a database query that is used when similar derivations are looked up.

2.0.11

Release date: 2025-01-14

Fixed

  • Fixed a regression causing substitution from self-hosted Cachix instances to use incorrect urls when fetching narinfos.

  • Fixed bad performance of a database query that is used to find non-deterministic builds.

2.0.10

Release date: 2025-01-09

Fixed

  • In builds for fixed-output derivations (that have network access), the file /etc/ssl/certs/ca-certificates.crt was a broken symlink. This caused some (but not all) downloads that uses SSL to fail.

  • If a build of a fixed-output derivation failed, this was incorrectly reported as an "internal error" and not as a normal build failure.

  • Build status messages are now stripped from any ANSI escape codes inserted by Nix.

  • Make build log GC deletion batch size configurable, using the nixbuild.gc-build-logs-deletion-batch-size configuration option. The default batch size is now set to 500 instead 100 that was the previous default. If you run large number of builds and log garbage collection can't keep up, you can try increasing this parameter.

  • If nixbuild-frontend crashed while running builds, those builds would not get registered in the table that keeps track on build log garbage collection. This meant that the garbage collector would never delete logs produced by such builds. This has been fixed by correcting such inconsistencies when starting up. If you have existing logs that are not deleted even though they are older than your GC setting you can simply drop such logs manually from the build_log table.

  • Fixed a bug where the build restart counter could be incremented incorrectly, leading to not restarting builds.

2.0.9

Release date: 2024-12-19

Changed

  • Update nixpkgs from nixpkgs-24.05 e8c38b73aeb218e27163376a2d617e61a2ad9b59 to nixpkgs-24.11 394571358ce82dff7411395829aa6a3aad45b907.

  • Update the Linux kernel used inside KVM-based build sandboxes from 5.10.142 to 6.12.5.

2.0.8

Release date: 2024-12-06

Added

  • When uploading store paths to external caches, we now show in the build log when the upload is done for inputs and outputs, respectively.

Fixed

  • Also store profiling data on the volume configured to handle nixbuild.net state. This avoids filling up root disks with profiling data.

  • Make S3 upload compress buffer a bit larger by default (from 10 MB to 24 MB), giving some more margin to S3's minimum part size of 5 MB, which can help if the data is very compressible.

  • S3 upload concurrency fixes: Avoid reading many small nar files into memory at the same time during upload, and allow more AWS requests to be performed concurrently.

2.0.7

Release date: 2024-11-27

Added

  • Make awscli2 available to cloud-init scripts on EC2 instances.

Fixed

  • ec2: Make termination of unresponsive nodes more reliable.

2.0.6

Release date: 2024-11-20

Changed

  • Update nixpkgs-24.05 from rev cd3e8833d70618c4eea8df06f95b364b016d4950 to rev e8c38b73aeb218e27163376a2d617e61a2ad9b59. This fixes a kernel issue on builders that could cause npm to hang (see https://github.com/NixOS/nixpkgs/issues/353709).

Fixed

  • Update the AWS EC2 instance type definitions. This makes new instance types like m8g etc available.

  • Fix the handling of EC2 InsufficientInstanceCapacity errors that was introduced in the previous release. A number of bugs caused node managament to be unreliable.

2.0.5

Release date: 2024-11-17

Added

  • During creation of new AWS EC2 builder instances, if an InsufficientInstanceCapacity error is triggered, we will now disable all non-active builder instances that use the same EC2 instance type and subnet as was used when triggering the error. The build will then be retried with another builder instance. The disabled builders will be re-enabled after 12 hours (at which point we may retry creation of such instances).

2.0.4

Release date: 2024-11-14

Changed

  • Update the Nix version used inside the builder sandbox from 2.18 to 2.24. If you use nxb-server together with a nxb-builder of version <= 2.0.3, you might now see a message saying unknown setting 'build-dir'. This message is harmless. Updating your nxb-builder AMI will remove the message.

  • Tweak memory allocation so that builds that fail immediately (perhaps due to builder provisioning errors) doesn't get memory increases.

Fixed

  • Fix issue where the SSH keys in the database was not properly synchronized with the predefined-accounts configuration.

  • Don't restart builds that Nix report as transient, since Nix seems to report the transient status incorrectly for failing fixed output builds.

2.0.3

Release date: 2024-11-04

Added

  • When narinfo files are uploaded to S3, any identically named narinfo files that already exists in the remote bucket will be "touched" (modification time updated). This allows performing garbage collection based on modification times.

Changed

  • Update nixpkgs-24.05 from rev 4a92571f9207810b559c9eac203d1f4d79830073 to rev cd3e8833d70618c4eea8df06f95b364b016d4950.

Fixed

  • Fix issue that caused the "page cache" used inside the build sandbox to be very small. Also tweak the caching logic a bit; avoid caching large reads, saving cache space.

  • Improved performance of the Nix Base32 decoder that caused significant CPU usage when parsing store paths.

2.0.2

Release date: 2024-10-22

Fixed

  • Fixed space (memory) leak in S3 upload that caused nard to use up all memory when uploading large outputs.

  • Fixed possible double-compression that could happen during S3 uploads. It is likely that the fixed code path actually never could be triggered, though.

2.0.1

Release date: 2024-10-21

Added

  • Initial web UI added, together with simplified sign-up flow using Hanko.

Changed

  • Limit the number of concurrent S3 fetches performed during substitution to 16. This matches the limit we already had on HTTP/Cachix substitutions. However, this will be revised again in the near future, and made configurable.

Fixed

  • Made S3 substitution more efficient not running the library auth and environment setup on each fetch.

2.0.0

Release date: 2024-10-08

Fixed

  • Strip trailing slashes from S3 bucket URLs.

Added

  • Support S3 session tokens. You can now add access-tokens looking like this: s3://mybucket=<KEY ID>:<SECRET>:<SESSION TOKEN>. If you do, the session token will be used in addition to the access key when accessing S3.

  • Support for setting the following settings inside the NixBuild shell: default-cpu, default-mem-per-cpu, max-build-restarts, max-mem, max-silent-time, min-cpu, min-mem, timeout. These settings were previously only possible to set using the SSH environment or derivation attributes.

  • Support bzip2 compressed store paths when substituting from binary caches. This compression format was previously not supported. The supported compression formats are now: xz, brotli, zstd and bzip2.

  • A database migration was added, making older versions of NixBuild incompatible with databases that have been used with this version of NixBuild. Therefore, the major version number has been increased.

1.1.7

Release date: 2024-10-04

Fixed

  • The user running the nard.service has been added to the nixbuild-secrets group, fixing a problem where nard was unable to read config files.

  • The S3 region was not set correctly when fetching nar files during substitution, only when fetching narinfo files.

Added

  • Added more logging when S3 errors are triggered during store path uploads.

  • On nxb-ec2-server, a new service has been added: nixbuild-9p.service. It listens on 127.0.0.1:564 and serves 9P clients with the contents of the NixBuild Nix store. This service is highly experimental for the moment, and added to nxb-ec2-server for early testing. For example is auth controls completely missing, meaning that you can access the Nix store of any NixBuild account if you have can connect to the 9P listening port. This is why the service is only listening on 127.0.0.1.

1.1.6

Release date: 2024-09-30

Fixed

  • Fixed memory leak in the byte stream that provides build sandboxes with instructions and data. It is not clear exactly how big effect, if any, this had on builds. The memory leak was found when using the same way of transferring bytes in a test unrelated to the build sandbox.

1.1.5

Release date: 2024-09-27

Added

  • Added the Prometheus metric nixbuild_build_post_time_seconds to nixbuild-frontend. This is a summary metric that measures the time spent after a build has finished but have not yet been reported back to the Nix client. This mostly includes time spending uploading build inputs and outputs to any configured caches. The metric has two tags, system (build platform) and status (build status).

  • Added the tag status (build status) to the Prometheus summary metrics nixbuild_build_duration_seconds and nixbuild_build_queue_time_seconds.

Fixed

  • The Prometheus metrics nixbuild_build_duration_seconds and nixbuild_build_queue_time_seconds that nixbuild-frontend tracks was not correctly registered, or in some cases not registered at all.

  • Restarted builds could have incorrect stop times and build statuses registered in certain situations.

1.1.4

Release date: 2024-09-27

Fixed

  • narinfo parser did not handle narinfos without a Sig line.

1.1.3

Release date: 2024-09-26

Fixed

  • Parsing of narinfo files during substitution did not work if the store paths had another store directory than /nix/store.

1.1.2

Release date: 2024-09-25

Added

  • S3 substituters now accept the setting require-sigs=false. If added, substitution will happen even if the narinfos found in the S3 bucket has no trusted signatures. Instead, the narinfos will be signed using the key defined in the signing-key-for-uploads setting (see https://docs.nixbuild.net/settings/#signing-key-for-uploads), before they are added to the NixBuild database. This makes substitution behave just like copying paths into NixBuild using nix copy behaves.

Changed

  • The configuration specified in nixbuild.predefined-accounts will now replace the configuration also of existing accounts, if the account ids match. This means nixbuild.predefined-accounts configuration can be updated after the initial creation of accounts. It also means that any manual changes you make to those accounts will be overwritten.

Fixed

  • The NixBuild configuration was not parsed correctly, making some settings specified in nixbuild.predefined-accounts.*.user-settings go missing.

1.1.1

Release date: 2024-09-23

Fixed

  • The region parameter on S3 buckets is now used in AWS operations. Specified like this: s3://my-bucket?region=us-east-2.

Added

  • Path prefixes now supported for S3 buckets used for substitution and caching. Syntax: s3://my-bucket/prefix. Access tokens must include the prefix, which means you can have different access tokens for different prefixes within the same bucket.

1.1.0

Release date: 2024-09-11

Added

  • Support substituting from S3 buckets.
  • Support pushing build inputs and outputs to S3 buckets.

1.0.3

Release date: 2024-08-15

Fixed

  • Fixed several issues in the nxb-builder-ec2 AMI for aarch64-linux that prevented instances from starting correctly.

1.0.2

Release date: 2024-08-14

Added

  • Explicitly VACUUM postgresql tables after GC events.

  • Added Lix 2.90 to the set of clients used in automated integration tests. Also expanded the number of versions of Nix used in the tests. Now testing the following versions: 2.3, 2.10, 2.11, 2.14, 2.15, 2.16, 2.17, 2.18, 2.19, 2.20, 2.21, 2.22 and 2.23.

Changed

  • Update nixpkgs-24.05 from rev 706eef542dec88cc0ed25b9075d3037564b2d164 to rev 4a92571f9207810b559c9eac203d1f4d79830073.

  • Adjusted reading of nix.conf internally in the builder environment. This also allowed us to remove the /etc/nix directory.

1.0.1

Release date: 2024-08-02

Changed

  • nxb-server-ec2: Configure security.sudo.wheelNeedsPassword = false.

1.0.0

Release date: 2024-07-27

Added

  • This CHANGELOG file