Nix 2.4 came out on November 1, 2021, and that’s sort of a big deal.

It’s been more than two years since the last major Nix release, and there are some pretty important changes. Including some breaking changes, which is why I have waited so long to upgrade.

Because I think that, in order to keep using Nix, I’m finally going to have to learn what “flakes” are, and do a bunch of other stuff just to restore the functionality that I was enjoying before.

Or maybe not. It might be painless. Let’s find out!

$ nix-env -iA nixpkgs.nix
replacing old 'nix-2.3.16'
installing 'nix-2.4'
...
$ nix --version
nix (Nix) 2.4

Well that was easy.

The closest thing to “documentation” about Nix 2.4 is the release notes, so let’s try reading through them to learn what’s changed.

The big picture highlights are:

• The nix command is better, apparently, but it is also now disabled by default, so anyone using Nix has to add the line experimental-features = nix-command to their Nix config file.

  I guess this is a nice way to teach people where the Nix config file lives…? But I would consider Nix basically unusable without nix search, so this is a very strange choice.

  But even after you do this, nix search no longer works, so… yeah okay.

• Flakes, a controversial new feature that I don’t fully understand but am personally very bullish about, has taken over much of the UI.

• Content-addressable store paths. I’m interested to see how these work and what this is, because it’s not obvious to me how you implement this seamlessly on top of Nix’s existing situation.

• The manual has been subjected to an automated conversion to mdbook that makes it completely unsearchable and much more difficult to read, conspiring to make Nix harder to learn than ever before. But it also makes it much easier for people to contribute changes to the documentation, so perhaps given time the positive slope may be able to overcome the negative y-intercept that this change inflicted.

  Fortunately, the Nixpkgs manual and NixOS manual have not undergone the same migration… at least, not yet.

• Single-user installs on macOS are no longer supported.

  This is very sad for me, as I never would have tried Nix in the first place if I had to do a multi-user install, and I imagine that there are many people who feel similarly to me.

  I understand that it is possible to hack the install script to do a single-user install on macOS, but it was already pretty difficult to install in the first place, and this makes it basically impossible to market Nix to macOS users who are not already “sold” on Nix.

The release of 2.4 feels bittersweet for me. On the one hand, I’m excited about the UX improvements in the nix command line – mostly nix profile, I guess, because nix-env has been the source of most of my Nix UX complaints. On the other hand, it feels like a huge step backward for “macOS users who want to learn Nix by themselves.” Which described me, earlier this year. If my first introduction to Nix had been Nix 2.4, this blog series never would have existed, and I’d still be stuck with Homebrew.

But anyway. First things first: let’s try to restore all of the functionality that we had before we upgraded to Nix 2.4. That means:

nix search
nix repl
nix path-info
nix show-derivation

nix repl seems to work out of the box just fine, once I enable the nix binary again. And so does nix show-derivation. nix search and nix path-info both yell about flakes, though, and their API seems to have changed.

So I guess I have to learn something about flakes. I opted into it in my config:

$ grep 'experimental' ~/.config/nix/nix.conf
experimental-features = nix-command flakes

And now when I run nix search:

$ nix search git
error: cannot find flake 'flake:git' in the flake registries

Hmm. Nothing about how to fix that, so I suppose I’ll have to start studying.

The release announcement links to this blog post as an introduction to flakes, so I suppose I’ll start there.

But first: I have heard of flakes over the course of my regular Nix usage this year. I am not coming in blind; I am vaguely aware of the feature already. My shaky half-understanding going into all of this is:

• it’s a new way to download Nix expressions from the internet that replaces “channels”
• it’s a way to import Nix expressions from elsewhere on the internet, instead of just NIX_PATH
• it makes it easy to pin/lock dependencies without resorting to a third-party tool like niv
• it makes it easy to install software that is not packaged in Nixpkgs, so you can easily distribute your own software without getting it merged into the Nixpkgs monorepo
• it’s ergonomically very unpleasant to write flake.nix files, and a lot of people are upset about that

I am hopeful that flakes will make it possible to break up the Nixpkgs monolith, so that you can depend on, like, helper functions or the build infrastructure without also having to download every package description for every Haskell package ever published or whatever.

But I don’t actually know any of those things. So let’s dive in and try to find out more.

Nix Flakes, Part 1: An introduction and tutorial

A flake is simply a source tree (such as a Git repository) containing a file named flake.nix that provides a standardized interface to Nix artifacts such as packages or NixOS modules. Flakes can have dependencies on other flakes, with a “lock file” pinning those dependencies to exact revisions to ensure reproducible evaluation.

Okay, simple enough. Nothing about the “flake registry” yet… but it’s going into a simple example of a project called dwarffs. Let’s take a look at its flake.nix file:

{
  description = "A filesystem that fetches DWARF debug info from the Internet on demand";

  inputs.nixpkgs.follows = "nix/nixpkgs";

  outputs = { self, nix, nixpkgs }:

    let
      supportedSystems = [ "x86_64-linux" "i686-linux" "aarch64-linux" ];
      forAllSystems = f: nixpkgs.lib.genAttrs supportedSystems (system: f system);
      version = "0.1.${nixpkgs.lib.substring 0 8 self.lastModifiedDate}.${self.shortRev or "dirty"}";
    in

    {

      overlay = final: prev: {

        dwarffs = with final; let nix = final.nix; in stdenv.mkDerivation {
          name = "dwarffs-${version}";

          buildInputs = [ fuse nix nlohmann_json boost ];

          NIX_CFLAGS_COMPILE = "-I ${nix.dev}/include/nix -include ${nix.dev}/include/nix/config.h -D_FILE_OFFSET_BITS=64 -DVERSION=\"${version}\"";

          src = self;

          installPhase =
            ''
              mkdir -p $out/bin $out/lib/systemd/system

              cp dwarffs $out/bin/
              ln -s dwarffs $out/bin/mount.fuse.dwarffs

              cp ${./run-dwarffs.mount} $out/lib/systemd/system/run-dwarffs.mount
              cp ${./run-dwarffs.automount} $out/lib/systemd/system/run-dwarffs.automount
            '';
        };

      };

      defaultPackage = forAllSystems (system: (import nixpkgs {
        inherit system;
        overlays = [ self.overlay nix.overlay ];
      }).dwarffs);

      checks = forAllSystems (system: {
        build = self.defaultPackage.${system};

        test =
          with import (nixpkgs + "/nixos/lib/testing-python.nix") {
            inherit system;
          };

          makeTest {
            nodes = {
              client = { ... }: {
                imports = [ self.nixosModules.dwarffs ];
                nixpkgs.overlays = [ nix.overlay ];
              };
            };

            testScript =
              ''
                start_all()
                client.wait_for_unit("multi-user.target")
                client.succeed("dwarffs --version")
                client.succeed("cat /run/dwarffs/README")
                client.succeed("[ -e /run/dwarffs/.build-id/00 ]")
              '';
          };
      });

      nixosModules.dwarffs =
        { pkgs, ... }:
        {
          nixpkgs.overlays = [ self.overlay ];

          systemd.packages = [ pkgs.dwarffs ];

          system.fsPackages = [ pkgs.dwarffs ];

          systemd.units."run-dwarffs.automount".wantedBy = [ "multi-user.target" ];

          environment.variables.NIX_DEBUG_INFO_DIRS = [ "/run/dwarffs" ];

          systemd.tmpfiles.rules = [ "d /var/cache/dwarffs 0755 dwarffs dwarffs 7d" ];

          users.users.dwarffs =
            { description = "Debug symbols file system daemon user";
              group = "dwarffs";
              isSystemUser = true;
            };

          users.groups.dwarffs = {};
        };

    };
}

Wow! That’s a lot. I don’t… I don’t think this a very good starting point. I can sort of squint and ignore most of it:

{
  description = "A filesystem that fetches DWARF debug info from the Internet on demand";

  inputs.nixpkgs.follows = "nix/nixpkgs";

  outputs = { self, nix, nixpkgs }:
    let
      supportedSystems = [ "x86_64-linux" "i686-linux" "aarch64-linux" ];
      forAllSystems = f: nixpkgs.lib.genAttrs supportedSystems (system: f system);
      version = "0.1.${nixpkgs.lib.substring 0 8 self.lastModifiedDate}.${self.shortRev or "dirty"}";
    in
    {
      overlay = final: prev: { dwarffs = ...; };
      defaultPackage = forAllSystems (system: (import nixpkgs {
        inherit system;
        overlays = [ self.overlay nix.overlay ];
      }).dwarffs);
      checks = forAllSystems (system: { ... });
      nixosModules.dwarffs = ...;
    };
}

So it has description, inputs, and outputs. Remember that that input line is just shorthand for:

inputs = { nixpkgs = { follows = "nix/nixpkgs"; }; };

We can also see all the boilerplate that I have heard complaints about – the forAllSystems stuff. So in order to reasonably define a flake, you sort of have to include Nixpkgs as one of your dependencies, because the helpers you need in order to define flakes live in Nixpkgs. Even though they could just live in a separate flake-helpers repo or something.

Of course, there is nothing stopping us from making our own flake-helpers repo and using it instead, as we’re no longer restricted to treating Nixpkgs like the only source of Nix expressions.

There is also a flake.lock file:

{
  "nodes": {
    "lowdown-src": {
      "flake": false,
      "locked": {
        "lastModified": 1598695561,
        "narHash": "sha256-gyH/5j+h/nWw0W8AcR2WKvNBUsiQ7QuxqSJNXAwV+8E=",
        "owner": "kristapsdz",
        "repo": "lowdown",
        "rev": "1705b4a26fbf065d9574dce47a94e8c7c79e052f",
        "type": "github"
      },
      "original": {
        "owner": "kristapsdz",
        "repo": "lowdown",
        "type": "github"
      }
    },
    "nix": {
      "inputs": {
        "lowdown-src": "lowdown-src",
        "nixpkgs": "nixpkgs"
      },
      "locked": {
        "lastModified": 1610390819,
        "narHash": "sha256-XE2ajw0BOtmEksRfeY5+CKN8PNlZnsMGL73sxykdQdM=",
        "owner": "NixOS",
        "repo": "nix",
        "rev": "6254b1f5d298ff73127d7b0f0da48f142bdc753c",
        "type": "github"
      },
      "original": {
        "id": "nix",
        "type": "indirect"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1602702596,
        "narHash": "sha256-fqJ4UgOb4ZUnCDIapDb4gCrtAah5Rnr2/At3IzMitig=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "ad0d20345219790533ebe06571f82ed6b034db31",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "ref": "nixos-20.09-small",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "nix": "nix",
        "nixpkgs": [
          "nix",
          "nixpkgs"
        ]
      }
    }
  },
  "root": "root",
  "version": 7
}

Which appears to be JSON.

I’m not really sure what lowdown-src is, or where that came from? There’s no mention of that in the flake.nix file. But it seems that nixpkgs depends on nix, which depends on lowdown-src. Huh. And indeed, we can see that if we look at the flake.nix file for Nix itself:

{
  description = "The purely functional package manager";

  inputs.nixpkgs.url = "nixpkgs/nixos-21.05-small";
  inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };

  outputs = ...;
}

Interesting! Lowdown seems to be a markdown parser, presumably one that Nix uses to generate its new manual/man pages.

That’s weird, right?

Like, it sounds like a build-time dependency; it sounds like an implementation detail of Nix. It does not sound like something that anyone depending on Nixpkgs should pull in.

And maybe they don’t; I don’t really have any sense of what this means yet. But it seems weird.

Oh, I’m supposed to be using nix flake commands to inspect all of this. I’m not supposed to be looking at the raw files. Let’s give that a shot:

$ nix flake metadata github:edolstra/dwarffs
Resolved URL:  github:edolstra/dwarffs
Locked URL:    github:edolstra/dwarffs/f691e2c991e75edb22836f1dbe632c40324215c5
Description:   A filesystem that fetches DWARF debug info from the Internet on demand
Path:          /nix/store/769s05vjydmc2lcf6b02az28wsa9ixh1-source
Revision:      f691e2c991e75edb22836f1dbe632c40324215c5
Last modified: 2021-01-21 06:41:26
Inputs:
├───nix: github:NixOS/nix/6254b1f5d298ff73127d7b0f0da48f142bdc753c
│   ├───lowdown-src: github:kristapsdz/lowdown/1705b4a26fbf065d9574dce47a94e8c7c79e052f
│   └───nixpkgs: github:NixOS/nixpkgs/ad0d20345219790533ebe06571f82ed6b034db31
└───nixpkgs follows input 'nix/nixpkgs'

We only have an explicit dependency on Nixpkgs, but nix is listed here. So I guess that this line:

inputs.nixpkgs.follows = "nix/nixpkgs";

Means “I depend on whatever version of Nixpkgs that Nix itself depends on?” Okay. So Nixpkgs doesn’t depend on Nix; it’s the other way around. So lowdown-src is probably not infecting any other projects. Got it got it. Still getting my bearings here.

Let’s see if I’m right:

$ nix flake metadata github:nixos/nixpkgs
Resolved URL:  github:nixos/nixpkgs
Locked URL:    github:nixos/nixpkgs/e98afa97d3554e00661e436ba5ab5938d40bc761
Description:   A collection of packages for the Nix package manager
Path:          /nix/store/xh4gyysicnjk9kbsxcxykmk85chqc12f-source
Revision:      e98afa97d3554e00661e436ba5ab5938d40bc761
Last modified: 2021-12-05 12:16:45
Inputs:

Okay, so no inputs. Good good. The world is in order.

You can’t see it in this blog post, but these commands come with nice little animated download progress spinner thingies.

That command took a really long time to run, even after the download completed. But subsequent invocations were basically instantaneous. So… huh. I don’t know how to explain that.

Anyway. Then we learn:

$ nix flake show github:edolstra/dwarffs
github:edolstra/dwarffs/f691e2c991e75edb22836f1dbe632c40324215c5
├───checks
│   ├───aarch64-linux
│   │   ├───build: derivation 'dwarffs-0.1.20210121.f691e2c'
│   │   └───test: derivation 'vm-test-run-unnamed'
│   ├───i686-linux
│   │   ├───build: derivation 'dwarffs-0.1.20210121.f691e2c'
│   │   └───test: derivation 'vm-test-run-unnamed'
│   └───x86_64-linux
│       ├───build: derivation 'dwarffs-0.1.20210121.f691e2c'
│       └───test: derivation 'vm-test-run-unnamed'
├───defaultPackage
│   ├───aarch64-linux: package 'dwarffs-0.1.20210121.f691e2c'
│   ├───i686-linux: package 'dwarffs-0.1.20210121.f691e2c'
│   └───x86_64-linux: package 'dwarffs-0.1.20210121.f691e2c'
├───nixosModules
│   └───dwarffs: NixOS module
└───overlay: Nixpkgs overlay

Okay. That’s about what I hacked the flake.nix file into showing. Interesting that some of these things are derivations (under the checks attribute), but some of these things are packages. I have no idea what “package” means in the Nix world, but hopefully I will find out.

While a flake can have arbitrary outputs, some of them, if they exist, have a special meaning to certain Nix commands and therefore must have a specific type. For example, the output defaultPackage.<system> must be a derivation; it’s what nix build and nix shell will build by default unless you specify another output.

Okay, so… why is it called a package, then? If it’s just a derivation? That’s… this is confusing.

I look back at the original source, which defines these as:

defaultPackage = forAllSystems (system: (import nixpkgs {
  inherit system;
  overlays = [ self.overlay nix.overlay ];
}).dwarffs);

So… this is a little bit of strange indirection, but basically we’re adding nixpkgs.dwarffs using an overlay, and then looking up the result of applying that overlay.

The overlay just defines it as a normal derivation:

overlay = final: prev: {

  dwarffs = with final; let nix = final.nix; in stdenv.mkDerivation {
    name = "dwarffs-${version}";
    ...
  };
};

So no idea why it reports it as a package, or what makes a package different from a derivation. Very confusing.

Maybe we’ll get to it if we keep reading.

The nix CLI allows you to specify another output through a syntax reminiscent of URL fragments:

$ nix build github:edolstra/dwarffs#checks.aarch64-linux.build

Okay. Do we have to type out the platform name, though? I guess I don’t really know when I would ever do this, so maybe it isn’t cumbersome in practice. I don’t really know how to reason about “multi-output flakes” when we already have “multi-output derivations” or how they might differ. I guess that something like nixpkgs probably doesn’t have a defaultPackage thing. That makes sense.

By the way, the standard checks output specifies a set of derivations to be built by a continuous integration system such as Hydra. Because flake evaluation is hermetic and the lock file locks all dependencies, it’s guaranteed that the nix build command above will evaluate to the same result as the one in the CI system.

I was wondering about checks. That’s pretty neat? I like that this is a first-class thing.

The Flake Registry

Okay, here we go.

Flake locations are specified using a URL-like syntax such as github:edolstra/dwarffs or git+https://github.com/NixOS/patchelf. But because such URLs would be rather verbose if you had to type them all the time on the command line, there also is a flake registry that maps symbolic identifiers such as nixpkgs to actual locations like https://github.com/NixOS/nixpkgs. So the following are (by default) equivalent:

$ nix shell nixpkgs#cowsay --command cowsay Hi!
$ nix shell github:NixOS/nixpkgs#cowsay --command cowsay Hi!

Alright. This section introduces us to the nix registry subcommand, and I learn that I can try nix registry list, because I am curious if I have a nixpkgs entry “by default” after upgrading.

$ nix registry list
global flake:blender-bin github:edolstra/nix-warez?dir=blender
global flake:dwarffs github:edolstra/dwarffs
global flake:hydra github:NixOS/hydra
global flake:mach-nix github:DavHau/mach-nix
global flake:nimble github:nix-community/flake-nimble
global flake:nix github:NixOS/nix
global flake:nixops github:NixOS/nixops
global flake:nixos-hardware github:NixOS/nixos-hardware
global flake:nixos-homepage github:NixOS/nixos-homepage/flake
global flake:nur github:nix-community/NUR
global flake:nixpkgs github:NixOS/nixpkgs
global flake:templates github:NixOS/templates
global flake:patchelf github:NixOS/patchelf
global flake:nix-serve github:edolstra/nix-serve
global flake:nickel github:tweag/nickel

Wow, what? That’s a lot! blender-bin?? What… what is all of this stuff? What is nix-warez? It is obviously a GitHub repository, and it contains… Baldur’s Gate? As two of the five things here. Weird weird weird.

Okay, so we have a pretty large and robust default flake registry.

I thought that my search was failing because I needed to do something to initialize my package registry. But apparently not?

$ nix search git
error: cannot find flake 'flake:git' in the flake registries

So apparently I need to search nixpkgs now…?

$ nix search nixpkgs

I tried running that by itself but it’s kind of just… hanging. Oh, and then it printed out every package in the universe. Okay.

$ nix search nixpkgs git

So initially I thought that it felt slower than the old search, and then I realized that it was respecting the fact that I have NIX_PAGER= set, so I could see the speed that it printed results. When I pipe into less, I can see that it’s printing output faster than I can scroll through it. And all without needing to update a cache? That’s pretty great.

Okay. I realize that nix search git is a little more intuitive than nix search nixpkgs git. But I personally prefer the explicitness of this; I never really understood what nix search was actually searching, with Nix 2.3. Just nixpkgs? Everything on my NIX_PATH? I never knew. But now it’s clear.

Let’s try searching another “flake:”

$ nix search nur git
error: cannot write modified lock file of flake 'flake:nur' (use '--no-write-lock-file' to ignore)

What? Why? Other flakes seem to work:

$ nix search nixops git
error: no results for the given search term(s)!

$ nix search nimble git
error: no results for the given search term(s)!

What’s different about nur?

$ nix search nur git --show-trace
error: cannot write modified lock file of flake 'flake:nur' (use '--no-write-lock-file' to ignore)

       … while updating the lock file of flake 'github:nix-community/NUR/a47ee757c61690b442324d60812c3269fecb9916'

That doesn’t really tell me anything.

What’s going on here? Where do these lockfiles live? Where does the registry live?

I don’t see anything likely in my home directory. Nor are there any new subdirectories in /nix/var/nix.

$ sqlite3 /nix/var/nix/db/db.sqlite .schema

CREATE TABLE ValidPaths (
    id               integer primary key autoincrement not null,
    path             text unique not null,
    hash             text not null,
    registrationTime integer not null,
    deriver          text,
    narSize          integer,
    ultimate         integer, -- null implies "false"
    sigs             text, -- space-separated
    ca               text -- if not null, an assertion that the path is content-addressed; see ValidPathInfo
);
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE Refs (
    referrer  integer not null,
    reference integer not null,
    primary key (referrer, reference),
    foreign key (referrer) references ValidPaths(id) on delete cascade,
    foreign key (reference) references ValidPaths(id) on delete restrict
);
CREATE INDEX IndexReferrer on Refs(referrer);
CREATE INDEX IndexReference on Refs(reference);
CREATE TRIGGER DeleteSelfRefs before delete on ValidPaths
  begin
    delete from Refs where referrer = old.id and reference = old.id;
  end;
CREATE TABLE DerivationOutputs (
    drv  integer not null,
    id   text not null, -- symbolic output id, usually "out"
    path text not null,
    primary key (drv, id),
    foreign key (drv) references ValidPaths(id) on delete cascade
);
CREATE INDEX IndexDerivationOutputs on DerivationOutputs(path);

Some new stuff in there for content-addressed paths, but nothing about flakes. Which is expected: the store shouldn’t really know anything about them.

But I’m sort of running out of places to look. I know Nix stores stuff in ~/.cache, but nothing… important, right?

$ ls ~/.cache/nix
binary-cache-v5.sqlite
binary-cache-v5.sqlite-journal
binary-cache-v6.sqlite
binary-cache-v6.sqlite-journal
eval-cache-v2/
fetcher-cache-v1.sqlite
fetcher-cache-v1.sqlite-journal
flake-registry.json@ -> /nix/store/qi48i5dkbffrdxc15rhdz1x4mpm6q6n3-flake-registry.json
gitv2/
package-search.json
tarballs/

Aha. flake-registry.json certainly sounds promising.

$ cat /nix/store/qi48i5dkbffrdxc15rhdz1x4mpm6q6n3-flake-registry.json

{
  "flakes": [
    {
      "from": {
        "id": "nur",
        "type": "indirect"
      },
      "to": {
        "owner": "nix-community",
        "repo": "NUR",
        "type": "github"
      }
    },
    {
      "from": {
        "id": "nixpkgs",
        "type": "indirect"
      },
      "to": {
        "owner": "NixOS",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    ...
  ],
  "version": 2
}

So nur doesn’t look any different from nixpkgs. Hmm. This is all very strange. Perhaps it will explain where the lock files live, if I keep reading.

Writing your first flake

The next thing the blog post describes is writing my own flake.nix file. It looks… easy?

Let’s try it. I wrote a little tool called sd that is 1) useful and 2) not in Nixpkgs. Maybe I can make a flake out of it, so that people could install it with Nix, if they wanted to.

It’s literally just a shell script, so hopefully it won’t be too difficult to package.

$ cd ~/src/sd

$ nix flake init

$ cat flake.nix

{
  description = "A very basic flake";

  outputs = { self, nixpkgs }: {

    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;

    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;

  };
}

Okay. Strange. I’m on x86_64-darwin, first of all. Second of all… where the heck does nixpkgs come from? There are no inputs here. Does it… implicitly depend on nixpkgs?

$ nix flake metadata
warning: Git tree '/Users/ian/src/sd' is dirty
warning: creating lock file '/Users/ian/src/sd/flake.lock'
warning: Git tree '/Users/ian/src/sd' is dirty
Resolved URL:  git+file:///Users/ian/src/sd
Locked URL:    git+file:///Users/ian/src/sd
Description:   A very basic flake
Path:          /nix/store/ikqp3gx8c796gpc3s8ggsjd5xzz1sk84-source
Last modified: 2021-12-05 09:05:48
Inputs:
└───nixpkgs: github:NixOS/nixpkgs/e1e7c171a258fc15424b22a1566d9af530784aa7

Apparently… yes? This behavior has not been described yet, though. It certainly feels weird to me. I would expect it to be just explicitly listed in the new flake template, instead of defaulted by whatever part of Nix is evaluating this. Hmm.

The outputs attribute is the heart of the flake: it’s a function that produces an attribute set. The function arguments are the flakes specified in inputs.

The self argument denotes this flake. Its primarily useful for referring to the source of the flake (as in src = self;) or to other outputs (e.g. self.defaultPackage.x86_64-linux).

So that’s interesting. src = self; – so a flake is like… a first class value, that is an attribute set but also a path? Or something? I don’t really know how to reason about this. It goes on to say:

Every flake has some metadata, such as self.lastModifiedDate, which is used to generate a version string like hello-20191015.

So flakes are like… they’re like derivations, in that they are magic things that sort of look like they’re just attribute sets but secretly they’re something slightly different? And there are no types anywhere, so we just have to guess and check until we gain some kind of intuition for how they behave.

But how do I observe a flake? Let’s try the repl.

$ nix repl
nix-repl> import ./.
error: opening file '/Users/ian/src/sd/default.nix': No such file or directory

Okay, so not quite. And if I just:

nix-repl> import ./flake.nix
{ description = "A very basic flake"; outputs = «lambda @ /Users/ian/src/sd/flake.nix:4:13»; }

Then I just import it as a raw attribute set – aka, as the thing that it looks like. It would be weird if it didn’t import it as a regular file – how does it know that it’s a flake?

But how do any other commands know? Are flakes built into the language? Or… if not, what is adding the magic lastModifiedDate metadata thing, and how can I observe it?

Presumably by using the nix flake subcommands. But they aren’t showing me the value of the flake. They’re just showing me these pretty human-readable things. I want to see the raw thing, somehow.

Here are my options, from nix flake --help:

· nix flake archive - copy a flake and all its inputs to a store
· nix flake check - check whether the flake evaluates and run its tests
· nix flake clone - clone flake repository
· nix flake info - show flake metadata
· nix flake init - create a flake in the current directory from a template
· nix flake lock - create missing lock file entries
· nix flake metadata - show flake metadata
· nix flake new - create a flake in the specified directory from a template
· nix flake prefetch - download the source tree denoted by a flake
  reference into the Nix store
· nix flake show - show the outputs provided by a flake
· nix flake update - update flake lock file

Hmm.

$ nix flake info
warning: 'nix flake info' is a deprecated alias for 'nix flake metadata'
warning: Git tree '/Users/ian/src/sd' is dirty
Resolved URL:  git+file:///Users/ian/src/sd
Locked URL:    git+file:///Users/ian/src/sd
Description:   A very basic flake
Path:          /nix/store/ikqp3gx8c796gpc3s8ggsjd5xzz1sk84-source
Last modified: 2021-12-05 09:05:48
Inputs:
└───nixpkgs: github:NixOS/nixpkgs/e1e7c171a258fc15424b22a1566d9af530784aa7

Aw. Worth a shot.

I can try trace? I’ve never actually used it before, but I think it’s my only hope of trying to understand what the heck these values actually are.

My first attempt failed:

$ cat flake.nix

{
  description = "A very basic flake";

  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;

    defaultPackage.x86_64-linux = builtins.trace self self.packages.x86_64-linux.hello;
  };
}

$ nix flake show
warning: Git tree '/Users/ian/src/sd' is dirty
git+file:///Users/ian/src/sd
├───defaultPackage
error: invalid value

I have to give it a string, it seems. But how do I create a string that represents it? I just toString it, and I get a path:

$ nix flake show
warning: Git tree '/Users/ian/src/sd' is dirty
git+file:///Users/ian/src/sd
├───defaultPackage
trace: /nix/store/vzgc0bww6i611sqv1j54c35nf61rii5l-source
│   └───x86_64-linux: package 'hello-2.10'
└───packages
    └───x86_64-linux
        └───hello: package 'hello-2.10'

But it seems to be caching the evaluation result somewhere, which means that it only traces on the first invocation. If I run it again, the trace does not run:

$ nix flake show
warning: Git tree '/Users/ian/src/sd' is dirty
git+file:///Users/ian/src/sd
├───defaultPackage
│   └───x86_64-linux: package 'hello-2.10'
└───packages
    └───x86_64-linux
        └───hello: package 'hello-2.10'

So… a fun debugging experience. But as long as I’m changing the value I’m tracing, it isn’t too confusing. I learn the following things:

isAttrs: true
isPath: false

Well that’s a relief, I guess. But why does it toString as a path? It doesn’t have a __toString. It doesn’t have a drvPath, which I know is also special to toString. It doesn’t have a type attribute.

I try:

defaultPackage.x86_64-linux = builtins.trace "${nixpkgs.lib.generators.toPretty {multiline = true;} self}" self.packages.x86_64-linux.hello;

But I am met with:

$ nix flake show
warning: Git tree '/Users/ian/src/sd' is dirty
git+file:///Users/ian/src/sd
├───defaultPackage
error: infinite recursion encountered

       at /nix/store/rjmn9kzjnz0n89411hr2yqxrdhyqmwjv-source/flake.nix:7:5:

            6|
            7|     defaultPackage.x86_64-linux = builtins.trace "${nixpkgs.lib.generators.toPretty {multiline = true;} self}" self.packages.x86_64-linux.hello;
             |     ^
            8|   };
(use '--show-trace' to show detailed location information)

Sigh. Great looking error message, though!

I am still mystified by how difficult it is to just observe values in Nix. It took me weeks of reading documentation to figure out how to just print out a derivation. And now there’s a new magic type of thing that seems even more wily. Maybe in a few more weeks, I will understand how to get hold of one…

Anyway. I looked through the Nix source, but like who knows. It’s its own bespoke thing, but it can coerce to an attribute set, I guess? And also has a custom toString. I don’t know what I’m doing here. It seems that there is a type of thing called a tree, and that flakes are also trees, and maybe there are ways to fetch “trees” built into Nix. I don’t remember anything about this when I was reading the manual, so presumably this is a new 2.4 concept.

I try to search the manual for “trees” but, obviously, it thinks I want to search for “tree” and gosh there is nothing worse than a client-side JavaScript full-text search when all I want to do is ⌘F. Remember when you could just ⌘F through the manual? I remember. I am still salty about this.

Anyway. Let’s assume, for now, that it’s magic, and let our minds crystallize around the fact that flakes are difficult and mysterious, until we are afraid to try to understand them further.

What was I trying to do?

Oh, package sd as a flake. Okay. This isn’t really a prerequisite. I’ll just… pretend that self is a path but also remember that it isn’t. No problem.

The template that it created for me seems kind of… insufficient. I liked what dwarffs did; that makes sense to me. I want to publish an overlay, so that you can install it as nixpkgs.sd or whatever. I assume that’s what the overlay thing is for. So I’m going to copy that instead, and then modify it to suit my purposes:

$ cat flake.nix

{
  description = "A simple command dispatch tool";

  outputs = { self, nixpkgs }:
    let
      supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];
      forAllSystems = f: nixpkgs.lib.genAttrs supportedSystems (system: f system);
      version = "0.1.${nixpkgs.lib.substring 0 8 self.lastModifiedDate}.${self.shortRev or "dirty"}";
    in
    {
      overlay = final: prev: {
        sd = with final; stdenv.mkDerivation {
          pname = "sd";
          inherit version;

          src = self;

          installPhase =
            ''
              install -D sd -m 0555 "$out/bin/sd"
              install -D _sd -m 0444 "$out/share/zsh/site-functions/_sd"
            '';
        };
      };

      defaultPackage = forAllSystems (system: (import nixpkgs {
        inherit system;
        overlays = [ self.overlay ];
      }).sd);
    };
}

I had heard people of complaining about flake boilerplate before, and now I can see what they were talking about.

This is also something that I don’t already have a default.nix for. If I did have a default.nix, it would presumably start with something like:

with import <nixpkgs> {}; mkDerivation { ... }

How would I translate that into a flake? I guess I would make a function that took nixpkgs and returned a derivation, then call that from my flake.nix and my default.nix? Do I still need default.nix files, now that there are flakes? I don’t really understand.

Anyway, let’s see if this works.

Can I, umm… build it?

$ nix build

$ ll result/bin
total 16
-r-xr-xr-x  1 ian   5.4K Dec 31  1969 sd*

Okay! It had to download 700mb of something for some reason first, but it did work! Eventually.

So two of those lines are lines that I wrote. And the other 29 are… boilerplate that I copied. So that’s a little… bad. That’s bad.

But! With the power flakes, we are able to improve our lot. Since it’s now possible to write helper functions that don’t live in Nixpkgs, there are helper functions for us that don’t live in Nixpkgs.

One of them is called flake-utils, and it is a collection of helper functions of exactly the sort I am after. So let’s try it?

We have to include it as an input, and then it seems like I just write:

{
  description = "A simple command dispatch tool";

  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.simpleFlake {
      inherit self nixpkgs;
      name = "sd";
      overlay = final: prev: {
        sd = with final; stdenv.mkDerivation {
          pname = "sd";
          version = "0.1.${nixpkgs.lib.substring 0 8 self.lastModifiedDate}.${self.shortRev or "dirty"}";

          src = self;

          installPhase =
            ''
              install -D sd -m 0555 "$out/bin/sd"
              install -D _sd -m 0444 "$out/share/zsh/site-functions/_sd"
            '';
        };
      };
    };
}

Hmm, but no. That does not work.

$ nix build
warning: Git tree '/Users/ian/src/sd' is dirty
error: flake 'git+file:///Users/ian/src/sd' does not provide attribute 'packages.x86_64-darwin.defaultPackage.x86_64-darwin', 'legacyPackages.x86_64-darwin.defaultPackage.x86_64-darwin' or 'defaultPackage.x86_64-darwin'

Because although there is flake-utils.lib.defaultSystems, those are not… the default systems. When you use simpleFlake. So we have to write:

outputs = { self, nixpkgs, flake-utils }:
  flake-utils.lib.simpleFlake {
    inherit self nixpkgs;
    systems = flake-utils.lib.defaultSystems;
    ...
  };

But that still doesn’t work. It fails with the same error. What?

$ nix flake show

warning: Git tree '/Users/ian/src/sd' is dirty
git+file:///Users/ian/src/sd
└───legacyPackages
warning:     ├───aarch64-darwin: omitted (use '--legacy' to show)
warning:     ├───aarch64-linux: omitted (use '--legacy' to show)
warning:     ├───i686-linux: omitted (use '--legacy' to show)
warning:     ├───x86_64-darwin: omitted (use '--legacy' to show)
warning:     └───x86_64-linux: omitted (use '--legacy' to show)

Wha? So… it doesn’t work? It generates something that is actually incompatible with flakes?

$ nix flake show --legacy
warning: Git tree '/Users/ian/src/sd' is dirty
git+file:///Users/ian/src/sd
└───legacyPackages
    ├───aarch64-darwin: package 'sd-0.1.20211205.dirty'
    ├───aarch64-linux: package 'sd-0.1.20211205.dirty'
    ├───i686-linux: package 'sd-0.1.20211205.dirty'
    ├───x86_64-darwin: package 'sd-0.1.20211205.dirty'
    └───x86_64-linux: package 'sd-0.1.20211205.dirty'

Okay, so whatever this is… is useless to me. Let’s try something else.

I find flake-utils-plus, but it is… the documentation is a single 123-line long Nix file. That is obviously too complicated for what I am trying to do.

The library is meant to be easy to understand and use.

Sigh.

Even just reading the documentation requires learning some new terminology – it uses the word “channels” here to mean something that I assume is incompatible with “Nix channels,” because… what do channels have to do with flakes? Don’t you replace channels with flakes? You can use flakes to… export channels? This makes no sense to me yet.

Sigh. Okay. So I have no idea what the right way to write flakes is, then.

Copy 30 lines of boilerplate around? Write out that bizarre overlay-application-lookup thing manually every time? This seems awful, and I am left with a very sour taste in my mouth.

I guess the answer is “write my own helper library that actually works the way I want it to,” but… come on.

I settle for this disappointing compromise:

{
  description = "A simple command dispatch tool";

  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }: {
    overlay = final: prev: {
      sd = with final; stdenv.mkDerivation {
        pname = "sd";
        version = "0.1.${nixpkgs.lib.substring 0 8 self.lastModifiedDate}.${self.shortRev or "dirty"}";

        src = self;

        installPhase =
          ''
            install -D sd -m 0555 "$out/bin/sd"
            install -D _sd -m 0444 "$out/share/zsh/site-functions/_sd"
          '';
      };
    };
  } //
  flake-utils.lib.eachDefaultSystem (system: {
    defaultPackage = (import nixpkgs {
      inherit system;
      overlays = [ self.overlay ];
    }).sd;
  });
}

It’s better than not using flake-utils at all, but I’m sad that I still have to write that (import nixpkgs { overlays = ...; }).sd nonsense.

Alllllright.

Did I learn flakes?

I got to the end of that blog post. I wrote my first flake. It was… not very pleasant, but I think in order to make it better I will have to make my own helper library, and I don’t want to do that right now.

I’m not really sure what to do with my flake now that I’ve written it, since I still don’t know how to install a flake, or how to import it from a Nix expression, or how to actually apply my overlay to my copy of Nixpkgs. So I have a flake.nix file, and nothing more.

I did not learn much about how flakes actually work, but there are still two blog posts in this series, and then the man pages, so there should be plenty of time to fix that.

• Why is lowdown a dependency of the Nix flake?
• What is the difference between a “derivation” and a “package”?
• Why does nix search nur git fail?
• Where is the flake registry written down? Is it a per-user thing, or a global thing…?
• Where are flake lockfiles stored?
• Why is nixpkgs an implicit flake dependency? Is the URL of this GitHub repo hardcoded in Nix…?
• What is a flake in the Nix expression language? What type of value is it? How do I print one?
• How do I translate an existing default.nix into a flake?
• Why does Nix randomly decide to download things when I run nix search? What is it downloading? How can I make it stop? Please, please don’t tell me it’s “automatically” updating my flakes.