When I was learning Nix for the first time, I never found myself frustrated with it.

My brain was in learning mode: everything was strange and new and exciting. I was more interested in what Nix was than in how to actually use it; I was more interested in how it worked than what I could do with it.

And more importantly: I expected it to be difficult to use. Nix has a reputation, after all, and it didn’t really bother me that it kept plunging me into a bash shell, or that it didn’t have a way to upgrade packages, or that I had to type crazy-looking commands all the time. I knew what I was signing up for.

But many months have passed since my first dance with Nix, and in that time our relationship has evolved. Nix is no longer an interesting curio; it is now a very important tool for me. My mind left learning mode, and entered the working mode.

And when Nix 2.4 came out, I wasn’t expecting it to be difficult to use. I was looking forward to the new UI; I was looking forward to recommending it to people without having to attach a hundred asterisks. I was very surprised when I saw that the profile UI was not an answer to nix-env, and that commands would sometimes – randomly! – add one full minute of latency before they started doing anything.

So I reverted to Nix 2.3. Not because I’m giving up on Nix, or done trying to learn about flakes. But because it will be much easier to study Nix 2.4 from arms-length. I need to treat Nix 2.4 as another strange artifact, and I need to learn how it works without it getting in the way of my useless side projects real work.

So let’s get started. Again.

I had the chance to speak with someone very knowledgeable about Nix recently, and I was able to ask a question that has been on my mind: can I do everything that I could do before, if I go all in on flakes? If I abandon backwards-compatibility, if I delete my channels, if I lean into the experimental features?

And the answer was a resounding no.

At least, as of today: it is not possible to do everything with flakes. In order to use Nix in all of the ways that I use it, you have to use both channel-based Nix and flake-based Nix at the same time. And there is no built-in way to unify the two – to make a channel that tracks a flake, or to make a flake that tracks a channel.

So if you install Nix 2.4 (or 2.5, which has emerged since my last post), you have to use flake-based Nix, because there is no way to make nix search work without flakes. And you also have to use channel-based Nix, because the new Nix has no equivalent of nix-shell -i, and only a limited version of nix-shell -p.

So that’s sort of upsetting, and it makes it pretty awkward to recommend Nix right now. “You should install Nix, but make sure you install the Nix from two versions ago, because, well, there’s a new way to do everything, but not everything, so if you install the latest version you’ll have to understand these two mutually incompatible models of how Nix expressions work, and Nix was hard enough to understand already.” It’s not… a great pitch. But hopefully this intermediate state won’t last too long.

That was my high-level takeaway. Let’s get a little more specific.

I currently use Nix in four different ways:

  1. shell.nix files
  2. installing tools I use interactively (like git or rg)
  3. nix-shell -p (to try stuff out)
  4. nix-shell -i (writing portable, self-contained scripts)

Flakes can currently do… some of these things. Let’s go through the list:

shell.nix files. In theory, this is the big one, and the use case that seems most improved by flakes. You can replace shell.nix files with flake.nix files that expose a devShell output, and get nice pinning behavior and faster evaluation and no more weird hacks to add GC roots. I haven’t actually tried this yet, though.

User software? Theoretically this is the idea behind nix profile. In practice, it’s not really usable yet, as seen previously.

nix-shell -p has a replacement – the new nix shell command. But it’s more limited: it can install specific derivations, but it cannot evaluate arbitrary expressions, apparently. This means you can’t write something like:

nix-shell -p 'python2.withPackages (pkgs: [ pkgs.psycopg2 ])'

Which is a real thing that I have done before. I’m not sure if there’s some unofficial workaround for this, but we might be able to figure something out.

nix-shell -i has no equivalent; this is not something that you can do with flakes at all yet.

So we still need nix-shell and nix-env and all that. We can’t give up on channels altogether. Even if we decide to embrace flakes, we’ll need some sort of compatibility shim for the time being.

I tried to come up with one in the last post, and I failed.

But the Nix expert I spoke to showed me a partial approximation. Because it turns out that my attempt before – to add a flake that pointed to my channel – that should have worked. The fact that it didn’t is just a weird bug in Nix.

It has to do with the fact that the path to my channel contained a symlink. Observe:

$ nix search path:$HOME/.nix-defexpr/channels/nixpkgs git
error: access to path '/nix/store/dd99fd7ik49hfc7ywiy85n2wbylhamjr-nixpkgs-22.05pre335173.56cbe42f166/nixpkgs/flake.nix' is forbidden in restricted mode

But if I resolve the symlink for it, it works correctly:

$ nix search path:$(realpath ~/.nix-defexpr/channels/nixpkgs) git
copying '/nix/store/dd99fd7ik49hfc7ywiy85n2wbylhamjr-nixpkgs-22.05pre335173.56c
...

The fact that those are different is surprising, but bugs are bugs. It happens.

So does that mean we’re done? We can write a flake that tracks our channel, and have seamless backwards compatibility?

Well… not quite. Because it’s incredibly slow:

$ time nix search path:$(realpath ~/.nix-defexpr/channels/nixpkgs) git
copying '/nix/store/dd99fd7ik49hfc7ywiy85n2wbylhamjr-nixpkgs-22.05pre335173.56c
...
5.04s user 14.44s system 71% cpu 27.142 total

It hangs for about fifteen seconds (copying something that is already in the Nix store into the Nix store (?)). Then it spends… twelve seconds printing out output. Which is a bit too long.

Subsequent runs are faster, though.

$ time nix search path:$(realpath ~/.nix-defexpr/channels/nixpkgs) git
copying '/nix/store/dd99fd7ik49hfc7ywiy85n2wbylhamjr-nixpkgs-22.05pre335173.56c
...
4.62s user 11.17s system 93% cpu 16.921 total

Presumably this is because of the evaluation cache: it still spends fifteen seconds copying everything, but then manages to print output in only two seconds – which is as fast as nix search nixpkgs git.

So this isn’t really a viable alternative. I don’t want to type all of that every time I search something, and if I add the realpath to my registry then I would have to remember to re-sync it whenever I updated my channel. I could make an alias for this, and resolve the symlink on demand, but a seventeen second search is not reasonable anyway. Especially when you consider that, by rolling back to Nix 2.3, I can search in only one second.

I don’t claim to understand what’s happening there, but in the opinion of the person I spoke to, the fact that it’s copying anything is a bug. So it is entirely possible that the obvious flake workaround will work in a future version of Nix. Neat.

Until then, however, let’s try to come up with something else.

I wanted to add a flake that reflected my channel. But I feel like that’s spiritually against the rules. My channel is a mutable target.

I could add a flake that points to a local clone of Nixpkgs, and benefit from the git-based rev checking immutability stuff – but then my nix-channel --update would become (cd ~/src/nixpkgs && git fetch && git merge --ff-only). Which is fine… I mean, I can make an alias to do that, I guess. But it’s not a very satisfying solution in general.

But perhaps instead of trying to make a flake that reflects the contents of my channel, I could make a channel that reflects the contents of my flake?

I happen to have come across another piece of forbidden knowledge since last week: someone emailed me to talk about flakes, and they pointed out a function that would have been very useful for me back when I first encountered them: builtins.getFlake.

Here’s what the manual has to say about it:

$ rg 'getFlake' ~/src/nix/doc
/Users/ian/src/nix/doc/manual/src/release-notes/rl-2.4.md
189:  - `builtins.getFlake` fetches a flake and returns its output

Ah. Okay. It’s undocumented, except for a line in the release notes. Let’s expand that:

New built-in functions:

  • builtins.fetchTree allows fetching a source tree using any backends supported by the fetcher infrastructure. It subsumes the functionality of existing built-ins like fetchGit, fetchMercurial and fetchTarball.

  • builtins.getFlake fetches a flake and returns its output attributes. This function should not be used inside flakes! Use flake inputs instead.

  • builtins.floor and builtins.ceil round a floating-point number down and up, respectively.

This is pretty important, and something that I completely missed while I was scanning through the release notes. But it gives me the in into the flake world that I was missing – the ability to grab hold of one in motion.

Let’s try it out and see what happens.

$ nix repl
Welcome to Nix 2.4. Type :? for help.

nix-repl> nixpkgs = builtins.getFlake "nixpkgs"

nix-repl> nixpkgs.git
[15.6/0.0 MiB DL] downloading 'https://api.github.com/repos/NixOS/nixpkgs/tarball/dd89ff

One full minute later…

nix-repl> nixpkgs.git
error: attribute 'git' missing

       at «string»:1:1:

            1| nixpkgs.git
             | ^
            2|
[26.8 MiB DL]

Okay! So yeah it really is downloading things from the internet as a result of evaluating Nix expressions.

The first time I encountered fetchurl, I guessed that it worked this way. Then I learned that I was actually looking at a derivation, and my mental model changed to say that derivations are like side-effect-thunks.

But I recently learned that my first guess was actually correct, and that there is a builtins.fetchurl that behaves differently than nixpkgs.fetchurl – which is what I was looking at at the time. Here’s Nix 2.3:

$ nix repl
Welcome to Nix version 2.3.16. Type :? for help.

nix-repl> something = builtins.fetchurl "https://ianthehenry.com"

nix-repl> something
downloading 'https://ianthehenry.com'"/nix/store/a9c7r27a2jdp58bsl9d4fplrn7h30gps-ianthehenry.com"

[0.0 MiB DL]

The output is weird because it tried to do an animated progress bar thing and kind of botched it, but you can see that evaluating that expression did download a file from the internet and put it in my Nix store.

So anyway, this not-really-lazy evaluation model has actually been part of Nix for a while. But now I get to see it.

Let’s keep seeing it:

nix-repl> nixpkgs = builtins.getFlake "nixpkgs"

nix-repl> nixpkgs
{ checks = { ... };
  htmlDocs = { ... };
  inputs = { ... };
  lastModified = 1639847300;
  lastModifiedDate = "20211218170820";
  legacyPackages = { ... };
  lib = { ... };
  narHash = "sha256-ZBa69DnJX0TubUgwf7ENMylX0h92smvUXsR2Jt1qCqQ=";
  nixosModules = { ... };
  outPath = "/nix/store/a9hp244h16li4c3ja454jkczc7gfmwn4-source";
  outputs = { ... };
  rev = "dd89ff03f3ce876086ec254610005949ca0818d0";
  shortRev = "dd89ff0";
  sourceInfo = { ... }; }

(No; nix repl hasn’t learned about pretty-printing; I added some newlines manually.)

Okay. So the “documentation” for builtins.getFlake said that it “fetches a flake and returns its output attributes.” But fortunately this is not true: it fetches a flake and returns the flake itself. Which is far more useful.

And now I can see that which I could not see before: what a flake is. I’ve finally got a hold of one. Let’s check it out.

We can see all of the magic properties that were documented before – excepting revCount, which is not available for GitHub flakes. And I can see that outPath is the thing that builtins.trace was showing me, and I can see why:

nix-repl> builtins.toString nixpkgs
"/nix/store/a9hp244h16li4c3ja454jkczc7gfmwn4-source"

I thought that flakes had their own custom toString, but it appears that’s not the case. I just forgot that the outPath key is special to Nix:

nix-repl> builtins.toString { outPath = "hello"; }
"hello"

What else is in here? I don’t want to recursively print everything, but:

nix-repl> :p nixpkgs.inputs
{ }

As I remember.

How about:

nix-repl> nixpkgs.outputs.legacyPackages.x86_64-darwin
{ AAAAAASomeThingsFailToEvaluate = «error: error: Please be informed that this pseudo-package is not the only part of
       Nixpkgs that fails to evaluate. You should not evaluate entire Nixpkgs
       without some special measures to handle failing packages, like those taken
       by Hydra.»; AMB-plugins = «derivation «error: error: Package ‘AMB-plugins-0.8.1’ in /nix/store/a9hp244h16li4c3ja454jkczc7gfmwn4-source/pkgs/applications/audio/AMB-plugins/default.nix:23 is not supported on ‘x86_64-darwin’, refusing to evaluate.

       a) To temporarily allow packages that are unsupported for this system, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowUnsupportedSystem = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowUnsupportedSystem = true; }
       to ~/.config/nixpkgs/config.nix.»; ArchiSteamFarm = «derivation «error: error: Package ‘lttng-ust-2.12.2’ in /nix/store/a9hp244h16li4c3ja454jkczc7gfmwn4-source/pkgs/development/tools/misc/lttng-ust/generic.nix:37 is not supported on ‘x86_64-darwin’, refusing to evaluate.

       a) To temporarily allow packages that are unsupported for this system, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowUnsupportedSystem = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowUnsupportedSystem = true; }
       to ~/.config/nixpkgs/config.nix.»; AusweisApp2 = «derivation «error: error: Package ‘AusweisApp2-1.22.2’ in /nix/store/a9hp244h16li4c3ja454jkczc7gfmwn4-source/pkgs/applications/misc/ausweisapp2/default.nix:20 is not supported on ‘x86_64-darwin’, refusing to evaluate.

    ... THOUSANDS OF LINES OF ERROR MESSAGES ...

    ^C^C^C^C^C^C^C

    ... THOUSANDS MORE ERROR MESSAGES ...

Oh gosh oh gosh oh no. It just started spewing errors, trying to evaluate every single package, and ignoring all attempts to interrupt it. I eventually had to kill -9 it, because it wouldn’t even respond to SIGTERM. Not great. Let’s try that again:

$ nix repl
Welcome to Nix 2.4. Type :? for help.

nix-repl> nixpkgs = builtins.getFlake "nixpkgs"

nix-repl> nixpkgs.outputs.legacyPackages.x86_64-darwin.git
«derivation /nix/store/1s02f3hsyvcx7kpyzj15ja3kklwj86b6-git-2.34.0.drv»

Okay. Everything is fine.

So this makes me curious: what is happening when I type nixpkgs#git at the command line? What is that actually referring to, and why?

nix flake --help says:

# nix build github:NixOS/nixpkgs#hello

github:NixOS/nixpkgs is a flake reference (while hello is an output attribute).

Okay. “Output attribute.”

I’m not really sure how to square that with the contents of the outputs attribute set. This flake does not have an attribute named git. Presumably it’s looking up legacyPackages.x86_64-darwin.git – although that same derivation might exist at some other path. But this behavior isn’t explained anywhere in nix flake --help.

But perhaps it wouldn’t be – perhaps this is explained in the documentation for nix build or nix profile install or the other places that actually take this. Let’s see.

nix build --help says:

Synopsis

    nix build [option...] installables...

So nixpkgs#git is an “installable.” This seems to be a new Nix term; it’s used throughout the help text. It doesn’t explain how installables work or what they are. It does say:

Options that change the interpretation of installables:

  · --derivation
    Operate on the store derivation rather than its outputs.

  · --expr expr
    Interpret installables as attribute paths relative to the Nix expression expr.

  · --file / -f file
    Interpret installables as attribute paths relative to the Nix expression stored
    in file.

So that’s interesting, but I really want to know how installables are treated by default.

$ rg installable ~/src/nix/doc
/Users/ian/src/nix/doc/manual/src/package-management/basic-package-mgmt.md
61:to show the “available” (i.e., installable) packages, as opposed to the

/Users/ian/src/nix/doc/manual/src/quick-start.md
18:1. See what installable packages are currently available in the

Hmm. Interesting. But that doesn’t include any --help text output… and maybe it is explained there, somewhere?

Where does the --help output live?

$ rg 'Interpret installables'
src/libcmd/installables.cc
136:        .description = "Interpret installables as attribute paths relative to the Nix expression stored in *file*.",
145:        .description = "Interpret installables as attribute paths relative to the Nix expression *expr*.",

Ah. I see. Well, that’s probably going to be my best bet…

There’s nothing in installables.hh that explains what these are – I have had some success reading header files in the past, but not this time.

But reading the source reveals parseInstallables:

Strings SourceExprCommand::getDefaultFlakeAttrPathPrefixes()
{
    return {
        // As a convenience, look for the attribute in
        // 'outputs.packages'.
        "packages." + settings.thisSystem.get() + ".",
        // As a temporary hack until Nixpkgs is properly converted
        // to provide a clean 'packages' set, look in 'legacyPackages'.
        "legacyPackages." + settings.thisSystem.get() + "."
    };
}

Aha. Okay. I’m not sure what the difference is between packages and legacyPackages. Ah, but grepping for legacyPackages shows me the documentation I was looking for before: this is all explained in nix --help.

Installables

Many nix subcommands operate on one or more installables. These are command line arguments that represent something that can be built in the Nix store. Here are the recognised types of installables:

  • Flake output attributes: nixpkgs#hello

    These have the form flakeref[#attrpath], where flakeref is a flake reference and attrpath is an optional attribute path. For more information on flakes, see [the nix flake manual page](./nix3-flake.md). Flake references are most commonly a flake identifier in the flake registry (e.g. nixpkgs) or a path (e.g. /path/to/my-flake or .).

    If attrpath is omitted, Nix tries some default values; for most subcommands, the default is defaultPackage.system (e.g. defaultPackage.x86_64-linux), but some subcommands have other defaults. If attrpath is specified, attrpath is interpreted as relative to one or more prefixes; for most subcommands, these are packages.system, legacyPackages.*system* and the empty prefix. Thus, on x86_64-linux nix build nixpkgs#hello will try to build the attributes packages.x86_64-linux.hello, legacyPackages.x86_64-linux.hello and hello.

Wow, this is a really nice overview! I escaped some markdown for clarity – note that the --help text also reproduces that filename that it’s supposed to link to. Which is a little goofy but whatever. That legacyPackages.*system* bit makes more sense in the raw markdown:

`packages.`*system*, `legacyPackages.*system*`

Anyway. Interesting that nixpkgs#git will try to find packages.x86_64-darwin.git before it tries to find git. That seems… wrong to me. I have no idea how you’d specify that you meant git in the case that you had both of these outputs.

Other installables:

  • store paths
  • store derivations (with a definition and explanation of this term!)
  • Nix attributes, which is to say, “attributes in regular non-flake Nix files.”
  • Nix expressions, which look like, well, expressions.

Okay neat. Let’s play with these for a minute.

I wonder if the “attribute” expression will automatically call functions the way that the old UI used to do. That is, can I write:

$ nix build git --file ~/.nix-defexpr/channels/nixpkgs/default.nix

$ readlink result
/nix/store/f12mf054h7r0i4vjvvcjh0ymjvcbwpa6-git-2.33.1

I can indeed! Okay. I wonder if it does the same automatic directories-become-attribute-sets translation that nix-env --file does:

$ nix build nixpkgs.git --file ~/.nix-defexpr/channels
error: opening file '/nix/store/26vpc8czyr4gak7b1dvqpra159y1x9vd-user-environment/default.nix': No such file or directory

Nope. Okay, well that’s fine. This is already a step in the right direction: a way to use all of the new-style commands with expressions from my channel.

Except for nix search, of course…

Let’s look at the literal expression style. There was an example and a caveat in the --help text:

  • Nix expressions:
--expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'

When the --expr option is given, all installables are interpreted as Nix expressions. You may need to specify --impure if the expression references impure inputs (such as <nixpkgs>).

And indeed:

$ nix build --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
error: cannot look up '<nixpkgs>' in pure evaluation mode (use '--impure' to override)

       at «string»:1:9:

            1| (import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })
             |         ^

I am really loving those new error messages.

Wow, there’s like… fancy animated build progress now?

$ nix build --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })' --impure
[1/0/1 built, 1 copied (0.7/0.7 MiB), 0.7 MiB DL] building my-hello (configurePhase): checking whether strcasestr is d

It’s chopped off at one line and I had to expand my shell to see… part of what it’s doing. Interesting. How did it know what phase of the build it was in? Is this something that changed in stdenv to… prefix all output like that? Or is Nix like… parsing the output to figure that out?

Huh. Let’s compare what this looks like in Nix Classic.

$ nix-build --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
/nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello

Oh, right, duh.

$ rm result

$ nix-collect-garbage

I had a lot of garbage to collect from when I accidentally evaluated nixpkgs.outputs.legacyPackages.x84_64-darwin – remember that evaluating derivations at the REPL (or anywhere?) has the side effect of putting .drv files in your store, which is very surprising to me. But something that I have been surprised by previously, and thus am not surprised by anymore.

So I got to see a lot of fun stuff like this:

deleting '/nix/store/yvh50wy12sf2l6g3w1sfww72h7wdxxmf-libutil-47.30.1.tar.gz.drv'
deleting '/nix/store/bnjllha45nkw7qg1l6jix128aqr4g2pp-embedfile.r54865.tar.xz.drv'
deleting '/nix/store/gd1c2l66lamhr89f673jzbsq1j8mrvzy-expand-brackets-2.1.4.tgz.drv'
deleting '/nix/store/1iv4czqa4yic4sbl2lf7z1yygrlwpp5a-diffutils-3.8'
deleting '/nix/store/s79j4k2d7n1dzpsil5wl6c09phydhsx9-azure-mgmt-rdbms-10.0.0.zip.drv'
deleting '/nix/store/whj853b0h33bwp9bfqvf5ifzzph6w4k3-ballerburg-1.2.0.tar.gz.drv'
deleting '/nix/store/a50jhaz3zkx9a1iv87kfa0bzj9dn5yhl-newspaper.r15878.tar.xz.drv'
deleting '/nix/store/9acf6z50qqingjpn8i6bw9r5d85nxzhg-fix-qemu-ga.patch'
deleting '/nix/store/7mrgg6wk9fbq809jrccpmdasb5hmvb8f-vancouver.r55423.tar.xz.drv'

Anyway, let’s try that again:

$ nix-build --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
/nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello

Huh! Still nothin'. That’s weird. Why is that alive?

$ nix-store --query --referrers /nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello

Right, but that’s not really the question I want to ask. Hmm. man nix-store teaches me --roots:

$ nix-store --query --roots /nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello
/Users/ian/src/nix/result -> /nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello

Aha! I can see that it’s because I’m an idiot: when I switched from Nix 2.4 to Nix 2.3, I was in a different directory. Ha. Maybe I shouldn’t set PS1="$ " while I’m writing these blog posts…

One more try:

$ nix-build --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
these derivations will be built:
  /nix/store/w1w2gixcykpjyw08g5n8ql51iv1ldkhx-my-hello.drv
these paths will be fetched (128.35 MiB download, 638.54 MiB unpacked):
  /nix/store/0z8am7gyngj9104lsgcpigk165nqzdsj-patch-2.7.6
  /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
  /nix/store/5cd9pbkyvj941w15cm7bidf8nkpxygmd-findutils-4.8.0
  /nix/store/66crk4qirnyz08zw91nhlz4x8lrxprc0-clang-7.1.0-lib
  /nix/store/a2v97g7g9dq9vy8v3094lsdgjmwn35yd-adv_cmds-119-locale
  /nix/store/aiv11nnqmcyvbq33vx2aiaf4a2n6wkds-ed-1.17
  /nix/store/aqwnkakcgz0h8bn1g7pya9x0mknfpnfx-gnumake-4.3
  /nix/store/bx4prwlqjf856w4swv6vdz7v9avhwixy-cctools-binutils-darwin-949.0.1
  /nix/store/dhhbwn09yhr7jqvwkvi0v4m62xzsvdi9-stdenv-darwin
  /nix/store/gsfxxazc51sx6vjdrz6cq8s19x7h5mwh-clang-wrapper-7.1.0
  /nix/store/hxylb8a54skddgam8ip6lg6f8ahmdpw7-cctools-binutils-darwin-wrapper-949.0.1
  /nix/store/jjqvaxg6xsz31ln57ri4ssasxxm4rj9i-expand-response-params
  /nix/store/jxh690i6zwbx53s04493diwkcmz36fzv-libtapi-1100.0.11
  /nix/store/kd3qrxfgws2kwyhkamm52dsjfrdbrlw1-binutils-2.35.2
  /nix/store/lxd1nlzcdfih205yyjnizqgwdpydfc66-llvm-7.1.0
  /nix/store/r0i9j5m3xnqlwmri7g0kiqnxy21vpq5i-gawk-5.1.1
  /nix/store/s9r2s5qcrpzdx2maijhwl5vf33rfiqzw-cctools-port-949.0.1
  /nix/store/w4a9ljyrkai9y9bv3gwm21hr6s6n4g0l-llvm-7.1.0-lib
  /nix/store/z3p39y1nj2w6db8myfp1bcg8vq6yic78-diffutils-3.8
  /nix/store/zfh3npfhfjjgwi0dqpriklip5k15ppmj-clang-7.1.0
copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/a2v97g7g9dq9vy8v3094lsdgjmwn35yd-adv_cmds-119-locale' from 'https://cache.nixos.org'...
copying path '/nix/store/kd3qrxfgws2kwyhkamm52dsjfrdbrlw1-binutils-2.35.2' from 'https://cache.nixos.org'...
copying path '/nix/store/z3p39y1nj2w6db8myfp1bcg8vq6yic78-diffutils-3.8' from 'https://cache.nixos.org'...
copying path '/nix/store/aiv11nnqmcyvbq33vx2aiaf4a2n6wkds-ed-1.17' from 'https://cache.nixos.org'...
copying path '/nix/store/jjqvaxg6xsz31ln57ri4ssasxxm4rj9i-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/5cd9pbkyvj941w15cm7bidf8nkpxygmd-findutils-4.8.0' from 'https://cache.nixos.org'...
copying path '/nix/store/r0i9j5m3xnqlwmri7g0kiqnxy21vpq5i-gawk-5.1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/aqwnkakcgz0h8bn1g7pya9x0mknfpnfx-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/jxh690i6zwbx53s04493diwkcmz36fzv-libtapi-1100.0.11' from 'https://cache.nixos.org'...
copying path '/nix/store/w4a9ljyrkai9y9bv3gwm21hr6s6n4g0l-llvm-7.1.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/s9r2s5qcrpzdx2maijhwl5vf33rfiqzw-cctools-port-949.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/zfh3npfhfjjgwi0dqpriklip5k15ppmj-clang-7.1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/66crk4qirnyz08zw91nhlz4x8lrxprc0-clang-7.1.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/lxd1nlzcdfih205yyjnizqgwdpydfc66-llvm-7.1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/0z8am7gyngj9104lsgcpigk165nqzdsj-patch-2.7.6' from 'https://cache.nixos.org'...
copying path '/nix/store/bx4prwlqjf856w4swv6vdz7v9avhwixy-cctools-binutils-darwin-949.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/hxylb8a54skddgam8ip6lg6f8ahmdpw7-cctools-binutils-darwin-wrapper-949.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/gsfxxazc51sx6vjdrz6cq8s19x7h5mwh-clang-wrapper-7.1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/dhhbwn09yhr7jqvwkvi0v4m62xzsvdi9-stdenv-darwin' from 'https://cache.nixos.org'...
warning: unknown setting 'experimental-features'
warning: unknown setting 'flake-registry'
building '/nix/store/w1w2gixcykpjyw08g5n8ql51iv1ldkhx-my-hello.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello
checking for a BSD-compatible install... /nix/store/jyv20ip5h954zsygg1xhg4n0jfcy82ck-coreutils-9.0/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/jyv20ip5h954zsygg1xhg4n0jfcy82ck-coreutils-9.0/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ISO C89... none needed
checking whether clang understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of clang... none
checking how to run the C preprocessor... clang -E
checking for grep that handles long lines and -e... /nix/store/wnq3vcvg8sfxc938f70q6s6n5j9ran1x-gnugrep-3.7/bin/grep
checking for egrep... /nix/store/wnq3vcvg8sfxc938f70q6s6n5j9ran1x-gnugrep-3.7/bin/grep -E
checking for Minix Amsterdam compiler... no
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
...

I much prefer the explicitness of this output, but there might be a way to configure it.

I guess it’s possible that stdenv is like… detecting that I’m in Nix 2.4, and doing something different?

Or Nix 2.4 is parsing this output… but that seems crazy.

Hmm.

I can’t find anything in the --help text about “progress” or “output” or anything. Let’s try --verbose:

$ nix build --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })' --impure --verbose
copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'...
copying path '/nix/store/a2v97g7g9dq9vy8v3094lsdgjmwn35yd-adv_cmds-119-locale' from 'https://cache.nixos.org'...
copying path '/nix/store/kd3qrxfgws2kwyhkamm52dsjfrdbrlw1-binutils-2.35.2' from 'https://cache.nixos.org'...
copying path '/nix/store/z3p39y1nj2w6db8myfp1bcg8vq6yic78-diffutils-3.8' from 'https://cache.nixos.org'...
copying path '/nix/store/aiv11nnqmcyvbq33vx2aiaf4a2n6wkds-ed-1.17' from 'https://cache.nixos.org'...
copying path '/nix/store/jjqvaxg6xsz31ln57ri4ssasxxm4rj9i-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/5cd9pbkyvj941w15cm7bidf8nkpxygmd-findutils-4.8.0' from 'https://cache.nixos.org'...
copying path '/nix/store/r0i9j5m3xnqlwmri7g0kiqnxy21vpq5i-gawk-5.1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/aqwnkakcgz0h8bn1g7pya9x0mknfpnfx-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/jxh690i6zwbx53s04493diwkcmz36fzv-libtapi-1100.0.11' from 'https://cache.nixos.org'...
copying path '/nix/store/w4a9ljyrkai9y9bv3gwm21hr6s6n4g0l-llvm-7.1.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/s9r2s5qcrpzdx2maijhwl5vf33rfiqzw-cctools-port-949.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/zfh3npfhfjjgwi0dqpriklip5k15ppmj-clang-7.1.0' from 'https://cache.nixos.org'...
[0/2 built, 1/12/20 copied (330.8/638.5 MiB), 68.8/128.3 MiB DL] fetching clang-7.1.0 from https://cache.nixos.org

That’s better. But it’s still hiding the build output from me. Hmm.

I search through the Nix codebase, and find that the progress bar display is a type of “logger.” Okay.

Logger * makeDefaultLogger() {
    switch (defaultLogFormat) {
    case LogFormat::raw:
        return makeSimpleLogger(false);
    case LogFormat::rawWithLogs:
        return makeSimpleLogger(true);
    case LogFormat::internalJSON:
        return makeJSONLogger(*makeSimpleLogger(true));
    case LogFormat::bar:
        return makeProgressBar();
    case LogFormat::barWithLogs:
        return makeProgressBar(true);
    default:
        abort();
    }
}

Okay! Now we’re talking. nix --help has this to say about “logging:”

Logging-related options:

  · --debug
    Set the logging verbosity level to 'debug'.

  · --log-format format
    Set the format of log output; one of raw, internal-json, bar or bar-with-logs.

  · --print-build-logs / -L
    Print full build logs on standard error.

  · --quiet
    Decrease the logging verbosity level.

  · --verbose / -v
    Increase the logging verbosity level.

It would not have occurred to me that “logging” meant “the thing Nix prints to stderr when run interactively.” I assumed that this meant, you know, the log files that Nix creates in the temporary build directories that I’ve seen before, or the contents of /nix/var/log. But clearly the term is overloaded.

Also amusing: I did think to look here. But I searched for the string “progress,” when it is only called “bar.”

Let’s try some other formats.

$ nix build --log-format raw --impure --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'

Huuuuh. It didn’t print anything! It just swallowed all output. Okay. Unexpected. Is that because I didn’t also say --print-build-logs? I don’t… what?

$ nix build --log-format raw --print-build-logs --impure --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
[0/2 built, 1/10/20 copied (124.6/638.5 MiB), 25.1/128.3 MiB DL] fetching llvm-7.1.0-lib from https://cache.nixos.org

Okay, this has brought the progress bar back. I guess that --print-build-logs… ignores the --log-format? Or… what?

It did eventually start printing stuff, though:

$ nix build --log-format raw --print-build-logs --impure --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
[0/2 built, 1/13/20 copied (575.0/638.5 MiB), 119.0/128.3 MiB DL] fetching clang-7.1.0-lib from https://cache.nixos.or
my-hello> unpacking sources
my-hello> unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
my-hello> source root is hello-2.10
my-hello> setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
my-hello> patching sources
my-hello> configuring
my-hello> configure flags: --disable-dependency-tracking --prefix=/nix/store/80vpk09z38yh30b5q4rgjr86c3gj7412-my-hello
my-hello> checking for a BSD-compatible install... /nix/store/jyv20ip5h954zsygg1xhg4n0jfcy82ck-coreutils-9.0/bin/install -c
my-hello> checking whether build environment is sane... yes
my-hello> checking for a thread-safe mkdir -p... /nix/store/jyv20ip5h954zsygg1xhg4n0jfcy82ck-coreutils-9.0/bin/mkdir -p
my-hello> checking for gawk... gawk
my-hello> checking whether make sets $(MAKE)... yes
my-hello> checking whether make supports nested variables... yes
my-hello> checking for gcc... clang
my-hello> checking whether the C compiler works... yes
my-hello> checking for C compiler default output file name... a.out
my-hello> checking for suffix of executables...
my-hello> checking whether we are cross compiling... no
my-hello> checking for suffix of object files... o

I actually really like this output. The my-hello> bit renders with dim text to show me that it’s not really part of the output. I don’t know what’s raw about this output. It looks pretty well-done.

Let’s try some more.

$ nix build --log-format internal-json --impure --expr '(import <nixpkgs> {}).hello.overrideDerivation (prev: { name = "my-hello"; })'
@nix {"action":"start","id":69711614181376,"level":0,"text":"","type":102}
@nix {"action":"start","id":69711614181377,"level":0,"text":"","type":104}
@nix {"action":"start","id":69711614181378,"level":0,"text":"","type":103}
@nix {"action":"result","fields":[0,1,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,0],"id":69711614181376,"type":106}
@nix {"action":"start","id":69711614181379,"level":6,"text":"querying info about missing paths","type":0}
@nix {"action":"stop","id":69711614181379}
@nix {"action":"result","fields":[0,2,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[0,3,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[0,4,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[0,3,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,0],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[0,3,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,2,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,726160],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,726064],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[0,3,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,2,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,738140],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,769040],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[0,3,0,0],"id":69711614181377,"type":105}
@nix {"action":"result","fields":[0,29,0,0],"id":69711614181378,"type":105}
@nix {"action":"result","fields":[101,775108],"id":69711614181376,"type":106}
@nix {"action":"result","fields":[100,1780264],"id":69711614181376,"type":106}
...
@nix {"action":"result","fields":[4587520,164382360,0,0],"id":69711614181411,"type":105}
^C@nix {"action":"result","fields":[4595712,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"result","fields":[4603904,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"result","fields":[4612096,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"result","fields":[4620288,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"result","fields":[4628480,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"result","fields":[4636672,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"result","fields":[4644864,164382360,0,0],"id":69711614181411,"type":105}
@nix {"action":"stop","id":69711614181411}
@nix {"action":"stop","id":69711614181410}
@nix {"action":"stop","id":69711614181378}
@nix {"action":"stop","id":69711614181377}
@nix {"action":"stop","id":69711614181376}
@nix {"action":"result","fields":[3685826,30041472,0,0],"id":69711614181412,"type":105}
@nix {"action":"stop","id":69711614181412}
@nix {"action":"msg","column":null,"file":null,"level":0,"line":null,"msg":"\u001b[31;1merror:\u001b[0m interrupted by the user","raw_msg":"interrupted by the user"}

Wow. Wild. I did, in fact, eventually interrupt it, because it was completely blowing away my scrollback. internal-json indeed.

--log-format bar seems to be the default – the compact one line thing that you can’t read unless your terminal is at least 120 characters wide.

And --log-format bar-with-logs seems to be the same as --log-format raw --print-build-logs.

Okay, yeah, --print-build-logs seems to be an alias for --log-format bar-with-logs. --log-format internal-json --print-build-logs also gives me the same (nice) output. I get why they have a short flag for this output type, but it’s surprising to me that it isn’t documented that way.

addFlag({
    .longName = "print-build-logs",
    .shortName = 'L',
    .description = "Print full build logs on standard error.",
    .category = loggingCategory,
    .handler = {[&]() {setLogFormat(LogFormat::barWithLogs); }},
});

How do I change that to be my default output format? man 5 nix.conf has no answers. I search through the code, and conclude that there is currently no way to do this. But I could be wrong, or that could change in the future, so I’m still adding that as an open question to come back to later.

Okay.

Where were were? What are we doing here?

Oh, I was wondering how it knew that it was in the configurePhase.

Okay, so I looked through the source, and found that this is something the stdenv builder is doing:

if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { \"action\": \"setPhase\", \"phase\": \"$curPhase\" }" >&$NIX_LOG_FD
fi

It’s that same “internal JSON” syntax, but here the builder is printing those log lines to a file descriptor. This is very interesting! My perception of the environment with which derivations are run is changing. I thought all of the environment variables were documented in the .drv file. But here is a magic, extra one?

Apparently this is nothing new; this has been the case since Nix 2.0 (according to the release notes). I’ve just never used the nix build command before.

So, alright. Cool. Installables.

That whole thing was sort of a tangent. Let’s get back on track: I was originally trying to use builtins.getFlake in order to make a “channel” that reflected a “flake.”

Let’s try this:

$ cat ~/.nix-defexpr/flakgs/default.nix

(builtins.getFlake "nixpkgs").outputs.legacyPackages.${builtins.currentSystem}

Will that work? It seems like it does, from a repl:

nix-repl> flakgs = import ./.

nix-repl> flakgs.git
[26.8 MiB DL]«derivation /nix/store/1s02f3hsyvcx7kpyzj15ja3kklwj86b6-git-2.34.0.drv»
«derivation /nix/store/1s02f3hsyvcx7kpyzj15ja3kklwj86b6-git-2.34.0.drv»

Again, weird botched animated progress bar output.

So that (effectively) gives me a flakgs entry on my NIX_PATH. But what I really need this for is nix-shell -p – and nix-shell -p is hardcoded to look up nixpkgs on my NIX_PATH. But that shouldn’t be a problem…

$ NIX_PATH="nixpkgs=$HOME/.nix-defexpr/flakgs" nix-shell -p git
error: attempt to call something which is not a function but a set

       at «string»:1:18:

            1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (git) ]; } ""
             |                  ^

Ah, I see. nix-shell -p doesn’t “auto-call” the Nixpkgs result function, it always calls it. So instead:

$ cat ~/.nix-defexpr/flakgs/default.nix

{ ... }: (builtins.getFlake "nixpkgs").outputs.legacyPackages.${builtins.currentSystem}

Okay, great. I’m just ignoring all arguments, which I might regret I guess. I could at least take system instead of hardcoding it. But whatever; this’ll get me there for now.

Does it work?

$ NIX_PATH="nixpkgs=$HOME/.nix-defexpr/flakgs" nix-shell -p git --command 'git --version'
git version 2.34.0

$ nix-shell -p git --command 'git --version'
git version 2.33.1

Alright! Great. It certainly seems like it’s doing something.

So I think that this will be completely workable for me. And my “channel” also works with commands that expect ~/.nix-defexpr:

$ nix-env -qaA nixpkgs.git
git-2.33.1

$ nix-env -qaA flakgs.git
git-2.34.0

So, armed with this workaround, I think I can try switching back to Nix 2.4, and officially replace my nixpkgs channel with this flake-based alternative. That should allow me to do everything I was doing before – apart from nix-channel --update – even with the newer version.

And then I can take the time to explore flakes at my leisure.

  • Does derivation evaluation always create a .drv file, or is this something that the REPL adds?
  • How can I specify that I really mean outputs.foo if a flake also has outputs.x86_64-darwin.foo?
  • What is --log-format raw?
  • How can I change the default “log format” to bar-with-logs?
  • Can I make builds --verbose by default?