Oh gosh. I’ve been really putting off chapter five because, well, because look at it. It’s massive. It’s intimidating.

But as I recall from the Nix manual, the reference sections do not make the most exciting reading, but they are usually very educational. So I can’t just skip them.

Even if I really want to skip them.

5.1 Nixpkgs Library Functions

I’ve seen lib before, and wondered what it was, and why it was different from builtins. I suppose that now I’m going to find out.

This section is broken up into multiple subsections. In this order, they are:

Assert functions
Attribute-set functions
String manipulation functions
Miscellaneous functions
List manipulation functions
Debugging functions
NixOS / nixpkgs option handling

This is the first time I have seen the term “attribute-set.” The Nix manual exclusively used the bare term “set,” and I was surprised by the builtins.isAttrs function (instead of builtins.isSet). I thought at the time that this was just because isSet is a very ambiguous name – it’s not clear if “set” is a verb or a noun in there.

But now I think that I’m seeing some archaeological evidence that this type was originally called “attribute-set.” I can imagine that, over time, Nix users got tired of saying “attribute-set” all the time, so they started just saying “set,” and everyone knew that it was short for “attribute-set,” and peace reigned throughout the kingdom. But this went on for so long that eventually the abbreviation was reflected in the documentation, and the manual, and everywhere else, and the name “attribute set” fell into legend. And one day newcomers arrived from the neighboring kingdom, and they kept hearing the word “set,” and they were very confused that the Nixites chose a word that already has a pretty well-established and different meaning for this core concept, and many of them returned to their homes, repulsed, but some others stayed, because so many of the customs here were strange and foreign, and they were determined to broaden their minds. And eventually, after befriending some of the locals, they heard whisper of the term “attribute set,” and they realized that perhaps the Nixites were not completely crazy, just victims of an unfortunate linguistic evolution that was deeply ingrained in their culture.

Sigh I guess I actually have to read this chapter now.

I am delighted to report that these functions actually have type signatures, unlike the functions in the Nix manual. That will make it much easier to follow along at home.

5.1.1 Assert functions

The assertion functions are pretty simple – they extend assert expressions with error messages. It looks like they’re implemented in terms of trace, which appears to be the only way to print things in Nix? I’m basing that on this example:

assert lib.asserts.assertMsg ("foo" == "bar") "foo is not bar, silly"
stderr> trace: foo is not bar, silly
stderr> assert failed

Wait okay this is not as simple as I thought. How do I… parse that? There’s no semicolon there. I thought assert expressions needed semicolons. What’s going on here?

$ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> lib = (import <nixpkgs>).lib

nix-repl> assert lib.asserts.assertMsg (1 == 2) "message"

Aha. Okay. Good. I hit enter, but it’s still expecting input, because there was no semicolon, so it’s still scanning. I am reassured. The example is just wrong.

nix-repl> assert lib.asserts.assertMsg (1 == 2) "message"; 1
error: value is a function while a set was expected, at (string):1:2

Hmmm. Well that’s… what? The type signature given is assertMsg :: Bool -> String -> Bool . So… no sets that I can see? The only function in question here is lib.asserts.assertMsg. Which… is a function, right?

nix-repl> lib.asserts.assertMsg
error: value is a function while a set was expected, at (string):1:2

Oh. Ha. Okay, I don’t feel that dumb here.

Have you spotted it yet? The error refers not to the line we typed, but to the line we started with:

nix-repl> lib = (import <nixpkgs>).lib

I forgot to call import <nixpkgs>. But because laziness, it didn’t actually evaluate that expression until we used it. So the error didn’t show up until later.

A little confusing! I could see this confusing someone for quite some time, but fortunately it was but a minor setback on my road towards asserting nonsense. Once more, with feeling:

nix-repl> lib = (import <nixpkgs> {}).lib

nix-repl> assert (lib.asserts.assertMsg (1 == 2) "message")
          ; 1
trace: message
error: assertion (((lib).asserts.assertMsg  (1 == 2))  "message") failed at (string):1:1

Yes, okay; the example is a little stale.

The only other assert function is given as:

assertOneOf :: String -> String -> StringList -> Bool

Which is a strange signature – the first string is the name of the variable, and the second is the value. The example says:

let sslLibrary = "bearssl";
in lib.asserts.assertOneOf "sslLibrary" sslLibrary [ "openssl" "bearssl" ];
=> false
stderr> trace: sslLibrary must be one of "openssl", "libressl", but is: "bearssl"

Now if you look closely you can see that this example is nonsensical; the list does not match the error message, and in fact that expression does not fail the assertion at all. If we modify it so that it does:

nix-repl> let sslLibrary = "bearssl"; in lib.asserts.assertOneOf "sslLibrary" s
slLibrary [ "openssl" "libressl" ]
trace: sslLibrary must be one of [
  "openssl"
  "libressl"
], but is: "bearssl"
false

We can see the output has gotten a little bit fancier.

I confirm that lib.asserts still only contains these two functions, even though I believe the Nixpkgs manual is several months older than the Nixpkgs that I am using today. I assume it’s only updated after Nix releases, and 2.3.10 came out in December 2020 (it is now the beginning of April 2021).

5.1.2. Attribute-Set Functions

Lots of functions for dealing with “paths” – lists of string key names. Pretty simple. Also dealing with lists of keys… honestly I have no idea when I would expect these functions to live in builtins or in lib. These seem to be high-level helpers? Like, maybe builtins is the minimum necessary functionality out of which all the helpers can be built?

And maybe lib mirrors every function from builtins, so you don’t need to remember which live in which and you can always just use lib.attrsets and everything will just work?

Nope. I checked. You have to remember which functions are in lib and which are in builtins. Alas.

I weakly try a few examples to see if they still work.

nix-repl> catAttrs "a" [{a = 1;} {b = 0;} {a = 2;}]
error: undefined variable 'catAttrs' at (string):1:1

nix-repl> lib.catAttrs "a" [{a = 1;} {b = 0;} {a = 2;}]
[ 1 2 ]

nix-repl> lib.attrsets.catAttrs "a" [{a = 1;} {b = 0;} {a = 2;}]
[ 1 2 ]

A lazy copy and paste taught me that I don’t need to type lib.attrsets.function. I can just type lib.function. Is that true… for all of these functions? Some of these functions? The manual has not yet said anything about this.

Wow. We get into some pretty specific helpers here.

mapAttrsRecursiveCond :: (AttrSet -> Bool) -> ([ String ] -> Any -> Any) -> AttrSet -> AttrSet

I’m just… you don’t need to use that function; you don’t need to understand it. If you do need to use it, you’ll know.

Whoa! We have just hit a major breakthrough.

isDerivation :: Any -> Bool 
Check whether the argument is a derivation. Any set with { type = "derivation"; } counts as a derivation.

Is that what a derivation is? Have we finally caught a live one?

nix-repl> { x = 1; }
{ x = 1; }

nix-repl> { x = 1; type = "derivation"; }
«derivation ???»

!!!

We did it. It only took us until part 26 to finally have an answer to the question “what is a derivation.” The answer is… well, you just saw. The answer is it’s a set. With a particular key.

That’s – well, that’s what it means for the isDerivation, and that’s what it means for nix repl, apparently. But is it actually?

nix-repl> (import <nixpkgs> {}).hello
«derivation /nix/store/6dh5ds3dnrkl995wg767mq9iw6pfg8f1-hello-2.10.drv»

nix-repl> (import <nixpkgs> {}).hello // { type = ""; }
{
  __darwinAllowLocalNetworking = false;
  __ignoreNulls = true;
  __impureHostDeps = [ ... ];
  __propagatedImpureHostDeps = [ ... ];
  __propagatedSandboxProfile = [ ... ];
  __sandboxProfile = "";
  all = [ ... ];
  args = [ ... ];
  buildInputs = [ ... ];
  builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash";
  configureFlags = [ ... ];
  depsBuildBuild = [ ... ];
  depsBuildBuildPropagated = [ ... ];
  depsBuildTarget = [ ... ];
  depsBuildTargetPropagated = [ ... ];
  depsHostHost = [ ... ];
  depsHostHostPropagated = [ ... ];
  depsTargetTarget = [ ... ];
  depsTargetTargetPropagated = [ ... ];
  doCheck = true;
  doInstallCheck = false;
  drvAttrs = { ... };
  drvPath = "/nix/store/6dh5ds3dnrkl995wg767mq9iw6pfg8f1-hello-2.10.drv";
  inputDerivation = «derivation /nix/store/rglwjg1sqfwwnskgi7h7h7kj6d3pxmrj-hello-2.10.drv»;
  meta = { ... };
  name = "hello-2.10";
  nativeBuildInputs = [ ... ];
  out = «derivation /nix/store/6dh5ds3dnrkl995wg767mq9iw6pfg8f1-hello-2.10.drv»;
  outPath = "/nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10";
  outputName = "out";
  outputUnspecified = true;
  outputs = [ ... ];
  override = { ... };
  overrideAttrs = «lambda @ /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/customisation.nix:85:73»;
  overrideDerivation = «lambda @ /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/customisation.nix:84:32»;
  passthru = { ... };
  patches = [ ... ];
  pname = "hello";
  propagatedBuildInputs = [ ... ];
  propagatedNativeBuildInputs = [ ... ];
  src = «derivation /nix/store/njch6qh4l5k42bz14fya766l0piyvril-hello-2.10.tar.gz.drv»;
  stdenv = «derivation /nix/store/v1fmdbwdgqds6b4icqzyin0anag03dz3-stdenv-darwin.drv»;
  strictDeps = false;
  system = "x86_64-darwin";
  type = "";
  userHook = null;
  version = "2.10";
}

Aha! We have finally managed to observe our first actual derivation in the wild (cosmetics by yours truly; I still have no idea how to pretty-print a Nix expression). I could never figure out how to get nix repl to show me that before now.

All kinds of good stuff in there. But it’s a lot to read. I don’t know how much of that is primitive stuff common to any derivation and how much of it is niceties like overrideAttrs and friends.

So let’s go back to my first derivation, which is just about the “simplest” derivation I can think of. But first, a refresher of where we left it:

cat ~/scratch/hello.nix

{ stdenv }:

derivation {
  inherit stdenv;
  system = "x86_64-darwin";
  name = "my-hello-1.0";
  builder = stdenv.shell;
  args = [ "-e" ./builder.sh ];
  messagefile = ./message.txt;
  outputs = ["foo" "out"];
  shellHook = "PS1='sup$ '";
}

And in motion:

nix-repl> (import ~/scratch/hello.nix { inherit (nixpkgs) stdenv; }) // { type = "divination"; }
{
  all = [ ... ];
  args = [ ... ];
  builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash";
  drvAttrs = { ... };
  drvPath = "/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv";
  foo = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»;
  messagefile = /Users/ian/scratch/message.txt;
  name = "my-hello-1.0";
  out = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»;
  outPath = "/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo";
  outputName = "foo";
  outputs = [ ... ];
  shellHook = "PS1='sup$ '";
  stdenv = «derivation /nix/store/v1fmdbwdgqds6b4icqzyin0anag03dz3-stdenv-darwin.drv»;
  system = "x86_64-darwin";
  type = "divination";
}

Well that’s a lot tamer looking. So tame that I expect we can afford to look at the whole thing:

nix-repl> :p (import ~/scratch/hello.nix { inherit (nixpkgs) stdenv; }) // { type = "divination"; }
{
  all = [ «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» ];
  args = [ "-e" /Users/ian/scratch/builder.sh ];
  builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash";
  drvAttrs = {
    args = «repeated»;
    builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash";
    messagefile = /Users/ian/scratch/message.txt;
    name = "my-hello-1.0";
    outputs = [ "foo" "out" ];
    shellHook = "PS1='sup$ '";
    stdenv = «derivation /nix/store/v1fmdbwdgqds6b4icqzyin0anag03dz3-stdenv-darwin.drv»;
    system = "x86_64-darwin";
  };
  drvPath = "/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv";
  foo = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»;
  messagefile = /Users/ian/scratch/message.txt;
  name = "my-hello-1.0";
  out = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»;
  outPath = "/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo";
  outputName = "foo";
  outputs = «repeated»;
  shellHook = "PS1='sup$ '";
  stdenv = «repeated»;
  system = "x86_64-darwin";
  type = "divination";
}

That’s still not too bad. Look at that «repeated» thing that nix repl is doing – that’s kind of neat? Possibly very annoying? How does that work?

nix-repl> { x = "hi"; y = "hi"; }
{ x = "hi"; y = "hi"; }

nix-repl> let hi = "hi"; in { x = hi; y = hi; }
{ x = "hi"; y = "hi"; }

nix-repl> { x = { hi = "hello"; }; y = { hi = "hello"; }; }
{ x = { ... }; y = { ... }; }

nix-repl> let hi = { hi = "hello"; }; in { x = hi; y = hi; }
{ x = { ... }; y = «repeated»; }

So it seems to be reference equality, but strings have value semantics (as they should).

Anyway, back to our divination.

The only attributes that are not just copied from our input attributes are:

{
  all = [ «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» ];
  drvAttrs = { ... };
  drvPath = "/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv";
  foo = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»;
  out = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»;
  outPath = "/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo";
  outputName = "foo";
  type = "divination";
}

It’s very interesting to me that all contains two things, repeated. But obviously it doesn’t really (the second is not listed as «repeated», implying they are distinct sets). I’m assuming they differ only in outputPath and outputName. Let’s find out:

nix-repl> hello = (import ~/scratch/hello.nix { inherit (nixpkgs) stdenv; }) // { type = "divination"; }

nix-repl> hello.all...

Dang, how do you index a list in Nix? Did I ever learn this? I remember reading something about “arrays” at some point, but I remember it being a lie… I search my diary, and yeah, I find it in part 16:

In addition to attribute names, you can also specify array indices. For instance, the attribute path foo.3.bar selects the bar attribute of the fourth element of the array in the foo attribute of the top-level expression.

Yeah, I don’t know what arrays are, but that sure doesn’t work for lists.

nix-repl> hello.all.0
error: attempt to call something which is not a function but a list, at (string):1:1

Hmm. I search the manual and find buildints.elemAt. I guess I just forgot about it.

nix-repl> :p builtins.elemAt hello.all 0 // { type = "divination"; }
{ all = [ «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» ]; args = [ "-e" /Users/ian/scratch/builder.sh ]; builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash"; drvAttrs = { args = «repeated»; builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash"; messagefile = /Users/ian/scratch/message.txt; name = "my-hello-1.0"; outputs = [ "foo" "out" ]; shellHook = "PS1='sup$ '"; stdenv = «derivation /nix/store/v1fmdbwdgqds6b4icqzyin0anag03dz3-stdenv-darwin.drv»; system = "x86_64-darwin"; }; drvPath = "/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv"; foo = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»; messagefile = /Users/ian/scratch/message.txt; name = "my-hello-1.0"; out = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»; outPath = "/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo"; outputName = "foo"; outputs = «repeated»; shellHook = "PS1='sup$ '"; stdenv = «repeated»; system = "x86_64-darwin"; type = "divination"; }

nix-repl> :p builtins.elemAt hello.all 1 // { type = "divination"; }
{ all = [ «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv» ]; args = [ "-e" /Users/ian/scratch/builder.sh ]; builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash"; drvAttrs = { args = «repeated»; builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash"; messagefile = /Users/ian/scratch/message.txt; name = "my-hello-1.0"; outputs = [ "foo" "out" ]; shellHook = "PS1='sup$ '"; stdenv = «derivation /nix/store/v1fmdbwdgqds6b4icqzyin0anag03dz3-stdenv-darwin.drv»; system = "x86_64-darwin"; }; drvPath = "/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv"; foo = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»; messagefile = /Users/ian/scratch/message.txt; name = "my-hello-1.0"; out = «derivation /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv»; outPath = "/nix/store/pppj7nh5s8l9wbmv6i3avyb9skzayyg9-my-hello-1.0"; outputName = "out"; outputs = «repeated»; shellHook = "PS1='sup$ '"; stdenv = «repeated»; system = "x86_64-darwin"; type = "divination"; }

There has got to be a better way to see this than my goofy // { type = "divination"; } hack.

Anyway, diffing those in fact reveals:

$ diff -U 0 elem-{0,1}
--- elem-0      2021-04-08 10:33:58.000000000 -0700
+++ elem-1      2021-04-08 10:32:39.000000000 -0700
@@ -20,2 +20,2 @@
-outPath = "/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo";
-outputName = "foo";
+outPath = "/nix/store/pppj7nh5s8l9wbmv6i3avyb9skzayyg9-my-hello-1.0";
+outputName = "out";

That my theory was correct. It’s very interesting to me that nix repl stringifies these as “the same derivation,” instead of reporting their outputPath. These seem like different things to me!

It makes me wonder how nix repl is choosing to report the name of a derivation.

I had previously seen this:

nix-repl> { type = "derivation"; }
«derivation ???»

So I should be able to reverse engineer exactly what it cares about.

nix-repl> { type = "derivation"; outputPath = "/nix/store/foo"; }
«derivation ???»

That just confirms that it is not looking at outputPath – although it certainly seems like it should.

Oh, right, duh. I just missed the .drv at the end of that giant string. It’s probably printing the drvPath.

nix-repl> { type = "derivation"; drvPath = "foo"; }
error: string 'foo' doesn't represent an absolute path, at (string):1:24
«derivation
nix-repl> { type = "derivation"; drvPath = "/foo"; }
«derivation /foo»

Yep. Okay. That’s… that’s not really what I want? But maybe this makes sense? I still don’t really understand why I would care about a .drv file. Or even really… what those are. Or what to do with them. Or why they exist.

Kinda fun that it still prints the string «derivation, even though it errored afterwards? It makes me think that nix repl prints "«derivation" and then evaluates the drvPath. You can also see the spacing is different – nix repl usually prints a blank line between the result of the previous expression and the next prompt. But here it seems that blank line is the only newline, since it never got to print the closing "»\n". Kinda fun. Little peek under the hood.

Anyway. We might be slightly off track. But I feel like I’ve learned more about derivations in this chapter than in any other so far, so let’s keep this good thing going. Maybe this post will be about the nature of derivations, instead of the function reference. Also this is more fun than reading the infinite list of functions.

I wonder if I can construct my own derivation whole cloth. I already know what a derivation looks like. If I just inline the result of the derivation call as a value, can I install that? Or is there some hidden side effect that derivation (really derivationStrict) has to make this possible?

Let’s find out. It’ll be fun. Laziness makes it easy to construct the self-referential value without needing mutability:

$ cat default.nix

rec {
  hello = import ./hello-bare.nix;
}

$ cat hello-bare.nix

let stdenv = (import <nixpkgs> {}).stdenv; in
let recursiveBindingHack = rec {
  foo = {
    all = [ foo out ];
    args = [ "-e" /Users/ian/scratch/builder.sh ];
    builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash";
    drvAttrs = {
      args = [ "-e" /Users/ian/scratch/builder.sh ];
      builder = "/nix/store/l25gl3siwmq6gws4lqlyd1040xignvqw-bash-4.4-p23/bin/bash";
      messagefile = /Users/ian/scratch/message.txt;
      name = "my-hello-1.0";
      outputs = [ "foo" "out" ];
      shellHook = "PS1='sup$ '";
      inherit stdenv;
      system = "x86_64-darwin";
    };
    drvPath = "/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv";
    inherit foo;
    messagefile = /Users/ian/scratch/message.txt;
    name = "my-hello-1.0";
    outputs = [ "foo" "out" ];
    outPath = "/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo";
    outputName = "foo";
    shellHook = "PS1='sup$ '";
    inherit stdenv;
    system = "x86_64-darwin";
    type = "derivation";
  };

  out = foo // {
    outPath = "/nix/store/pppj7nh5s8l9wbmv6i3avyb9skzayyg9-my-hello-1.0";
    outputName = "out";
  };

};
in
recursiveBindingHack.foo

It’s a little long, but it’s basically exactly what we’ve seen before.

And now, the moment of truth…

$ nix-env -f default.nix -iA hello
installing 'my-hello-1.0'

$ hello
Hello, Nix!

Okay! So derivations really are just sets. Nothing magic; nothing weird.

Good. Good good.

Next I wonder what a “minimal” derivation looks like. I try deleting things from this set and seeing if I can still install the derivation. It continues to work suspiciously well even as I strip it down to just a couple fields. Which makes me think that it isn’t actually doing any work here, because the output already exists. I need to collect garbage and try again.

And in doing so I learn something very upsetting:

$ nix-collect-garbage --dry-run

That doesn’t do anything. That prints nothing, even when there’s a lot of garbage to collect. In order to actually see what will be garbage collected, I have to run:

$ nix-store --gc --print-dead
(thousands of lines of output)

It seems nix-collect-garbage --dry-run only does anything if you run it with --delete-old – it seems like it just calls nix-env --delete-generations old --dry-run and then exits. Or, if you didn’t pass --delete-old, it does nothing and then exits. I would definitely expect it to run nix-store --gc --print-dead.

And yes, I do realize that it’s a tiny bit complicated because if you run nix-collect-garbage --delete-old --dry-run then you won’t get a reasonable response unless you take into account the roots that will be removed. You’d need like nix-store --gc --print-dead --without-roots /path/to/old/profile ... or something, in order to get an accurate summary. But that doesn’t seem like a very difficult thing to add? nix-collect-garbage only has two jobs here.

Anyway, I remember that I can delete things manually, so I calculate my own paths and nuke them:

$ nix-store --delete /nix/store/pppj7nh5s8l9wbmv6i3avyb9skzayyg9-my-hello-1.0 /nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv
finding garbage collector roots...
deleting '/nix/store/0im5w3kbi8zz96mpavsa40lmdfii988b-user-environment'
deleting '/nix/store/393mi8mzvdyvddg64f710ip9r9jxy003-user-environment'
deleting '/nix/store/5mbgwc5fszpnspw1lfggpvnyypv7ryiv-user-environment.drv'
deleting '/nix/store/qfgjqfqhsbg2jx090c5751kgml2vdpmb-user-environment'
deleting '/nix/store/b3ayv8lxc2a96rpap1v34g2p7yq2afsx-env-manifest.nix'
deleting '/nix/store/3dcsr90w86w0mpmcav4mm2yy31gdk1i0-user-environment.drv'
deleting '/nix/store/f83cdyd1dwmfbp3kanmqr4h4vagl3a5y-env-manifest.nix'
deleting '/nix/store/di7r5855zdn1gg7w6ly7mc0sx6ksz2fn-user-environment.drv'
deleting '/nix/store/lhs0hv63vzqaqvajny3j4prrnynrf08h-env-manifest.nix'
deleting '/nix/store/8x0zx1p2k76abcmmp6la239ybwa7p1nd-my-hello-1.0-foo'
deleting '/nix/store/pppj7nh5s8l9wbmv6i3avyb9skzayyg9-my-hello-1.0'
deleting '/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 0.00 MiB
12 store paths deleted, 0.41 MiB freed

Recall that nix-store --delete needs to calculate GC roots to prevent me from accidentally breaking stuff, unless I pass --ignore-liveness.

Random side note: this was my first time using tmux-thumbs to copy those store paths and paste them to my command line and it was incredible. It felt great; I loved it. I have written about using tmux to select and copy text before, which is a pretty important part of my workflow in writing these posts. After publishing that, someone pointed me at tmux-thumbs, and I eagerly installed it.

But I was disappointed to find that it didn’t actually work very well out of the box. But I liked the idea so much that I forked it to handle the Unicode characters in my prompt properly, and to not strip colors/formatting from the screen (which I found very jarring). If anyone knows enough Rust to tell me how to do that correctly and wants to help me get those changes worthy of a PR, please get in touch. Otherwise it will probably be months before I get around to it.

Anyway, I’m really not sure why this deleted things I didn’t ask it to. I guess it deleted anything that referenced those paths as well? Yeah, that makes sense. It doesn’t want to leave them broken.

Okay. Now let’s try again, and see if we’re actually able to install this thing as a “bare” derivation.

$ nix-env -f default.nix -iA hello
installing 'my-hello-1.0'
don't know how to build these paths:
  /nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv
cannot build missing derivation '/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv'
error: build of '/nix/store/pyskbsiixv28jj1s01gijh0b0jpaymkb-my-hello-1.0.drv' failed

Aw. Darn. It seems it only worked before because I already had the .drv file in my store.

Okay. So I take back what I said before: derivation is magical, somehow, in a way I don’t fully understand. We’ll have to circle back. Let’s keep reading.

The very next function is also intriguing:

toDerivation :: Path -> Derivation
Converts a store path to a fake derivation.

I don’t know what that means. Let’s see it.

nix-repl> lib.attrsets.toDerivation "/foo"
«derivation ???»

nix-repl> lib.attrsets.toDerivation "/foo" // { type = "divination" }
error: syntax error, unexpected '}', expecting ';', at (string):1:59

nix-repl> lib.attrsets.toDerivation "/foo" // { type = "divination";
 }
error: getting status of '/foo': No such file or directory

Every single time I type a singleton set I leave off the semicolon. Usually I edit those errors out, but that’s sort of dishonest, and does not reflect my actual experience with Nix. Which involves a lot of being really annoyed about the semicolon strictness.

nix-repl> lib.attrsets.toDerivation "/nix/store/70pxcwpdiq7ddrk4w8ax
fl51s9xh9ahn-hello-2.10"
«derivation ???»

nix-repl> :p lib.attrsets.toDerivation "/nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10" // { type = "divination"; }
{
  name = "hello-2.10";
  out = «derivation ???»;
  outPath = "/nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10";
  outputName = "out";
  outputs = [ "out" ];
  type = "divination";
}

Okay? I guess? Don’t really know when I would use this. I probably never will.

Let’s see… lib.attrsets.optionalAttrs has the wrong type signature – it says optionalAttrs :: Bool -> AttrSet but it is actually optionalAttrs :: Bool -> AttrSet -> AttrSet. Pretty minor.

Honestly it seems like any helper I could possibly imagine is in here. There are so many functions. Incredibly specific ones:

zipAttrsWithNames :: [ String ] -> (String -> [ Any ] -> Any) -> [ AttrSet ] -> AttrSet 
recursiveUpdateUntil :: ( [ String ] -> AttrSet -> AttrSet -> Bool ) -> AttrSet -> AttrSet -> AttrSet

recursiveUpdate might be useful; it’s like the // operator but… recursive.

nix-repl> :p lib.attrsets.recursiveUpdate 
               { person = { name = "ian"; age = 15; }; }
               { person = { name = "young ian"; }; }
{ person = { age = 15; name = "young ian"; }; }

Random side note: tab completion in the nix repl is so fast. Compared to like shell tab completion, where I have to wait for noticeable milliseconds, it feels really good to press tab and have tab just do something. Why is my shell so slow? Is this oh-my-zsh’s fault?

Random other side note: I formatted that myself to make it more readable; as far as I can tell you can’t actually do that in nix repl:

nix-repl> :p lib.attrsets.recursiveUpdate \
error: syntax error, unexpected $undefined, expecting $end, at (string):1:30

Oh well.

The last “attribute set” function is very mysterious.

recurseIntoAttrs :: AttrSet -> AttrSet
Make various Nix tools consider the contents of the resulting attribute set when looking for what to build, find, etc.

This function only affects a single attribute set; it does not apply itself recursively for nested attribute sets.

attrs - an attribute set to scan for derivations.

Wha?

The example doesn’t exactly clear anything up. This is labeled “Example 5.34. Making Nix look inside an attribute set.”

{ pkgs ? import <nixpkgs> {} }:
{
  myTools = pkgs.lib.recurseIntoAttrs {
    inherit (pkgs) hello figlet;
  };
}

So, okay, we have a set that contains derivations. Sure.

Hmmmm. Something is occurring to me that might make this make sense.

Back in part 22, I found out – or I thought I found out – that I could make my “top level” Nix expression (i.e. nix-env -f expression) be a list or a nested list or whatever. And in fact, when I made it a list, every element of the list had to be a derivation (or a list or a set or whatever).

It now occurs to me that I didn’t really explore what happens when the top-level expression is a set – as it would be in real life all the time. And in fact clearly not everything has to be a derivation or a list of derivations or whatever, because the real top-level Nixpkgs expression contains, well, lib – a set full of functions.

So let’s return to that really quick, and see if we can understand the semantics a little better. And then we’ll take a look at recurseIntoAttrs.

Where we left it last time…

$ cat packages.nix

with import <nixpkgs> {};
[ [[hello] hello] { attribute = git; } ]

$ nix-env -qaf packages.nix
git-2.30.1
hello-2.10

Okay. So first let’s try some more stuff in attributes.

So we already saw this kind of failure:

with import <nixpkgs> {};
[
  [[hello] hello]
  [ hello (x: x + 1) ]
  { attribute = git; }
]

$ nix-env -qaf packages.nix
error: expression does not evaluate to a derivation (or a set or list of those)

But it turns out you can do this:

with import <nixpkgs> {};
[
  [[hello] hello]
  { attribute = git;
    inc = (x: x + 1);
  }
]

$ nix-env -qaf packages.nix
git-2.30.1
hello-2.10

For some reason you can put a function in a set, but you can’t put it in a list.

That’s very unexpected! I didn’t tried this before because it never occurred to me that Nix would treat them differently – especially given the error message I was getting: “expression does not evaluate to a derivation (or a set or list of those).”

So let’s see how that works…

with import <nixpkgs> {};
[
  { attribute = git;
    nested = { inherit hello; };
  }
]

$ nix-env -qaf packages.nix
git-2.30.1

Huh! It didn’t recurse into the set at all. Does it not recurse into lists either?

with import <nixpkgs> {};
[
  [hello [git]]
]

$ nix-env -qaf packages.nix
git-2.30.1
hello-2.10

Okay, it does recurse into lists. Just not sets.

Does it actually invoke functions with no arguments?

with import <nixpkgs> {};
[
  { attribute = git; }
  ({}: { inherit hello; })
]

$ nix-env -qaf packages.nix
git-2.30.1
hello-2.10

Yes. It does invoke functions with no arguments.

Even if they’re nested?

with import <nixpkgs> {};
[
  { inherit git;
    thunk = ({}: hello);
  }
]

$ nix-env -qaf packages.nix
git-2.30.1

No! Not if they’re nested. That’s kind of surprising to me. So functions of no arguments in lists get evaluated, but functions of no arguments in sets do not.

Why? Why is this? I can’t think of a principled explanation for this behavior.

Which is annoying, because I know I’m not going to remember all of these rules. Even already, I couldn’t give you a concise summary of what I’ve learned. The idea does not fit into a small number of words, which means that it is going to leak out of my head unless I spend a long time experiencing these rules and reinforcing them.

But I probably won’t. So I will just expect to be confused at some point in the future.

But okay. Lists are kind of– in practice, I don’t think lists show up that often. So let’s make the top-level expression a set, and just focus on the set rules for now.

with import <nixpkgs> {};
{
  nixpkgs = { inherit git; };
}

So this is something sort of like “reality.” Really the top-level expression should be a function, not a set, but whatever; let’s start here.

Now if the nesting rules I’ve learned so far still apply, this should give me nothing, right?

$ nix-env -qaf packages.nix

Okay! Good. I mean, weird, but also good.

But of course I would like for this to work. This certainly works for the “real” nixpkgs.

So at this point I know that I have lib.attrsets.recurseIntoAttrs, and presumably it will solve this problem for me.

But I also know that that’s just a library function. It takes a set and returns a set. So I should be able to write such a function myself.

But… what on earth would it look like?

let recurseIntoAttrs = attrs:
  ??? attrs;
in
with import <nixpkgs> {};
{
  nixpkgs = recurseIntoAttrs { inherit git; };
}

Presumably I don’t want to break the existing structure. I still want it to be a set. But I don’t really know how I would make this…

Hmm. I know that it invoke functions… but presumably it will also call sets with the __functor attribute? And then if that returns a list, then I can have this sort of dual-natured thing… I dunno this sounds dumb but let’s try it.

let recurseIntoAttrs = attrs:
  attrs // { __functor = ({}: builtins.attrValues attrs); };
in
with import <nixpkgs> {};
{
  nixpkgs = recurseIntoAttrs { inherit git; };
}

Odd observation: this file completely breaks the Nix syntax highlighting in Sublime Text. Although Nix itself has no problem with it:

$ nix-env -qaf packages.nix

Unfortunately, it did not work. Still nothing. (Note that I didn’t bother to actually recurse, just flatten one level, but whatever).

Oh, right, but I can’t actually make a __functor of “no arguments.” Because it will be called with the set itself. So let’s try that…

let recurseIntoAttrs = attrs:
  attrs // { __functor = (self: {}: builtins.attrValues attrs); };
in
with import <nixpkgs> {};
{
  nixpkgs = recurseIntoAttrs { inherit git; };
}

$ nix-env -qaf packages.nix

Still nothin'. Okay, I give up. Let’s see how the real thing works.

with import <nixpkgs> {};
{
  nixpkgs = lib.attrsets.recurseIntoAttrs { inherit git; };
}

$ nix-env -qaf packages.nix
git-2.30.1

Yes, okay, it did work… would have been surprising if it hadn’t. But how did it work?

nix-repl> lib.attrsets.recurseIntoAttrs { inherit git; }
{ git = «derivation /nix/store/fdx964jvsli6bj2lawxd1zkwhbzcsdxl-git-2.30.1.drv»;
  recurseForDerivations = true; }

Oh come on. That’s cheating. You cheated. The function definition:

/* Make various Nix tools consider the contents of the resulting
   attribute set when looking for what to build, find, etc.

   This function only affects a single attribute set; it does not
   apply itself recursively for nested attribute sets.
 */
recurseIntoAttrs =
  attrs: attrs // { recurseForDerivations = true; };

That’s just… yeah, that hardly seems fair.

Anyway, we get a little bit more insight into the nature of nix repl’s stringification of derivations. Apparently there are certain magical attributes that it will also print out? So far we know drvPath and recurseForDerivations. Are there others? I don’t know.

Let’s keep going. We’ve finally arrived at:

5.1.3. String manipulation functions

Hopefully I will have a lot less to say about these.

intersperse :: a -> [a] -> [a]

Weird that that is considered a “string function,” when it is actually a generic list manipulation function.

Functions for making PATHs and RPATHs.

stringToCharacters :: string -> [string]
Convert a string to a list of characters (i.e. singleton strings). This allows you to, e.g., map a function over each character. However, note that this will likely be horribly inefficient; Nix is not a general purpose programming language. Complex string manipulations should, if appropriate, be done in a derivation. Also note that Nix treats strings as a list of bytes and thus doesn’t handle unicode.

Emphasis mine.

escapeShellArg :: string -> string
Quote string to be used safely within the Bourne shell.
escapeShellArg "esc'ape\nme"
=> "'esc'\\''ape\nme'"

And:

escapeShellArgs :: [string] -> string
Quote all arguments to be safely passed to the Bourne shell.
escapeShellArgs ["one" "two three" "four'five"]
=> "'one' 'two three' 'four'\\''five'"

Those seem pretty handy. Also functions for quoting strings for use in Nix.

lib.strings.addContextFrom
Appends string context from another string. This is an implementation detail of Nix.

Strings in Nix carry an invisible context which is a list of strings representing store paths. If the string is later used in a derivation attribute, the derivation will properly populate the inputDrvs and inputSrcs.

There is no type signature given, but from the example it seems to be:

addContextFrom :: Derivation -> string -> string

I dunno; seems interesting? Maybe I will return to this one day. I choose not to investigate it right now.

Most of these string functions are missing type signatures, actually. But they’re usually pretty obvious.

lib.strings.isStorePath works on a string or a derivation.

lib.strings.readPathsFromFile appears to be, like, find $path -type f, but with some massaging of the output. Weird.

Another function with side effects: lib.strings.fileContents :: path -> string. Weird. I would think of that as an IO function rather than a string function, but whatever.

5.1.4. Miscellaneous functions

Amusingly, this section is called “miscellaneous functions,” but the section’s ID is sec-functions-library-trivial, and it seems to be documenting lib.trivial, not lib.misc.

Basic utilities: the identity function, lib.trivial.id, the const function, etc.

There is this mysterious one:

lib.trivial.concat
note please don’t add a function like compose = flip pipe. This would confuse users, because the order of the functions in the list is not clear. With pipe, it’s obvious that it goes first-to-last. With compose, not so much.

x Function argument

y Function argument

No explanation of what it does! Just a note not to add compose. I assume it’s reverse function application, based on that, but I have to look in the source…

Nope! It’s just the function version of the ++ operator. Just concatenates lists. Huh. I think something went wrong in the documentation here. That note should be attached to the previous definition, lib.trivial.pipe, and the arguments are just wrong.

Oh, no, ha, by “function arguments” it means “argument to the function” not “a function that is the argument.” Haha. All the rest of the lifted operators have the same argument documentation.

We have bitwise operations (which are not available as infix operators).

Nix calls the function version of // mergeAttrs. (I would call it update, and the updateRecursive function seems to agree with me.)

Then we get some things that are not simple function functions:

nix-repl> lib.trivial.version
"21.05pre274251.f5f6dc053b1"

nix-repl> lib.trivial.release
"21.05"

nix-repl> lib.trivial.codeName
"Okapi"

nix-repl> lib.trivial.versionSuffix
"pre274251.f5f6dc053b1"

nix-repl> lib.trivial.inNixShell
false

And then:

lib.trivial.splitByAndCompare :: 
  (a -> bool) ->
  (a -> a -> int) ->
  (a -> a -> int) ->
  (a -> a -> int)

(Formatting mine.)

I wouldn’t really call that trivial. Like, compared to const, you know.

And that’s followed by:

lib.trivial.importJSON :: path -> any

Which I have a really hard time calling trivial.

Okay, here’s a weird one:

lib.trivial.setFunctionArgs
Add metadata about expected function arguments to a function. The metadata should match the format given by builtins.functionArgs, i.e. a set from expected argument to a bool representing whether that argument has a default or not. setFunctionArgs : (a → b) → Map String Bool → (a → b)

This function is necessary because you can’t dynamically create a function of the { a, b ? foo, ... }: format, but some facilities like callPackage expect to be able to query expected arguments.

Huh. Note, like, Map, note Unicode arrows, note OCaml-style single colon type specifier, but Haskell-style type parameters? I don’t know. This one is a bit of an outlier, formatting-wise. Let’s see what it does:

nix-repl> { num, add ? 1 }: num + add
«lambda @ (string):1:1»

nix-repl> lib.trivial.setFunctionArgs ({ num, add ? 1 }: num + add) { num = false; add = true; }
{ __functionArgs = { ... }; __functor = «lambda @ /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/trivial.nix:324:19»; }

Hmm. Okay. So do things like callPackage have to check for a function or a set with a __functionArgs? Or does builtins.functionArgs, like, know this convention? Is __functionArgs special?

nix-repl> ({ num, add ? 1 }: num + add).__functionArgs
error: value is a function while a set was expected, at (string):1:1

So functions are distinct from sets with __functor, at least in this way.

nix-repl> builtins.functionArgs ({ num, add ? 1 }: num + add)
{ add = true; num = false; }

Okay.

nix-repl> builtins.functionArgs
            { __functor = (self: { num, add ? 1 }: num + add); }
error: 'functionArgs' requires a function, at (string):1:1

Hmm. But…

nix-repl> builtins.functionArgs {
            __functor = (self: { num, add ? 1 }: num + add);
            __functionArgs = { add = true; num = false; }; 
          }
error: 'functionArgs' requires a function, at (string):1:1

So still no. I guess that callPackage (and any other site) has to do some dynamic inspection to tease out functions and things what look like functions. I mean, there’s probably lib.trivial function that does exactly that, and I should be preferring it to builtins.functionArgs at all times. Right?

Yep; it’s the very next one listed:

lib.trivial.functionArgs
Extract the expected function arguments from a function. This works both with nix-native { a, b ? foo, ... }: style functions and functions with args set with ‘setFunctionArgs’. It has the same return type and semantics as builtins.functionArgs. setFunctionArgs : (a → b) → Map String Bool.

Whoever wrote these type signatures is gonna be heartbroken when they learn the word “set.”

And similarly we have lib.trivial.isFunction, which we should presumably prefer to builtins.isFunction in most cases.

5.1.5. List manipulation functions

Interestingly, there is a lib.lists.foldl, although I remember there being a builtins.foldl'.

foldl = op: nul: list:
let
  foldl' = n:
    if n == -1
    then nul
    else op (foldl' (n - 1)) (elemAt list n);
in foldl' (length list - 1);

Not sure why you would ever want that. But sure.

We also have lib.lists.foldl', which is just an alias for builtins.foldl'.

foldl' = builtins.foldl' or foldl;

Some more archaeology: apparently builtins.foldl' was called builtins.foldlat some point, before Nix decided to be more explicit about strictness.

Weird that imap is called lib.lists.imap0 and there is also a function called imap1, which iterates starting from 1. Huh.

We got a lotta good stuff here. lib.lists.toposort, lib.lists.listDfs, which… I dunno, it’s sort of a confusing API.

We have lib.lists.naturalSort, which sorts numbers the way you want, even if they are not padded with leading 0s:

naturalSort ["disk11" "disk8" "disk100" "disk9"]
=> ["disk8" "disk9" "disk11" "disk100"]

The rest are basic list functions. Not a ton to say.

5.1.6. Debugging functions

Lotta trace functions – lib.debug.traceIf, traceSeq, etc.

This one is worth mentioning:

lib.debug.runTests
Evaluate a set of tests. A test is an attribute set {expr, expected}, denoting an expression and its expected result. The result is a list of failed tests, each represented as {name, expected, actual}, denoting the attribute name of the failing test and its expected and actual results.

Used for regression testing of the functions in lib; see tests.nix for an example. Only tests having names starting with “test” are run.

Add attr { tests = ["testName"]; } to run these tests only.

I’m not sure what tests.nix it’s referring to:

$ find ~/src/nixpkgs -name tests.nix
/Users/ian/src/nixpkgs/pkgs/tools/misc/phoronix-test-suite/tests.nix
/Users/ian/src/nixpkgs/pkgs/development/interpreters/python/tests.nix
/Users/ian/src/nixpkgs/pkgs/development/libraries/physics/geant4/tests.nix

None of those sound very plausible. I assume this is stale.

I find ~/src/nixpkgs/lib/tests/misc.nix, which seems to contain some examples of tests written in this style:

testConst = {
  expr = const 2 3;
  expected = 2;
};

Simple enough.

That’s about all there is to say about debugs.

5.1.7. NixOS / nixpkgs option handling

We’re so close. It’s the last part. I can almost taste it.

lib.options.mkOption is so complicated that it doesn’t get a type signature and instead a page of text about its eleven optional arguments.

It doesn’t describe what this is, though.

nix-repl> lib.options.mkOption { defaultText = "foo"; }
{ _type = "option"; defaultText = "foo"; }

But it implies these are for… documentation generation? I grep around for uses of this, and it seems to be mostly for, like, /etc/nixos/configuration.nix – a NixOS thing that we’re not going to talk about because we haven’t gotten there yet.

So we’re gonna mostly skip this section.

Presumably these functions generate this massive web page, which is “Appendix A” in the NixOS manual but which gets it own page because it’s so big.

$ curl https://nixos.org/manual/nixos/stable/options.html | wc -c
10261076

Ten megabytes of documentation! Wow.

we did it

Okay. That’s the end. Wow. That was… that was a long section. We read a lot of functions. Turns out lib is pretty comprehensive. My main takeaway is that if I ever need a helper function, I should just grep this manual for the type signature and I will probably find it already defined in here.

Phew. I wonder what the sky looks like. I’m gonna go… not be on my computer for a while.

How do I print the contents of a derivation without setting the type to something else?
Why does the repl print the drvPath instead of the outputPath?
What’s the point of .drv files?
Why doesn’t nix-collect-garbage --dry-run do the thing I expect?
How do I add a shell.nix file to my gcroots?
What derivation attributes are special enough that nix repl prints them out?
What’s up with lib.strings.addContextFrom?