14.3. Arguments and Variables

We open on another annotated code snippet:

rec {

  hello = import ../applications/misc/hello/ex-1 {
    inherit fetchurl stdenv perl;
  };

  perl = import ../development/interpreters/perl {
    inherit fetchurl stdenv;
  };

  fetchurl = import ../build-support/fetchurl {
    inherit stdenv; ...
  };

  stdenv = ...;

}

This is explained as a simplified excerpt from the file pkgs/top-level/all-packages.nix, which seems to be a path in the nixpkgs repository.

I open that file to see what the full version looks like.

And it’s not… very similar. It’s a function, not a “set” expression. There’s a bunch of stuff that I can’t understand. I grep for hello and find this:

hello = callPackage ../applications/misc/hello { };

Hmm. Not very similar at all. The manual claims that file is “where all Nix expressions for packages are imported and called with the appropriate arguments,” but I sure don’t… see that.

Let’s look in the ../applications/misc/hello file and see if it’s any better.

Oh, it’s not a file. It’s a directory, with a default.nix. Fair enough. It looks a little bit like the example from the previous chapter:

{ lib, stdenv, fetchurl }:

stdenv.mkDerivation rec {
  pname = "hello";
  version = "2.10";

  src = fetchurl {
    url = "mirror://gnu/hello/${pname}-${version}.tar.gz";
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };

  doCheck = true;

  meta = with lib; {
    description = "A program that produces a familiar, friendly greeting";
    longDescription = ''
      GNU Hello is a program that prints "Hello, world!" when you run it.
      It is fully customizable.
    '';
    homepage = "https://www.gnu.org/software/hello/manual/";
    changelog = "https://git.savannah.gnu.org/cgit/hello.git/plain/NEWS?h=v${version}";
    license = licenses.gpl3Plus;
    maintainers = [ maintainers.eelco ];
    platforms = platforms.all;
  };
}

It seems that in real life (or perhaps in a newer version of Nix since the manual was written) the name attribute is split into pname and version. doCheck and meta are here now, and there is no perl – instead we take an argument called lib, which is used in a very weird-looking syntactic construct that I can’t begin to understand. It’s not really like anything I am familiar with, so my brain doesn’t know what to think of it.

No explicit builder either, but the manual did say that it was basically showing us the default builder, so it makes sense that it’s omitted in real life.

Anyway, there’s nothing here about how those arguments are provided. We’ll have to look at callPackage.

But I don’t know how to do that. I don’t know what that function is – it’s not declared in all-packages.nix, and I don’t know where it comes from, or how to trace/debug Nix expressions.

I am now sad about this section of the manual, because it appears to be lying to me, or to be very outdated. I don’t know how outdated, or how relevant anything of this is, or what I should believe right now. It makes me sad about the entire manual, too: I wonder what else I’ve learned is no longer true.

But I press on:

rec is familiar to me – like OCaml’s let rec or lisp’s let*, which we need so that packages can refer to one another. Sure. Makes sense. import does the thing you expect.

foo = import bar { inherit qux; }; looks like some weird new syntax where the import is scoped over the following block or something, because my brain is so used to parsing curly braces as blocks. I have to remind myself that this is just another function call: foo = (import bar) { inherit qux }, since our hello/default.nix contains a function declaration.

Aha! The manual actually explains callPackage:

Nixpkgs has a convenience function callPackage that imports and calls a function, filling in any missing arguments by passing the corresponding attribute from the Nixpkgs set, like this:

hello = callPackage ../applications/misc/hello/ex-1 { };

If necessary, you can set or override arguments:

hello = callPackage ../applications/misc/hello/ex-1 { stdenv = myStdenv; };

So, okay. The same thing is going on; the manual was not just lying to me. I resolve to be more patient in the future.

But this explanation raises a few new questions. This implies that Nix has some kind of reflection capability, to know which arguments to pass. That’s weird. Or maybe Nix ignores “extra” attributes when it pattern matches on the set (or whatever), so callPackage just passes it everything. That feels more plausible.

It also says it passes them “from the Nixpkgs set,” which is presumably the set that we are in the process of defining, right? So that’s… a little magical. I am at peace with lazy evaluation, though: mutually recursive values are familiar to me, so my brain is not broken by this notion. I could imagine someone being pretty confused by how this works, though, so it would be nice if the manual explained it (if that’s actually what’s happening here).

Moving on, I am intrigued by this bit:

fetchurl = import ../build-support/fetchurl { inherit stdenv; ... };

Naturally, I open up ~/src/nixpkgs/pkgs/build-support/fetchurl/default.nix to see what it looks like.

And it’s a big file! With a lot of comments. It actually inspires me to install Nix syntax highlighting to be able to parse it all more easily. It really helps to distinguish comments, I find. Maybe I should try a color scheme that just colors strings and comments differently, and doesn’t try to distinguish between reserved words and types or whatever else in my code. Is that a thing?

It is not a Nix thing. Focus, Ian.

Huh. Okay. So fetchurl is a function with a lot of optional arguments (I assume that’s what question marks mean), and it basically looks like this:

{ # URL to fetch.
  url ? ""

, # Alternatively, a list of URLs specifying alternative download
  # locations.  They are tried in order.
  urls ? []
, ...
}:

...

let
  urls_ =
    if urls != [] && url == "" then
      (if lib.isList urls then urls
       else throw "`urls` is not a list")
    else if urls == [] && url != "" then [url]
    else throw "fetchurl requires either `url` or `urls` to be set";
    ...
in

stdenvNoCC.mkDerivation {
    name =
      if showURLs then "urls"
      else if name != "" then name
      else baseNameOf (toString (builtins.head urls_));

    builder = ./builder.sh;

    ...
}

So it returns… a derivation?

I don’t really understand.

Except that I do, when I think about it for a minute.

In the example we saw in the last section, perl was a derivation, and apparently Nix built and installed it in the Nix store before it called our build script. And now I learn that src was also a derivation, although I didn’t realize it at the time, so of course Nix also built and installed it in the Nix store before it called our build script.

But what does it mean to “build” src? Easy: whatever it says in ~/src/nixpkgs/pkgs/build-support/fetchurl/builder.sh. Which does exactly the thing you’d expect: basically curl $url --output $out.

So, okay, great. Now this should mean that I have the source of hello in my store, somewhere, with the name hello-2.10.tar.gz – because that’s the url attribute declared in hello/default.nix.

But I don’t.

$ ls /nix/store | grep hello-2.10.tar.gz
8g94wcv0wjx2rldf25gvrrwdj6hgifiy-hello-2.10.tar.gz.drv

That’s very close, though. But it’s a file – I don’t know what kind of file – and it looks something like this:

Derive(
  [ ( "out"
    , "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz"
    , "sha256"
    , "31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b"
    ) ]
, [ ("/nix/store/5n184bf8dg2n7lbq0cvsl9dn4npk3508-bash-4.4-p23.drv", ["out"])
  , ("/nix/store/7nw4bgvwl4w03s6g15279hq52gbc30iy-stdenv-darwin.drv", ["out"])
  , ("/nix/store/qyy39sw31s9zkx1qq23jg1df79m71rqa-mirrors-list.drv", ["out"])
  , ("/nix/store/xn0d02v02gncbfp811ca0h4iq2qw5wff-curl-7.74.0.drv", ["dev"])
  ]
, ...much, much more elided...
)

I “pretty printed” it by hand, because I don’t know what format this is, but presumably it’s some kind of description of every input to the fetchurl “anonymous derivation” we created.

I realize that I probably installed hello from cache, so of course I don’t actually have the sources. I’ll need to build it.

I check man nix-env, again. There’s nothing in the --INSTALL section on how to force it to build from scratch. But I think I might know:

$ nix-env -iA nixpkgs.hello --option substituters ''
replacing old 'hello-2.10'
installing 'hello-2.10'

Ugh. No. That didn’t work. I return to the man page. It tells me how to fail if it can’t find a prebuilt version, but not how to ignore the cache. Nothing in here seems helpful, and after checking man 5 nix.conf I really do think that should work.

It occurs to me that this probably is a no-op, and I need to remove the package first. I do that, but get the same result. Then it occurs to me that I need to collect garbage, and try again:

$ nix-env -iA nixpkgs.hello --option substituters ''
DOWNLOADING MILLIONS OF PACKAGES

Oh gosh. Oh gosh. Mistake. I just wanted to build hello from source, but this is triggering a cascade of building everything from source – everything that hello needs, including such vital packages as:

/nix/store/jwm6dyvh483x60k47yn7dpm9gg39f0qc-bash44-013.drv
/nix/store/la57hnb0dn1w4bwym7y55klxpk33vzms-bash44-004.drv
/nix/store/m2dwqchar62dp6f1llplrbzy6rwy75n9-bash44-014.drv
/nix/store/n17k51dick3gp7ms6dzprmvp56q3rsnr-bash44-011.drv
/nix/store/s9jzppqzbi2ra6nhza6m2v8c2chbbqxs-bash44-018.drv
/nix/store/sc81n3y4wkzhk5br6f16waiqmnxyd996-bash44-009.drv
/nix/store/v9vfflw7j170q5qzv056a0gswzz1jx8n-bash44-015.drv
/nix/store/wfqvhn8bycl29011lmja7870lyq0grq9-bash44-023.drv

Don’t wanna. Maybe if I install some trivial program first with substituters, and then turn them off, I can make this work…

$ nix-env -iA nixpkgs.cat
error: attribute 'cat' in selection path 'nixpkgs.cat' not found

Boo. I’m vaguely afraid of installing all of coreutils. I try the next best thing:

$ nix-env -iA nixpkgs.sl
installing 'sl-5.05'
these paths will be fetched (3.16 MiB download, 15.43 MiB unpacked):
  /nix/store/a5mixw1swqprm1jm897kdl6nqfmx1j4k-gettext-0.21
  /nix/store/ps4yifn6srsqb2x1g3r3h7v0k91h88j1-sl-5.05
  /nix/store/rvp4bvhqbjnkb2zpshdifvkdscj328cy-ncurses-6.2
  /nix/store/vxr67g3i7m7d5gmfq231g2i4zfnqzj6d-gcc-10.2.0-lib
copying path '/nix/store/rvp4bvhqbjnkb2zpshdifvkdscj328cy-ncurses-6.2' from 'https://cache.nixos.org'...
copying path '/nix/store/a5mixw1swqprm1jm897kdl6nqfmx1j4k-gettext-0.21' from 'https://cache.nixos.org'...
copying path '/nix/store/vxr67g3i7m7d5gmfq231g2i4zfnqzj6d-gcc-10.2.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/ps4yifn6srsqb2x1g3r3h7v0k91h88j1-sl-5.05' from 'https://cache.nixos.org'...
building '/nix/store/wsnz4hwx5y6lbqis47snmrblxjxqvk9f-user-environment.drv'...
created 38 symlinks in user environment

After a mandatory sl viewing – always a delight – I realize the foolishness of my plan: that didn’t install all of its build time dependencies, because it didn’t need to. Hrm.

I mean, how long could a full build of hello take from scratch? Let’s find out.

I reach for my power cord, as my laptop attempts to achieve powered flight.

I’m thinking this is a “leave it overnight” situation.

I’m only doing this to prove to myself that the hello source will end up in the Nix store, but I’m pretty sure I’m right about that, and the cost of this is not totally worth confirming. But still I’m curious to see the error message I will inevitably get. Is there any chance this actually works? That I can build this, and all of its build dependencies, completely from scratch? No, right?

I could manually add a bunch of its build dependencies to the store, if I knew how to do that – but nix-env is all I know right now, and that really does not seem like the right way to do this. I resolve to just leave this running to see what happens, and carry on with it in the background.

Although, hmm. Looks like it broke already.

A curl invocation is hanging while trying to download:

http://www.opensource.apple.com/tarballs/bootstrap_cmds/bootstrap_cmds-121.tar.gz

It seems like a transient problem, but Nix’s fetchurl doesn’t notice that nothing is happening, so it just hangs. Annoying.

I kill it. And I check /nix/store, and I can see some sources here – /nix/store/rv68r40mwx7xa7vrlmanczkkjcnkc452-bash-4.4.tar.gz is a 9 meg file that I assume was produced by this.

So, okay, good enough: they’re just files.

Which makes me suspect that what I thought before was wrong: the thing that calls builder.sh doesn’t “create the destination directory.” I now think that the hello directory only existed because make install created it. And if it had created a file, that would be just fine too: so long as it puts something at $out, I think Nix will be happy.

Neat.

14.4. Building and Testing

This section opens with the words:

You can now try to build Hello. Of course, you could do nix-env -i hello, but you may not want to install a possibly broken package just yet. The best way to test the package is by using the command nix-build, which builds a Nix expression and creates a symlink named result in the current directory.

What do you mean “a possibly broken package? What do you mean “test the package”? I haven’t authored hello. I haven’t put files anywhere. I’ve run that command many times before, and I know that it just installs the package from Nixpkgs.

It seems that the manual is trying to walk me through creating my own package, but it is not doing that. It’s just showing me some code snippets. I have no idea how I would actually do anything with those snippets. What files they should go in, how to get nix-env to look at them, whatever.

But I think I might have enough information at this point to figure out what I need to do on my own. So I attempt to set up my own little test package.

First I’ll set up ~/scratch as a custom “channel.”

$ ln -s ~/scratch ~/.nix-defexpr/ianpkgs

Then I create a few files:

$ cd ~/scratch

$ cat builder.sh

source $stdenv/setup
mkdir $out
echo $'#!/usr/bin/env bash\necho "Hello, Nix!"' > $out/hello
chmod +x $out/hello

$ cat hello.nix

{ stdenv }:

stdenv.mkDerivation {
  name = "my-hello";
  builder = ./builder.sh;
}

$ cat default.nix

rec {
  hello = import ./hello.nix { inherit stdenv; };
  stdenv = import /Users/ian/src/nixpkgs/pkgs/stdenv;
}

Yes, I cheated at the end there. What else was I supposed to do?

Let’s see if this works. The manual says I can build it with nix-build instead of installing it.

$ nix-build -A ianpkgs.hello
error: attribute 'ianpkgs' in selection path 'ianpkgs.hello' not found

Okay umm hmm. I don’t know what that means. Ah: it seems nix-build does not use ~/.nix-defexpr at all, so my channel name is irrelevant. It uses whatever the default.nix is in the current directory. Got it:

$ cd ~/scratch
$ nix-build -A hello
error: value is a function while a set was expected, at /Users/ian/scratch/hello.nix:3:1

Hrm. I’m encouraged that it, like, got as far as it did? I basically just copied the example, though, so I am discouraged that it did not work.

Value is a function while a set was expected. Very mysterious.

Ah. It’s referring to stdenv, of course. I never called it.

But how do I call it? What do I pass it?

I try to figure this out by tracing what the “real” Nixpkgs does. The default.nix in Nixpkgs points me to pkgs/top-level/impure.nix. That file imports ./., which I take to mean default.nix. That file calls a function called boot – defined in ../stdenv/booter.nix – and passes it allPackages – which is a function – that is itself the result of calling the function ./stage.nix – with a couple arguments, one of which is lib, the other nixpkgsFun. Not so far, it isn’t. nixpkgsFun has a 25-line comment on its head explaining what it exists for, but not in terms that clarify anything here.

In the process of looking at nixpkgsFun I lose the call stack, and forget what I was doing, and give up.

This cannot be my first encounter with the Nix language.

So instead I cheat harder:

$ cat default.nix

rec {
  hello = import ./hello.nix { inherit stdenv; };
  stdenv = (import /Users/ian/src/nixpkgs).stdenv;
}

This is so close to working – but nixpkgs is apparently actually not a value, but a function. I have to call it like this:

$ cat default.nix

rec {
  hello = import ./hello.nix { inherit stdenv; };
  stdenv = (import /Users/ian/src/nixpkgs {}).stdenv;
}

That doesn’t make any sense to me, given that nix-env -iA nixpkgs.hello seems to treat it as a plain old “set.” Does nix-env actually expect a unit function that returns a set? If so, why? Why aren’t we just passing values around here?

I don’t know. But my thing works now!

$ nix-build -A hello
these derivations will be built:
  /nix/store/707rfjxpfxhbjqp1i6d2x1f0s654c1m5-my-hello.drv
these paths will be fetched (130.32 MiB download, 653.59 MiB unpacked):
    (long list that I've elided)
copying path '/nix/store/18k3k2pnqj0xqlnw9nk3mxldi939byx4-gawk-5.1.0' from 'https://cache.nixos.org'...
...many more "copying path lines"...
building '/nix/store/707rfjxpfxhbjqp1i6d2x1f0s654c1m5-my-hello.drv'...
/nix/store/my9k2rj6syj018mn9fnscy81amzw5sgs-my-hello

It worked! nix-build placed a symlink to the result in the current directory, which is… weird, but kind of neat? It could have just… printed it, or something? It also printed it. In fact:

$ nix-build -A hello 2>/dev/null
/nix/store/my9k2rj6syj018mn9fnscy81amzw5sgs-my-hello

It’s the only thing it prints to stdout. So, okay. I have my result. And I can…

$ result/hello
Hello, Nix!

Excellent.

My first Nix… thing.1 Despite Nix’s best efforts to prevent me from making a package, I eventually overcame. Proud of myself, I decide to install it, too, that it may grace my PATH forevermore:

$ nix-env -iA ianpkgs.hello
installing 'my-hello'
building '/nix/store/2yfxccscbx0rm1v51hr4lwhlpila25l1-user-environment.drv'...
created 39 symlinks in user environment

$ hello
zsh: command not found: hello

Gasp. I’ve installed the package – and I know the path in the Nix store works – but there’s apparently nothing about the package that lets nix-env know that it contains an executable, so it doesn’t get a symlink.

How do I fix that?

The example in the manual certainly didn’t do anything special. I open up a few packages. I swap name for pname on the off chance it means “program name” instead of, as I assumed, “printable name” or “pretty name” or something like that.

Nope.

I run find ~/src/nixpkgs/pkgs -name builder.sh, and look for interesting candidates that do not use the generic builder (which I suspect is somehow doing something to register executables, or something). I see pkgs/games/keen4, and nostalgia washes over me as I remember the thrill of temporarily turning potato guards into flowers. I mean, I played the original too, but something about throwing those little grenades was kisses fingers emoji. Don’t think I ever tried Goodbye Galaxy.

Curiosity gets the better of me – is this really? – so I give it a shot.

$ nix-env -iA nixpkgs.keen4
installing 'keen4'
error: Package ‘keen4’ in /nix/store/ys2x968psphylqb0gri928km5bny25ww-nixpkgs-21.05pre271444.9816b99e71c/nixpkgs/pkgs/games/keen4/default.nix:17 has an unfree license (‘unfree’), refusing to evaluate.

a) To temporarily allow unfree packages, you can use an environment variable
   for a single invocation of the nix tools.

     $ export NIXPKGS_ALLOW_UNFREE=1

b) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

Alternatively you can configure a predicate to whitelist specific packages:
  { nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
      "keen4"
    ];
  }

c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)

Huh. Umm. Okay?

$ NIXPKGS_ALLOW_UNFREE=1 nix-env -iA nixpkgs.keen4
installing 'keen4'
...
$ keen4

And bam. It just worked! DOSBox started up, I selected my sound card – hell yeah – loaded the main menu, and then quit. So I could get back to work.

The galaxy can wait.

Why is this in here? Was this game released into the public domain? No; it’s explicitly marked “unfree.” Did I just… did I just pirate something? Am I cybercriminal now?

You can’t prove anything.

Well, I didn’t learn what I wanted to learn. There’s nothing different about that package. I look through builder.sh and don’t notice it doing anything different from what I’m doing. By random luck, it also just writes a shell script (that launches DOSBox and does some other stuff) and chmod +xs it. Nostalgia has guided me well this night.

As a result, I learn the right way to shebang: I should have used $SHELL, not /usr/env/bin bash. Now you’re thinking with Nix.

But why is keen4 on my PATH, when hello is not?

I double check ~/.nix-profile/bin and confirm – again – that it still isn’t there. But in the process of doing so, I just happen to notice that there is a ~/.nix-profile/hello – a symlink to /nix/store/vyn32fklp5ign13w21s9hl58a62dwlky-my-hello-1.0/hello!

Aha, I realize: it’s symlinking everything I install, but with the same directory structure as the directory. I modify my builder.sh:

$ cat builder.sh

source $stdenv/setup

mkdir -p $out/bin

cat >$out/bin/hello <<EOF
#!$SHELL
echo "Hello, Nix!"
EOF

chmod +x $out/bin/hello

Try again:

$ nix-env -iA ianpkgs.hello
replacing old 'my-hello-1.0'
installing 'my-hello-1.0'
these derivations will be built:
  /nix/store/0jld2igd44rgs1mdi59h34zbx6i04qrh-my-hello-1.0.drv
building '/nix/store/0jld2igd44rgs1mdi59h34zbx6i04qrh-my-hello-1.0.drv'...
building '/nix/store/ns4gzk4al99ki5glsfgm7h2bha48zkmk-user-environment.drv'...
created 41 symlinks in user environment

And hold my breath:

$ hello
Hello, Nix!

Like looking into your child’s eyes for the very first time.

This realization was actually random luck. I don’t know how long that would have taken me if I didn’t tab-complete so aggressively, or if I hadn’t set up my tab completion to show a full directory listing. For posterity, this is what my shell looks like when I tab complete:

$ nix-env --rollback
switching from generation 29 to 28

$ ll ~/.nix-profile/<TAB> # <- my cursor is still on this line
Library@       bin/           etc/           hello@         lib@
libexec@       manifest.nix@  share/

I just happened to notice that hello@ while tabbing over to bin/. Phew. (The rollback was to put it back in the “bad” state, for this historical reenactment, before I fixed the builder.sh.)

I have to say: this seems obvious, now that I know how it works. There’s nothing “telling” Nix to add a symlink to bin/: when you install something, everything in the build’s output is symlinked, no matter what it looks like. (I tested it by creating a non-executable file as well, just to be sure). This also explains all the other stuff in ~/.nix-profile – presumably it’s just there because it’s there. The directory structure isn’t significant to nix-env, it’s just the way the installed subset of my /nix/store happens to look.

But this was completely not obvious to me before I discovered it. I really wish that some part of the manual had explained how Nix assembles the user environment before I got this far. Who knows how long that might have taken me to figure out, had fate not smiled this day.

But okay. We ran the sample code in the manual, sort of. After much deliberation. Are we done? No. Now we get to read the words that come after it. This is referring to the command nix-build -A hello:

The -A option selects the hello attribute. This is faster than using the symbolic package name specified by the name attribute (which also happens to be hello) and is unambiguous (there can be multiple packages with the symbolic name hello, but there can be only one attribute in a set named hello).

Okay: by “symbolic name” I assume this is the prefix of the name attribute that precedes the version number – I chose "my-hello" for that to see what would happen. But there doesn’t seem to be any way to call nix-build with a “symbolic name,” so this paragraph feels weird and confusing. I read man nix-build to confirm, and yeah. That’s not a thing nix-build can do.

I assume that this is supposed to be referring to the 30-second-long way to install packages with nix-env -i hello, and that somehow over the course of changes to the manual it now exists in a section talking about nix-build. And I now understand why nix-env -i hello is slower than nix-env -iA nixpkgs.hello: in the first command, hello is a string, and we have to search through every derivation for a matching string. In the second, hello is an attribute name, and we just have to look it up. And yes, it is still absolutely insane that the default behavior is to use the symbolic name.

I also learn:

nix-build registers the ./result symlink as a garbage collection root, so unless and until you delete the ./result symlink, the output of the build will be safely kept on your system. You can use nix-build’s -o switch to give the symlink another name.

Indeed:

$ ll /nix/var/nix/gcroots/auto
qqrfq5975mbpf5p9xpsqwhfpyj7q5jkk@ -> /Users/ian/scratch/result

$ rm /nix/var/nix/gcroots/auto/qqrfq5975mbpf5p9xpsqwhfpyj7q5jkk

And a previous theory of mine is confirmed:

Nix has transactional semantics. Once a build finishes successfully, Nix makes a note of this in its database: it registers that the path denoted by out is now “valid”. If you try to build the derivation again, Nix will see that the path is already valid and finish immediately. If a build fails, either because it returns a non-zero exit code, because Nix or the builder are killed, or because the machine crashes, then the output paths will not be registered as valid. If you try to build the derivation again, Nix will remove the output paths if they exist (e.g., because the builder died half-way through make install) and try again. Note that there is no “negative caching”: Nix doesn’t remember that a build failed, and so a failed build can always be repeated. This is because Nix cannot distinguish between permanent failures (e.g., a compiler error due to a syntax error in the source) and transient failures (e.g., a disk full condition).

I thought that whole paragraph was nice and clear and well-written so I copied it verbatim.

The manual then tells me that there are per-store-path locks, so I don’t need to worry about running multiple nix-foo commands at the same time. I sort of figured this was the case, but good. I learn that – because the locks are per store path – I can have Nix build multiple store paths in parallel with -j. Ain’t nobody got time to type that all the time, so man 5 nix.conf teaches me to set max-jobs = auto in my nix.conf if I want to live life to the fullest. But I still haven’t made a nix.conf, so I do nothing.

And that’s the end of the section.

I feel pretty good. I feel like I learned a lot here – I got to build my first package; I got to see how my user-environment is actually put together. I learned that “derivation” appears to just mean “thing that puts files in /nix/store,” and exists at a more generic level than “software packages.” I think this is a really important insight that the manual has so far not said anything about. I am very glad that I thought to look up fetchurl.

I do wish the manual had made all of this a bit easier, but hey. That’s why we’re here, right?

14.5. Generic Builder Syntax

This post should be over, but there’s only one section left before we hit a very different chapter, so I’m tacking it on here.

We basically just learn how to use the “generic builder,” for packages that do the classic configure/make/make install dance.

buildInputs="$perl"

source $stdenv/setup

genericBuild

This is equivalent to the original example from the last post. I see here a reason why we source $stdenv/setup ourselves, instead of having it be automatic: so that we can set variables and stuff before we do so.

I learn that I can set buildInputs to be a list of things and Nix will infer if they should be binaries on our PATH or header search path elements or “so on.” “So on?” What does that mean? A footnote expands:

How does it work? setup tries to source the file pkg/nix-support/setup-hook of all dependencies. These “setup hooks” can then set up whatever environment variables they want; for instance, the setup hook for Perl sets the PERL5LIB environment variable to contain the lib/site_perl directories of all inputs.

Huh. I find perl in ~/src/nixpkgs/pkgs/development/interpreters/perl. There is no nix-support subdirectory, but there is a file called setup-hook.sh. I suspect documentation rot: this file does, in fact, configure PERL5LIB:

addPerlLibPath () {
    addToSearchPath PERL5LIB $1/lib/perl5/site_perl
}

addEnvHooks "$hostOffset" addPerlLibPath

No idea what $hostOffset is. Interesting that this doesn’t configure the environment, but “schedules” an environment configuration to take place at the appropriate time. Sure. Still not totally clear on “hooks,” but being familiar with the term and seeing it (apparently) used in the familiar way here, I assume I understand what’s happening here.

Continuing on, I learn that you can specify buildInputs as a list (? array?) in the argument to mkDerivation instead of the script, which makes the script itself unnecessary – I can now omit the builder attribute, and the default build will take place, and everything will be the same.

I don my obnoxious pedant hat to say this: “Generic Builder Syntax” is a bad section title. We didn’t learn any syntax here. We learned the “Generic Builder Semantics.” I take my obnoxious pedant hat off. So there.

Okay. Now we’re actually done.

  • What exactly does callPackage call functions with?
  • Why does import /Users/ian/src/nixpkgs give me a function back?
  • Why is there a Commander Keen game in nixpkgs? How is that allowed?
  • How do I make my own ianpkgs without importing things from the “main” nixpkgs? What would a minimal default.nix look like?
  • What is $hostOffset in a setup hook?

  1. Did I actually get this right on my first try? No. I didn’t. I forgot that bash needs $'...' quoted strings to interpret escape characters, because zsh does not. But I went back and changed the historical record because this just seemed like distracting noise. I already feel terrible that I lied to you. ↩︎