The section we’ve all been waiting for…

15.4. Derivations

The most important built-in function is derivation, which is used to describe a single derivation (a build action). It takes as input a set, the attributes of which specify the inputs of the build.

Okaaaaaay. I have only seen mkDerivation, which I assume is a thin wrapper around derivation that fills in the system argument for me. We see the term “build action” again – I remember that from the glossary. Still don’t understand what that term means.

I wish that this section would just show me a type signature for derivation, but it doesn’t – instead it uses English prose to describe the arguments.

I try to piece it into something more familiar:

derivation ::
    { system   : String
    , name     : String
    , builder  : Path | Derivation
    , ?args    : [String]
    , ?outputs : [String]
    } -> Derivation

Okay, maybe that’s not any more clear. Lotta strings. I’m using the ? prefix to denote optional arguments, and [String] to mean “list of Strings,” in case that wasn’t clear.

Note that I am using -> in the “returns” sense and not the “implication” sense of the last post, although it’s fun to think about the equivalence between the two.

Anyway, the manual actually does a really good job describing all of these arguments, so I won’t bother repeating all that here.

Here’s what gave me pause:

The manual calls the system argument a “platform identifier,” and gives examples like "i686-linux" or "x86_64-darwin". There is a footnote on this that says:

To figure out your platform identifier, look at the line “Checking for the canonical Nix system name” in the output of Nix’s configure script.

What? I have to build Nix to find out my platform identifier? That can’t be true.

The build can only be performed on a machine and operating system matching the platform identifier. (Nix can automatically forward builds for other platforms by forwarding them to other machines; see Chapter 16, Remote Builds.)

Okay. Very excited to learn about remote builds, as my laptop is, as previously stated, ancient. It would be nice to be able to use it as a thin client to some beefy machine set up in my haunted basement. Maybe throw in some Tailscale, take those builds to go, finally get a setup that can keep up with my metropolitan lifestyle. I am probably not actually going to do any of that, but it’s nice to know that I could.

Interesting that builder can be a derivation. I am into that.

Every attribute is passed as an environment variable to the builder. Attribute values are translated to environment variables as follows:

Strings and numbers are just passed verbatim.

A path (e.g., ../foo/sources.tar) causes the referenced file to be copied to the store; its location in the store is put in the environment variable. The idea is that all sources should reside in the Nix store, since all inputs to a derivation should reside in the Nix store.

A derivation causes that derivation to be built prior to the present derivation; its default output path is put in the environment variable.

Lists of the previous types are also allowed. They are simply concatenated, separated by spaces.

true is passed as the string 1, false and null are passed as an empty string.

Okay! This is neat. We are finally learning something about derivations. To test my knowledge, I want to find the builder.sh file that I have referenced in a path back when I was building my-hello.

$ ls /nix/store | grep builder.sh
4k7n61apbmyw8wxh0914nyixpig7cia7-builder.sh
720ikgx7yaapyb8hvi8lkicjqwzcx3xr-builder.sh
95sl3n27s7rnx6l2kdd5yr83kfrx6574-builder.sh
9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
bzh096qsv3rrvflsx6jd8jh7yqmxnjck-builder.sh
c38g59q6rq924qbry9r4s1lrjxx57lrh-builder.sh
d50riha84mzs9kk64ppcvx0qjhv5v5s3-builder.sh
dsyj1sp3h8q2wwi8m6z548rvn3bmm3vc-builder.sh
hrdhq9yxx8ajjyqr83jri3x7kn5yk78c-builder.sh
l62skckvzzlv3nb7klhcqcrs7ni1pi1w-builder.sh
qf3mzpvsmkrw963xchbivcci06078n13-builder.sh
r5n0lagqhs6hbaqm0sizqq1hw07wp5y7-builder.sh
vh1cpdrgrbipscys6lzr0sfcamr0r2vk-builder.sh
x1l9ji3aynq3ckvsmr7gjq2r5x57ybqc-builder.sh
ygkmp0ydpmg01f7bhx95ck02kzx7sxij-builder.sh

Ah. Ha. Okay. Right.

$ find /nix/store -maxdepth 1 -name '*builder.sh' -exec grep -q 'Hello, Nix' {} \; -print
/nix/store/r5n0lagqhs6hbaqm0sizqq1hw07wp5y7-builder.sh
/nix/store/ygkmp0ydpmg01f7bhx95ck02kzx7sxij-builder.sh
/nix/store/hrdhq9yxx8ajjyqr83jri3x7kn5yk78c-builder.sh
/nix/store/95sl3n27s7rnx6l2kdd5yr83kfrx6574-builder.sh
/nix/store/d50riha84mzs9kk64ppcvx0qjhv5v5s3-builder.sh

Okay! Those are all the different versions of the script I used as I was playing around with this the other day – before and after I used $SHELL; evidence of when I got my string escaping wrong and then lied about it… neat. Alright.

I said args was a list of strings up there when I was writing my type signature, but now that I think about it, it’s probably a list of things that become strings. The manual just says “It should be a list.” Wonderfully explicit. I bet if I used a path, that path would be copied to the store. Let’s find out:

$ echo "Hello, Nix!" > message.txt

$ cat hello.nix

{ stdenv }:

stdenv.mkDerivation {
  pname = "my-hello";
  version = "1.0";
  builder = ./builder.sh;
  args = [ ./message.txt ];
}

$ cat builder.sh

source $stdenv/setup

mkdir -p $out/bin

messagefile=$1

cat >$out/bin/hello <<EOF
#!$SHELL
cat $messagefile
EOF

chmod +x $out/bin/hello

And the moment of truth:

$ nix-build -A hello
these derivations will be built:
  /nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv
building '/nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv'...
/nix/store/nmmdalhj24wsz2zi342ymavzqklywk97-message.txt: line 1: Hello,: command not found
builder for '/nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv' failed with exit code 127
error: build of '/nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv' failed

Huh. Why didn’t that work? Why did it try to run message.txt? I was right that it copied the file into the store:

$ cat /nix/store/nmmdalhj24wsz2zi342ymavzqklywk97-message.txt
Hello, Nix!

But I have no idea why it treated it like a script, or what point of the process tried to run that. I re-read my builder.sh, and convince myself I didn’t just do something idiotic. I sure don’t think I did…

Hmm. Very weird.

I try nix-build -A hello -v, but it adds no information that is relevant. I’m not really sure how to debug this – I’m not sure where to look.

Let’s try to see if it fails before or after it starts running builder.sh, to try to figure out if it’s my fault:

$ mv builder.sh backup

$ echo 'exit 1' > builder.sh

$ nix-build -A hello
these derivations will be built:
  /nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv
building '/nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv'...
/nix/store/nmmdalhj24wsz2zi342ymavzqklywk97-message.txt: line 1: Hello,: command not found
builder for '/nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv' failed with exit code 127
error: build of '/nix/store/9jq2pcra3bnb80aqi18jvl9mlg2qk9rw-my-hello-1.0.drv' failed

Huh! So no. Not my doing. Something in Nix itself is causing it to try to execute this like a script. What the heck?

I realize that maybe this is a difference between the derivation function and the mkDerivation function, but that seems crazy to me and I dismiss the idea.

Is this something about using args, or is every file treated as executable? I try to just include it as an environment variable:

$ cat hello.nix

{ stdenv }:

stdenv.mkDerivation {
  pname = "my-hello";
  version = "1.0";
  builder = ./builder.sh;
  messagefile = ./message.txt;
}

$ nix-build -A hello
these derivations will be built:
  /nix/store/dl4lrbicrw7gqnymxg4lnp5gg7zs75cm-my-hello-1.0.drv
building '/nix/store/dl4lrbicrw7gqnymxg4lnp5gg7zs75cm-my-hello-1.0.drv'...
/nix/store/kjjs6wpb4b8rj8rbvw3zc82hyqzxzf4s-my-hello-1.0

$ cat result/bin/hello
#!/nix/store/5y6lcfjghk5kbv4782vi7w79vz60gsyn-bash-4.4-p23/bin/bash
cat /nix/store/nmmdalhj24wsz2zi342ymavzqklywk97-message.txt

(Not shown: I also updated builder.sh in the obvious way.)

So… okay. Weird. Something weird about args. Maybe it is a mkDerivation thing? I hope to find out soon.

The manual describes how I can have multiple outputs – neat. I assume that the “default” value of outputs is ["out"] and I can override that – that is, if I specify something else, I will no longer get the $out environment variable. But it’s possible that will still exist, and point to what the manual calls the “default output path.”

To test this, I added outputs = ["foo"]; to my hello.nix, and it failed with a super weird error:

$ nix-build -A hello
these derivations will be built:
  /nix/store/k5r28886afz00l485rsajmzprsp9273r-my-hello-1.0.drv
building '/nix/store/k5r28886afz00l485rsajmzprsp9273r-my-hello-1.0.drv'...
Error: _assignFirst found no valid variant!
builder for '/nix/store/k5r28886afz00l485rsajmzprsp9273r-my-hello-1.0.drv' failed with exit code 1
error: build of '/nix/store/k5r28886afz00l485rsajmzprsp9273r-my-hello-1.0.drv' failed

No idea what that means. Nix has variants? I don’t know if that’s an internal error from within the Nix binary or if it’s some Nix function in Nixpkgs failing. I’m guessing at this point that mkDerivation is much more than just a thin wrapper around derivation, and that trying to apply this section to my hello.nix is dumb.

So I try using derivation instead:

$ cat hello.nix

derivation {
  system = "x86_64-darwin";
  name = "my-hello-1.0";
  builder = ./builder.sh;
  messagefile = ./message.txt;
  outputs = ["foo"];
}

$ cat default.nix

{ hello = import ./hello.nix; }

Wow! Much simpler, because I don’t need stdenv anymore, which means hello.nix doesn’t even need to be a function. Definitely wish I had started here, and worked my way up to the conveniences that mkDerivation provides. Obviously I don’t actually want to hardcode my system, but it’s nice to start here, you know?

It doesn’t quite work, though:

$ nix-build -A hello
these derivations will be built:
  /nix/store/0rpwly8bj4fww51l25dw5a0g21f2p7b7-my-hello-1.0.drv
building '/nix/store/0rpwly8bj4fww51l25dw5a0g21f2p7b7-my-hello-1.0.drv'...
sandbox-exec: execvp() of '/nix/store/s9fsa5qqvxy0z6zaqchdq5scidpkb01r-builder.sh' failed: Permission denied
builder for '/nix/store/0rpwly8bj4fww51l25dw5a0g21f2p7b7-my-hello-1.0.drv' failed with exit code 71

Okay fine:

$ chmod +x ./builder.sh

$ nix-build -A hello
these derivations will be built:
  /nix/store/1ypqmpdpv0pqpbv4xsq342w8cyk2gis6-my-hello-1.0.drv
building '/nix/store/1ypqmpdpv0pqpbv4xsq342w8cyk2gis6-my-hello-1.0.drv'...
/nix/store/wpqa59i0akzibpdcbpgz0xpgc68lmdmc-builder.sh: line 3: mkdir: command not found
/nix/store/wpqa59i0akzibpdcbpgz0xpgc68lmdmc-builder.sh: line 7: /nix/store/g2dwvn98qciaj087nf82xn99qidhv87m-my-hello-1.0-foo/bin/hello: No such file or directory
/nix/store/wpqa59i0akzibpdcbpgz0xpgc68lmdmc-builder.sh: line 12: chmod: command not found
builder for '/nix/store/1ypqmpdpv0pqpbv4xsq342w8cyk2gis6-my-hello-1.0.drv' failed with exit code 127
error: build of '/nix/store/1ypqmpdpv0pqpbv4xsq342w8cyk2gis6-my-hello-1.0.drv' failed

And, ah, right. Fair enough. I got rid of source $stdenv/setup, and as a result… I have no PATH. Fair enough! I put stdenv back:

$ cat hello.nix

{ stdenv }:

derivation {
  inherit stdenv;
  system = "x86_64-darwin";
  name = "my-hello-1.0";
  builder = ./builder.sh;
  messagefile = ./message.txt;
  outputs = ["foo"];
}

And try again:

$ nix-build -A hello
these derivations will be built:
  /nix/store/zqxm1nw8rixr2wxrs0fisa55vkckrnds-my-hello-1.0.drv
building '/nix/store/zqxm1nw8rixr2wxrs0fisa55vkckrnds-my-hello-1.0.drv'...
/nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh: line 27: syntax error near unexpected token `<'
builder for '/nix/store/zqxm1nw8rixr2wxrs0fisa55vkckrnds-my-hello-1.0.drv' failed to produce output path '/nix/store/4bmh4ifmd4px1jj5bg9ic9dw3r1b2r0a-my-hello-1.0-foo'

Ummmm. Well that’s… pretty unexpected. Let’s take a looksee:

$ sed -n '27p' /nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh
    done < <(find $prefix -type l -print0)

Ah. Okay. I could see myself being very confused at this point, but I happen to know that subshell expansion does not exist in sh, although it is does exist in bash and zsh and every shell anyone would actually ever actually use. So I assume that by using derivation Nix is defaulting my shell to sh instead of bash – which I honestly think is totally reasonable. But it seems that this script is not “POSIX shell compliant,” which is something that I would never ask a script to be but that I understand some people get very upset about. This would seem fine to me in pretty much any case except this one: if Nix doesn’t work without bash, Nix shouldn’t default to using sh.

If that’s even what’s happening here.

If it is, I would consider this to be the first bug that I’ve found. It might seem sort of annoying and nit-picky to call this a bug, but it’s too late. The words are already out of my mouth.

I wonder how to change this, and glance at the manual – and fortunately the very next thing explains both of the problems I have:

The function mkDerivation in the Nixpkgs standard environment is a wrapper around derivation that adds a default value for system and always uses Bash as the builder, to which the supplied builder is passed as a command-line argument.

Ah. Okay. Nix, I am sorry I called it a bug. I feel bad and dumb. Obviously Nix is not doing anything with sh. It’s just invoking my executable, which has no shebang, so macOS is defaulting to using sh. I chmod +x’d without thinking. I briefly think about editing the historical record so that I come off looking a little smarter, but I take the noble road.

And I now realize that make-symlinks-relative is not something that Nix is calling out to in the process of building a derivation. It’s something that my script is calling, when I source $stdenv/setup. This wasn’t clear to me from the error message, but it became clear as soon as I read the above quoted paragraph. Nix isn’t written in shell. It doesn’t need to execute shell scripts in the course of doing a build. That’s on me.

It also explains why my args thing didn’t work when I used it with mkDerivation: my message.txt was being passed as an argument to bash, so of course this happened. I think that that’s really confusing behavior: I would expect mkDerivation to do something to adjust args at the same time it’s adjusting builder so that it can be used the way I expected – but whatever. I get what’s going on now.

Well, I can adjust it myself, right?

$ cat default.nix

rec {
  hello = import ./hello.nix { inherit stdenv bash; };
  stdenv = (import /Users/ian/src/nixpkgs {}).stdenv;
  bash = (import /Users/ian/src/nixpkgs {}).bash;
}

$ cat hello.nix

{ stdenv, bash }:

derivation {
  inherit stdenv;
  system = "x86_64-darwin";
  name = "my-hello-1.0";
  builder = bash;
  args = [ ./builder.sh ];
  messagefile = ./message.txt;
  outputs = ["foo"];
}

But this doesn’t work, even though the documentation explicitly said the builder could be a derivation:

$ nix-build -A hello
these derivations will be built:
  /nix/store/2xan4hspswqjh5lq3bc12ibrls49pxvg-my-hello-1.0.drv
building '/nix/store/2xan4hspswqjh5lq3bc12ibrls49pxvg-my-hello-1.0.drv'...
sandbox-exec: execvp() of '/nix/store/5y6lcfjghk5kbv4782vi7w79vz60gsyn-bash-4.4-p23' failed: Permission denied
builder for '/nix/store/2xan4hspswqjh5lq3bc12ibrls49pxvg-my-hello-1.0.drv' failed with exit code 71
error: build of '/nix/store/2xan4hspswqjh5lq3bc12ibrls49pxvg-my-hello-1.0.drv' failed

Because apparently the bash derivation refers to the bash directory, and of course I can’t execute that. I’m not sure how to say that my builder is bash/bin/bash. Unless…

$ cat hello.nix

{ stdenv, bash }:

derivation {
  inherit stdenv;
  system = "x86_64-darwin";
  name = "my-hello-1.0";
  builder = bash + "/bin/bash";
  args = [ ./builder.sh ];
  messagefile = ./message.txt;
  outputs = ["foo"];
}

Huh. There’s no way that that will work, right? Nix isn’t going to be able to coerce that derivation into a string – and builder is supposed to be a path anyway, not a string. But let’s try it, just in case.

$ nix-build -A hello
these derivations will be built:
  /nix/store/bcmb0jix0ml925bkxqfxp4nh2fp47g16-my-hello-1.0.drv
building '/nix/store/bcmb0jix0ml925bkxqfxp4nh2fp47g16-my-hello-1.0.drv'...
Error: _assignFirst found no valid variant!
builder for '/nix/store/bcmb0jix0ml925bkxqfxp4nh2fp47g16-my-hello-1.0.drv' failed with exit code 1
error: build of '/nix/store/bcmb0jix0ml925bkxqfxp4nh2fp47g16-my-hello-1.0.drv' failed

That actually almost makes it seem like it worked? Surprisingly? Maybe paths are not actually a separate type but just a different syntax for writing strings?

Or maybe it didn’t. I dunno. We’re back to the error I got the first time I tried to set outputs. Huh. Let’s look at mkDerivation and try to see how it does this.

Er… yeah. We can do that, right?

$ find ~/src/nixpkgs -name 'mkDerivation.nix'
/Users/ian/src/nixpkgs/pkgs/development/libraries/qt-5/mkDerivation.nix

Um, no. I eventually find it at ~/src/nixpkgs/pkgs/stdenv/generic/make-derivation.nix, though. Fair enough.

Wow this file is 380 lines long. It’s like… it’s like a whole thing.

I see that it doesn’t actually use bash; it uses:

builder = attrs.realBuilder or stdenv.shell;
args = attrs.args or ["-e" (attrs.builder or ./default-builder.sh)];

Aha. I learn that there is a stdenv.shell. And I learn that it always passes through args literally if it’s set, even if realBuilder is not specified. I think this is super confusing, but okay. At least I understand what went wrong.

I recreate the conditions of mkDerivation as well as I can:

$ cat hello.nix

{ stdenv }:

derivation {
  inherit stdenv;
  system = "x86_64-darwin";
  name = "my-hello-1.0";
  builder = stdenv.shell;
  args = [ "-e" ./builder.sh ];
  messagefile = ./message.txt;
  outputs = ["foo"];
}

And, unsurprisingly, get the same _assignFirst error. -v tells me nothing and I have no idea how to like… make Nix show me a call stack (?). I think about googling it, but hold off. Instead I rg around the tree to see if it’s a helper, and I get a hit: ~/src/nixpkgs/pkgs/build-support/setup-hooks/multiple-outputs.sh:

# Assign the first string containing nonempty variable to the variable named $1
_assignFirst() {
    local varName="$1"
    local REMOVE=REMOVE # slightly hacky - we allow REMOVE (i.e. not a variable name)
    shift
    while (( $# )); do
        if [ -n "${!1-}" ]; then eval "${varName}"="$1"; return; fi
        shift
    done
    echo "Error: _assignFirst found no valid variant!"
    return 1 # none found
}

Nice! I assumed that was a Nix function. But it’s just a shell function. That means I know how to debug this:

args = [ "-x" "-e" ./builder.sh ];

And after looking through stderr, I see:

_assignFirst outputDev dev out

Aha. Okay. So outputs did cause Nix not to set $out – but $stdenv/setup needs $out to be set. So… that’s dumb.

Incidentally, I check in man nix-build and see that -vvvv would have given me debug info, if this had been a Nix function.

So okay. Should we… set $out? Add it to our list of outputs? Should we not use $stdenv/setup? Let’s see what other builders do.

Huh!

$ rg 'outputs\s*=\s*\[' ~/src/nixpkgs | head
/Users/ian/src/nixpkgs/pkgs/servers/sql/postgresql/default.nix:    outputs = [ "out" "lib" "doc" "man" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/postgresql/ext/postgis.nix:  outputs = [ "out" "doc" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/cockroachdb/default.nix:  outputs = [ "out" "man" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mysql/5.7.x.nix:  outputs = [ "out" "static" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mariadb/default.nix:  outputs = [ "out" "man" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mariadb/default.nix:  outputs = [ "out" "man" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mariadb/connector-c/default.nix:  outputs = [ "out" "dev" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/virtuoso/6.x.nix:  outputs = [ "out" "dev" "doc" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mysql/8.0.x.nix:  outputs = [ "out" "static" ];
/Users/ian/src/nixpkgs/pkgs/servers/hitch/default.nix:  outputs = [ "out" "doc" "man" ];
/Users/ian/src/nixpkgs/nixos/tests/taskserver.nix:    outputs = [ "out" "cacert" "cert" "key" "crl" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/cockroachdb/default.nix:  outputs = [ "out" "man" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mysql/8.0.x.nix:  outputs = [ "out" "static" ];
/Users/ian/src/nixpkgs/pkgs/servers/sql/mariadb/default.nix:  outputs = [ "out" "man" ];

Okay. Investigation reveals that there is a single package that does not have an out as one of its outputs:

$ grep outputs ~/src/nixpkgs/pkgs/tools/compression/zstd/default.nix
  outputs = [ "bin" "dev" ]

And it sets dev, so… yeah. I have no idea what dev should be or what outputDev is gonna be used for. But the example in the manual shows a package that has neither dev nor out – which is, apparently, incompatible with $stdenv/setup. Super cool.

So, okay. That’s something that I learned, that is not called out anywhere. I guess I’ll just add out and see what happens? I assume it will complain that I don’t write anything to out. Yep:

$ nix-build -A hello
these derivations will be built:
  /nix/store/6nz24fd7f00hq0yi2mjabgwzdfyvsj1l-my-hello-1.0.drv
building '/nix/store/6nz24fd7f00hq0yi2mjabgwzdfyvsj1l-my-hello-1.0.drv'...
builder for '/nix/store/6nz24fd7f00hq0yi2mjabgwzdfyvsj1l-my-hello-1.0.drv' failed to produce output path '/nix/store/x0yjnl7rbgzy3f64fnw8vdmighcjkp93-my-hello-1.0'
error: build of '/nix/store/6nz24fd7f00hq0yi2mjabgwzdfyvsj1l-my-hello-1.0.drv' failed

So, okay. I added echo "whatever" > $out to my builder. And now it works!

$ nix-build -A hello
these derivations will be built:
  /nix/store/kyhgjvkf73h8yf8xqzcds6sikbq6y95q-my-hello-1.0.drv
building '/nix/store/kyhgjvkf73h8yf8xqzcds6sikbq6y95q-my-hello-1.0.drv'...
/nix/store/b36nq1fqc7iw812pvgif9czsammi6pqv-my-hello-1.0-foo

It prints the “default output path,” the one with the -foo suffix, instead of the unsuffixed path. Which has a different hash:

$ cat /nix/store/pd66andz12y4klq1cwcz7nw75n43q7w8-my-hello-1.0
whatever

It’s interesting to me that the hash is different – clearly it’s not a hash of the inputs but a hash of like… the inputs and the output name? Or the inputs modulo the output name? I will need to investigate further.

So this is all fine. I learned that $out does not always refer to the default output path, although it… seems like it should.

I continue through the manual. It describes exactly how Nix invokes builder, and what environment variables it provides. NIX_BUILD_TOP is the working directory for the build. All of TMPDIR, TEMPDIR, TMP, and TEMP are also set to that, to reduce the chances of polluting somewhere else. Isn’t Unix beautiful?

PATH is set to /path-not-set to prevent shells from initialising it to their built-in default value.

I like this, but /PATH-not-set would be a lot more clear in error messages – there are lots of different paths, but only one PATH.

HOME is set to /homeless-shelter to prevent programs from using /etc/passwd or the like to find the user’s home directory, which could cause impurity. Usually, when HOME is set, it is used as the location of the home directory, even if it points to a non-existent path.

I am not usually one to stifle whimsy but I think if I got an error containing the string /homeless-shelter it would take me a lot longer to figure out what was happening than if I got /HOME-not-set. But this is probably not something I will ever encounter in practice.

For each output declared in outputs, the corresponding environment variable is set to point to the intended path in the Nix store for that output. Each output path is a concatenation of the cryptographic hash of all build inputs, the name attribute and the output name. (The output name is omitted if it’s out.)

Got it, thank you. Although I wouldn’t be me if I didn’t say something about the lack of an Oxford comma.

A log of the combined standard output and error is written to /nix/var/log/nix.

I look in this directory and find a pretty weird structure.

$ tree /nix/var/log/nix | head
/nix/var/log/nix
└── drvs
    ├── 03
    │   └── abqbaf96ngihm2zlai0pvfnw8a6zx8-bootstrap-stage0-coreutils.drv.bz2
    ├── 0g
    │   └── 79v3nbrz76m5x2dnwarpvd4xcj778c-bash44-019.drv.bz2
    ├── 0j
    │   └── ld2igd44rgs1mdi59h34zbx6i04qrh-my-hello-1.0.drv.bz2
    ├── 0r
    │   └── pwly8bj4fww51l25dw5a0g21f2p7b7-my-hello-1.0.drv.bz2

I assume this is like the thing git does in .git/objects so that you don’t have a single directory with millions of inodes.

I tree to look at the logs from the build I just did, but…

$ find /nix/var/log/nix -name '*7iw812pvgif9czsamm*'

I cannot find them. I give up quickly, and move on.

If the build was successful, Nix scans each output path for references to input paths by looking for the hash parts of the input paths. Since these are potential runtime dependencies, Nix registers them as dependencies of the output paths.

Neat. I actually really like this? Like, you could try to “defeat” this if you wanted to, and write software that is deliberately incompatible with Nix. But I assume if you were packaging that package, Nix would give you some other way to explicitly register runtime dependencies? Gotta be.

Note that possible setuid and setgid bits are cleared. Setuid and setgid programs are not currently supported by Nix. This is because the Nix archives used in deployment have no concept of ownership information, and because it makes the build result dependent on the user performing the build.

My last job involved a fair amount of weird low-level userspace networking stuff, and we mostly used capabilities, but I’m pretty sure we had to distribute some helpers as setuid executables for reasons I can’t remember. It’s been a while. Anyway, the manual says nothing about capabilities or extended attributes in here.

15.4.1. Advanced Attributes

In which I learn that my type signature at the beginning of this post was woefully incomplete.

There’s some neat stuff here. You can whitelist runtime dependencies with allowedReferences to prevent dependency creep. Love that. allowedRequisites for a whitelist on the entire dependency closure. disallowedReferences and disallowedRequisites – the thing you expect.

exportReferencesGraph is useful for builders that want to do something with the closure of a store path. Examples include the builders in NixOS that generate the initial ramdisk for booting Linux (a cpio archive containing the closure of the boot script) and the ISO-9660 image for the installation CD (which is populated with a Nix store containing the closure of a bootable NixOS configuration).

I doubt I will ever need that, and don’t bother to play around with it. But it sounds neat.

impureEnvVars is interesting – apparently fetchurl uses this to keep HTTP proxy stuff when you’re building. That’s nice.

This attribute is only allowed in fixed-output derivations, where impurities such as these are okay since (the hash of) the output is known in advance. It is ignored for all other derivations.

Very cool. How do I make my own “fixed-output derivation”?

Ah, it’s the very next thing listed: outputHash, outputHashAlgo, and outputHashMode.

For fixed-output derivations, on the other hand, the name of the output path only depends on the outputHash* and name attributes, while all other attributes are ignored for the purpose of computing the output path. (The name attribute is included because it is part of the path.)

So presumably a fixed-output derivation cannot specify multiple outputs. But that’s okay: I assume this is mostly useful for things like fetchurl that are not “actual” packages.

passAsFile is interesting – in case an attribute value is particularly long, or contains special characters or something, you can receive it as a path instead of a literal value. What happens if I passAsFile = [ "passAsFile" ]? Nothing. That’s not weird. That is not a strange loop.

preferLocalBuild is for “trivial builders where the cost of doing a download or remote build would exceed the cost of building locally”. It doesn’t seem to disable the binary cache, but something called “distributed building,” which I have not gotten to yet.

However I could use allowSubstitutes to disable the binary cache. “This is useful for very trivial derivations (such as writeText in Nixpkgs) that are cheaper to build than to substitute from a binary cache.”

Alright.

I still don’t know what a derivation is. I mean, I get it – I get how it works, what it does, how to make them. But I don’t know what it is in the Nix language: is it a separate type? Is it a “set” with some magic __i_am_derivation attribute? What is it?

I don’t know. Hopefully I will soon.

How do I find my platform identifier without building Nix from scratch?
If builder is a derivation that produces multiple files, what binary does it run?
Are paths actually a different type at runtime, or are they really strings?
Why does mkDerivation pass through args literally?
Why does $stdenv/setup require one of $dev or $out to exist?
How do I actually find the logs I want in /nix/var/log/nix?
How can I explicitly register a runtime dependency?
How do extended file system attributes work in Nix?

How to Learn Nix, Part 13:Derivations

15.4. Derivations

15.4.1. Advanced Attributes

How to Learn Nix, Part 13:
Derivations