The next few chapters take place in the standard environment “part,” but we have left the standard environment “chapter.” I suppose that that these are details only relevant to stdenv.mkDerivation? Or I shouldn’t read too much into the grouping here.

Chapter 7. Meta-attributes

We learn that meta attributes are not passed to the derivation’s builder, and thus they do not trigger a rebuild when they change. So that’s nice. It would be pretty annoying if changing the maintainer caused the hash of a package to change.

We already learned about passthru, whose attributes are also not passed along to the derivation, but which are slightly different than meta in that passthru attributes still wind up attached to the top-level derivation. meta attributes get no such treatment:

nix-repl> pkgs.mercurial.maintainers
error: attribute 'maintainers' missing, at (string):1:1

nix-repl> pkgs.mercurial.meta.maintainers
[ { ... } ]

Nixpkgs teaches us the --json flag to nix-env -q. That’s something we discovered for ourselves, way back in the day. No mention of the apparently-vestigial --meta flag.

Then we see a list of “expected” meta attributes.

description
longDescription
branch
homepage
downloadPage
changelog
license
maintainers
priority
platforms
tests
timeout
hydraPlatforms
broken
updateWalker

Mostly very straightforward. priority is bad and unnecessary and it should feel bad. But we don’t really have to use it.

tests is weird because it’s not meta.tests, it’s passthru.tests, but for some reason it’s documented here, as a “meta attribute,” instead of documented when we learned about passthru. So.

timeout seems like… a weird thing to be here. Who respects timeout? All Nix operations? I hope not. I hope that’s only, like, a CI thing.

$ cat sleeper.nix

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "timeout-tester-1.0";
  builder = ./sleeper.sh;
}

$ cat sleeper.sh

source $stdenv/setup

sleep 5

echo "hi" > "$out"

$ nix-build sleeper.nix
these derivations will be built:
  /nix/store/whgkpqbpm9mzzknvdbkd8xinx7akrl2r-timeout-tester-1.0.drv
building '/nix/store/whgkpqbpm9mzzknvdbkd8xinx7akrl2r-timeout-tester-1.0.drv'...
/nix/store/lnd1qc1liv7bcr505kz3hpccgvyfx80c-timeout-tester-1.0

Okay. That works. But when I add a timeout:

$ cat sleeper.nix

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "timeout-tester-1.0";
  builder = ./sleeper.sh;
  meta = { timeout = 1; };
}

$ nix-build sleeper.nix
/nix/store/lnd1qc1liv7bcr505kz3hpccgvyfx80c-timeout-tester-1.0

Er, right. Changing meta attributes doesn’t change the hash, so the package is the same…

$ rm result

$ nix-store --delete /nix/store/lnd1qc1liv7bcr505kz3hpccgvyfx80c-timeout-tester-1.0
finding garbage collector roots...
removing stale link from '/nix/var/nix/gcroots/auto/qqrfq5975mbpf5p9xpsqwhfpyj7q5jkk' to '/Users/ian/scratch/result'
...
1 store paths deleted, 0.00 MiB freed

Second time’s the charm…

$ nix-build sleeper.nix                                             these derivations will be built:
  /nix/store/whgkpqbpm9mzzknvdbkd8xinx7akrl2r-timeout-tester-1.0.drv
building '/nix/store/whgkpqbpm9mzzknvdbkd8xinx7akrl2r-timeout-tester-1.0.drv'...
/nix/store/lnd1qc1liv7bcr505kz3hpccgvyfx80c-timeout-tester-1.0

Okay. So this value is not magic to Nix. This is something that certain systems – I’m guessing just Hydra – can choose to look at.

hydraPlatforms says:

The list of Nix platform types for which the Hydra instance at hydra.nixos.org will build the package. (Hydra is the Nix-based continuous build system.) It defaults to the value of meta.platforms. Thus, the only reason to set meta.hydraPlatforms is if you want hydra.nixos.org to build the package on a subset of meta.platforms, or not at all.

In my head “Hydra” means “the thing that populates cache.nixos.org.” But I don’t really know why I think that. Maybe it isn’t? Maybe it’s really just a piece of test infrastructure? I hope that the manual will go into that a bit, eventually.

updateWalker is the last interesting one – it’s only relevant to a script called update-walker.sh, which appears to exist to automate version bumping of packages.

Lastly the chapter talks about different “generic” licenses to use if you don’t know the actual license a package is available under (?). I don’t know why you would ever do that. Nothing too interesting here.

Chapter 8. Multiple-output packages

We already know that a derivation can specify multiple outputs. Now we learn why:

The main motivation is to save disk space by reducing runtime closure sizes; consequently also sizes of substituted binaries get reduced. Splitting can be used to have more granular runtime dependencies, for example the typical reduction is to split away development-only files, as those are typically not needed during runtime. As a result, closure sizes of many packages can get reduced to a half or even much less.

Hmm. Okay. I can definitely understand, like, “I depend on perl but I don’t depend on the perl man pages,” or something. I don’t really understand why “development-only files” would be included in the first place. I feel like I don’t really understand the distinction between runtime and build-time dependencies in Nix.

The attribute meta.outputsToInstall is used to determine the default set of outputs to install when using the derivation name unqualified.

That seems like kind of an important meta field! Maybe it could have been documented in the section about meta fields, that we just read?

I remember from my testing that this defaulted to the first entry in the output list. But that does not appear to be the case. The manual links directly to the source:

# If the packager hasn't specified `outputsToInstall`, choose a default,
# which is the name of `p.bin or p.out or p`;
# if he has specified it, it will be overridden below in `// meta`.
#   Note: This default probably shouldn't be globally configurable.
#   Services and users should specify outputs explicitly,
#   unless they are comfortable with this default.
outputsToInstall =
  let
    hasOutput = out: builtins.elem out outputs;
  in [( lib.findFirst hasOutput null (["bin" "out"] ++ outputs) )]
    ++ lib.optional (hasOutput "man") "man";

As a native English speaker, it feels very jarring to see a gendered pronoun applied to “the packager.” I had to re-read that a couple times before I parsed it; the strange line break after the semicolon didn’t help.

Anyway, it seems that I remembered incorrectly. Interesting that you can specify a bin output and it will take precedence over the default output name out.

There’s a little bit about NixOS here that I don’t really understand; I skipped it. Then we get:

nix-env lacks an easy way to select the outputs to install. When installing a package, nix-env always installs the outputs listed in meta.outputsToInstall, even when the user explicitly selects an output.
nix-env silenty disregards the outputs selected by the user, and instead installs the outputs from meta.outputsToInstall. For example,
$ nix-env -iA nixpkgs.coreutils.info
installs the "out" output (coreutils.meta.outputsToInstall is [ "out" ]) instead of the requested "info".

Well I’ll be darned. It then says that the only way to actually install specific outputs is to add an overlay that changes that meta attribute.

But hang on! I remember a weird meta-specific override facility hidden deep in the Nix command reference. I couldn’t really come up with a decent use-case for it at the time. But could I use that to change the outputs that nix-env installs? That seems like exactly what this might be useful for.

Let’s test it. We’ll use coreutils, as we did already:

$ nix-env -iA nixpkgs.coreutils.info -p ~/scratch/profile
installing 'coreutils-8.32'
building '/nix/store/8z1s1xm3b3ljy5mkw30r216n9197mcjs-user-environment.drv'...
created 2 symlinks in user environment

$ tree -l ~/scratch/profile/
/Users/ian/scratch/profile
├── bin -> /nix/store/cpvjym13fdglv6zmdr5xav20g5rbafbx-coreutils-8.32/bin
│   ├── [ -> coreutils
│   ├── b2sum -> coreutils
│   ├── base32 -> coreutils
│   ├── base64 -> coreutils
│   ├── basename -> coreutils
│   ├── basenc -> coreutils
│   ├── cat -> coreutils
│   ├── chcon -> coreutils
│   ├── chgrp -> coreutils
│   ├── chmod -> coreutils
│   ├── chown -> coreutils
│   ├── chroot -> coreutils
│   ├── cksum -> coreutils
│   ├── comm -> coreutils
│   ├── coreutils
│   ├── cp -> coreutils
│   ├── csplit -> coreutils
│   ├── cut -> coreutils
│   ├── date -> coreutils
│   ├── dd -> coreutils
│   ├── df -> coreutils
│   ├── dir -> coreutils
│   ├── dircolors -> coreutils
│   ├── dirname -> coreutils
│   ├── du -> coreutils
│   ├── echo -> coreutils
│   ├── env -> coreutils
│   ├── expand -> coreutils
│   ├── expr -> coreutils
│   ├── factor -> coreutils
│   ├── false -> coreutils
│   ├── fmt -> coreutils
│   ├── fold -> coreutils
│   ├── groups -> coreutils
│   ├── head -> coreutils
│   ├── hostid -> coreutils
│   ├── id -> coreutils
│   ├── install -> coreutils
│   ├── join -> coreutils
│   ├── kill -> coreutils
│   ├── link -> coreutils
│   ├── ln -> coreutils
│   ├── logname -> coreutils
│   ├── ls -> coreutils
│   ├── md5sum -> coreutils
│   ├── mkdir -> coreutils
│   ├── mkfifo -> coreutils
│   ├── mknod -> coreutils
│   ├── mktemp -> coreutils
│   ├── mv -> coreutils
│   ├── nice -> coreutils
│   ├── nl -> coreutils
│   ├── nohup -> coreutils
│   ├── nproc -> coreutils
│   ├── numfmt -> coreutils
│   ├── od -> coreutils
│   ├── paste -> coreutils
│   ├── pathchk -> coreutils
│   ├── pinky -> coreutils
│   ├── pr -> coreutils
│   ├── printenv -> coreutils
│   ├── printf -> coreutils
│   ├── ptx -> coreutils
│   ├── pwd -> coreutils
│   ├── readlink -> coreutils
│   ├── realpath -> coreutils
│   ├── rm -> coreutils
│   ├── rmdir -> coreutils
│   ├── runcon -> coreutils
│   ├── seq -> coreutils
│   ├── sha1sum -> coreutils
│   ├── sha224sum -> coreutils
│   ├── sha256sum -> coreutils
│   ├── sha384sum -> coreutils
│   ├── sha512sum -> coreutils
│   ├── shred -> coreutils
│   ├── shuf -> coreutils
│   ├── sleep -> coreutils
│   ├── sort -> coreutils
│   ├── split -> coreutils
│   ├── stat -> coreutils
│   ├── stdbuf -> coreutils
│   ├── stty -> coreutils
│   ├── sum -> coreutils
│   ├── sync -> coreutils
│   ├── tac -> coreutils
│   ├── tail -> coreutils
│   ├── tee -> coreutils
│   ├── test -> coreutils
│   ├── timeout -> coreutils
│   ├── touch -> coreutils
│   ├── tr -> coreutils
│   ├── true -> coreutils
│   ├── truncate -> coreutils
│   ├── tsort -> coreutils
│   ├── tty -> coreutils
│   ├── uname -> coreutils
│   ├── unexpand -> coreutils
│   ├── uniq -> coreutils
│   ├── unlink -> coreutils
│   ├── uptime -> coreutils
│   ├── users -> coreutils
│   ├── vdir -> coreutils
│   ├── wc -> coreutils
│   ├── who -> coreutils
│   ├── whoami -> coreutils
│   └── yes -> coreutils
├── libexec -> /nix/store/cpvjym13fdglv6zmdr5xav20g5rbafbx-coreutils-8.32/libexec
│   └── coreutils
│       └── libstdbuf.so
└── manifest.nix -> /nix/store/jsmdf1ik7lqm5ajsrjc7bll5158m167g-env-manifest.nix

Indeed: no info pages. But I should be able to use…

$ nix-env --set-flag outputsToInstall '["info"]' coreutils -p ~/scratch/profile
setting flag on 'coreutils-8.32'
error: this derivation has bad 'meta.outputsToInstall'

Uhhhh. Well that’s… what? I’m guessing that --set-flag only takes a literal string, doesn’t it. I can’t give it a list. Anyway, it didn’t do anything:

$ nix-env -qaA nixpkgs.coreutils --json -p ~/scratch/profile | jq '."nixpkgs.coreutils".meta.outputsToInstall'
[
  "out"
]

Note the annoying nested quoting in that jq expression because the key in the result object is the literal string nixpkgs.coreutils.

So, okay. No. It’s not a thing. Still no idea what the point of --set-flag is.

Alright, the next section is a little dense:

In the Nix language the individual outputs can be reached explicitly as attributes, e.g. coreutils.info, but the typical case is just using packages as build inputs.

Okay, sure. Just not if you’re using nix-env. Got it.

When a multiple-output derivation gets into a build input of another derivation, the dev output is added if it exists, otherwise the first output is added. In addition to that, propagatedBuildOutputs of that package which by default contain $outputBin and $outputLib are also added. (See Section 8.4.2, “File type groups”.)

Okay. No idea what $outputBin or $outputLib are, but we’ll get to that very soon.

In some cases it may be desirable to combine different outputs under a single store path. A function symlinkJoin can be used to do this. (Note that it may negate some closure size benefits of using a multiple-output package.)

Okay. No explanation of that function or how to use it, but ⌘F tells me we’ll get to it in a few more chapters.

In nixpkgs there is a framework supporting multiple-output derivations. It tries to cover most cases by default behavior. You can find the source separated in <nixpkgs/pkgs/build-support/setup-hooks/multiple-outputs.sh>; it’s relatively well-readable. The whole machinery is triggered by defining the outputs attribute to contain the list of desired output names (strings).
outputs = [ "bin" "dev" "out" "doc" ];

Wait.

This is a Nixpkgs thing?

No. I’ve written a multi-output derivation myself. I didn’t use mkDerivation or anything else. outputs is respected by the builtin.derivation function. Hmm. I don’t understand. Let’s keep going.

I learn that I should usually make the first output the one with the binaries in it, because apparently derivations stringify to the path of their first output, so this lets me write $drv/bin/exe. I learn an exception to this is glibc, whose first output is libs. Because you’re more likely to be referencing a library. Easy enough.

Now we learn about “file type groups.”

The support code currently recognizes some particular kinds of outputs and either instructs the build system of the package to put files into their desired outputs or it moves the files during the fixup phase. Each group of file types has an outputFoo variable specifying the output name where they should go. If that variable isn’t defined by the derivation writer, it is guessed – a default output name is defined, falling back to other possibilities if the output isn’t defined.

The manual really doesn’t describe what this means. What are “variables” here? Environment variables? Are these supposed to be derivation attributes? It lists a bunch of examples of the form:

$outputDev is for development-only files. These include C(++) headers, pkg-config, cmake and aclocal files. They go to dev or out by default.

I don’t know what it means by “they go to dev or out by default.” What does… what?

$outputMan is for man pages (except for section 3). They go to man or $outputBin by default.

Am I supposed to say, like outputBin = "my-executables"; or something? Are aliases for output “roles”? I don’t understand.

I am very suspicious, however, about outputDev and the error I got back in part 13. Oh, yeah! That’s exactly it – it seems that dev and out are not particularly special. From multiple-outputs.sh:

_overrideFirst outputDev "dev" "out"
_overrideFirst outputBin "bin" "out"

If I had specified dev (but not out), it would have complained about “bin or out” next. And then lib, and then doc… but by providing an output named out, it’s used as a fallback for every one of these.

And yes, I now see what these things are. These are either shell variables or environment variables that are relevant to the $stdenv/setup.sh script. And it seems I could have avoided those errors by specifying every one of outputDev, outputBin, outputLib, and outputDoc to be the string name of my weird unconventional output, if I’m reading the script correctly.

Neat.

The chapter ends with some “common caveats:” no circular references. Don’t needlessly split things up if depending on one output will cause you to depend on all the other outputs. Nothing too exciting.

And that’s it! Pretty good chapters. Made some things more concrete for me.

Is Hydra responsible for populating https://cache.nixos.org?
Does Nix not explicitly distinguish between runtime and build-time dependencies?
If I use an override or an overlay or whatever, does that only affect nix-env? Or does that affect any Nix command?

How to Learn Nix, Part 29:Derivations in detail

Chapter 7. Meta-attributes

Chapter 8. Multiple-output packages

How to Learn Nix, Part 29:
Derivations in detail