I don’t know about the word “pills.”

The word “pills” makes me think of medicine; it makes me think of the idiom “a bitter pill to swallow;” it makes me think of sweaters that just came out of the wash. It does not make me think of… documentation, or learning.

I assume that the title is meant to evoke “bite-sized” or something like that. I don’t know. It’s weird a weird choice, for a series.

But it’s distinctive. I’ll give it that. If you say “Nix pills,” people know what you’re talking about. No matter how unsavory the connotation, I suppose it’s well-marketed.

Anyway. Nix Pills started life as a series of blog posts in 2014, written by a real human person writing for the greater good of all humankind, and I already feel a little bad for saying I think the title is weird.¹ But it has since become a part of the official documentation – at least, there is an updated(?) version of the blog series hosted on https://nixos.org and linked to from the official “Learn” page, so I’m calling it official documentation, and I’m gonna read it.

But… the audience of the Pills series appears to be someone who has not encountered Nix before. And I have been encountering Nix for the better part of four months now, so I am not really going to be a good judge of what’s confusing, what could be clearer, etc. In other words, I am not really doing this to audit the documentation – I don’t have any intention of trying to improve these blog posts from 2014 – but rather because I think I might learn something.

So this is going to go a lot faster than a typical post: most of these concepts are things that I’ve encountered before.

Let’s dive in.

Chapter 1. Why You Should Give it a Try

Most of this chapter is concerned with extolling the virtues of Nix and motivating the series itself; there is little in the way of information for an existing Nix user.

This series aims to complement the existing explanations from the more formal documents.

This is interesting. I’ve heard Nix Pills described as a “textbook, from first principles” approach to explaining Nix – i.e., something I would consider more traditionally formal than the actual manuals. Which didn’t seem very formal at all. But maybe they were more so, when this was written?

Considering Nixpkgs is a completely new repository of all the existing software, with a completely fresh concept, and with few core developers but overall year-over-year increasing contributions, the current state is more than acceptable and beyond the experimental stage. In other words, it’s worth your investment.

It is weird to think of Nixpkgs – one of the largest package repositories in the world – as being new in 2014. 2014 wasn’t that long ago! But I look through the Nixpkgs history, and see that the first commit was created in 2003. Okay. That’s closer to what I expected. Using the term to mean novel, I suppose.

Chapter 2. Install on Your Running System

Installation. Actually our first nugget of information:

Yes, Nix also has a database. It’s stored under /nix/var/nix/db. It is a sqlite database that keeps track of the dependencies between derivations.

The schema is very simple: there’s a table of valid paths, mapping from an auto increment integer to a store path.

Then there’s a dependency relation from path to paths upon which they depend.

You can inspect the database by installing sqlite (nix-env -iA sqlite -f '<nixpkgs>') and then running sqlite3 /nix/var/nix/db/db.sqlite.

I don’t remember the manual ever describing where to find the database, or what was in it, or really anything about it. It’s nice to call that out up front. It doesn’t really explain why the database exists, but this is just the installation chapter – perhaps it will go into more detail later (although it’s already gone into more detail than the manual ever did).

Apparently this chapter is quite a bit more than installation. It describes the user environment used by nix-env commands, it describes channels and Nixpkgs. It even teases:

We’ll talk about manifest.nix more in the next article.

Neat! That’s something I still don’t totally understand, and that the manual certainly never explained.

Chapter 3. Enter the Environment

Explains that a user environment is a directly full of symlinks to every file in all of your installed derivations, which I think is very useful to point out: that was something I did not understand from reading the manual, and had to discover for myself in the process of installing the first derivation I wrote.

There isn’t anything like apt which solves a SAT problem in order to satisfy dependencies with lower and upper bounds on versions. There’s no need for this because all the dependencies are static: if a derivation X depends on a derivation Y, then it always depends on it. A version of X which depended on Z would be a different derivation.

This is a pretty important statement. Because Nix can have multiple versions of a package installed at once, there’s no need to have version bounds. Chapter one made this point as well, but it feels worth repeating.

The manifest.nix file contains metadata about the environment, such as which derivations are installed. So that nix-env can list, upgrade or remove them. And yet again, the current manifest.nix can be found at ~/.nix-profile/manifest.nix.

That’s interesting: so presumably nix-env -q operates by looking at manifest.nix. I wonder what that file actually looks like. I feel like I could understand it now…

$ cat ~/scratch/ugh.nix
(import <nixpkgs> {}).lib.generators.toPretty { multiline = true; } (import /Users/ian/.nix-profile/manifest.nix)

$ nix-instantiate --eval --strict ~/scratch/ugh.nix
error: attribute 'drvPath' missing, at /nix/store/4rvsrjbd2f351zgdh38as0xzwrlmvzkm-nixpkgs-21.05pre287374.1c16013bd6e/nixpkgs/lib/generators.nix:251:24

Hmm. No. The toPretty function seems to assume that any record with { type = "derivation"; } must also have a drvPath. Which is not… looking at the source here, it seems this is not going to pretty-print an expression, but give me output like <derivation /nix/store/whatever>. I don’t want that. It’s a set. Just… just pretty-print it, please…

And no, there is no option to enable that behavior.

So… I still have no idea how to pretty-print a Nix expression, without resorting to third-party tools.

But I’ve read the manuals already. I’m open to the idea of third-party tools, at this point, as I’m pretty sure there is no first-party support. I google, and find something called nix-beautify.

It’s not a part of Nixpkgs, apparently, but there are instructions to install it from their default.nix expression:

nix-env -i nix-beautify -f https://github.com/nixcloud/nix-beautify/archive/master.tar.gz

But I don’t want to install it, of course. I’d rather just nix-shell -p it. But I’ve never done that with anything but nixpkgs before. I know that nix-shell -p doesn’t accept -f and works by -I, so without really thinking about it I try this:

$ nix-shell -p nix-beautify -I nixpkgs=https://github.com/nixcloud/nix-beautify/archive/master.tar.gz
unpacking 'https://github.com/nixcloud/nix-beautify/archive/master.tar.gz'...
[1]    7103 segmentation fault  nix-shell -p nix-beautify -I

Hmm. I suspect this is because the first thing that default.nix does is import <nixpkgs>, and this causes a stack overflow. Didn’t really think about it, but yeah. Okay.

Fine. I install it, annoyed, and remember to uninstall it afterwards.

Something is weird with my shell completion, though, and I have to manually run rehash after installing it in order to not have to type out the whole executable name. Weird. Can’t remember the last time I had to do that.

Anyway, does it work?

$ nix-beautify manifest.nix

It’s just hanging. It’s just hanging there. It can only read from stdin, can’t it.

$ nix-beautify <manifest.nix
(extremely long, unprettified string of Nix expression)

Sigh. So that’s… not a thing. Looking at the example in the README, I suspect that this can only indent Nix expressions, not actually… beautify them. Sigh.

I find another project, from a name I trust. The README is… not reassuring, but it can’t be any worse than nix-beautify. This one is already in nixpkgs, so…

$ nix-shell -p nixfmt

[nix-shell:~]$ nixfmt ~/.nix-profile/manifest.nix
nixfmt: /Users/ian/.nix-profile: openTempFileWithDefaultPermissions: permission denied (Permission denied)

Hmmmmm. I suspect this is a symlink issue? Maybe?

[nix-shell:~]$ cp ~/.nix-profile/manifest.nix ~/scratch

[nix-shell:~]$ nixfmt ~/scratch/manifest.nix

Ah. No. It literally was permission denied, because this was trying to reformat in-place, instead of just printing out the formatted version. Okay.

Well anyway we can now look at the pretty-printed manifest.nix:

$ cat ~/scratch/manifest.nix

[
  {
    meta = {
      available = true;
      broken = false;
      changelog =
        "https://git.savannah.gnu.org/cgit/hello.git/plain/NEWS?h=v2.10";
      description = "A program that produces a familiar, friendly greeting";
      homepage = "https://www.gnu.org/software/hello/manual/";
      insecure = false;
      license = {
        fullName = "GNU General Public License v3.0 or later";
        shortName = "gpl3Plus";
        spdxId = "GPL-3.0-or-later";
        url = "https://spdx.org/licenses/GPL-3.0-or-later.html";
      };
      longDescription = ''
        GNU Hello is a program that prints "Hello, world!" when you run it.
        It is fully customizable.
      '';
      maintainers = [{
        email = "...";
        github = "edolstra";
        githubId = 1148549;
        name = "Eelco Dolstra";
      }];
      name = "hello-2.10";
      outputsToInstall = [ "out" ];
      platforms = [ ... ];
      position =
        "/nix/store/4rvsrjbd2f351zgdh38as0xzwrlmvzkm-nixpkgs-21.05pre287374.1c16013bd6e/nixpkgs/pkgs/applications/misc/hello/default.nix:15";
      unfree = false;
      unsupported = false;
    };
    name = "hello-2.10";
    out = {
      outPath = "/nix/store/pl4yqgxabnwf56lm0yf9hzy1gmsmwrkr-hello-2.10";
    };
    outPath = "/nix/store/pl4yqgxabnwf56lm0yf9hzy1gmsmwrkr-hello-2.10";
    outputs = [ "out" ];
    system = "x86_64-darwin";
    type = "derivation";
  }
  ...
]

It’s long. Incredibly long. So long that I only included the very first entry, and even then I omitted the platforms key.

So it’s a list of every installed derivation in the environment, and nothing else. Which kind of surprises me – I thought there’d be something in there – but okay. Whatever. Good enough.

What were we talking about?

Chapter 4. The Basics of the Language

Attempting to perform division in Nix can lead to some surprises.
nix-repl> 6/3
/home/nix/6/3

I’m glad the chapter calls this out. I haven’t made this mistake yet,² and I totally understand (and agree with) Nix’s choice, but this is still a gotcha. A very small gotcha, since I imagine you just aren’t doing a lot of division in your Nix expressions, but still worth noting.

Let’s see… points out kebab-case identifiers, something that either wasn’t explained or that I missed when I read the manual. That’s nice.

Talks about string escapes. I remember the manual giving an absolutely unintelligible description of these:

Since ${ and '' have special meaning in indented strings, you need a way to quote them. $ can be escaped by prefixing it with '' (that is, two single quotes), i.e., ''$. '' can be escaped by prefixing it with ', i.e., '''. $ removes any special meaning from the following $.

Meanwhile, Nix Pills explains it with an actual example:

Escaping ${...} within double quoted strings is done with the backslash. Within two single quotes, it’s done with '':
nix-repl> "\${foo}"
"${foo}"
nix-repl> ''test ''${foo} test''
"test ${foo} test"

Doesn’t give an example of the “$ removes any special meaning from the following $” bit, though. Maybe we can observe that?

nix-repl> ''hello ${world}''
error: undefined variable 'world' at (string):1:11

nix-repl> ''hello $${world}''
"hello $${world}"

Okay. Weird. Not really… I mean I guess it makes sense that it’s different from ''${world}, but… huh? It’s weird. It’s a weird thing. It’s weird that there’s a redundant escape syntax. You can get the same result like this:

nix-repl> ''hello $''${world}''
"hello $${world}"

Anyway.

I learn from this chapter that you can bind multiple identifiers in a single let expression:

nix-repl> let a = 3; b = 4; in a + b
7

Which… sort of explains the insane choice to have a semicolon before the in, but not… really. This reminds me of requiring terminal semicolons in attribute sets. Which is to say: why would you do this.

The chapter explains with, but not the weird multiple-scope with-not-shadowing-explicitly-bound-identifiers thing that the manual described.

Laziness gets a couple sentences, but no mention of gotchas or patterns or anything. Just an intro.

Chapter 5. Functions and Imports

Explains currying and function application, which the manual… just didn’t? I don’t remember it talking about that. But maybe I’ve forgotten.

Lotsa function stuff. Pattern matching, optional arguments, etc. Nothing we haven’t seen already.

Talks about import, but no mention of <path> syntax yet. I would probably introduce those around the same time, since most import expressions I’ve seen have been import <nixpkgs>.

Chapter 6. Our First Derivation

Let’s try to fake the name of the system:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»
Oh oh, what’s that? Did it build the derivation? No it didn’t, but it did create the .drv file. nix repl does not build derivations unless you tell to do so.

Whoa. That’s weird. I didn’t really think about… nix repl actually instantiating .drv files. Why…? Why does it do that? Does it actually…?

nix-repl> derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»

$ cat /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
Derive([("out","/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname","","")],[],[],"mysystem","mybuilder",[],[("builder","mybuilder"),("name","myname"),("out","/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"),("system","mysystem")])

Just… what? What possible use… is this? This feels like a weird implementation leak of the repl. This can’t be intentional, can it? There’s no reason you would ever want this behavior, is there?

Anyway, the chapter spends some time discussing .drv files, which I appreciate. They were given absolutely no treatment in the Nix manual, which is crazy to me.

I learn something very valuable from this: nix show-derivation. Basically a way to pretty-print those horrible “ATerm” files, which is something that I was looking for recently. But it doesn’t just pretty-print it, it also annotates each field with a name, and prints it in JSON – an actual format with actual tools for working with it.

Let’s look at the derivation I was trying to inspect last time, that I ended up seding into a sexp just to see it more clearly:

$ nix show-derivation /nix/store/4n40rm31n58ga0xl62nanq13a34axwwx-nix-shell.drv
{
  "/nix/store/4n40rm31n58ga0xl62nanq13a34axwwx-nix-shell.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/5i9c80x5h0mjpxwfnj6iz8ykivzjrcyq-nix-shell"
      }
    },
    "inputSrcs": [
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "inputDrvs": {
      "/nix/store/8mgqiwshdp8pvl7agmi1ls6m2ziwp4ag-stdenv-darwin.drv": [
        "out"
      ],
      "/nix/store/gngpiflribp4h76hz4yrfm68vhs50d23-python3.9-cram-0.7.drv": [
        "out"
      ],
      "/nix/store/s5mjnf3b6nmxpyy640mx0s78zspihn2y-bash-4.4-p23.drv": [
        "out"
      ],
      "/nix/store/sqs3pqphn9jbizwxmp8zvfcfg1lp2prm-souffle-2.0.2.drv": [
        "out"
      ]
    },
    "platform": "x86_64-darwin",
    "builder": "/nix/store/30njb8l701pwnm5ya749fh2cgyc2d70m-bash-4.4-p23/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "__darwinAllowLocalNetworking": "",
      "__impureHostDeps": "/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom /bin/sh",
      "__propagatedImpureHostDeps": "",
      "__propagatedSandboxProfile": "",
      "__sandboxProfile": "",
      "buildInputs": "",
      "builder": "/nix/store/30njb8l701pwnm5ya749fh2cgyc2d70m-bash-4.4-p23/bin/bash",
      "configureFlags": "",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "",
      "doInstallCheck": "",
      "name": "nix-shell",
      "nativeBuildInputs": "/nix/store/n0wm6l7qk9ygzjd76ashf9xx7f0z3kh6-souffle-2.0.2 /nix/store/axzlbh5ji9pr97mi04f70lizn22bkxj5-python3.9-cram-0.7",
      "nobuildPhase": "echo\necho \"This derivation is not meant to be built, aborting\";\necho\nexit 1\n",
      "out": "/nix/store/5i9c80x5h0mjpxwfnj6iz8ykivzjrcyq-nix-shell",
      "outputs": "out",
      "patches": "",
      "phases": "nobuildPhase",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "shellHook": "",
      "stdenv": "/nix/store/59hdixd6qf2jq5mj0bqiwrhy7621wa7j-stdenv-darwin",
      "strictDeps": "",
      "system": "x86_64-darwin"
    }
  }
}

Ahhh. Much better. It seems that my guesses were right about the different values, also – inputSrcs and inputDrvs. Nothing about the weird empty strings, but I happened to stumble upon an answer, while re-reading an old blog post of mine:

[ ( "out"
, "/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz"
, "sha256"
, "31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b"
) ]

It seems that those values are meaningful only for a “fixed-output derivation,” but instead of just being omitted in that case (as they are in the show-derivation output), they appear as empty strings:

[ ( "out"
  , "/nix/store/5i9c80x5h0mjpxwfnj6iz8ykivzjrcyq-nix-shell"
  , ""
  , ""
  ) ]

Okay, yeah, I guess it is a tuple – probably, who knows what data types this format allows – but, like, gross.

I’m very glad that this explains these files and why they exist and what they look like. I happened to have gone on this journey myself, but it feels important enough that I wish the manual spent some time explaining it.

Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It’s possible however to have content-addressable derivations for e.g. tarballs as we’ll see later on.

Are they actually content-addressable? I know they’re “fixed output” but… I thought that was just an assertion, not actually reflected in the path itself.

Let’s look at one. Let’s see… I’ve got:

/nix/store/bd5v53gnpwcgdlgq9hc2x8l8xm8rsz9h-pcre-ocaml-7.2.3.tar.gz.drv

That sounds like it oughtta be fixed-output.

$ nix show-derivation /nix/store/bd5v53gnpwcgdlgq9hc2x8l8xm8rsz9h-pcre-ocaml-7.2.3.tar.gz.drv \
> | jq '.[] | .outputs'
{
  "out": {
    "path": "/nix/store/67wja6vzp1da9adlbimb6839malmnlm0-pcre-ocaml-7.2.3.tar.gz",
    "hashAlgo": "sha256",
    "hash": "6339694dbeb706c5097180ed1d79b2dae681bf155a4780a7909af49b0e6f4666"
  }
}

So… it’s hard to tell, because one is in hex, and the other is in… checks notes… base32. But my notes also tell me this handy helper:

$ nix to-base32

$ nix to-base32 6339694dbeb706c5097180ed1d79b2dae681bf155a4780a7909af49b0e6f4666
error: hash '6339694dbeb706c5097180ed1d79b2dae681bf155a4780a7909af49b0e6f4666' does not include a type

$ nix to-base32 sha256 6339694dbeb706c5097180ed1d79b2dae681bf155a4780a7909af49b0e6f4666
error: hash 'sha256' does not include a type

$ nix to-base32 --help
Usage: nix to-base32 <FLAGS>... <STRINGS>...

Summary: convert a hash to base-32 representation.

Flags:
      --type <TYPE>  hash algorithm ('md5', 'sha1', 'sha256', or 'sha512')

Note: this program is EXPERIMENTAL and subject to change.

Why… does a conversion function need to know the algorithm? Aren’t we just converting bytes into a different representation? What…? Whatever:

$ nix to-base32 --type sha256 6339694dbeb706c5097180ed1d79b2dae681bf155a4780a7909af49b0e6f4666
0rj6dw79px4sj2kq0iss2nzq3rnsn9wivvc0f44wa1mppr6njfb3

Hmm. That’s still more characters than occur in the store path, and it doesn’t seem like the store path is a prefix or anything. So… I’m gonna say it’s not content-addressed? At least not by default. But the chapter did say that it’s only “possible” to have content-addressable derivations. It did not say that fixed-output derivations were content-addressable. So that’s an open question, I guess.

Or I don’t really understand what “content-addressable” means. That’s… not unlikely.

Okay. Overall, this chapter seems great. This is a ton of useful information, most of which did not exist in the manual in any form. I’m not going to write all of my thoughts, but this is certainly something I would recommend new Nix users to read. I can’t really judge how approachable it is, because I already know all these things, but it seems to cover basically everything I had questions about before.

I learned also about the magic value set.outPath:

nix-repl> { outPath = "hello" }
error: syntax error, unexpected '}', expecting ';', at (string):1:21

nix-repl> { outPath = "hello"; }
{ outPath = "hello"; }

nix-repl> builtins.toString { outPath = "hello"; }
"hello"

Feels like that’s worth mentioning in a discussion of toString. The manual described the special key __toString, but not this.

We’re going through a lot of stuff without any mention of stdenv, which… might be confusing? I can’t really tell. Like, if this was your absolute first exposure, you wouldn’t have any questions here. But if you’ve seen some Nix before, I could imagine this being confusing because of how different it is to what you’ve already seen? I admire that we’re building up things from scratch, with only Nix, and not really discussing Nixpkgs at all. But it never explicitly says that’s what it’s doing.

I dunno. I shouldn’t speculate on how understandable this is. I have lost all ability to judge that.

I learned this repl idiom:

nix-repl> :l <nixpkgs>
Added 14075 variables.

Instead of:

nix-repl> pkgs = import <nixpkgs> {}

As I have been doing. That’s a neat trick.

Chapter 7. Working Derivation

It also took me two posts to get to a working derivation. And I was following the manual!

I dunno; I like this chapter too? This seems good. I don’t learn anything, but I think I would recommend it to others. It starts off in a nix repl, which is kind of neat – declaring inline derivations, building them with :b… it’s weird, sure, but probably a decent first intro? Although I must repeat: it’s not my first intro, so I don’t really know what I’m talking about. And it doesn’t linger there, but moves on quickly to writing a “real” default.nix file.

nix-build does two jobs:

nix-instantiate: parse and evaluate simple.nix and return the .drv file corresponding to the parsed derivation set

nix-store -r: realise the .drv file, which actually builds it.

I think it’s nice to call out the primitive operations when discussing the porcelain. Good work.

Chapter 8. Generic Builders

Okay. So I think what’s happening here is: Nix Pills is going to walk us through mkDerivation and the $stdenv/setup script, and motivate all of the complexities there, starting from scratch. This sort of gets us started, showing how it puts buildInputs on our PATH and makes sed available by default and all that.

But it doesn’t say that’s what it’s doing. And although I can sort of appreciate that the individual steps here do make sense in isolation, I think that having the end goal in mind would make it all more… concrete.

Chapter 9. Automatic Runtime Dependencies

NAR is the Nix ARchive. First question: why not tar? Why another archiver? Because commonly used archivers are not deterministic. They add padding, they do not sort files, they add timestamps, etc.. Hence NAR, a very simple deterministic archive format being used by Nix for deployment. NARs are also used extensively within Nix itself as we’ll see below.

That’s actually really nice to call out. I assumed NAR files were something more than just bundles of files. But this makes me think that NARs are just… you know. Just the thing that it says. Why, then, is it called Nix Archive? There’s nothing Nix-specific, there, is there? Hmm.

Build dependencies are automatically recognized by Nix once they are used in any derivation call, but we never specify what are the runtime dependencies for a derivation.

Okay. Yes. I’m on board here.

There’s really black magic involved. It’s something that at first glance makes you think “no, this can’t work in the long term”, but at the same time it works so well that a whole operating system is built on top of this magic.

Steps:

Dump the derivation as NAR, a serialization of the derivation output. Works fine whether it’s a single file or a directory.

For each build dependency .drv and its relative out path, search the contents of the NAR for this out path.

If found, then it’s a runtime dependency.

This… does not sound like black magic to me. This sounds like exactly how I would expect it to work.

Actually, I originally expected it would just search for anything that looked like a store path in the output. But as soon as I tried manually adding a text file full of store paths to the store, I realized why that doesn’t make sense: derivations aren’t allowed to get their hands on any random paths. The only way it can know a store path is if that store path is a dependency of the .drv file.

So, okay. This makes me wonder how cross-compilation works, since it sounds like all runtime dependencies are subsets of build-time dependencies, but the architectures might be different… but I guess that doesn’t matter, though? Different architectures are different derivations. Different paths. Not really a problem. Okay.

Now it’s motivating the patchelf and strip thing, by showing how a trivial C program ends up with (by this heuristic) a runtime dependency on gcc. This is actually really great: a better description of why the fixupPhase exists than the Nixpkgs manual gave, if I’m remembering right.

Chapter 10. Developing with `nix-shell`

First thing to notice, we call nix-shell on a nix expression which returns a derivation. We then enter a new bash shell, but it’s really useless. We expected to have the GNU hello world build inputs available in PATH, including GNU make, but it’s not the case.

But, we have the environment variables that we set in the derivation, like $baseInputs, $buildInputs, $src and so on.

Innnnteresting. I didn’t really think about the PATH thing before. I guess this is something mkShell gives me? No; looking back on the source, it must be something the standard environment’s shellHook thing?

Hmm.

I feel like this chapter is… not a good intro for a beginner. This is not a good first “how to use Nix.” But it’s a very good “I’ve used Nix a bit, and now I want to go in deeper.”

Anyway, thinking about how little nix-shell actually does makes it even weirder to me that it executes bash. Like, the more generic version of nix-shell is to just execute a different builder with the same environment. Let me pick the builder. It’s already separate in the drv file, right?

Hmm.

Chapter 11. Garbage Collector

Doesn’t really add anything over the manual. Talks about adding roots, and indirect roots. No mention of keep-outputs. I didn’t learn anything from this chapter.

Chapter 12. Inputs Design Pattern

This chapter is… motivating the idea of making packages take their inputs as function arguments?

This is something that seems so natural to me I wouldn’t have thought to motivate it. But starting from first principles, it is an invention, I guess?

I dunno.

Chapter 13. Callpackage Design Pattern

callPackage. It seems a little odd to call this a “design pattern,” just as it felt odd to call… functions a design pattern. It’s a pattern! It’s… I don’t know what “design pattern” means in this day and age, I suppose.

We renamed the old pkgs of the previous pill to nixpkgs. Our package set is now instead named pkgs. Sorry for the confusion.

This shouldn’t really… hmm. That’s a weird thing to keep in the “official” mirror of this post. Maybe these chapters haven’t actually changed since the original publication?

Anyway. callPackage. I’ve seen it.

Chapter 14. Override Design Pattern

We walk through an implementation of the override “method.” I don’t really get this. It seems straightforward to me, but I guess that if you’re used to object-oriented languages, it might be weird?

Chapter 15. Nix Search Paths

Explains NIX_PATH and .nix-defexpr. I dunno; this is valuable information, but it also seems like something that you’ve probably already encountered if you made it 15 chapters into this guide. Maybe not! But I dunno. Since we’ve only really talked about writing default.nix files, and not really using nix-env yet, I guess maybe it makes sense to introduce it this late. But the order is not really… I dunno. Seems weird.

Chapter 16. Nixpkgs Parameters

I’m sure on the wiki or other manuals you’ve read about ~/.nixpkgs/config.nix and I’m sure you’ve wondered whether that’s hardcoded in nix. It’s not, it’s in nixpkgs.

Huh. Mine is ~/.config/nixpkgs/config.nix. I’m sure this looks at a million possible files. This section links to a source line in the Nixpkgs repo, but not a specific revision, so of course it refers to something nonsensical now.

I’m not sure if the Nix Pills source lives in the Nixpkgs repo or the Nix repo or what, but keeping this up to date seems silly. Anyway; here’s the source as it stands right now:

# Fallback: The contents of the configuration file found at $NIXPKGS_CONFIG or
# $HOME/.config/nixpkgs/config.nix.
config ? let
    configFile = getEnv "NIXPKGS_CONFIG";
    configFile2 = homeDir + "/.config/nixpkgs/config.nix";
    configFile3 = homeDir + "/.nixpkgs/config.nix"; # obsolete
  in
    if configFile != "" && pathExists configFile then import configFile
    else if homeDir != "" && pathExists configFile2 then import configFile2
    else if homeDir != "" && pathExists configFile3 then import configFile3
    else {}

# obsolete indeed, I guess.

Okay. That’s kind of all that’s in this chapter? Seems a little weird. But it’s probably motivating, like, the existence of Nixpkgs, if you’re starting from scratch? But who is starting from that degree of scratch, at this point?

Chapter 17. Nixpkgs Overriding Packages

Talking about “the Nix packages fixpoint” is actually very useful/interesting. But I don’t really think the treatment here… I’m glad it gives an example of fix, I guess. That’s useful. That’s good. I can’t really judge if this is a useful explanation.

The fixed point with lazy evaluation is crippling but about necessary in a language like Nix. It lets us achieve something similar to what we’d do imperatively.

This is a weird thing to say. It does not explain what we would do imperatively, or what this lets us achieve, or why it’s crippling. I don’t understand the sentiment this is trying to convey.

I assume that it means imperatively re-assigning properties of a derivation, in a way such that they propagate to other derivations that reference ours. This how a “normal” imperative language with reference semantics would work. It doesn’t say that, though, or really explain why this is equivalent. So, I don’t know.

Whereas in an imperative setting, like with other package managers, a library is installed replacing the old version and applications will use it, in Nix it’s not that straight and simple. But it’s more precise.

I very mildly object to this statement. I think it’s true; I just think it deserves some justification. The does not really try to substantiate this claim. Anyway. This isn’t interesting.

Okay.

I don’t know if what I’m about to say is true, but I don’t think that this chapter did a very good job of explaining how overrides work to someone who doesn’t already have a good idea of how they work.

It tries more than the official manual does to explain what’s going on, but I have a feeling the explanation here would leave a newcomer to lazy evaluation more confused than when they started. Or maybe not! I’m a terrible judge of this now.

Chapter 18. Nix Store Paths

The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the Nix source code.

Okay. This seems… not super important to know.

Finally the comments tell us to compute the base-32 representation of the first 160 bits (truncation) of a sha256 of the above string

Ah. Maybe this is why my “convert this hash to base32” thing back in chapter 6 didn’t work.

Let’s see… 160 bits = 20 bytes = 40 hex characters, so…

$ nix to-base32 --type sha256 6339694dbeb706c5097180ed1d79b2dae681bf15
error: hash '6339694dbeb706c5097180ed1d79b2dae681bf15' has wrong length for hash type 'sha256'

Well, yeah, of course it does, but… can’t you just, like, do a base conversion of a number? Why… what?

Anyway; paying more attention to the contents of this chapter, this wouldn’t have worked anyway, because the hash we’re talking about is not the hash of the contents but rather the hash of a string containing the contents. Okay?

My eyes glaze over; I do not understand why it is important to know precisely where these hashes come from. This chapter has not motivated why it’s explaining this, and I am having a hard time coming up with a reason.

Ah, here we go: it walks through a concrete example of a fixed-output derivation. And I learn the --truncate flag to nix hash. Which lets me get all the way there:

$ nix-shell -p coreutils

[nix-shell:~]$ nix-hash --type sha256 --base32 --truncate --flat <(printf 'output:out:sha256:%s:/nix/store:pcre-ocaml-7.2.3.tar.gz' $(echo
 -n 'fixed:out:sha256:6339694dbeb706c5097180ed1d79b2dae681bf155a4780a7909af49b0
e6f4666' | sha256sum | cut -d' ' -f1))
r24wj6nsw3s9sli2wkfj6nb2rmzr2gbw

No. Still not right. Okay. I have exhausted my ability to care. But theoretically, this chapter is telling me that fixed-output paths really are content-addressable. It’s just a weird and convoluted address. Okay.

And that was the whole chapter. Interesting. I mean, it is interesting, to me, how this works, but this is like… this is a deep cut, here. This is not something that anyone needs to know, in order to use Nix. Probably not even to contribute to Nix. This is like… something you can figure out, if you ever actually care. I mean, I recognize that that’s probably what this chapter is: the author wanting to understand this. But for a newcomer’s guide to Nix, this feels pretty unnecessary.

Although… is that the intent of the Nix Pills series? Is this supposed to be a newcomer’s guide? Or did this begin something like my weird series, a chronicle of an the author following the fun, that has since morphed into recommended reading?

I cannot know.

Chapter 19. Fundamentals of Stdenv

Remember our generic builder.sh in Pill 8? It sets up a basic PATH, unpacks the source and runs the usual autotools commands for us.

The stdenv setup file is exactly that. It sets up several environment variables like PATH and creates some helper bash functions to build a package. I invite you to read it, it’s only 860 lines at the time of this writing.

It is now 1,338 lines long, but, you know, whatever. It’s long. It’s complicated. I looked through it, as a new Nix user, and got very little out of it. Perhaps you will get more out of it? I don’t know. I feel like I get the gist, now, and don’t look through it again.

And here comes the reveal: that the builders we were writing in previous chapters were secretly building up to stdenv/setup all along. That’s sort of all of the information here. It’s a strange presentation order. This is a pretty hard downshift from “implementation details of Nix’s hashing algorithm.”

Chapter 20. Basic Dependencies and Hooks

This chapter begins with a pleasant introduction, followed by this concerning note:

Note: The complexity of the dependencies and hooks infrastructure has increased, over time, to support cross compilation.

Has it ever.

Once you learn the core concepts, you will be able to understand the extra complexity. As a starting point, you might want to refer to nixpkgs commit 6675f0a5, the last version of stdenv without cross-compilation complexity.

That is probably a very interesting diff to read, actually. Of course I don’t care about 6675f0a5, I care about the “child” commits, which is… how do you see that? That is definitely not something I could recite off the top of my head. The internet tells me:

$ git log 6675f0a5..HEAD --ancestry-path --reverse

To see the commits made after that. Why is --ancestry-path required? I don’t know. I tried to look at the explanation in man git-log and just… wow.

Suppose you specified foo as the <paths>. We shall call commits that modify foo !TREESAME, and the rest TREESAME. (In a diff filtered for foo, they look different and equal, respectively.)

I spent five minutes trying to figure out what that parenthetical is trying to tell me, but I have given up.

Anyway. That does show the commits that introduce the cross-compilation stuff, but I don’t actually bother reading the diff. Seems… enormously complicated.

Okay so. What are we doing here? Reading chapter 20. Okay.

We learn about buildInputs. It explains the bash function findInputs in the stdenv/setup script, which adds the bin/ directories of our buildInputs to our PATH. It walks through some code that does that, although it’s exactly the code you’d expect.

Next we learn propagatedBuildInputs. This is something I sort of remember learning about, maybe in the Nixpkgs manual, but it was in the extremely complicated cross-compilation section, and I could not actually tell you what it… is. Maybe it was explained well, and I’ve just forgotten? But I do not remember.

The chapter gives a very good explanation, though:

The buildInputs covers direct dependencies, but what about indirect dependencies where one package needs a second package which needs a third? Nix itself handles this just fine, understanding various dependency closures as covered in previous builds. But what about the conveniences that buildInputs provides, namely accumulating in pkgs environment variable and inclusion of pkg/bin directories on the PATH? For this, stdenv provides the propagatedBuildInputs.

This is interesting: it describes that this works by actually creating files in the out path, in a special nix-support subdirectory.

$ find /nix/store -name 'nix-support'
(thousands of results)

$ cd /nix/store/m9n29mydxdjvjqj7rw9idwiqhm2bww1d-ghc-8.10.4/nix-support

$ ls
propagated-build-inputs
propagated-target-target-deps

$ cat propagated-build-inputs
/nix/store/47vpv5i10dwfg1cf5wca1k40f982g5fm-clang-wrapper-7.1.0

$ cat propagated-target-target-deps
/nix/store/dy2rb80nvksrcsbm4hggvayzdv8fwvhx-ncurses-6.2 /nix/store/3068jzr60jjd3xagmly4b06nh5kcb0cs-libffi-3.3 /nix/store/w6iysk0w7j8qnzljp8467yw03ppq3bvk-gmp-6.2.1 /nix/store/4gngk4048zq7fcyx9959vhv09pwy4d6h-libiconv-50

Interesting! So basically, the output path now references on these other packages. We can think of these as being promoted to “runtime” dependencies, although we recognize that Nix does not think in terms like that.

More importantly, the contents of these files can be read back by the stdenv script, such we don’t just put buildInputs in our PATH, but also each of these dependencies, and so on recursively. You get it. I don’t need to explain this.

in general, a dependency might affect the packages it depends on in arbitrary ways. Arbitrary is the key word here. We could teach setup.sh things about upstream packages like pkg/nix-support/propagated-build-inputs, but not arbitrary interactions.

Setup hooks are the basic building block we have for this. In nixpkgs, a “hook” is basically a bash callback, and a setup hook is no exception.

Okay. I do remember setup hooks, as basically allowing packages to do things to packages that depend on them. So like, yeah, propagating dependencies is a specific case of this, right? That could be implemented as a setup hook. But it’s special-cased by stdenv/setup.

One can almost think of this as an escape hatch around Nix’s normal isolation guarantees, and the principle that dependencies are immutable and inert. We’re not actually doing something unsafe or modifying dependencies, but we are allowing arbitrary ad-hoc behavior. For this reason, setup-hooks should only be used as a last resort.

I don’t… follow this reasoning at all. An “escape hatch around … the principle that dependencies are immutable and inert.” I mean, they’re still immutable and inert, right? We didn’t escape that. In no way can this mutate a dependency. And the dependency itself isn’t doing anything; we’re choosing to search for these setup-hook files and run them. This is something we could choose not to do, if we didn’t want to.

I dunno; this doesn’t seem like a “last resort” thing. It seems like a useful tool to have when you need it. Anyway.

Lastly, we learn about environment hooks. So basically… we find every (direct or “propagated”) dependency. And every time we find a dependency, we source its setup-hook file, if it exists. And each setup-hook may or may not register “environment” hooks.

Once we’ve done that for every package, we go through and add each one to our PATH, if it has a bin/ directory. And then we invoke every “environment hook” that we have so far – basically, the union of “environment hooks we explicitly defined” and “environment hooks registered by our dependencies' setup-hooks.”

So every dependency gets a chance to see every other dependency, basically.

Why?

The introduction to this section motivated this, before explaining what it was:

Recall in Pill 12 how we created NIX_CFLAGS_COMPILE for -I flags and NIX_LDFLAGS for -L flags, in a similar manner to how we prepared the PATH.

I don’t really remember that because I was kind of… skimming, if we’re being honest. Chapter 12 seemed boring.

One point of ugliness was how anti-modular this was. It makes sense to build the PATH in generic builder, because the PATH is used by the shell, and the generic builder is intrinsically tied to the shell. But -I and -L flags are only relevant to the C compiler. The stdenv isn’t wedded to including a C compiler (though it does by default), and there are other compilers too which may take completely different flags.

That makes sense. So basically, the derivation for gcc has a setup-hook which registers an environment hook which accumulates include paths to eventually pass as -I. Neat. Okay.

That’s the end!

That’s the end of Nix Pills series. It actually has a “coming soon” teaser at the bottom for the “Next pill,” as every chapter has so far:

…I’m not sure! We could talk about the additional dependency types and hooks which cross compilation necessitates, building on our knowledge here to cover stdenv as it works today. We could talk about how nixpkgs is bootstrapped. Or we could talk about how localSystem and crossSystem are elaborated into the buildPlatform, hostPlatform, and targetPlatform each bootstrapping stage receives. Let us know which most interests you!

It’s a shame that the series ended there, because I feel like the author of the Pills could probably explain the cross-compilation stuff more clearly than the Nixpkgs manual did.

We did it

There we have it! I read the Pills. I know the things, now.

I learned some good stuff. It was definitely not completely redundant with the Nix and Nixpkgs manuals, and I felt that last chapter especially gave a nice treatment of a part of the genericBuilder that was a little buried in the Nixpkgs manual. Or that I just forgot about. And 6 and 7 were just great introductions to defining your own derivation. I think. Hard to really judge, at this point.

Phew. That was a long one. Why did I do this in a single post? Why didn’t I break this up? What was I thinking?

Why does the repl instantiate .drv files and put them into the actual store?
Is there anything Nix-specific about NAR files?

Something about the lack of attribution and collaborative nature of the Nix and Nixpkgs manual made it easy for me to criticize, even though I do realize there are real people with real feelings who wrote those also. I hope that I haven’t been being mean without thinking about it this entire time. ↩︎
I would never, ever snuggle my operands in actual code, but I could see myself lazily typing that at a repl. ↩︎