This is it. We’ve reached the end. After a month and a half of reading the Nixpkgs manual in excruciating detail, I have finally defeated it.

Or, I’m about to defeat it. We’re about to defeat it. We’re going to do this together.

Chapter 17. Quick Start to Adding a Package

At this point I feel that I’ve… kind of got a pretty good handle on this. Step 1: write a default.nix file. Step 2: put it somewhere reasonable. Step 3: add it to the giant monolithic list of every package in the universe.

And, yeah. That’s pretty much what the manual says.

It then gives a list of 9 packages to serve as good examples for writing said default.nix files.

Aha!

You can use nix-prefetch-url url to get the SHA-256 hash of source distributions. There are similar commands as nix-prefetch-git and nix-prefetch-hg available in nix-prefetch-scripts package.

I was just wondering about that. Now I have my answer.

A list of schemes for mirror:// URLs can be found in pkgs/build-support/fetchurl/mirrors.nix.

Wha? What are mirror:// URLs?

It doesn’t say, but it links to the code, so let’s take a look.

And there is… no explanation of this. I have never encountered a mirror:// URL, although ⌘F tells me that the Nixpkgs manual did use some of them (without explanation) in the section on packaging Perl programs.

It just says:

# Mirrors for mirror://site/filename URIs, where "site" is
# "sourceforge", "gnu", etc.

And then there’s a big long list of attributes like this:

 # SourceForge.
sourceforge = [
  "https://downloads.sourceforge.net/"
  "https://prdownloads.sourceforge.net/"
  "https://heanet.dl.sourceforge.net/sourceforge/"
  "https://surfnet.dl.sourceforge.net/sourceforge/"
  "https://dfn.dl.sourceforge.net/sourceforge/"
  "https://osdn.dl.sourceforge.net/sourceforge/"
  "https://kent.dl.sourceforge.net/sourceforge/"
];

# CPAN mirrors.
cpan = [
  "https://cpan.metacpan.org/"
  "https://cpan.perl.org/"
  "http://backpan.perl.org/"  # for old releases
];

So I’m assuming – this is not explained in the code or the manual – that there is some Nix machinery in fetchurl to try these lists of mirrors, when you use the mirror “protocol.”

But there is also one mysterious one – at the very top of the file:

# Content-addressable Nix mirrors.
hashedMirrors = [
  "http://tarballs.nixos.org"
];

I don’t really know what that’s about. It certainly seems different than all the others.

But I remember hearing – I don’t remember reading this in the manual, or in some random blog post, or what – about some sort of mechanism that fetchurl can use to transparently mirror sources, as a way to make Nix builds reproducible even as The Internet changes out from under us.

Aha! Grepping my blog reveals the answer. The Nix manual mentioned it, in passing, in the documentation for a configuration value called hashed-mirrors.

So presumably this is just the default value, and I can configure my own hashed-mirrors with all the sources that my software depends on. Which I like!

I wonder how I can set that up.

I also wonder if hashed-mirrors is like… a fallback? Or a preferred cache, like a CDN situation? I would guess that it’s just a fallback. But maybe not? Since you’d only use it in the case that you aren’t using a substitute. And Nix is paying for that bandwidth already. Anyway, if I had my own private version, I would definitely want it to be the primary source. Hmm. I wonder if I can configure that.

Anyway. Yeah. That’s how you contribute. And then you open a PR. Seems straightforward.

Chapter 18. Coding conventions

This is a chapter about conventions.

It begins with… syntax. Like, where to put your braces. How many spaces to use. Stuff like that. Stuff that, in 2021, actual humans are still spending actual human effort on. Stuff that a computer can– okay. I won’t go on a rant about autoformatters. Or linters, at the very least – that’s not even a controversial ask, is it? Anyway. Definitely bigger fish to fry here.

I’m not going to read all of these detailed instructions for how to format source code because it’s, you know, life is too short.

Actually, I will rant a little bit, because I thought of something almost interesting to say about this.

One of the beautiful things about Nix is that it takes “build instructions” – documents written for humans – and turns them into expressions written for computers.

Consider, for example, this document. It is a document that describes how to build the music programming environment SuperCollider. It is a lot of words. It contains instructions for human people to follow, if they want to build SuperCollider for themselves. Many of the words are just about how to install Qt.

Now consider this alternative. That’s the Nix expression to build SuperCollider.

The ability to lift “instructions for humans” into “instructions for computers” is one of the great superpowers that Nix gives you.

I don’t mean to pick on SuperCollider; that’s just something I happened to be playing with recently, and I thought its build instructions made a very good Nix case study. And I recognize that I’m not being fair, really: the official instructions contain a lot of information about conditional compilation of different features, which the Nix expression does not support.

So I will pick on myself instead:

This project relies on cairo and pango. If you don’t have them already, you can install them with Homebrew like this:

$ brew install cairo --without-x11
$ brew install pango --without-x11

Afterwards, I’m pretty sure you need gtk2hs-buildtools for something.

$ cabal install gtk2hs-buildtools

Make sure you don’t install that into a sandbox! It adds some binaries that you need. So make sure ~/.cabal/bin is on your PATH.

Afterwards, we do haskell things:

$ cabal sandbox init

We have to install the Haskell pango bindings separately because clang can’t build it. Look, don’t ask me.

$ cabal install pango -j --with-gcc=gcc-4.8

Then we install the rest:

$ cabal install -j --only-dependencies

Those are the build instructions I wrote once for a little toy project designed to waste company time and money. Are the instructions correct? Complete? I have no idea. It’s not like I tested them. How could I? It worked on my machine. Did I capture every dependency? I don’t know. The language I used there is certainly not reassuring.

Anyway, we’re in part 35 of a series about learning Nix. So I don’t really think I need to convince you of the value of actually repeatable build instructions.

Now consider this document:

Function formal arguments are written as:

{ arg1, arg2, arg3 }:

but if they don’t fit on one line they’re written as:

{ arg1, arg2, arg3
, arg4, ...
, # Some comment...
  argN
}:

It’s just as infuriating, right? As opposed to:

$ nix fmt

Or whatever.

Okay. End of rant.

The manual then describes package naming conventions. Which begins with the wonderful:

The key words must, must not, required, shall, shall not, should, should not, recommended, may, and optional in this section are to be interpreted as described in RFC 2119. Only emphasized words are to be interpreted in this way.

You know you’re in for a good time whenever someone links to RFC 2119.

This is followed by a couple simple rules for naming packages. No uppercase letters. Versions start with numbers. I dunno. I thought I could be snarky about the naming police intensely overspecifying every little detail, but there are only actually six different rules.

One of them is:

If a package is not a release but a commit from a repository, then the version part of the name must be the date of that (fetched) commit. The date must be in “YYYY-MM-DD” format. Also append “unstable” to the name - e.g., “pkgname-unstable-2014-09-23”.

Which seems odd to me. I dunno. Kinda arbitrary, whether a commit is “a release” or just “a commit.”

Anyway, then we describe the layout of the Nixpkgs repository. What folders to put stuff in. I don’t really know the exact difference between a tool/ and an application/, but I could probably distinguish between them pretty well in practice. I dunno. servers/ is for both HTTP servers and X.org, which is very funny to me.

18.4. Fetching Sources

Right now there is only one fetcher which has mirroring support and that is fetchurl.

Aha. So the hashed-mirrors thingy doesn’t work with fetchgit? Or fetchFromGitHub? That seems… problematic.

Or, no… it seems fetchFromGitHub is implemented with fetchurl, not fetchgit – it fetches a source archive over HTTP; it doesn’t fetch via the git protocol directly. Okay; that’s good to know.

The manual gives an example here of how to use the nix-prefetch-scripts:

Find the value to put as sha256 by running

nix run -f '<nixpkgs>' nix-prefetch-github -c nix-prefetch-github --rev 1f795f9f44607cc5bec70d1300150bfefcef2aae NixOS nix

or

nix-prefetch-url --unpack https://github.com/NixOS/nix/archive/1f795f9f44607cc5bec70d1300150bfefcef2aae.tar.gz.

I tried running that first one:

$ nix run -f '<nixpkgs>' nix-prefetch-github -c nix-prefetch-github --rev 1f795f9f44607cc5bec70d1300150bfefcef2aae NixOS nix
[4 copied (1.2 MiB), 0.3 MiB DL]
{
    "owner": "NixOS",
    "repo": "nix",
    "rev": "1f795f9f44607cc5bec70d1300150bfefcef2aae",
    "sha256": "1i2yxndxb6yc9l6c99pypbd92lfq5aac4klq7y2v93c9qvx2cgpc",
    "fetchSubmodules": true
}

Pretty neat!

This is my first encounter with nix run. Well, since the quick start guide, anyway.

Let’s try to dissect that a little. From nix run --help:

Usage: nix run <FLAGS>... <INSTALLABLES>...

-f, --file <FILE>               evaluate FILE rather than the default
-c, --command <COMMAND> <ARGS>  command and arguments to be executed; defaults to 'bash'

So we provide one INSTALLABLE, nix-prefetch-github. Presumably that’s the name of a top-level attribute:

$ sd nix info nix-prefetch-github
nix-prefetch-github-4.0.3 Prefetch sources from github

Indeed. There is also:

$ sd nix info nix-prefetch-scripts
nix-prefetch-scripts Collection of all the nix-prefetch-* scripts which may be used to obtain source hashes

But I don’t try to run that one.

Then the --file we pass is <nixpkgs>. Although this doesn’t look like a “file,” remember that it is – to Nix, <nixpkgs> is another way to write the path /Users/ian/.nix-defexpr/channels/nixpkgs, which is a directory with a default.nix file.

And then -c is the command to run with that INSTALLABLE on our PATH, I guess. I assume it just adds /nix/store/xxx-nix-prefetch-github-4.0.3/bin to the PATH and then tries to execute that command?

Let’s find out.

$ nix run -f '<nixpkgs>' nix-prefetch-github -c env | grep '^PATH='
PATH=/nix/store/y1c2zbzirfhv29ix4hdsxp6ksnxl2q3q-python3.8-nix-prefetch-github-4.0.3/bin:...

Indeed. Okay. Cool.

18.5. Obtaining source hash

We just learned how to do this, but now there’s a whole section on how to do this? In more detail, but I’m not sure that any of the extra detail is necessary, given the above example.

I learn how to turn a hexadecimal hash into a Base32 hash.

You can convert between formats with nix-hash, for example:

nix-hash --type sha256 --to-base32 HASH

Okay. I think that’s the first time I’ve seen nix-hash.

It then says “if all else fails, put a fake hash and run some command and copy the error,” and provides examples of well-formed but incorrect hashes: lib.fakeSha256 and friends. Huh!

Actually really it says don’t do this, because downloading a package is not a very good way to find out what the hash should be, because man in the middle attacks.

It adds this mystifying explanation:

https:// URLs are not secure in method 5. When obtaining hashes with fake hash method, TLS checks are disabled. So refetch source hash from several different networks to exclude MITM scenario. Alternatively, use fake hash method to make Nix error, but instead of extracting hash from error, extract https:// URL and prefetch it with method 1.

…what? Why on earth would TLS checks be disabled when fetching sources with a known fake hash? What? This must be… wrong, right? What is this trying to say?

I don’t know. I am upset and afraid.

Chapter 19. Submitting changes

Alright. Now we know how to make changes. Now we learn how to submit them.

This is… instructions for how to make good commits? Update release notes. How to test stuff. Good information, I guess? Link to contribution guidelines. How to mark security releases and get them out sooner. “There is a pull request template.”

This is interesting:

Sandboxing is not enabled by default in Nix due to a small performance hit on each build. In pull requests for nixpkgs people are asked to test builds with sandboxing enabled (see Tested using sandboxing in the pull request template) because in https://nixos.org/hydra/ sandboxing is also used.

It then tells me how to enable sandboxes. By editing my global /etc/nix/nix.conf? That seems dumb. If I just wanna test one derivation… there’s no --sandbox flag I can pass to nix-build, or something? Come on.

If you are updating a package’s version, you can use nixpkgs-review to make sure all packages that depend on the updated package still compile correctly. The nixpkgs-review utility can look for and build all dependencies either based on uncommited changes with the wip option or specifying a github pull request number.

nix run nixpkgs.nixpkgs-review -c nixpkgs-review pr 12345

Pretty neat!

Also, that’s much nicer looking than nix run -f '<nixpkgs>'. How does that work, then? Does nix run use ~/.nix-defexpr? I attempt to answer empirically:

$ nix run nixpkgs.hello -c hello
Hello, world!

$ nix run ianpkgs.hello -c hello
error: attribute 'ianpkgs' in selection path 'ianpkgs.hello' not found

Hmm. Maybe not. Or I just broke my ianpkgs thing that I set up forever ago.

Yes, okay. It seems I did break it, at some point:

$ nix-env -iA ianpkgs.hello
error: expression does not evaluate to a derivation (or a set or list of those)

I fix it – my default.nix wasn’t fully applying callPackage – and try again:

$ nix run ianpkgs.hello -c hello
error: attribute 'ianpkgs' in selection path 'ianpkgs.hello' not found

So, no. I thought there was a chance that it was just a bad error message, but it seems it is not using ~/.nix-defexpr at all.

So… what is it using? nix run --help doesn’t explain this. It does say:

To start a shell providing youtube-dl from your 'nixpkgs' channel:
$ nix run nixpkgs.youtube-dl

But… what? Is it literally defaulting to ~/.nixpkgs-defexpr/channels? That would be insane.

I have no idea.

I guess I have to look in the source.

Let’s see… src/nix/run.cc does not document the -f flag, but it seems to be an “installable” command. That leads me to installables.cc.

Which is sort of inscrutable. But it seems to rely on NIX_PATH elements, and duplicate some subset of the nix-env directory-traversing thing?

$ echo $NIX_PATH
/Users/ian/.nix-defexpr/channels:/Users/ian/.nix-defexpr/channels

Why is that my NIX_PATH? Because apparently I’ve sourced my ~/.zshenv since I started this shell session, and every time I do that, it adds an entry here.

$ source ~/.zshenv

$ echo $NIX_PATH
/Users/ian/.nix-defexpr/channels:/Users/ian/.nix-defexpr/channels:/Users/ian/.nix-defexpr/channels

How am I to express my feelings, with only the Latin alphabet at my disposal? I feel as though I must turn, in these trying times, to Unicode’s Supplemental Symbols and Pictographs block: 🤦‍♀️

Anyway, I find this issue by looking at the blame for installables.cc, which… does nothing to give me confidence in… anything. My takeaway is: no one knows how this works, and no one knows how this is supposed to work, because no one has decided how this is supposed to work, and that’s what you get for using nix run when it’s marked experimental.

Yeah; I have no idea here. Let’s do some experiments:

$ NIX_PATH=ianpkgs=$HOME/.nix-defexpr/ianpkgs nix run ianpkgs.hello -c hello
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I), at /Users/ian/scratch/default.nix:1:13

Ah. Right. That’s on me: I import <nixpkgs> in my default.nix.

$ NIX_PATH=ianpkgs=$HOME/.nix-defexpr/ianpkgs:nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs nix run ianpkgs.hello -c hello
Hello, Nix!

Okay. So that works. I guess. So this… treats the first component as a NIX_PATH element?

I don’t know.

I guess the -f '<nixpkgs>' version – while quite a mouthful – at least I could explain how it worked.

Anyway, moving on: description of branches. Where to put your PR. Simple stuff.

Chapter 20. Reviewing contributions

Be nice. Be polite.

GitHub provides reactions as a simple and quick way to provide feedback to pull requests or any comments. The thumb-down reaction should be used with care and if possible accompanied with some explanation so the submitter has directions to improve their contribution.

I heartily approve of this language.

Some examples of common tasks, and how to do them. Seems pretty nice. Updating packages. Adding new ones. Not a lot to say about this one.

Chapter 21. Contributing to this documentation

Now we’re talking.

$ cd /path/to/nixpkgs/doc
$ nix-shell
[nix-shell]$ make

I try it out, and just get a massive pile of error messages.

Starts like this:

pandoc preface.chapter.md -t docbook \
        --top-level-division=chapter \
        --extract-media=media \
        --lua-filter=/diagram-generator.lua \
        -f markdown+smart \
| cat  > preface.chapter.xml
Error running filter /diagram-generator.lua:
/diagram-generator.lua: openBinaryFile: does not exist (No such file or directory)

Ends like this:

^
manual.xml:40: element include: XInclude error : could not load contributing/contributing-to-documentation.chapter.xml, and no fallback was found
make: *** [Makefile:77: manual-full.xml] Error 1

With hundreds of similar error messages along the way.

If you experience problems, run make debug to help understand the docbook errors.

Okay. This gives me less output…

[nix-shell:~/src/nixpkgs/doc]$ make debug
nix-shell --run "xmloscopy --docbook5 ./manual.xml ./manual-full.xml"
trace: lib.crossLists is deprecated, use lib.cartesianProductOfSets instead
trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
trace: warning: lib.readPathsFromFile is deprecated, use a list instead
trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
trace: lib.zip is deprecated, use lib.zipAttrsWith instead
    Feasability check on files without xincludes...

A document is feasibly valid if it could be transformed into a valid
document by inserting any number of attributes and child elements
anywhere in the tree.

This is equivalent to pretending every element is optional.

This option may be useful while a document is still under
construction.

This option also disables checking that references are valid.
./release-notes.xml: OK
./preface.chapter.xml: Not feasibly valid:
Line 1:
make: *** [Makefile:8: debug] Error 1

I mean, it’s not… wrong, I guess.

[nix-shell:~/src/nixpkgs/doc]$ cat preface.chapter.xml

It’s just an empty file. Which is, in fact, not feasibly valid.

After making modifications to the manual, it’s important to build it before committing. You can do that as follows:

$ cd /path/to/nixpkgs/doc
$ nix-shell
[nix-shell]$ make clean
[nix-shell]$ nix-build .

Okay. I’ll bite.

I ran this, and it… succeeded? Just fine?

If the build succeeds, the manual will be in ./result/share/doc/nixpkgs/manual.html.

And indeed. That worked. Very different CSS – I’m so used to way the manual looks, after all these months, that it’s sort of jarring to see it using these alternative styles – but yeah, it’s all here.

So… I have no idea why it said to run make, when, in fact, the secret is to run nix-build. That’s… that’s extremely confusing.

Do you think this is a test? This is like, if you want to make contributions to the Nixpkgs documentation, the first thing you need to do is fix the documentation for how to make documentation contributions?

Ugh.

Well, one day, maybe. I read the whole Nixpkgs manual. Although I feel like I have a lot more to say about the Nix manual than the Nixpkgs manual, it’s not impossible that I’ll be back here some day, fixing typos or something at least.

For now? Farewell, Nixpkgs manual. You were… a lot longer and a lot harder to read than the Nix manual, if we’re being honest with each other.

  • How do hashed-mirrors work?
  • Why does Nix… disable TLS when fetching sources? What?
  • How do I enable the sandbox for a single build?