Well, I took a bit of a break from learning Nix, but I’m back to continue my journey through the documentation.

I already read the whole Nix manual, so now it’s time read through the Nixpkgs manual. Wish me luck.

Chapter 1: Preface

This manual primarily describes how to write packages for the Nix Packages collection (Nixpkgs). Thus it’s mainly for packagers and developers who want to add packages to Nixpkgs.

Hmm. At this point in my life I don’t really want to contribute to Nixpkgs. I was hoping that by reading this manual I would learn something about how to specify, like, opam dependencies or something. How to actually write shell.nix files for nontrivial projects. We shall see if this hope is in vain.

We get a little overview of channels, including the nixpkgs channels vs the nixos channels.

Hmm. The manual describes channels named nixpkgs and nixos-unstable. But the default Nix installation uses a channel called nixpkgs-unstable. I assume this is documentation rot, as the manual’s description of nixpkgs (that it follows master) seems to match my understanding of nixpkgs-unstable.

I learn the word Hydra, which was a word I already remembered, and learn that it produces builds for x86_64-linux, i686-linux and x86_64-darwin. What about the new “M1” Macs? I guess they aren’t cached yet.

Chapter 2. Global configuration

I learn various reasons that Nix will refuse to install packages. It lists a few reasons:

First off, if meta.broken = true. It doesn’t say anything about why this would be the case. Is this intended for like… package maintainers to easily mark the latest release as broken? Is this something Nix will override with a mechanism like the nix-env --set-flag thing? What’s the point of this?

The next two reasons make sense: if a package is not supported on the current platform, or if it has an unfree license – I encountered the latter already.

The last reason is if there are any meta.knownVulnerabilities. Why is this different from broken? Just better signalling? I don’t know.

Then we learn something fascinating:

Note that all this is checked during evaluation already, and the check includes any package that is evaluated. In particular, all build-time dependencies are checked. nix-env -qa will (attempt to) hide any packages that would be refused.

Innnnteresting. So nix-env -qa is not actually printing all the derivations it can find in an expression, as I thought I learned previously. It’s actually filtering things out. Let’s see it:

$ cat packages.nix
with import <nixpkgs> {}; [

$ nix-env -qa --file packages.nix

But if I break a package…

$ cat packages.nix
with import <nixpkgs> {}; [
  (hello // { meta.broken = true; }) git

$ nix-env -qa --file packages.nix

Nope. Doesn’t… doesn’t work. Why not? Because // { meta.broken = true; } didn’t do the thing I expected it to do? Because it doesn’t actually filter broken packages? I don’t know.

$ cat packages.nix
with import <nixpkgs> {}; [
  (hello // { name = "something"; })

$ nix-env -qa --file packages.nix

Well, hmm. It does seem like it’s doing something.

$ cat packages.nix
with import <nixpkgs> {}; [
  (hello // { meta.knownVulnerabilities = ["something bad"]; })

$ nix-env -qa --file packages.nix

But not this. I have no idea. It seems like nix-env -qa is not actually doing the filtering that the manual describes, or I have no idea what I’m doing. Which is… plausible.

Moving on, we learn about the Nixpkgs configuration. I have already configured Nixpkgs to allow me to install unfree programs, and the manual doesn’t say anything about any other configuration option.

This is a curious note:

Note that we are not able to test or build unfree software on Hydra due to policy. Most unfree licenses prohibit us from either executing or distributing the software.

Huh. I have definitely installed “unfree” packages, and I thought I was installing them from the binary cache. But apparently not:

$ nix-env -qasA nixpkgs.keen4
---  keen4

$ nix-env -qasA nixpkgs.ngrok
IP-  ngrok-2.3.35

I suspect that I didn’t notice that I was “building” these myself because they are just set up to download binary distributions from somewhere else. But I don’t know how to view the derivation description.

I tried:

$ nix-env -qaA nixpkgs.ngrok --json

But that just prints out, like, the name and meta fields and stuff. Nothing about the actual Nix expression or builder or anything interesting.

How do I see that? Previously I have just been rging through the Nixpkgs tree, but there’s gotta be a better way to do it. But I don’t know how yet, so I eventually find:

$ cat ~/src/nixpkgs/pkgs/tools/networking/ngrok-2/default.nix
{ lib, stdenv, fetchurl }:

with lib;

let versions = builtins.fromJSON (builtins.readFile ./versions.json);
    arch = if stdenv.isi686 then "386"
           else if stdenv.isx86_64 then "amd64"
           else if stdenv.isAarch32 then "arm"
           else if stdenv.isAarch64 then "arm64"
           else throw "Unsupported architecture";
    os = if stdenv.isLinux then "linux"
         else if stdenv.isDarwin then "darwin"
         else throw "Unsupported os";
    versionInfo = versions."${os}-${arch}";
    inherit (versionInfo) version sha256 url;

stdenv.mkDerivation {
  name = "ngrok-${version}";
  version = version;

  # run ./update
  src = fetchurl { inherit sha256 url; };

  sourceRoot = ".";

  unpackPhase = "cp $src ngrok";

  buildPhase = "chmod a+x ngrok";

  installPhase = ''
    install -D ngrok $out/bin/ngrok

  passthru.updateScript = ./update.sh;

  meta = { /* ... */ };

And yeah, it seems to just download a literal binary, run chmod a+x it, and then copy it into place – consulting a JSON file to find out the particular path to fetch.

I have never seen the install command before. I run man install to see what -D does, but my man page doesn’t list -D as an option. I realize that I’m looking at the BSD install that comes with macOS, and Nix is surely providing the GNU version to this script.

Hmm. I try nix-shell -p coreutils and run man install from there, but that doesn’t work. Is install not in coreutils? Where does it come from? I don’t know how to answer that. I google it, and find that it is in coreutils. Hmmmmm.

$ nix-shell -p coreutils

[nix-shell:~/scratch]$ man --path | tr ':' '\n' | grep /nix/store

That’s just weird, right? Why don’t I get coreutils in my MANPATH? Well, because I have no MANPATH:

[nix-shell:~/scratch]$ echo $MANPATH

Hmmm. I thought that Nix was setting that for me. But I guess not.

Then where are those man pages coming from? I’ve honestly never had to think about this before.

My /etc/man.conf is quite large. And no more illuminating.

man man tells me that if there is no explicit rule for where to find man pages for executables in a particular directory, it’s going to “search nearby directories.” I assumed that this meant, like, if I have /foo/bar/exe on my path, and I run exe, it would search for like /foo/bar/man, /foo/man, /foo/share, etc.

But apparently it’s much weirder. It appears to be checking through all elements in my PATH. Like, not just where the command is in my PATH. It’s just looking through every PATH entry I have and checking each of them, just in case.

That’s… that’s messed up. TIL.

Anyway, while (e.g.) findutils has manual entries:

[nix-shell:~/scratch]$ ls $(dirname $(which find))/../share/man/man1
find.1.gz  locate.1.gz  updatedb.1.gz  xargs.1.gz

coreutils does not.

[nix-shell:~/scratch]$ ls $(dirname $(which install))/..
bin  libexec

Huh. Why is this? I suspect coreutils has a different “output” for the man pages.

[nix-shell:~/scratch]$ grep outputs ~/src/nixpkgs/pkgs/tools/misc/coreutils/default.nix
  outputs = [ "out" "info" ];

Hrmmm. info. Surely I don’t have to…

[nix-shell:~/scratch]$ info install | head -n4
info: Writing node (*manpages*)install...
info: Done.
File: *manpages*,  Node: install,  Up: (dir)

INSTALL(1)                BSD General Commands Manual               INSTALL(1)

Nope. Still the macOS version. Well hmm.

[nix-shell:~/scratch]$ nix path-info coreutils
error: attribute 'coreutils' in selection path 'coreutils' not found

[nix-shell:~/scratch]$ nix path-info nixpkgs.coreutils

Hmm. That’s the default output. I look through nix path-info --help and can’t find anything about specifying a different output path. I try nix-path-info --json, which prints a single line of un-prettified JSON. So I have to re-run it piped to jq because, you know, that’s how the world works. And it still only contains info about the default output path.

So… hmm. I think… I remember a builtin Nix function that would give me the output path of a derivation. I never tried it, but maybe that’s the way to find this?

[nix-shell:~/scratch]$ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> builtins.placeholder

nix-repl> builtins.placeholder "something"

nix-repl> builtins.placeholder "out"

Okay that’s… clearly I misunderstood that one. That looks more like a weird gensym thing. Alright. I… I guess grep to the rescue?

[nix-shell:~/scratch]$ ls /nix/store | grep coreutils

Well, okay. I don’t have the info output installed? I guess? So I have all the GNU coreutils, but not their documentation. Super.

How do I… get that? I try this on a whim:

$ nix-shell -p coreutils.info
these paths will be fetched (0.18 MiB download, 0.91 MiB unpacked):
copying path '/nix/store/gg4347kccwbipad94cw8rh3ynmv8kahz-coreutils-8.32-info' from 'https://cache.nixos.org'...

And it worked! Why did it work? I don’t know. I still don’t understand what a derivation is, or what “attributes” it has, or… how any of this works.

Anyway, I have the info page, but I have no idea how to get info to actually consult this page.

[nix-shell:~/scratch]$ info install -d /nix/store/gg4347kccwbipad94cw8rh3ynmv8kahz-coreutils-8.32-info/share/info
info: dir: No such file or directory

[nix-shell:~/scratch]$ INFOPATH=/nix/store/gg4347kccwbipad94cw8rh3ynmv8kahz-coreutils-8.32-info/share/info info install
(displays the BSD one)


I don’t know. Screw it.

[nix-shell:~/scratch]$ info -f /nix/store/gg4347kccwbipad94cw8rh3ynmv8kahz-coreutils-8.32-info/share/info/coreutils.info

That works, and after navigating then info UI for a bit – why is this… just why – I finally find the documentation for install, which says:

     Create any missing parent directories of DEST, then copy SOURCE to
     DEST.  Explicitly specifying the ‘--target-directory=DIR’ will
     similarly ensure the presence of that hierarchy before copying
     SOURCE arguments.

     Create any missing parent directories, giving them the default
     attributes.  Then create each given directory, setting their owner,
     group and mode as given on the command line or to the defaults.

Well wasn’t that easy.1

Alright. That was a horrible detour. All to answer the question of what this little derivation is doing:

stdenv.mkDerivation {
  name = "ngrok-${version}";
  version = version;

  # run ./update
  src = fetchurl { inherit sha256 url; };

  sourceRoot = ".";

  unpackPhase = "cp $src ngrok";

  buildPhase = "chmod a+x ngrok";

  installPhase = ''
    install -D ngrok $out/bin/ngrok

  passthru.updateScript = ./update.sh;

  meta = { /* ... */ };

And honestly? I still don’t understand what the difference is between -d and -D. Both seem to be to do the mkdir -p thing. BSD install only has -d. I don’t know. I learned nothing. It does roughly the thing that I assumed that it did from the very beginning of this.

It seems weird to use install – which as far as I can tell exists to copy files while setting owner/group/permission bits to something – while also running chmod a+x in the buildPhase. Like… we could just… whatever. None of this matters.

So yes, the Nixpkgs manual is telling the truth: we did not get this binary distribution from the official NixOS cache.

Let’s keep reading.

In the same way that I can allowUnfree, I can also allowBroken, or allowUnsupportedSystem. Okay. I am pleased by this explanation:

The difference between a package being unsupported on some system and being broken is admittedly a bit fuzzy. If a program ought to work on a certain platform, but doesn’t, the platform should be included in meta.platforms, but marked as broken with e.g. meta.broken = !hostPlatform.isWindows. Of course, this begs the question of what “ought” means exactly. That is left to the package maintainer.

That gives me some intuition for what broken means, in a very human way. I like that, and I’m not even going to say anything pretentious about the idiom they employed.

I learn that I can set whitelistedLicenses and blacklistedLicenses, in case I am a lawyer with strong opinions about software licenses.

There is no blanket allowInsecure, but I can whitelist specific packages to install. That’s nice.

2.5. Modify packages via packageOverrides

Now we’re talking. This seems really useful. The example given is this:

  packageOverrides = pkgs: rec {
    foo = pkgs.foo.override { ... };

Which I can use to change derivations on the fly, I guess? What is .override? A method??

Unfortunately the example doesn’t actually tell us how to use it, but it must be intuitive, right? Let’s see if we can see this in action.

$ cat packages.nix
with import <nixpkgs> {}; [
  (hello.override { meta.knownVulnerabilities = ["something bad"]; })
$ nix-env -qa --file packages.nix
error: anonymous function at /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/pkgs/applications/misc/hello/default.nix:1:1
called with unexpected argument 'meta', at /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/customisation.nix:69:16

Okay nope. It seems only specific fields can be overridden. Okay…

$ cat packages.nix
with import <nixpkgs> {}; [
  (hello.override { name = "goodbye"; })
$ nix-env -qa --file packages.nix
error: anonymous function at /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/pkgs/applications/misc/hello/default.nix:1:1
called with unexpected argument 'name', at /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/customisation.nix:69:16

Okay, well, I have no idea how override works. Why does the manual introduce this without explaining how to use it? This is literally all that section 2.5 contains. Just this one example that doesn’t actually tell me how to override packages. I think this ranks as the worst thing I have encountered in the Nix documentation so far.

2.6. Declarative Package Management

Huh! Here we go. Okay, so in my last post I said that I thought I would find some function in Nixpkgs that would allow me to build a user environment. And I did! It’s here. This seems to be telling me exactly how to do the thing I did in my last post.

  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [

pkgs.buildEnv. Very interesting. It doesn’t really make sense to me how this works – so this basically means we only have one package available in our Nixpkgs now?

To install it into our environment, you can just run nix-env -iA nixpkgs.myPackages.

That makes sense, yeah.

If you want to load the packages to be built from a working copy of nixpkgs you just run nix-env -f. -iA myPackages.

What? I think that means, like, basically --file ./default.nix – I’m not sure why that statement is being made here. This seems like a confusing complication that isn’t really related to the topic at hand. I don’t see why this makes it more likely that you would not want to use channels. I don’t get it.

To explore what’s been installed, just look through ~/.nix-profile/. You can see that a lot of stuff has been installed. Some of this stuff is useful some of it isn’t. Let’s tell Nixpkgs to only link the stuff that we want:

  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [ /* ... */ ];
      pathsToLink = [ "/share" "/bin" ];

Okay, that’s pretty cool, I guess? I don’t think the extra symlinks are really hurting me, though. But sure.

pathsToLink tells Nixpkgs to only link the paths listed which gets rid of the extra stuff in the profile. /bin and /share are good defaults for a user environment, getting rid of the clutter. If you are running on Nix on MacOS, you may want to add another path as well, /Applications, that makes GUI apps available.

I quoted that mostly because I didn’t know that Nix had any GUI macOS apps. I’m kind of surprised. Maybe I can get rid of my Homebrew casks after all? I just assumed it wouldn’t do that. I’ll have to look into it.

The manual goes on to describe a lot more things that I can do to this – installing extra outputs to get documentation (the manual lists man and doc as relevant output names, but not our old friend info).

This provides us with some useful documentation for using our packages. However, if we actually want those manpages to be detected by man, we need to set up our environment. This can also be managed within Nix expressions.

My goodness. It goes on to describe, like, managing my etc/profile.d using Nix? By hand? I don’t need to do any of these things. My man, I have learned, detects those man pages just fine. I’m not really sure what is happening here – this seems like a much more complicated solution than just using nix-env in the way that I did. I don’t get it.

Aha! The next section describes setting up info pages correctly. Okay, it’s pretty complicated. Basically adds this expression:

postBuild = ''
  if [ -x $out/bin/install-info -a -w $out/share/info ]; then
    shopt -s nullglob
    for i in $out/share/info/*.info $out/share/info/*.info.gz; do
        $out/bin/install-info $i $out/share/info/dir

I guess a lack of install-info is the reason I couldn’t look at the coreutils info page? Which lives in the texinfoInteractive package, I learn. Although, like, surely there are man pages for GNU coreutils, right? I realize I am not a daily Linux user. But like… people don’t really use info, do they?

I ask a friend who runs Linux on his laptop and who – as of March 2021 – has a working webcam for the first time in his life.

 ian: doug can i do an experiment on you

And he reports that nope, he has man install, it works exactly like anything else, and it does not appear to be something specific to his distro – he quotes the end of the man page:

       GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
       Report any translation bugs to <https://translationproject.org/team/>
       Copyright © 2020 Free Software Foundation, Inc.  License GPLv3+: GNU 
       GPL version 3 or later <https://gnu.org/licenses/gpl.html>. This is 
       free software: you are free to change and redistribute it.  There is 
       NO WARRANTY, to the extent permitted by law.
       Full documentation <https://www.gnu.org/software/coreutils/install>
       or available locally via: info '(coreutils) install invocation'
GNU coreutils 8.32             March 2020                        INSTALL(1)

Which certainly seems like a normal GNU thing, and not Arch added to make his life better.

Alright, this is the end of the chapter. I didn’t try packageOverrides because… well, because it seems bad? It seems worse than the thing I did? I don’t know. I don’t know why I would do this from a Nix config file. This just seems like a weird abuse of the concept of configuration, and of packageOverrides as well. I don’t really know what I would be breaking if I made my nixpkgs only contain a single package. Would I lose the ability to nix-shell -p? nix-build? Does this only affect nix-env, for some reason? Could I still nix-env -qa, or would that only return my one environment package?

Sigh fine I guess I’ll test it.

$ cat ~/.config/nixpkgs/config.nix
  allowUnfree = true;
  packageOverrides = pkgs: with pkgs; {
      myPackages = pkgs.buildEnv {
        name = "ian-packages";
        paths = [
$ nix-env -qaA nixpkgs.myPackages

So it exists. But so does everything else:

$ nix-env -qa | wc -l

This is not how the manual said that packageOverrides behaved. To quote:

It must be a function that takes pkgs as an argument and returns a modified set of packages

So it does not return a modified set of packages. It returns, like, a set of modifications to apply to the set of packages? I don’t know. I don’t know how the result is being interpreted. Obviously it’s not just mapping a set of packages to a new set of packages, which was how I read that.

But anyway, it seems like I can use it to “add” my own custom packages to the set. And of course I could have the config file import this list from my dotfiles or whatever, so I wouldn’t actually need to maintain my environment here. But I really like the sd nix diff command that I have, and I don’t know how I would re-create that, so it seems like my solution is both simpler and nicer. So. I’m just gonna stick with that.

But I am curious about this buildEnv character, because back in part 20 I thought I learned that the construction of the user environment actually took place in C++ code. I might be misremembering; it was a while ago now. So I look up the function in my Nixpkgs checkout, and find that yep, it’s a derivation that runs a Perl script to just create a whole bunch of symlinks. Huh! But this is separate from the code built into Nix to build the environment. Because nix-env doesn’t have a dependency on the contents of Nixpkgs – that would be crazy.

This gives me a sort of crazy idea.

You may remember this bug I encountered when I was trying to switch from Homebrew to Nix, that prevented me from installing mercurial. But what if… could I just…?

$ cat ~/.config/nixpkgs/config.nix
  allowUnfree = true;
  packageOverrides = pkgs: with pkgs; {
      mercurialHack = pkgs.buildEnv {
        name = "mercurial-workaround";
        paths = [ mercurial ];
$ nix-env -iA nixpkgs.mercurialHack
installing 'mercurial-workaround'
these derivations will be built:
building '/nix/store/63nk5cmxwfx37ar17pzrknlh0j84akg7-builder.pl.drv'...
building '/nix/store/jba37vrkk4pb3sh0j4ff755nm2n4nhc6-mercurial-workaround.drv'...
created 3 symlinks in user environment
building '/nix/store/bwk367244phrmp6hn8i0a3s3jlicqgb0-user-environment.drv'...
created 990 symlinks in user environment

Ahahaha. I can’t believe that worked. But yeah, the bug had nothing to do with Mercurial itself, it was just that the derivation had too many meta fields. But this weird Perl script doesn’t care about that. Haha. Oh man.

Well, that was fun.

But packageOverrides doesn’t play nicely with my user.nix thing. I have no idea when these overrides take place or which commands are going to actually use this. So I can’t… just use this. But I can use buildEnv to do the same thing, declaratively:

$ head -n4 ~/dotfiles/user.nix
with import <nixpkgs> {}; [
  (buildEnv { name = "mercurial-workaround"; paths = [ mercurial ]; } )

Lovely. This gets my brew list down to a modest:

autoconf        nodenv          pkg-config      readline
node-build      openssl@1.1     pyenv

Plus, you know, all my casks. But perhaps we’ll be rid of those soon as well.

  • How do I quickly “go to definition” of a package?
  • Why don’t I have man pages for coreutils?
  • Why do I have to explicitly install info pages for coreutils?
  • How do I print all output paths of a derivation?
  • What on earth is builtins.placeholder for?
  • What is derivation.override? A method?
  • How do I use derivation.override?
  • Can I install my macOS GUI apps with Nix?
  • Why is there buildEnv – a function – in the top-level of Nixpkgs, when I learned before that nix-env required all top-level things to be derivations?

  1. After going through all this I looked at the coreutils definition, and while I still don’t really understand why there is no man page distributed, I learned that there is a coreutils-full package that does include man pages, for whatever reason. So the “easy” way to do all this, is:

    nix-shell -p coreutils-full --run 'man install'

    Obviously. ↩︎