Part III deals with using Nix as a package manager. The quick start guide already gave us a brief introduction to this, but now we get to go into detail.

Chapter 9. Basic Package Management

“Channels” come up again, and the manual teases that Chapter 12 is a whole chapter on channels, so we’ll presumably understand what they are soon.

It also tantalizingly implies that we don’t need channels to use Nix. We can just manually download the latest version of Nixpkgs. But it links to http://nixos.org/nixpkgs/download.html, which now just redirects to https://nixos.org/download.html, which seems to have no way to manually download Nixpkgs. So maybe it’s lying.

But this is something I am very eager to understand: “what is nixpkgs?” is second only to “what is a derivation?” on my list of Nix questions that keep me up at night.

So I downloaded Nixpkgs by hand. I think. I cloned the GitHub repo locally. I think that’s the same? We’ll find out.

The manual tells me I can run this:

$ nix-env -qaf ~/src/nixpkgs

To use my “local” copy. And man nix-env explains the -f/--file command as:

Specifies the Nix expression (designated below as the active Nix expression) used by the --install, --upgrade, and --query --available operations to obtain derivations. The default is ~/.nix-defexpr.

If the argument starts with http:// or https://, it is interpreted as the URL of a tarball that will be downloaded and unpacked to a temporary location. The tarball must include a single top-level directory containing at least a file named default.nix.

Fascinating. None of this makes sense. It’s called --file, but the default value is ~/.nix-defexpr, a cryptically named directory that just consists of two symlinks: one called channels and one called channels_root.

I still don’t know what the “active Nix expression” means.

And yet: the command works. I get a list of 32,586 packages, after the familiar 30 seconds of waiting for anything to happen. Meanwhile, because I haven’t updated my channel as recently as the clone (I assume) I only get 32,544 packages when I run without --file. Cool. So the flag is doing something.

But this is sort of weird, because ~/.nix-defexpr has a completely different structure than my clone of the Nixpkgs repo. This upsets me, and makes me feel afraid. I want to think about things in “types” or “interfaces” or something. I can wrap my head around “give it a directory containing a default.nix file.” I can’t quite wrap my head around “give it a directory containing a channels symlink to another directory that contains a directory called nixpkgs which itself contains a file called default.nix.” And the fact that it accepts either? That’s messed up.

Can I just give it a file? Let’s see.

$ nix-env -qaf ~/src/nixpkgs/default.nix | wc -l
32586

Yep.

At first I was sad and confused that it didn’t explain what types of things can be passed to --file, and I wrote a little thing about how it might seem trivial but these things are very important to feel comfortable using a tool and I felt all simultaneously hurt and righteous.

But then I looked harder and it turns out the man page does describe it, just in a completely different section nowhere near where the --file flag is described. And it doesn’t actually describe --file, but rather ~/.nix-defexpr, which is the default value of --file, so it’s… it’s hidden.

But it says:

If ~/.nix-defexpr is a file, it is loaded as a Nix expression. If the expression is a set, it is used as the default Nix expression. If the expression is a function, an empty set is passed as argument and the return value is used as the default Nix expression.

If ~/.nix-defexpr is a directory containing a default.nix file, that file is loaded as in the above paragraph.

If ~/.nix-defexpr is a directory without a default.nix file, then its contents (both files and subdirectories) are loaded as Nix expressions. The expressions are combined into a single set, each expression under an attribute with the same name as the original file or subdirectory.

For example, if ~/.nix-defexpr contains two files, foo.nix and bar.nix, then the default Nix expression will essentially be
{
  foo = import ~/.nix-defexpr/foo.nix;
  bar = import ~/.nix-defexpr/bar.nix;
}
The file manifest.nix is always ignored. Subdirectories without a default.nix file are traversed recursively in search of more Nix expressions, but the names of these intermediate directories are not added to the attribute paths of the default Nix expression.

Okay. So looking at my ~/.nix-defexprs, and trying to follow the rules laid out here…

$ find -L ~/.nix-defexpr -not -name '.*' -maxdepth 3
/Users/ian/.nix-defexpr/channels_root
/Users/ian/.nix-defexpr/channels
/Users/ian/.nix-defexpr/channels/manifest.nix
/Users/ian/.nix-defexpr/channels/nixpkgs
/Users/ian/.nix-defexpr/channels/nixpkgs/maintainers
/Users/ian/.nix-defexpr/channels/nixpkgs/README.md
/Users/ian/.nix-defexpr/channels/nixpkgs/COPYING
/Users/ian/.nix-defexpr/channels/nixpkgs/pkgs
/Users/ian/.nix-defexpr/channels/nixpkgs/flake.nix
/Users/ian/.nix-defexpr/channels/nixpkgs/lib
/Users/ian/.nix-defexpr/channels/nixpkgs/default.nix
/Users/ian/.nix-defexpr/channels/nixpkgs/doc
/Users/ian/.nix-defexpr/channels/nixpkgs/nixos
/Users/ian/.nix-defexpr/channels/nixpkgs/svn-revision

(-L causes it to follow symbolic links, and I excluded all the hidden files like .git in nixpkgs/ just to make it a little more readable.)

Okay. So, if I understood correctly: it’s going to search channels_root and channels for default.nix files or else other .nix files, if the directory doesn’t have default.nix. Then it’s going to return a “set” containing the expressions in those files as “attributes.” So if I understood the example correctly, we’ll wind up with an expression that looks like this:

{
  nixpkgs = import /Users/ian/.nix-defexpr/channels/nixpkgs/default.nix;
}

Because the directory channels/ didn’t contribute to the name of the attribute, and channels_root was completely empty. I assume that winds up looking something like this:

{
  nixpkgs = {
    hello = (some derivation);
    git = (some derivation);
    ...
  };
}

Which would explain why we say nix-env -iA nixpkgs.hello.

Neat! To check my understanding, I tried to make a differently “named” nixpkgs:

$ mkdir ~/scratch

$ ln -s ~/src/nixpkgs ~/scratch/ianpkgs

$ nix-env -f ~/scratch -iA ianpkgs.hello
replacing old 'hello-2.10'
installing 'hello-2.10'

Great! Okay. So that sort of makes sense. We see the string nixpkgs because that’s the name of a child directory of ~/.nix-defexpr. Presumably the rest of the structure there will make sense once we understand channels.

I also wanna say: this makes sense, right? Because presumably I could have nixpkgs and like proprietary-company-software at the same time, and I’d need to specify which I wanted.

But the common, single user case– I mean, like 99% of cases where a human invokes nix-env, they’re going to be referring to nixpkgs, right? I’m sort of tempted here to make my ~/.nix-defexpr just symlink to the nixpkgs/default.nix so I can just write nix-env -iA hello instead of nix-env -iA nixpkgs.hello, and then I could just override my expression to something else in the few cases where I wanted to something that wasn’t in nixpkgs. But I am afraid to do this just yet, before I have a better understanding of the purpose of the channels directory…

Anyway. Moving on, this section teaches me that nix-env -q accepts regular expressions, even though nix-env -u accepted globs. Weird, but yeah, I gathered that from the 30 second syntax error in the last post.

It also teaches me I can use nix-env -qas to see the “status” of all “available” packages. As I’ve said before, there are countless thousands of available packages, so this command does not feel very useful, but presumably it takes regexes too. Or globs. Who knows?

It says if a package is installed in my environment, whether it’s present on the system (like hello is, even after I nix-env --uninstall it), and whether or not there is a “substitution” available.

This command seems to take even longer than usual so I didn’t play around with it.

But the manual highlights an interesting thing here: if I use my local clone of nixpkgs, I might not get as many binary cached packages. But if I use the channel, I will only get updated packages once all the new binaries have been cached. So that’s a good thing to know. The “channel” lags the repo a little bit, to make time for caching.

It also explains the difference between -u and -i:

This will only upgrade Subversion if there is a “newer” version in the new set of Nix expressions, as defined by some pretty arbitrary rules regarding ordering of version numbers (which generally do what you’d expect of them). To just unconditionally replace Subversion with whatever version is in the Nix expressions, use -i instead of -u; -i will remove whatever version is already installed.

Okay, sure. I still sort of question the usefulness of -u, but I guess it’s a nice way to upgrade all my packages at once – it would be annoying to reinstall all of them (or would it? I’ve already downloaded them, right? Does it actually have to do anything to reinstall them?). But I would probably just use -i if I had a specific one to upgrade, I think.

The chapter closes with an example of a command that looks equivalent to brew outdated. It’s:

$ nix-env --upgrade --dry-run

With no arguments, --upgrade will upgrade all your packages, and --dry-run obviously just prints what it would do.

This is… fine, I guess, but I’m a little saddened. In my head “list outdated packages” is a single, like, thing that I do. I like having a command for it. I use it to look up patch notes for software that I care deeply about before I upgrade it. A dry run upgrade is sort of… meh. I wish there was a nix outdated command, is what I’m saying. And this intrudes on my fantasy of getting rid of -u altogether.

Why is ~/.nix-defexpr not just a symlink to nixpkgs?
Why does Nix call what seem to be associative arrays, or records, or something “sets”?
Does nix-env -iA nixpkgs.hello actually do anything if hello is already installed?

How to Learn Nix, Part 5:Basic package management

Chapter 9. Basic Package Management

How to Learn Nix, Part 5:
Basic package management