How to Learn Nix, Part 22:
Setting up a declarative user environment

We open on a chat conversation that I had with my good friend and former colleague “Doug,” in which we were discussing my experience reading through the Nix manual.

 ian: there were certain things that i wanted to learn how to do
 ian: and i still don't really know how to do them
 ian: like
 ian: it seems like
 ian: you should just have a file
 ian: called "ian.nix"
 ian: where you write down the things you want
 ian: and then there should be a way to say "install exactly this set
      of things"
 ian: but there isn't
 ian: so i still want to figure out how to write that
 ian: anyway

At this point I intended to change the topic, because I wanted to talk to Doug about videogames or something. But the topic was not to be changed:

doug: isn't that command nix-env -irf ian.nix? or am i missing something
 ian: wut
 ian: oh
 ian: i have no idea
 ian: maybe!
doug: that was roughly my takeaway from my adventures

I pause, and think about it. I remember nix-env --set, and how I think I can use that if I can figure out how to write an expression that is a derivation for a user environment – one of those good old /nix/store/xxx-user-environment directories that we’ve seen many times.

I had spent a little time thinking about this already, and my intention was to read through the Nixpkgs manual, as I suspected that somewhere in that massive repo I would find some mkUserEnvironmentDerivation function.¹ I could imagine how I would write this derivation myself – it’s basically just find $targets -mindepth 1 -exec ln -s {} $out \; (yes and a separate find to set up the directory structure and you’d need to get the relative path for the whatever I’m not getting sucked into this rabbit hole with you right now) – but there’s gotta be some built-in “official” way to do that. And if I could find that official way, I could probably learn a lot about how the environment actually gets set up.

Anyway. These thoughts pass through my head, and I reply succinctly:

 ian: i don't know how to write ian.nix
 ian: i guess
 ian: i will try things
 ian: and see what happens
 ian: no spoilers

But Doug is determined to spoil:

doug: have you considered a record? or a list? or maybe it doesn't matter
      and it's unclear why?

At this point I’m thinking “Oh, Doug doesn’t get it. See, nix-env -i requires a single derivation; I read the whole manual so I know that I need to give it an expression that resolves to a derivation.” I’m sure if I tried to “install” a list, that command invocation would just fail.

Or would it? I become unsure. There are no type signatures here. man nix-env’s section on --install seems to support my position: the documentation talks only about single derivations. But still: maybe? Dynamically typed languages love implicit overloading, right?

I politely disagree.

 ian: yeah that's i guess
 ian: i don't know WHY that would work
 ian: given my understanding of nix-env -i
doug: everything is lazy except for the thing where shit gets installed

Doug said a swear, and I apologize for repeating it. He didn’t know this was going to end up in a blog post.

 ian: but like i have always specified a single package
 ian: there's nothing about it being able to work on lists or sets or something
 ian: like is it going to recursively evaluate every derivation in
      any expression i give it?
 ian: that's insane and also probably exactly how it works
doug: my understanding is that it doesn't "work" on a list, it just evaluates
      the expression and the side-effect of doing that installs the packages
doug: and then discards the result

I meditate on this. It makes some sense, and actually tickles a bit of intuition that I remember from long ago: I remember thinking that Nix worked by evaluating expressions, and somehow collecting a list of derivations encountered during that evaluation – like a writer monad – and then installing them as a second pass once it had evaluated everything.

I had completely forgotten that I had this mental model before Doug said that, but it comes back to me. This was my intuition for how NixOS decided what packages it needed to install when I would change configuration.nix. I don’t think I ever thought to apply the intuition to nix-env, but maybe – it’s been a long time.

Still, that can’t be right. Because:

 ian: but that would mean that if you type nix-env -i it would install the
      whole world
 ian: which like

You know that scene at the end of Oldboy where you’re slowly realizing what’s happening right as the protagonist is slowly realizing what’s happening?

doug: yes
 ian: surely not
doug: oh yes
 ian: surely not

The curtain has been drawn back. I can see the entire scheme laid out before me. I am faced with a horror too great for me to comprehend. The mind rejects it.

 ian: oh man this is going in the post
 ian: this conversation

I respond flippantly, as I am wont to do.

doug: i mean my state was kind of fucked up the whole time
doug: so i might be remembering wrong or i typed something that i thought
      was right
 ian: i do remember you doing something and it was trying to install the
      whole world
 ian: but surely it wasn't THAT

At this point I have to apologize again for Doug’s language but also clarify something.

I had told Doug that I was writing this series before I had finished reading the Nix manual, and before I had published any of these blog posts. And he had pretty much the optimal response: “Oh, yeah, I did that once. Lemme send it to you.” He then proceeds to pull out his own Nix diary, a chronicle of his attempt to set up a brand new laptop using Nix as his primary package manager. (Sans NixOS, because he’s brave but not crazy.)

I had read through his diary when I was getting pretty close to the end of the Nix manual and thought it wouldn’t corrupt me too much. But I was sort of skimming it, and obviously overlooked this most fascinating of experiences.

Fortunately, he proceeded to quote from his own Nix diary:

doug: > But I don't know how to say "just go back to whatever the config is".
      > I thought nix-env -ir would be it (which is install and remove all,
      > equivalent to uninstalling everything first), but that seemed to...
      > want to install everything?

I trust him to remember this. But I also cannot accept it. I apply a heuristic to try to decide if this is true: technical Darwinism. If it were the case that typing nix-env -i would try to install every package in Nixpkgs, would Nix still exist? No: even if it ever worked that way, that is such an obvious bug that it would have been fixed years ago. Nix would not exist with that bug. And Nix still exists, so that bug must not exist. Right?

 ian: that's like
 ian: that's so
 ian: easy
 ian: to shoot yourself in the foot with

But the sad truth comes to light.

doug: https://github.com/NixOS/nix/issues/1334
doug: hahaha
doug: oh here's the original one https://github.com/NixOS/nix/issues/308
doug: 2014
doug: open
doug: > I've come to like this feature

And here we are.

I rush to get my thoughts down as quickly as I can, as I know that I will not remember the details of the conversational flow and my own thought processes come tomorrow. The date: March 4, 2021. The day that I published the first 4 posts in this series – which is why we were having this conversation in the first place.

A week later I pick this back up, and decide to actually get around to trying this. Because this conversation definitely shortened the time it would take me to figure this out, I included it above. Now we’re on the same page again. You are once again in sync with my Nix journey.

Because I don’t really want to mess up my own profile, I make a new one:

$ nix-env -iA nixpkgs.hello --profile ~/scratch/profile
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10
copying path '/nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/14p1q2z1shvjw6jkk5wd63l0bia0wx6v-user-environment.drv'...
created 2 symlinks in user environment

$ ll ~/scratch/profile*
/Users/ian/scratch/profile@ -> profile-1-link
/Users/ian/scratch/profile-1-link@ -> /nix/store/l8h2bfp2grx7c0ia3c4awmsf2m3666a0-user-environment

$ tree -l ~/scratch/profile
/Users/ian/scratch/profile/
├── bin -> /nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10/bin
│   └── hello
├── manifest.nix -> /nix/store/nhh2gf0cpiaw5rsnr4g3xnai1jflxmgx-env-manifest.nix
└── share -> /nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10/share
    ├── info
    │   └── hello.info
    └── man
        └── man1
            └── hello.1.gz

Neat. I wasn’t really sure if that would work.²

Now let’s try to make a “declarative” profile:

$ cat ian.nix

with import <nixpkgs> {}; [ hello git ]

$ nix-env -p ~/scratch/profile --set ian.nix
error: selector 'ian.nix' matches no derivations

Er, right. Uhh…

$ nix-env -p ~/scratch/profile --set 'import ./ian.nix'
error: getting status of '/Users/ian/scratch/import ./ian.nix': No such file or directory

???

I read the man nix-env and learn that --set only works if I provide the name of a package. So… I have no idea what to do about that. Not use --set, I guess. And here I thought… that it was anything? Nope. Let’s try it Doug’s way:

$ nix-env -p ~/scratch/profile -irf ian.nix
building '/nix/store/6mlp94f12lh71knbvki2qk4g8qrk8ala-user-environment.drv'...
created 186 symlinks in user environment

$ ls ~/scratch/profile/bin
git@                git-credential-netrc@       git-credential-osxkeychain@
git-cvsserver@      git-http-backend@           git-receive-pack@
git-shell@          git-upload-archive@         git-upload-pack@
hello@

Okay! Lotta symlinks. Mostly man pages for git. But… that was it! That was super easy.

Now I have no idea why that was super easy. I mean, I sort of do – some sort of mental image is forming here.

$ nix-env -qaf ian.nix
git-2.30.1
hello-2.10

So what it seems like nix-env is doing is recursively evaluating whatever expression you give it. Instead of just evaluating the argument you give it, it is “deeply” evaluating everything, and installing every derivation it can get its hands on – or so it seems.

Let’s test a few different things:

$ cat ian.nix

with import <nixpkgs> {}; [ [[hello] hello] { attribute = git; } ]

$ nix-env -qaf ian.nix
git-2.30.1
hello-2.10

$ nix-env -p ~/scratch/profile -irf ian.nix

Doesn’t seem to mind arbitrary nesting. Seems to deduplicate derivations. Evaluates to the exact same set, apparently, such that nix-env -irf gave me no output.

Let’s try something weirder:

with import <nixpkgs> {}; [ ({}: hello) { attribute = git; } ]

No problems with that. But if I try:

with import <nixpkgs> {}; [ (x: hello) { attribute = git; } ]

$ nix-env -qaf ian.nix
error: expression does not evaluate to a derivation (or a set or list of those)

Aha. Okay. And indeed, tossing other values in there – numbers, strings, nulls, etc – gives me the same error.

So okay – at least it’s like… at least it’s not just cherry-picking derivations out of an arbitrary tree. I can feel okay about this. Minus the part where it’s completely undocumented.³

Now I can write my own “real” ian.nix by basically copying the output of nix-env -q…

I remember, though, that nix-env -q reports derivation names, but I need attribute names to put in my file. But these are almost always the same thing, which is super confusing – you don’t really need to think about this distinction 90% of the time but then sometimes something goes wrong and suddenly you do.

Anyway, this works great except for nss-cacert, which is apparently just called nixpkgs.cacert.

I try to test this out, but I’m getting errors about an excellent package called ngrok, which I’ve added to my environment since last we spoke.

$ nix-env -p ~/scratch/profile -irf ian.nix
error: Package ‘ngrok-2.3.35’ in /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/pkgs/tools/networking/ngrok-2/default.nix:38
has an unfree license (‘unfree’), refusing to evaluate.

The rest of the error message basically tells me to do this:

$ mkdir -p ~/.config/nixpkgs

$ echo '{ allowUnfree = true; }' > ~/.config/nixpkgs/config.nix

And now it works just fine.

Neat! Okay.

This seems like a pretty good solution to the declarative user environment problem? That file is pretty simple:

$ cat ian.nix

with import <nixpkgs> {}; [
  cabal-install
  curl
  fzf
  git
  # ...
  xz
  yarn
  yasm
  zsh
]

Now, what else do I want to do?

Well, over time I expect that I will nix-env -iA things, and will want to see how my environment has diverged from my declarative environment so that I can bring them into sync.

But that’s easy to do:

$ nix-env -p ~/scratch/profile -iA nixpkgs.hello
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10
copying path '/nix/store/70pxcwpdiq7ddrk4w8axfl51s9xh9ahn-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/687kj91sv4cdvhf2z9rkrdaj2g7gb6g2-user-environment.drv'...
created 767 symlinks in user environment

$ diff -u --label ian.nix <(nix-env -qaf ian.nix) --label current <(nix-env -p ~/scratch/profile -q)
--- ian.nix
+++ current
@@ -3,6 +3,7 @@
 fzf-0.25.1
 git-2.30.1
 graphviz-2.42.2
+hello-2.10
 hexedit-1.2.13
 htop-3.0.5
 hugo-0.81.0

Handy command, maybe? It occurs to me that this is also a better version of nix-env -u --dry-run for seeing outdated packages – I tweak it a little bit to show me “the diff that will be applied to my environment if I resync,” which is the opposite of what I had above, and save it in my script directory.

Now it looks like this (after a nix-channel --update):

$ sd nix diff
--- current
+++ user.nix
-git-2.30.0
+git-2.30.1
-hello-2.10
-python3-3.8.7
+python3-3.8.8

For the record, in the “full version,” that’s:

$ sd cat nix diff

#!/usr/bin/env bash

set -euo pipefail

diff -U 0 \
  --label current <(nix-env -q) \
  --label user.nix <(nix-env -qaf ~/dotfiles/user.nix) \
| grep -v '^@'

Neat. Now any packages I manually install will show up as - lines to tell me that “re-syncing” with nix-env -irf ~/dotfiles/user.nix will uninstall them. I think that’s clear; I think I can remember that. I might end up hating it, or making the output fancier. For now? Seems great.

This is also much faster than nix-env -u --dry-run, because it’s not doing anything with package names, just attributes. So that’s pretty nice too. And it’s not going to end up trying to get me to install some weird alpha version of Python. Which is a bonus.

Alright! I feel pretty good about this. Thanks, Doug. I think it would have taken me much longer to get here without your help.

But before I go, I just have to see for myself…

$ nix-env -p ~/scratch/profile -i

Oh. Oh yeah. Oh yeah it does.

It blew out my scrollback, but I can show you the very end:

...
installing 'zxing-3.1.0'
installing 'zydis-3.1.0'
installing 'zynaddsubfx-3.0.5'
installing 'zz-unstable-2021-01-26'
installing 'zziplib-0.13.71'
installing 'zzuf-0.15'
error: Package ‘1oom-1.0’ in /nix/store/v43dzqk60bmd4rpksq5ix9gibcj18as9-nixpkgs-21.05pre273941.a2b0ea6865b/nixpkgs/pkgs/games/1oom/default.nix:27 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) To temporarily allow packages that are unsupported for this system, you can 
   use an environment variable for a single invocation of the nix tools.

     $ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

b) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)

Phew. Thanks, 1oom-1.0. Saved me there.

I do realize that I’m relying on exactly this same behavior whenever I run nix-env -irf ian.nix. And I like that I can do that. But I would really like it much more if it went something like this:

$ nix-env -irf ian.nix
error: no selector provided
If you want to install the entire package universe, re-run this
command with --yes-i-really-want-to-install-every-package

$ nix-env -irf ian.nix --yes-i-really-want-to-install-every-package

But oh well. There’s no way I’ll ever type nix-env -i by itself by accident, so what are the odds that I’ll ever have to think about this again he said his voice trailing off into a mumble.