So this isn’t great:

$ nix-env -i python3
installing 'python3-3.10.0a5'

Why isn’t this great? Well, because in the Python versioning scheme, python3-3.10.0a5 is short for “Python 3.10.0 alpha version 5.” The latest stable release of the Python reference implementation, right now, is actually 3.9.5. 3.10 is currently considered the “pre” branch, and 3.11 is currently the “dev” branch. I think; I’m not really a Python person.

So it’s not great that a new Nix user might try to install python3 and wind up with some random alpha version of Python (the latest in the 3.10 branch is actually 3.10.0b2, for whatever that’s worth).

But it’s even less great because – at the time that I am writing these words right now – the python3-3.10.0a5 derivation is broken, and will not build.

I mean, maybe that’s actually a good thing: maybe it’s better that you have to stop and think and don’t just silently end up with the “wrong” version of Python. But… that’s a small consolation.

And of course I have to caveat all of this with a huge: you shouldn’t be running nix-env -i python3. That’s a bad thing, for a few reasons:

1. Dev tools like this shouldn’t be installed in your environment; they should be listed as dependencies in your default.nix or shell.nix file.

Okay, but to refute that, python3 isn’t just a development dependency, it’s also a useful tool to be able to drop into. I often run python3 to start up the repl and do a little math or light string processing or whatever: I think it’s perfectly reasonable to want to install python3 in your environment in addition to your actual project dependencies.

2. You shouldn’t run nix-env -i python3, you should run nix-env -iA nixpkgs.python3.

Okay, sure, but every single new Nix user is going to run that first thing because that’s the “obvious” command and it’s what the quick start guide told them to run and on and on and on.

And even if you do run nix-env -iA nixpkgs.python3, and then you later run nix-env -u to upgrade your packages, you’re going to have the same problem: nix-env -u will try to replace your nice python3-3.8.9 with python3-3.10.0a5 – although, fortunately(?), this will fail.

$ nix-env -iA nixpkgs.python3 --profile ~/scratch/profile
installing 'python3-3.8.9'

$ nix-env -u --profile ~/scratch/profile
upgrading 'python3-3.8.9' to 'python3-3.10.0a5'
...
Hunk #1 FAILED at 170.
Hunk #2 FAILED at 187.
Hunk #3 succeeded at 231 (offset 23 lines).
...
(your python remains the same)

In fact you may remember that I ran into this exact issue some months ago, but got around it by never using nix-env -u again. But that was sort of a cop-out, wasn’t it? Let’s dive back in, and see if we can get to the bottom of this.

Of all the things we could be worrying about

I’ve heard from a lot of people since I started writing this series: I’ve received more “comments” about Nix than any other subject I’ve blogged about. Which, like, I do realize that I’ve now written far more about Nix than any other subject, but my other posts are generally better and more interesting and less rambling and contain fewer long weird run-on sentences like this one because I actually proof-read them before I hit publish.

But last week was a first, for me. Last week I received my first ever email from someone asking for help. With Nix. And, I mean, you can see exactly how much I know about Nix, by reading this series. It ain’t much. But still: rather than post on StackOverflow, or the NixOS Discourse, or any other more useful, reputable platform, this person asked me. And they basically asked me: what’s up with this Python thing?

I was flattered that someone thought I knew something about Nix, and then I realized I actually did know something about Nix, sort of, and in the process of replying I learned a few more things about Nix, so now we’re going to re-trace my steps a bit so that you can also learn something about Nix.

So.

If you know much about Nix, you probably understand the problem: there are lots of packages “named” python3:

$ nix-env -qa python3 --json | jq '.[] |= .name'
{
  "nixpkgs.python310": "python3-3.10.0a5",
  "nixpkgs.python36": "python3-3.6.13",
  "nixpkgs.python36Full": "python3-3.6.13",
  "nixpkgs.python37Full": "python3-3.7.10",
  "nixpkgs.python37": "python3-3.7.10",
  "nixpkgs.gnuradio3_8Packages.python": "python3-3.8.9",
  "nixpkgs.python38Full": "python3-3.8.9",
  "nixpkgs.python3Full": "python3-3.8.9",
  "nixpkgs.sourcehut.python": "python3-3.8.9",
  "nixpkgs.python39": "python3-3.9.4",
  "nixpkgs.python39Full": "python3-3.9.4"
}

There is also obviously one at nixpkgs.python3, but apparently nix-env -qa only shows the “first” example of each derivation, and python3 is reference-equal to one of the other python3-3.8.9s:

nix-repl> :l <nixpkgs>
Added 14312 variables.

nix-repl> python3 == python38Full
false

nix-repl> python3 == python3Full
false

nix-repl> python3 == sourcehut.python
true

And for some reason that gets enumerated first, I guess.

Now, in the case that there are multiple packages named "python3", we remember that nix-env will use meta.priority to differentiate them. So why isn’t that working?

$ nix-env -qa python3 --json | jq '.[] |= .meta.priority'
{
  "nixpkgs.python310": null,
  "nixpkgs.python36": null,
  "nixpkgs.python36Full": null,
  "nixpkgs.python37Full": null,
  "nixpkgs.python37": null,
  "nixpkgs.gnuradio3_8Packages.python": null,
  "nixpkgs.python38Full": null,
  "nixpkgs.python3Full": null,
  "nixpkgs.sourcehut.python": null,
  "nixpkgs.python39": null,
  "nixpkgs.python39Full": null
}

Ah, yes, well, that would do it.

So that’s unfortunate: this is exactly the mechanism designed to fix this issue, but Nixpkgs is not employing it, because… well, because who knows. It’s a complicated world.

And since all of those packages have the same priority, Nix is falling back to version comparison, and picking the package with the highest version. Which is, in this case, 3.10.0a5.

But just because Nixpkgs doesn’t set meta.priority doesn’t mean that we can’t set meta.priority. We’re smart, confident Nix users. We can change the world, if we believe enough.

So my first thought was: I remember this weird obscure command, nix-env --set-flag. From man nix-env:

The --set-flag operation allows meta attributes of installed packages to be modified. There are several attributes that can be usefully modified, because they affect the behaviour of nix-env or the user environment build script:

• priority […boring stuff elided; I probably get it…]
• keep […same…]
• active […same…]

Sounds perfect!

However, when I actually read the description of priority, I realize that I had completely misunderstood:

priority can be changed to resolve filename clashes. The user environment build script uses the meta.priority attribute of derivations to resolve filename collisions between packages. Lower priority values denote a higher priority. For instance, the GCC wrapper package and the Binutils package in Nixpkgs both have a file bin/ld, so previously if you tried to install both you would get a collision. Now, on the other hand, the GCC wrapper declares a higher priority than Binutils, so the former’s bin/ld is symlinked in the user environment.

Okay. So meta.priority apparently means two different things. It not only decides which packages you install, but also whose files emerge victorious in the case of a collision during user environment construction.

And --set-flag can only usefully affect the latter, because:

The --set-flag operation allows meta attributes of installed packages to be modified.

You must have already installed the package in order to change this attribute, so it’s no use to us. Since we’re trying to affect which package gets installed, we’ll need to go to the source. Er, the metaphorical source. Not the source code, necessarily.

Because we remember the non-intrusive way to make changes to derivations: overrides. Right?

Sure. So I open up my ~/.config/nixpkgs/config.nix, and write this:

$ cat ~/.config/nixpkgs/config.nix

{
  allowUnfree = true;
  packageOverrides = pkgs: {
    python3 = pkgs.python3.overrideAttrs (attrs: attrs // { meta.priority = 100; });
  };
}

That doesn’t work for several reasons, but let’s start with the most obvious mistake I made.

This doesn’t set meta.priority.

This sets meta.

And it sets it to the attribute set { priority = 100; }. That’s not what I wanted to do, but it is exactly what I wrote, so… I’m not mad, but I feel a little silly. I realize that { meta.priority = 100; } is just syntax sugar for { meta = { priority = 100; }; }, and obviously I wouldn’t have written attrs // { meta = { priority = 100; }; }, because that’s very clearly overwriting the entire meta attribute. But still, in my head, I wanted // { foo.bar = 10; } to only set that one nested key.

Anyway, this is the sort of mistake that you make once and never make again, hopefully, and this was the time I made it. And now maybe you will not make it, if you haven’t already.

Now, there’s no “deep” version of the // operator in Nix, so we have to turn to nixpkgs.lib. I look through the function reference to remind myself the right function:

$ cat ~/.config/nixpkgs/config.nix

{
  allowUnfree = true;
  packageOverrides = pkgs:
    {
      python3 = pkgs.python3.overrideAttrs (attrs:
        pkgs.lib.attrsets.recursiveUpdate attrs { meta.priority = 100; });
    };
}

Which is just… ugh. Not only is that a mouthful, but take note of the incorrect argument order, so that these functions don’t compose or allow me to write that function point-free. Annoying.

But even with all of those extra characters, this still doesn’t work. It still wants to install 3.10.0a5.

$ nix-env -i python3
installing 'python3-3.10.0a5'

At this point I am confused. It should work, right? I go back and check man nix-env:

If there are multiple derivations matching a name in args that have the same name (e.g., gcc-3.3.6 and gcc-4.1.1), then the derivation with the highest priority is used.

That just… doesn’t seem to be true?

I sanity check that my override is actually happening:

$ nix-env -qa python3 --json | jq '.[] |= { name, priority: .meta.priority }'
{
  "nixpkgs.python310": { "name": "python3-3.10.0a5", "priority": null },
  "nixpkgs.python36": { "name": "python3-3.6.13", "priority": null },
  "nixpkgs.python36Full": { "name": "python3-3.6.13", "priority": null },
  "nixpkgs.python37Full": { "name": "python3-3.7.10", "priority": null },
  "nixpkgs.python37": { "name": "python3-3.7.10", "priority": null },
  "nixpkgs.gnuradio3_8Packages.python": { "name": "python3-3.8.9", "priority": 100 },
  "nixpkgs.python38": { "name": "python3-3.8.9", "priority": null },
  "nixpkgs.python38Full": { "name": "python3-3.8.9", "priority": null },
  "nixpkgs.python3Full": { "name": "python3-3.8.9", "priority": 100 },
  "nixpkgs.sourcehut.python": { "name": "python3-3.8.9", "priority": 100 },
  "nixpkgs.python39": { "name": "python3-3.9.4", "priority": null },
  "nixpkgs.python39Full": { "name": "python3-3.9.4", "priority": null }
}

So my change did take, it just… doesn’t work?

But ah, I needed to keep reading:

A derivation can define a priority by declaring the meta.priority attribute. This attribute should be a number, with a higher value denoting a lower priority. The default priority is 0.

Of course. When the manual said “the derivation with the highest priority is used,” it meant “the derivation with the lowest priority.” Obviously. And since the default priority is 0, that means we need to set a negative priority for nixpkgs.python3. Obviously.

$ cat ~/.config/nixpkgs/config.nix

{
  allowUnfree = true;
  packageOverrides = pkgs:
    {
      python3 = pkgs.python3.overrideAttrs (attrs:
        pkgs.lib.attrsets.recursiveUpdate attrs { meta.priority = -100; });
    };
}

And there it is! That works. We can now install python3 by name, or upgrade python3 at all, and get the version we actually want:

$ nix-env -i python3 --profile ~/scratch/profile
warning: there are multiple derivations named 'python3-3.8.9'; using the first one
installing 'python3-3.8.9'

Beautiful. And:

$ nix-env -u --profile ~/scratch/profile

Does nothing. Good!

So that solves it. But it’s a little weird and gross and complicated. I would prefer to write something like this:

{
  allowUnfree = true;
  simpleOverrides = { python3.meta.priority = -100; };
}

So… let’s do that? It’ll be good Nix practice.

So, uh, how do I get a list of key-value pairs again? I would definitely expect that to be a built-in. I can see builtins.attrNames and builtins.attrValues… hmm. There’s a builtins.listToAttrs, but no builtins.attrsToList. Darn. I guess it’s a lib function.

I look through the Nixpkgs manual and I can’t find the function I want… but I find mapAttrs, which is a better fit here anyway.

let overridePackageAttrs = overrides: pkgs:
  let update = with pkgs.lib; (flip recursiveUpdate); in
  pkgs.lib.mapAttrs (packageName: attrUpdates:
    pkgs.${packageName}.overrideAttrs (update attrUpdates))
    overrides;
in
{
  allowUnfree = true;
  packageOverrides = overridePackageAttrs { python3.meta.priority = -100; };
}

I won’t walk through that in detail; I think it’s pretty straightforward. But I will point out that it’s kind of annoying that I have to declare the update helper inside the overridePackageAttrs function because I need access to pkgs.lib – I can’t import <nixpkgs> in here, since this file is imported by Nixpkgs itself.¹

That’s a little nicer looking, and will make it easy to make other trivial changes in the future. Maybe. I don’t know if there’s a built-in nice way to do this already – the manual never mentioned it, if so – but it’s so little code that I’m not too bothered.

Besides, it’s not like this code should exist. This is just putting a band-aid over a problem in Nixpkgs. The right solution is to set the priority in Nixpkgs itself.

But it feels weird to explicitly set a negative priority. I wonder if any existing packages do that?

$ rg 'priority = -'
nixos/modules/services/web-servers/apache-httpd/default.nix
15:  apachectl = pkgs.runCommand "apachectl" { meta.priority = -1; } ''

pkgs/development/interpreters/python/cpython/2.7/default.nix
335:      priority = -100;

pkgs/tools/security/gnupg/1compat.nix
28:    priority = -1;

pkgs/os-specific/linux/tbs/default.nix
60:    priority = -1;

pkgs/os-specific/linux/rtw88/default.nix
38:    priority = -1;

What’s this? Python 2.7 already does this. Is that why you don’t have any problems when you run nix-env -i python?²

$ nix-env -qa python --json | jq '.[] |= { name, priority: .meta.priority }'
{
  "nixpkgs.gnuradio3_7Packages.python": { "name": "python-2.7.18", "priority": -100 },
  "nixpkgs.python27Full": { "name": "python-2.7.18", "priority": -100 },
  "nixpkgs.python2Full": { "name": "python-2.7.18", "priority": -100 },
  "nixpkgs.pythonFull": { "name": "python-2.7.18", "priority": -100 }
}

Er, okay. Not exactly. I guess there just… aren’t multiple pythons. But! If there were, you wouldn’t run into this problem. Probably.

Anyway; this seems obviously wrong, so let’s fix it.

Uhhh… how?

I want to say: by replacing this line:

python3 = python38;

With this line:

python3 = python38.overrideAttrs (attrs: lib.recursiveUpdate attrs { meta.priority = -100; });

So that we keep the change local with the reason for the change: if anyone goes in and bumps python3 to python39;, then suddenly python39 will automatically gain this higher priority. But I don’t know how cool this is?

I look through for other examples, because it looks super gross to me, and I see some examples of lib.meta. Aha! I had forgotten about these functions. But it seems that you can use the even shorter:

python3 = lib.hiPrio python38;

But when I test that, I see that it doesn’t actually work. Because lib.hiPrio doesn’t call overrideAttrs; it just uses // notation. So it’s actually:

python3 = python38.overrideAttrs lib.hiPrio;

That’s not too bad, though? And a nix-env -qa --json confirms that that does, in fact, work.

So… we can try opening a PR now? And maybe by doing so, we’ll learn the correct way to make such a change.

Maybe I should file an issue first, and then submit a PR to fix the issue. That seems nicer.

https://github.com/NixOS/nixpkgs/issues/126036

There are 208 open issues for the search term "python3" right now. It asked me to notify the maintainer, but I feel kind of bad for doing so. I’m about to fix it, and I’d rather notify the maintainer on my PR, but whatever. I did what it asked.

Man. Nixpkgs is so big. I’m pulling changes right now, and it’s just… it’s taking a while. Only ~85mb of changes, but GitHub seems to be throttling me pretty hard. I don’t remember it taking this long for the initial clone. Maybe clones are easier, because protocols or something. Maybe I should’ve just cloned from scratch.

Anyway, I opened the PR:

https://github.com/NixOS/nixpkgs/pull/126038

Perhaps someone will take the time to tell me the “right” way to do this. Or it will be closed as “this doesn’t matter what are you doing don’t ever use nix-env -u.” I hope not, but I do realize that this is a pretty subjective thing, and the way I think it “should” work may not match the way that actual Nixpkgs maintainers think it “should” work.

Anyway. I did my part. I learned the thing. I did not try to fix the weird broken python3-3.10.0a5 derivation, but… that’s okay with me. I’ll update with the outcome of that PR. Although, you know, you’ll be able to see it. It’ll be in the PR.