I opened a PR a few days ago to try to fix the issue I ran into with nix-env -u trying to upgrade to a weird alpha pre-release version of Python.

In my head, I was making the world a better place. It was an issue that bit me; it was an issue that bit someone who emailed me; it seemed like an issue worth fixing. But the maintainer of Python responded to that PR with:

I don’t think we should be doing this. Yes, it resolves the issue for python3, but there are others out there as well. “It is known” that nix-env -i should not be used.

My gut reaction to that comment was: well, sure, but we shouldn’t let perfect be the enemy of good. python3 is such a common, important package! Who cares if there are some random little packages sprinkled here and there that have the same problem. It’s worth fixing for big packages like Python.

But this person knows a lot more about Nix than I do. And a lot more about Nixpkgs. So if they believe that this is not worth fixing, there is a very good chance that this is not worth fixing, regardless of what my gut has to say about it.

So: let’s find out? Let’s look into this, and figure out how widespread this issue actually is.

I had assumed that it was a rare issue, because python3 is the only problematic package I’ve encountered, and I have quite a few packages installed in my environment at this point. Add to that the fact that it’s a relatively recent problem – introduced some time between NixOS 20.09 and NixOS 21.05 – and it seems like this isn’t typical. python3 is a weird outlier, right?

Let’s find out!

So I did. I found out. I did the thing. And, having done the thing, I have changed my tune:

python3 is the least of our worries.

nix-env -i is broken for so many packages – big, important packages – that there’s no point fixing python3. nix-env -i will never work; nix-env -u will never work; we should give up and go home and focus all efforts on fixing the manual so no one ever runs those commands again. We should honestly just deprecate those commands and print a warning to stderr any time someone runs them that explains how broken they are and why you shouldn’t run them and what to do instead.1

So: how did I arrive at this conclusion? Let’s crack open my diary, and find out…

Okay, I want to do an experiment: I would like to find all of the derivations with names like nixpkgs.foo and see which ones are different from the derivation “named” foo, according to nix-env -i’s rules.

So like…

$ time (nix-env --dry-run -i python3 --profile ~/scratch/profile 2>&1 | sed -En -e "s/^installing '([^']+)'/\1/p")
python3-3.10.0a5
9.10s user 1.31s system 100% cpu 10.406 total

Huh. Ten seconds per derivation… times 12,850 derivations… it would take 36 hours to check this using nix-env -i --dry-run. So I’m not doin' that.

Instead, I can write my own implementation of the algorithm, and run it in one pass over the output of nix-env -qa --json. Right?

Right.

$ nix-env -qa --json >nix-env-qa.json

$ du -h nix-env-qa.json
 56M    nix-env-qa.json

Okay. I only care about pname, version, and meta.priority. So let’s simplify this:

$ jq <nix-env-qa.json >nix-env-qa.simple.json '.[] |= { pname, version, priority: .meta.priority }'

$ du -h nix-env-qa.simple.json
4.1M    nix-env-qa.simple.json

$ head nix-env-qa.simple.json
{
  "nixpkgs._0verkill": {
    "pname": "0verkill-unstable",
    "version": "2011-01-13",
    "priority": null
  },
  "nixpkgs._0x0": {
    "pname": "0x0",
    "version": "2018-06-24",
    "priority": null

Let’s see. I only care about the immediate nixpkgs.* packages; I will ignore derivations like nixpkgs.pythonDocs.text.python27. Well, from one half of my calculation. But I can’t ignore them completely, can I? Because nix-env -i python27 is going to consider nixpkgs.pythonDocs.text.python27 when it does does its check, right? Otherwise it wouldn’t have showed up in nix-env -qa. (I’m not 100% sure this is true, but it makes sense to me. Worst case scenario this will cause me to flag ambiguous packages that are not actually ambiguous. Maybe I’ll double check them afterwards.)

So, okay. I need to form a lookup table – basically invert this shape so that pname is the top key.

jq has group_by, but it annoyingly doesn’t produce an object? I get that you might not be grouping by a string property, but it’s kind of annoying there’s no specialized alternative. I wind up with a pretty hairy-looking:

$ jq <nix-env-qa.simple.json >by-name.json 'to_entries | group_by(.value.pname) | map({ key: .[0] | .value.pname, value: map(.value.attr = .key | .value) }) | from_entries'

Which is the lookup table I want:

$ jq <by-name.json '.python3'

[
  {
    "pname": "python3",
    "version": "3.10.0a5",
    "priority": null,
    "attr": "nixpkgs.python310"
  },
  {
    "pname": "python3",
    "version": "3.6.13",
    "priority": null,
    "attr": "nixpkgs.python36"
  },
  {
    "pname": "python3",
    "version": "3.6.13",
    "priority": null,
    "attr": "nixpkgs.python36Full"
  },
  {
    "pname": "python3",
    "version": "3.7.10",
    "priority": null,
    "attr": "nixpkgs.python37"
  },
  {
    "pname": "python3",
    "version": "3.7.10",
    "priority": null,
    "attr": "nixpkgs.python37Full"
  },
  {
    "pname": "python3",
    "version": "3.8.9",
    "priority": null,
    "attr": "nixpkgs.gnuradio3_8Packages.python"
  },
  {
    "pname": "python3",
    "version": "3.8.9",
    "priority": null,
    "attr": "nixpkgs.python3Full"
  },
  {
    "pname": "python3",
    "version": "3.8.9",
    "priority": null,
    "attr": "nixpkgs.sourcehut.python"
  },
  {
    "pname": "python3",
    "version": "3.8.9",
    "priority": null,
    "attr": "nixpkgs.python38Full"
  },
  {
    "pname": "python3",
    "version": "3.9.4",
    "priority": null,
    "attr": "nixpkgs.python39"
  },
  {
    "pname": "python3",
    "version": "3.9.4",
    "priority": null,
    "attr": "nixpkgs.python39Full"
  }
]

Nice.

Now we can sort these by priority and then version… but we might not use the same sort function as Nix! That’s an important caveat, and another reason to check our work with nix-env -i --dry-run afterwards.

I assume jq is going to sort asciibetically:

$ echo '["3.5.0", "3.10.0", "3.1.0"]' | jq 'sort'
[
  "3.1.0",
  "3.10.0",
  "3.5.0"
]

Indeed. So that’s no good. We’ll need to try to simulate what (I think) Nix is doing.

I can think of a super hacky way to get what we want, basically padding each version component with 0s…

But for some reason my jq doesn’t have a sub() or gsub() function? Huh?

$ jq 'sub("x", "y")'
jq: error: sub/1 is not defined at <top-level>, line 1:
sub("x", "y")
jq: 1 compile error

I look in the definition and see there is an argument called onigurumaSupport, which defaults to true… but I have no idea if nixpkgs.jq passes an explicit false or not.

This seems like something I should be able to see, without having to dive into all-packages.nix as I always have before. Like, we have jq.override – that must capture the arguments in a closure. But I can’t get it out of the closure, can I? Here’s a really hacky way to see it:

nix-repl> jq.override { onigurumaSupport = true; }
«derivation /nix/store/m3rs5m4826bxqkmhbh22a1201kqbgisk-jq-1.6.drv»

nix-repl> jq
«derivation /nix/store/m3rs5m4826bxqkmhbh22a1201kqbgisk-jq-1.6.drv»

Since those have the same hash, I can be confident that I have oniguruma support.

Hmm. I dunno why my jq doesn’t have sub(). I google it, and find that jq needs the arguments to be semicolon-separated? Oof. That’s a really bad error message. But sure.

Apparently jq only supports named capture groups; I flail a bit with the syntax…

$ echo '["3.5.0", "3.10.0", "3.1.0"]' | jq '
> def pad: (length | if (. >= 6) then "" else "0" * (6 - .) end) + .;
> def pad_version: gsub("^(?<major> \\d+) \\. (?<minor> \\d+) \\. (?<patch> .*)$"; "\(.major | pad).\(.minor | pad).\(.patch | pad)"; "x");
> map(pad_version) | sort'
[
  "000003.000005.000000",
  "000003.000010.000000",
  "000003.000001.000000"
]

Gosh. We have definitely exceeded the level of complexity at which jq is a reasonable choice. We should jump to a real language. However, I currently find myself between scripting languages, and don’t have any that I would like to reach for. Python, Ruby, Perl, JS… they spark no joy anymore. So I press on.

That has the version bit sorted, for the most part. It’s weird if you have like a date-based version and a semver-looking version, and it doesn’t work for packages with more than 3 version components. Actually it’s much cleaner to do the more general thing:

$ echo '["3.5.0", "3.10.0", "3.1.0"]' | jq '
> def pad: (length | if (. >= 6) then "" else "0" * (6 - .) end) + .;
> def pad_version: split(".") | map(pad) | join(".");
> map(pad_version) | sort'
[
  "000003.000001.000000",
  "000003.000005.000000",
  "000003.000010.000000"
]

Yeah. I should’ve just started there. The other way is terrifying. I was thinking regexes because I wanted to use the padding-replacement-feature of some regex engines, but jq doesn’t support that, so yeah. No reason to use a regex at all.

Okay. Let’s test some slightly harder cases:

$ echo '["3.5.0", "3.10.0a5", "1003.1-2008", "2021-04-01", "3.1.0"]' | jq '
> def pad: (length | if (. >= 6) then "" else "0" * (6 - .) end) + .;
> def pad_version: split(".") | map(pad) | join(".");
> map(pad_version) | sort | reverse'
[
  "2021-04-01",
  "001003.1-2008",
  "000003.000010.0000a5",
  "000003.000005.000000",
  "000003.000001.000000"
]

That’s not really what I want. Although I don’t know what Nix would do in that situation, I feel like the date-based ones should rank the lowest… and I have no idea if you have different versioning schemes. Let’s assume that doesn’t happen.

$ echo '["3.5.0", "3.10.0a5", "3.10.0-2008", "2021-04-01", "3.1.0"]' | jq '
> def pad: (length | if (. >= 6) then "" else "0" * (6 - .) end) + .;
> def pad_version: split(".") | map(pad) | join(".") | sub("(?<date>\\d+-\\d+-\\d+)"; "000000: \(.date)");
> map(pad_version) | sort | reverse'
[
  "000003.000010.0000a5",
  "000003.000010.0-2008",
  "000003.000005.000000",
  "000003.000001.000000",
  "000000: 2021-04-01"
]

I dunno; good enough, probably? Who knows.

So to do the priority sort, we coalesce null to 0 and then take advantage of this guarantee made by group_by:

group_by(.foo) takes as input an array, groups the elements having the same .foo field into separate arrays, and produces all of these arrays as elements of a larger array, sorted by the value of the .foo field.

Emphasis mine. Without all the insane version stuff:

$ jq <by-name.json '.[] |= (group_by(.priority // 0) | .[0] | sort_by(.version) | .[0].version)'
...

So you can see how the priority check works in isolation. That one’s pretty simple. Takes a long time to run, though!

Now let’s put it all together, and test an easy one:

$ jq <by-name.json '
> def pad: (length | if (. >= 6) then "" else "0" * (6 - .) end) + .;
> def pad_version: split(".") | map(pad) | join(".") | sub("(?<date>\\d+-\\d+-\\d+)"; "000000: \(.date)");
> .python3 | (group_by(.priority // 0) | .[0] | sort_by(.version | pad_version) | reverse | .[0].version)'
"3.10.0a5"

Okay. That one works. Let’s try the whole thing:

$ time jq <by-name.json >resolved-version.json '
> def pad: (length | if (. >= 6) then "" else "0" * (6 - .) end) + .;
> def pad_version: split(".") | map(pad) | join(".") | sub("(?<date>\\d+-\\d+-\\d+)"; "000000: \(.date)");
> .[] |= (group_by(.priority // 0) | .[0] | sort_by(.version | pad_version) | reverse | .[0].version)'
68.72s user 0.76s system 96% cpu 1:12.33 total

Pretty slow! But it’s less than 36 hours. And it looks like it worked:

$ head resolved-version.json
{
  "0verkill-unstable": "2011-01-13",
  "0x0": "2018-06-24",
  "1oom": "1.0",
  "1password": "8.0.34",
  "2048-in-terminal": "2017-11-29",
  "20kly": "1.5.0",
  "2bwm": "0.3",
  "2fa": "1.2.0",
  "3270font": "2.3.0",

Nice. Let’s spot check a few. We know python3 worked… but let’s find one with an explicit priority.

$ jq <by-name.json '.[] | select(any(.priority != null))'
(lots of output)

Hmm, this is too many. Let’s find one with an explicit priority and actually different versions…

$ jq <by-name.json '.[] | select(any(.priority != null)) | select(length > 3) | unique_by(.version) | select(length > 1)'
(still lots of output)

Okay, that gives me several interesting packages. For example:

[
  {
    "pname": "tesseract",
    "version": "3.05.00",
    "priority": null,
    "attr": "nixpkgs.tesseract"
  },
  {
    "pname": "tesseract",
    "version": "4.1.1",
    "priority": 10,
    "attr": "nixpkgs.tesseract4"
  }
]

So nix-env -i tesseract should install 3.05.00, not 4.1.1. What answer did I get?

$ jq <resolved-version.json .tesseract
"3.05.00"

Nice! Okay. Let’s try to find some more complicated version examples:

$ jq <by-name.json '.[] | unique_by(.version) | select(length > 10) | .[0].pname'
"boost"
"cudatoolkit"
"electron"
"luajit"
"octave"
"protobuf"

$ jq <by-name.json '.protobuf | .[] | .version'
"2.5.0"
"3.1.0"
"3.10.1"
"3.11.4"
"3.12.4"
"3.13.0.1"
"3.14.0"
"3.15.8"
"3.16.0"
"3.6.1.3"
"3.7.1"
"3.8.0"
"3.9.2"

So, 3.16.0? But different versions have different numbers of components – which shouldn’t matter to us.

$ jq <resolved-version.json '.protobuf'
"3.16.0"

Good! Okay.

So that was an extremely unscientific series of tests… but whatever this is good enough for now.

Now lets do the same thing in the other direction. We want to exclude things like nixpkgs.ocamlPackages.utop, so we only look at the top-level attributes.

Oh, but hmm. This might actually, sometimes, give you a differently named package, mightn’t it? We should include that as well. So we have:

$ jq <nix-env-qa.simple.json >resolved.by-attr.json '
>  with_entries( select(.key | split(".") | length == 2) 
>              | { key: (.key | split(".") | .[1]),
>                  value: (.value.pname + "-" + .value.version) })'

$ head resolved.by-attr.json
{
  "_0verkill": "0verkill-unstable-2011-01-13",
  "_0x0": "0x0-2018-06-24",
  "_1oom": "1oom-1.0",
  "_1password": "1password-1.9.1",
  "_1password-gui": "1password-8.0.34",
  "_2048-in-terminal": "2048-in-terminal-2017-11-29",
  "_20kly": "20kly-1.5.0",
  "_2bwm": "2bwm-0.3",
  "go-2fa": "2fa-1.2.0",

And let’s make resolved-version.json match this same format…

$ jq <resolved-version.json >resolved.by-name.json '
> with_entries(.value = (.key + "-" + .value))'

$ head resolved.by-name.json
{
  "0verkill-unstable": "0verkill-unstable-2011-01-13",
  "0x0": "0x0-2018-06-24",
  "1oom": "1oom-1.0",
  "1password": "1password-8.0.34",
  "2048-in-terminal": "2048-in-terminal-2017-11-29",
  "20kly": "20kly-1.5.0",
  "2bwm": "2bwm-0.3",
  "2fa": "2fa-1.2.0",
  "3270font": "3270font-2.3.0",

Okay! Looks good.

You can already see some problems with this approach: you can nix-env -iA nixpkgs._1password, but you cannot nix-env -i _1password.

But that’s fine. I wasn’t really thinking precisely enough about what I’m trying to find here. The important question is “if I nix-env -iA nixpkgs._1password, and then nix-env -u, does anything happen?”

That’s what I’m looking for, really. So I need to do a sort of jump: look up each attribute, and learn 1) what name they have and 2) what name-version pair. Then look up the name, and see if it gives me the same name-version pair. I can’t just, like, comm them. Which is sort of what I was thinking going into this.

Okay. Slight backpedaling:

$ jq <nix-env-qa.simple.json >resolved.by-attr.json '
> with_entries(select(.key | split(".") | length == 2)
>             | { key: (.key | split(".") | .[1]),
>                 value: { name: .value.pname, version: .value.version } })'

$ head resolved.by-attr.json
{
  "_0verkill": {
    "name": "0verkill-unstable",
    "version": "2011-01-13"
  },
  "_0x0": {
    "name": "0x0",
    "version": "2018-06-24"
  },
  "_1oom": {

And we can revert to the original "name": "version" format of our by-name map:

$ mv resolved-version.json resolved.by-name.json

$ head resolved.by-name.json
{
  "0verkill-unstable": "2011-01-13",
  "0x0": "2018-06-24",
  "1oom": "1.0",
  "1password": "8.0.34",
  "2048-in-terminal": "2017-11-29",
  "20kly": "1.5.0",
  "2bwm": "0.3",
  "2fa": "1.2.0",
  "3270font": "2.3.0",

Okay!

We can combine both of these into a single file with --slurp:

$ jq '{ byName: .[0], byAttr: .[1] }' --slurp resolved.by-name.json resolved.by-attr.json >packages.json

Which will simplify subsequent stuff. So let’s do the thing!

I am learning that |= makes jq like… several thousands times slower. Here’s a version without it:

$ jq <packages.json >bad-packages.json '
> .byName as $byName | .byAttr | to_entries | .[]
> | { attr: .key,
>     name: .value.name,
>     version: .value.version,
>     latest: (.value.name as $name | $byName | .[$name]) }
> | select(.version != .latest)'

And that’s the final answer! Maybe!

There are a lot of packages in here.

$ jq <bad-packages.json --slurp 'length'
626

Except, sadly… it seems I have made a foolish mistake.

This does not include python3.

Because python3 isn’t in my original nix-env-qa.json dump. Why? Because it also exists as the path nixpkgs.sourcehut.python3, and nix-env -qa doesn’t repeat elements, so only the first entry appears. Which wouldn’t matter, except that nixpkgs.sourcehut.python3 is filtered out of my thing, because it’s a “nested” key, and I only looked at simple top-level things.

Alas. I wish I’d thought of that earlier.

So this is clearly incomplete. But it’s still interesting to look at. It does include things like nixpkgs.python36 and friends, which of course probably aren’t going to get nix-env -i’d.

If we only look at problematic packages where the attribute is equal to the name, we’re left with 118. As a lower-bound, remember, because it does not include python3 or other packages that occur all over the tree. 118 is a small enough number that I can look through them.

And some of these seem pretty important!

bash, for one:

$ nix-env --dry-run -i bash --profile ~/scratch/profile 2>&1 | sed -En -e "s/^installing '([^']+)'/\1/p"
bash-5.1-p4

$ nix-env --dry-run -iA nixpkgs.bash --profile ~/scratch/profile 2>&1 | sed -En -e "s/^installing '([^']+)'/\1/p"
bash-4.4-p23

Here’s the list I have – note that this is not complete:

$ jq <bad-packages.json 'select(.name == .attr) | delpaths([["attr"]])' -c

{"name":"allegro","version":"4.4.3.1","latest":"5.2.7.0"}
{"name":"antlr","version":"2.7.7","latest":"4.8"}
{"name":"arangodb","version":"3.4.8","latest":"3.5.1"}
{"name":"aseprite","version":"1.1.7","latest":"1.2.16.3"}
{"name":"bash","version":"4.4-p23","latest":"5.1-p4"}
{"name":"bazel","version":"3.7.2","latest":"4.1.0"}
{"name":"bird","version":"1.6.8","latest":"2.0.8"}
{"name":"blas","version":"3","latest":"3.8.0"}
{"name":"boost","version":"1.69.0","latest":"1.75.0"}
{"name":"botan","version":"1.10.17","latest":"2.18.0"}
{"name":"cairomm","version":"1.12.2","latest":"1.16.1"}
{"name":"cde","version":"0.1","latest":"2.3.2"}
{"name":"cgal","version":"4.14.2","latest":"5.2.1"}
{"name":"clang-manpages","version":"7.1.0","latest":"12.0.0"}
{"name":"claws-mail","version":"3.17.8","latest":"3.99.0"}
{"name":"clisp","version":"2.49","latest":"2.50pre20171114"}
{"name":"coq","version":"8.13.2","latest":"8.5pl3"}
{"name":"crawl","version":"0.26.1","latest":"0.26.1-tiles"}
{"name":"cura","version":"4.9.0","latest":"15.04"}
{"name":"curaengine","version":"4.9.1","latest":"15.04.6"}
{"name":"db","version":"5.3.28","latest":"6.2.23"}
{"name":"dex","version":"0.9.0","latest":"2.28.1"}
{"name":"diamond","version":"0.8.36","latest":"3.10"}
{"name":"dotnet-sdk","version":"2.1.810","latest":"5.0.202"}
{"name":"elasticsearch","version":"6.8.3","latest":"7.5.1"}
{"name":"elasticsearch-oss","version":"6.8.3","latest":"7.5.1"}
{"name":"erlang","version":"24.0.2","latest":"16B02.basho10"}
{"name":"etcd","version":"3.3.25","latest":"3.4.16"}
{"name":"filebeat","version":"6.8.3","latest":"7.5.1"}
{"name":"fltk","version":"1.3.5","latest":"1.4.x-r13121"}
{"name":"fplll","version":"5.3.2","latest":"20160331"}
{"name":"fsharp","version":"4.0.1.1","latest":"4.1.34"}
{"name":"gcc-arm-embedded","version":"10-2020-q4-major","latest":"9-2020-q2-update"}
{"name":"gegl","version":"0.2.0","latest":"0.4.30"}
{"name":"geogebra","version":"5-0-644-0","latest":"6-0-644-0"}
{"name":"git-lfs","version":"2.13.3","latest":"1.5.6"}
{"name":"glibmm","version":"2.64.5","latest":"2.68.1"}
{"name":"gmime","version":"2.6.23","latest":"3.2.7"}
{"name":"goocanvas","version":"1.0.0","latest":"2.0.4"}
{"name":"gr-limesdr","version":"2.0.0","latest":"3.0.1"}
{"name":"gr-osmosdr","version":"0.1.5","latest":"0.2.2"}
{"name":"gr-rds","version":"1.1.0","latest":"3.8.0"}
{"name":"grantlee","version":"0.5.1","latest":"5.2.0"}
{"name":"hadoop","version":"2.7.7","latest":"3.1.1"}
{"name":"heartbeat","version":"6.8.3","latest":"7.5.1"}
{"name":"helm","version":"0.9.0","latest":"3.6.0"}
{"name":"imagemagick","version":"7.0.11-9","latest":"6.9.12-12"}
{"name":"influxdb","version":"1.8.6","latest":"2.0.6"}
{"name":"insync","version":"1.5.7.37371","latest":"3.2.4.40856"}
{"name":"journalbeat","version":"6.8.3","latest":"7.5.1"}
{"name":"julia","version":"1.0.4","latest":"1.5.4"}
{"name":"julia-bin","version":"1.0.5","latest":"1.6.1"}
{"name":"junit","version":"4.11","latest":"4.12"}
{"name":"kibana","version":"6.8.3","latest":"7.5.1"}
{"name":"kibana-oss","version":"6.8.3","latest":"7.5.1"}
{"name":"kubelogin","version":"0.0.9","latest":"1.23.2"}
{"name":"libav","version":"11.12","latest":"12.3"}
{"name":"libcxx","version":"7.1.0","latest":"12.0.0"}
{"name":"libcxxabi","version":"7.1.0","latest":"12.0.0"}
{"name":"libfprint","version":"1.90.7","latest":"2-tod1-goodix-0.0.6"}
{"name":"libftdi","version":"0.20","latest":"1.5"}
{"name":"libgnome-keyring","version":"2.32.0","latest":"3.12.0"}
{"name":"libmicrohttpd","version":"0.9.71","latest":"0.9.72"}
{"name":"libmusicbrainz","version":"3.0.3","latest":"5.1.0"}
{"name":"libwebsockets","version":"3.2.2","latest":"4.1.6"}
{"name":"lilyterm","version":"0.9.9.4","latest":"2019-07-25"}
{"name":"lld","version":"7.1.0","latest":"12.0.0"}
{"name":"lldb","version":"11.1.0","latest":"12.0.0"}
{"name":"llvm","version":"7.1.0","latest":"12.0.0"}
{"name":"llvm-manpages","version":"7.1.0","latest":"11.1.0"}
{"name":"logstash","version":"6.8.3","latest":"7.5.1"}
{"name":"love","version":"0.10.2","latest":"11.3"}
{"name":"lua","version":"5.2.4","latest":"5.4.3"}
{"name":"luajit","version":"2.1.0-2021-05-29","latest":"2.1.0-2021-05-29-vstruct-2.0.2-1"}
{"name":"metricbeat","version":"6.8.3","latest":"7.5.1"}
{"name":"miniupnpc","version":"1.9.20160209","latest":"2.1.20190625"}
{"name":"minizip","version":"1.2.11","latest":"2.10.6"}
{"name":"mps","version":"1.117.0","latest":"2020.3.3"}
{"name":"nccl","version":"2.7.8-1-cuda-10.2","latest":"2.7.8-1-cuda-11.2"}
{"name":"ncurses","version":"6.2","latest":"6.2-abi5-compat"}
{"name":"nginx","version":"1.20.1","latest":"1.21.0"}
{"name":"nodejs-slim","version":"14.17.0","latest":"16.3.0"}
{"name":"nomad","version":"1.0.6","latest":"1.1.0"}
{"name":"nootka","version":"1.4.7","latest":"1.7.0-beta1"}
{"name":"notify-osd","version":"0.9.34","latest":"0.9.35+16.04.20160415"}
{"name":"nv-codec-headers","version":"9.1.23.1","latest":"10.0.26.2"}
{"name":"obsidian","version":"0.12.3","latest":"47.04a"}
{"name":"openafs","version":"1.8.7","latest":"1.9.0"}
{"name":"openapi-generator-cli","version":"5.1.0","latest":"6.0.0-2021-01-18"}
{"name":"openimageio","version":"1.8.17","latest":"2.2.12.0"}
{"name":"packetbeat","version":"6.8.3","latest":"7.5.1"}
{"name":"perl","version":"5.32.1","latest":"5.35.0"}
{"name":"pgf","version":"2.00","latest":"6.14.12"}
{"name":"phantomjs","version":"1.9.8","latest":"2.1.1"}
{"name":"postgresql","version":"11.11","latest":"13.2"}
{"name":"prison","version":"1.0","latest":"5.81.0"}
{"name":"psqlodbc","version":"09.01.0200","latest":"10.01.0000"}
{"name":"quota","version":"1003.1-2008","latest":"4.05"}
{"name":"qwt","version":"5.2.3","latest":"6.1.6"}
{"name":"R","version":"4.0.4","latest":"4.0.4-wrapper"}
{"name":"racer","version":"1.1","latest":"2.1.44"}
{"name":"ragel","version":"6.10","latest":"7.0.0.12"}
{"name":"readline","version":"6.3p08","latest":"8.1p0"}
{"name":"ruby","version":"2.7.3","latest":"3.0.1"}
{"name":"saxonb","version":"8.8","latest":"9.1.0.8"}
{"name":"scribus","version":"1.4.8","latest":"1.5.7"}
{"name":"spandsp","version":"0.0.6","latest":"3.0.0"}
{"name":"sqlite","version":"3.35.2","latest":"3.27.2+replication3"}
{"name":"swagger-codegen","version":"2.4.19","latest":"3.0.25"}
{"name":"swig","version":"3.0.12","latest":"4.0.2"}
{"name":"terraform","version":"0.12.31","latest":"0.15.5"}
{"name":"tinc","version":"1.0.36","latest":"1.1pre17"}
{"name":"tinyxml","version":"2.6.2","latest":"2-6.0.0"}
{"name":"vdr","version":"2.4.7","latest":"2.4.7-skincurses"}
{"name":"vtk","version":"8.2.0","latest":"9.0.1"}
{"name":"wcm","version":"0.7.0-wrapped","latest":"0.7.0"}
{"name":"xcb-util-cursor","version":"0.1.3","latest":"0.1.1-3-unstable-2017-04-05"}
{"name":"zoom","version":"1.1.5","latest":"5.6.20278.0524"}

But even this incomplete list is very convincing. python3 is the least of our problems; I just happened to hit that one first. Other things you might reasonably expect to find in a typical user environment include antlr, bazel, lua, perl, ruby, sqlite, terraform… not to mention, you know, bash.

So okay. Are those actually right? This looks like an example where my hacky sort attempt didn’t work:

{"name":"sqlite","version":"3.35.2","latest":"3.27.2+replication3"}

But nope! That’s correct:

$ nix-env -qa sqlite --json | jq '.[] |= { name, priority: .meta.priority }'

{
  "nixpkgs.sqlite-replication": {
    "name": "sqlite-3.27.2+replication3",
    "priority": null
  },
  "nixpkgs.sqlite": {
    "name": "sqlite-3.35.2",
    "priority": 10
  }
}

nixpkgs.sqlite has an explicit lower priority set. Huh?

sqlite = lowPrio (callPackage ../development/libraries/sqlite { });

Weird. Okay? I’m guessing that Nixpkgs mostly uses meta.priority to resolve filename collisions, and doesn’t actually use it to differentiate which version should be installed.

I kinda sorta start to blame it, but it takes ages, and it wasn’t added in the most recent commit to touch that line, so I give up.

Ugh okay fine I’ll learn how to blame properly. I am very spoiled by having access to a fancy interactive blame thingy at my last job. Maybe magit can do this? Hmm. The docs don’t imply so.

Well, I’m sort of… crippled by the size of this repository, huh? It takes a while.

I don’t know if there’s a smoother way to do this than appending a bunch of --ignore-revs as I go.

I make it back to April 2013 before I give up again. I think there’s a good chance sqlite has been lowPrio since the dawn of time.

Not important.

More important: I have sqlite installed in my environment right now. Why doesn’t nix-env -u try to upgrade it?

$ nix-env -q sqlite
sqlite-3.35.2

$ nix-env -u sqlite --dry-run
(dry run; not doing anything)

Huh! But…

$ nix-env -i sqlite --dry-run
(dry run; not doing anything)
replacing old 'sqlite-3.35.2'
installing 'sqlite-3.27.2+replication3'

What! That’s crazy.

So I guess… nix-env -u doesn’t… think about priority? Or… what? No. I checked this! I did! Here’s an example, after I upped the python3 priority in an override:

cat ~/.config/nixpkgs/config.nix

let overridePackageAttrs = overrides: pkgs:
  let update = with pkgs.lib; (flip recursiveUpdate); in
  pkgs.lib.mapAttrs (packageName: attrUpdates:
    pkgs.${packageName}.overrideAttrs (update attrUpdates))
    overrides;
in
{
  allowUnfree = true;
  packageOverrides = overridePackageAttrs {
    python3.meta.priority = -100;
  };
}

$ nix-env -i python3 --profile ~/scratch/profile
warning: there are multiple derivations named 'python3-3.8.9'; using the first one
installing 'python3-3.8.9'
building '/nix/store/0mgi2imsgp55p268qvvfasjv8by9jmrk-user-environment.drv'...
created 4 symlinks in user environment

$ nix-env -u --profile ~/scratch/profile

$ nix-env -u python3 --profile ~/scratch/profile

See? It works correctly.

But it also doesn’t work correctly:

$ nix-env -u python3 --dry-run
(dry run; not doing anything)
upgrading 'python3-3.8.9' to 'python3-3.10.0a5'

Isn’t that fascinating. My real profile behaves differently from my little scratch profile. Presumably… because when I installed the python3 in my “real” environment, it had a null priority? What other difference could there be? Let’s try a weird experiment:

$ nix-env -iA nixpkgs.python3
replacing old 'python3-3.8.9'
installing 'python3-3.8.9'
building '/nix/store/136b4yn3k0yjk75fhhzcnpbvsfmvm8ww-user-environment.drv'...
error: packages '/nix/store/p6k14ihyfaq3z4sycxr09cvgpis2zjd5-python3-3.8.9/bin/idle' and '/nix/store/i3719gs514i6061s99rv6r9q5adnj8p9-python-2.7.18/bin/idle' have the same priority -100; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)
builder for '/nix/store/136b4yn3k0yjk75fhhzcnpbvsfmvm8ww-user-environment.drv' failed with exit code 1
error: build of '/nix/store/136b4yn3k0yjk75fhhzcnpbvsfmvm8ww-user-environment.drv' failed

Huh! Interesting. Okay sure. I change the python3’s priority to -99…

$ nix-env -iA nixpkgs.python3
replacing old 'python3-3.8.9'
installing 'python3-3.8.9'

Okay. So it reinstalled the same thing, but with a different priority. And now if I upgrade…

$ nix-env -u python3 --dry-run
(dry run; not doing anything)

Huh! Look at that. Yeah; it depends on the priority that each package was installed with (?).

So… I have no idea what the actual semantics of nix-env -u are. I assumed it was the same as nix-env -i, but clearly that was wrong. From man nix-env:

The upgrade operation creates a new user environment, based on the current generation of the active profile, in which all store paths are replaced for which there are newer versions in the set of paths described by args. Paths for which there are no newer versions are left untouched; this is not an error. It is also not an error if an element of args matches no installed derivations.

For a description of how args is mapped to a set of store paths, see --install. If args describes multiple store paths with the same symbolic name, only the one with the highest version is installed.

But like… clearly priority does matter as well, sort of – but only the priority of the installed package? I have no idea. This is fully crazy town, and undocumented crazy town at that.

Anyway.

I check nix-env -i --dry-run to confirm that bash and sqlite and ruby are all, in fact, problematic, and my script did not flag them incorrectly:

$ nix-env -iA nixpkgs.{ruby,bash,sqlite} --profile ~/scratch/profile
installing 'ruby-2.7.3'
installing 'bash-4.4-p23'
installing 'sqlite-3.35.2'

$ nix-env -u --profile ~/scratch/profile
upgrading 'ruby-2.7.3' to 'ruby-3.0.1'
upgrading 'bash-4.4-p23' to 'bash-5.1-p4'

$ nix-env -i sqlite --profile ~/scratch/profile
replacing old 'sqlite-3.35.2'
installing 'sqlite-3.27.2+replication3'

Head explodes. No idea. I have convinced myself: this is hopelessly broken, and I close my PR.

  • Is there a sane standard alternative to nix-env -u?
  • How can I see the arguments that a given package was invoked with, without combing through all-packages.nix?
  • What on earth is nix-env -u doing with package priorities?

    • Actually, I’m gonna go ahead and close that one right here. I look in the source of nix-env, and find the following comment:

      /* Find the derivation in the input Nix expression
   with the same name that satisfies the version
   constraints specified by upgradeType.  If there are
   multiple matches, take the one with the highest
   priority.  If there are still multiple matches,
   take the one with the highest version.
   Do not upgrade if it would decrease the priority. */

      And looking through the code explains everything we’re seeing. nix-env -u explicitly looks for something with both 1) a higher version than our installed package and 2) a higher-or-equal priority than our installed package. So even though sqlite-3.27.2+replication3 had a higher priority, it did not have a higher version, so the upgrade skipped it. If there were, say, sqlite-3.99.2+replication3, we would have upgraded to that.

      This makes some sense: it would be weird for an “upgrade” to actually “downgrade” a package. But my mind deeply expects, on some primordial level, nix-env -u package to be the same as nix-env -e package followed by nix-env -i package. The fact that those are so different is very upsetting to me. But, you know, it’s not like I’m ever going to run nix-env -u again, so who cares.

      Reading the code tells me that you can get the behavior I was expecting with nix-env -u --always, which appears to have the same behavior as running nix-env -i for all your packages:

      $ nix-env -u --always --dry-run
(dry run; not doing anything)
downgrading 'imagemagick-7.0.11-9' to 'imagemagick-6.9.12-12'
downgrading 'sqlite-3.35.2' to 'sqlite-3.27.2+replication3'

      As well as resolving the thing where my python3 would still upgrade because it had a lower priority when I initially installed it, although --leq would have resolved that as well. From man nix-env:

      --leq
    In addition to upgrading to newer versions, also "upgrade" to
    derivations that have the same version. Version are not a unique
    identification of a derivation, so there may be many derivations
    that have the same version. This flag may be useful to force
    "synchronisation" between the installed and available derivations.

  1. Uhh… what should you do instead? I know the answer for nix-env -i, but I honestly don’t know what the “correct” alternative is to nix-env -u. Is there one? I have my solution, with my little file thingy, but it’s not like I can recommend that to everyone. I don’t know the answer! ↩︎