It’s been several days since I reinstalled a bunch of stuff with Nix instead of Homebrew, and so far everything has been fine. I mean, the software I installed, like, works the same. As far as I can tell.
But I realize that I should probably update my packages, and see if there are any juicy new bytes to keep my hard drive entertained.
I really didn’t expect that this would be worthy of a blog post but, well, here we are: my first upgrade. Attempt.
First off, nix-env -u --dry-run takes a really long time to complete. So long that… yeah, it’d be easier to just run the upgrade and hope everything goes okay.
Second off, spoilers: it didn’t work.
$ nix-env -u
upgrading 'cabal-install-3.2.0.0-git' to 'cabal-install-3.4.0.0'
upgrading 'python3-3.8.7' to 'python3-3.10.0a5'
upgrading 'imagemagick-6.9.11-60' to 'imagemagick-6.9.12-1'
upgrading 'git-2.30.0' to 'git-2.30.1'
these derivations will be built:
/nix/store/9ilafr8whxz231xifls2xzj61lzg9b74-remove-references-to.drv
/nix/store/hzpbi7rggv12wjql7y3yqfbli9cvbd6x-tasty-expected-failure-0.12.3.drv
/nix/store/04k2sggvcvsscf50cv3fnmvn7v0k7pfw-lukko-0.1.1.3.drv
/nix/store/kk10j00ilz03l451dvz6rffgkmfmx9h5-Cabal-3.4.0.0.drv
/nix/store/q80n6ps5f70hm7l74d5l8h77wkixszq1-cryptohash-sha256-0.11.102.0.drv
/nix/store/0m0m93jfgvqh6zrk2h2zf4vq3yqyg66k-hackage-security-0.6.0.1.drv
/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv
/nix/store/6kz156y854kxdh761s8nf9h9gy368w9r-echo-0.1.4.drv
/nix/store/qc2p8j26g8r9gqymmr7rfhian8c1bicg-random-1.2.0.drv
/nix/store/psqfavhxqnjdldqi80apcyypcrfa9282-edit-distance-0.2.2.1.drv
/nix/store/z3mhc4z63b7vp1s8frich1xqhv0vrd1f-resolv-0.1.2.0.drv
/nix/store/vqvy1ladrmrm6x21299fjbkypv649b14-cabal-install-3.4.0.0.drv
building '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv'...
unpacking sources
unpacking source archive /nix/store/6qjykfvkri219wk15gjijkfjs9ci0588-Python-3.10.0a5.tar.xz
source root is Python-3.10.0a5
setting SOURCE_DATE_EPOCH to timestamp 1612298741 of file Python-3.10.0a5/Misc/NEWS
patching sources
applying patch /nix/store/345r2zz7pgiyk91j89qlf7mhs95jrv6f-no-ldconfig.patch
patching file Lib/ctypes/util.py
applying patch /nix/store/9kwzs3pplms8sijf55sdryypzvic4x1s-python-3.x-distutils-C++.patch
patching file Lib/_osx_support.py
patching file Lib/distutils/cygwinccompiler.py
Hunk #1 succeeded at 123 (offset -2 lines).
Hunk #2 succeeded at 140 (offset -2 lines).
Hunk #3 succeeded at 170 (offset -2 lines).
Hunk #4 succeeded at 310 (offset -2 lines).
patching file Lib/distutils/sysconfig.py
Hunk #1 FAILED at 170.
Hunk #2 FAILED at 187.
Hunk #3 succeeded at 231 (offset 23 lines).
2 out of 3 hunks FAILED -- saving rejects to file Lib/distutils/sysconfig.py.rej
patching file Lib/distutils/unixccompiler.py
Hunk #3 FAILED at 183.
1 out of 3 hunks FAILED -- saving rejects to file Lib/distutils/unixccompiler.py.rej
patching file Makefile.pre.in
Hunk #1 succeeded at 619 (offset 35 lines).
builder for '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv' failed with exit code 1
error: build of '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv', '/nix/store/vqvy1ladrmrm6x21299fjbkypv649b14-cabal-install-3.4.0.0.drv' failed
That last line there says that python3 and cabal-install both failed. But the only errors printed seem to be for python3. That’s weird. What’s the cabal-install error?
$ nix-env -u cabal-install
upgrading 'cabal-install-3.2.0.0-git' to 'cabal-install-3.4.0.0'
these derivations will be built:
/nix/store/9ilafr8whxz231xifls2xzj61lzg9b74-remove-references-to.drv
/nix/store/hzpbi7rggv12wjql7y3yqfbli9cvbd6x-tasty-expected-failure-0.12.3.drv
/nix/store/04k2sggvcvsscf50cv3fnmvn7v0k7pfw-lukko-0.1.1.3.drv
/nix/store/kk10j00ilz03l451dvz6rffgkmfmx9h5-Cabal-3.4.0.0.drv
/nix/store/q80n6ps5f70hm7l74d5l8h77wkixszq1-cryptohash-sha256-0.11.102.0.drv
/nix/store/0m0m93jfgvqh6zrk2h2zf4vq3yqyg66k-hackage-security-0.6.0.1.drv
/nix/store/6kz156y854kxdh761s8nf9h9gy368w9r-echo-0.1.4.drv
/nix/store/qc2p8j26g8r9gqymmr7rfhian8c1bicg-random-1.2.0.drv
/nix/store/psqfavhxqnjdldqi80apcyypcrfa9282-edit-distance-0.2.2.1.drv
/nix/store/z3mhc4z63b7vp1s8frich1xqhv0vrd1f-resolv-0.1.2.0.drv
/nix/store/vqvy1ladrmrm6x21299fjbkypv649b14-cabal-install-3.4.0.0.drv
building '/nix/store/9ilafr8whxz231xifls2xzj61lzg9b74-remove-references-to.drv'...
building '/nix/store/kk10j00ilz03l451dvz6rffgkmfmx9h5-Cabal-3.4.0.0.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/cwqkic80vn6695kj13cds8640hc07smq-ghc-8.10.4.
unpacking sources
unpacking source archive /nix/store/89gdr0dkhyxm4bdzgn92mbij5v4ws04n-Cabal-3.4.0.0.tar.gz
source root is Cabal-3.4.0.0
setting SOURCE_DATE_EPOCH to timestamp 1000000000 of file Cabal-3.4.0.0/tests/misc/ghc-supported-languages.hs
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/tmp/nix-build-Cabal-3.4.0.0.drv-0/setup-package.conf.d -j4 -threaded -rtsopts
[ 1 of 241] Compiling Distribution.Compat.Async ( Distribution/Compat/Async.hs, /private/tmp/nix-build-Cabal-3.4.0.0.drv-0/Distribution/Compat/Async.o )
Yes.
You read that right.
This is attempting to recompile cabal-install from source.
I interrupt it, because I have compiled enough Haskell to expect that a project of such a size will take several weeks for my laptop to build.
But: what? Why on earth did that happen? I remember that using channels means that I will always have binary caches of packages available. From Chapter 9 of the Nix manual:
The Nixpkgs channel is only updated after all binaries have been uploaded to the cache, so if you stick to the Nixpkgs channel (rather than using a Git checkout of the Nixpkgs tree), you will get binaries for most packages.
I also remember that if a binary cache isn’t available, Nix is supposed to fail – it won’t fallback to doing a (potentially very expensive) build unless I explicitly ask it to.
I double check my understanding in the man pages:
--fallback
Whenever Nix attempts to build a derivation for which substitutes are known
for each output path, but realising the output paths through the
substitutes fails, fall back on building the derivation.
The most common scenario in which this is useful is when we have registered
substitutes in order to perform binary distribution from, say, a network
repository. If the repository is down, the realisation of the derivation
will fail. When this option is specified, Nix will build the derivation
instead. Thus, installation from binaries falls back on installation from
source. This option is not the default since it is generally not desirable
for a transient failure in obtaining the substitutes to lead to a full
build from source (with the related consumption of resources).
Okay. I suppose maybe this does not fall under the “derivation for which substitutes are known for each output path” clause? Perhaps cabal-install does not have a substitute for “each output path”? Whatever that means?
I remember the incredibly slow nix-env -qas command, and try it here:
$ time nix-env -qas cabal-install
--- cabal-install-3.4.0.0
nix-env -qas cabal-install 29.64s user 7.34s system 60% cpu 1:01.05 total
Sigh. Yes. A full minute to tell me: ---. Obviously I have forgotten the output format of this, so it’s time for another trip to man nix-env:
--status, -s
Print the status of the derivation. The status consists of three
characters. The first is I or -, indicating whether the derivation is
currently installed in the current generation of the active profile. This
is by definition the case for --installed, but not for --available. The
second is P or -, indicating whether the derivation is present on the
system. This indicates whether installation of an available derivation will
require the derivation to be built. The third is S or -, indicating whether
a substitute is available for the derivation.
So: not installed (because I have an older version installed… wouldn’t it be nice if packages had some kind of unique identifiers so that Nix could tell me that?). Not present (same reason). And no substitute available!
Huh. Well, that’s not great.
I suspect that this is also the reason Python failed.
$ time nix-env -qas python3
--- python3-3.10.0a5
--S python3-3.6.13
--S python3-3.6.13
--S python3-3.7.10
--S python3-3.7.10
-PS python3-3.8.8
--S python3-3.8.8
--S python3-3.8.8
-PS python3-3.8.8
--S python3-3.9.2
--S python3-3.9.2
nix-env -qas python3 9.03s user 1.48s system 97% cpu 10.817 total
Not as slow that time!
I assume that these are sorted by “priority,” and that an --upgrade will try install that first one – the one without the substitute available. I scroll up, and yep, it was trying to upgrade to python3-3.10.0a5. Although… it could also just be sorting asciibetically. I have no idea.
So: why do I not have a substitute for these?
Did I just happen to nix-channel --update before, like, the binary caches were populated? Isn’t that – by virtue of using channels – not supposed to happen?
This makes me sad, because I strongly suspect that I didn’t just happen to try to upgrade on the one day that the python3 build is broken. It makes me think that I absolutely would not have been able to install python3 if I hadn’t used a substitute. How many of these builds actually work, and how many of them just magically appear to work because whoever is populating the binary cache has just the right Mac?
I do not know.
I once again acknowledge that I am on “unstable” Nixpkgs but like… I guess I assumed that that meant “bleeding edge.” Like I was gonna get all the latest stuff, and some of the latest stuff might not be stable. Not that the package definitions themselves were unstable.
I would also once again like to point out that another good name for the “unstable” channel would be the “default” channel since it is the channel that every single person who installs Nix is on.
But okay sure whatever: the --fallback switch is irrelevant here because Nix doesn’t “know” any substitutes – --fallback only matters for transient issues with substituters. Sure. But why don’t I have a substitute here?
I have no idea how to answer this question. I check https://cache.nixos.org/, but it’s just a cute little landing page. It doesn’t tell me anything about anything.
I feel like I’m not going to find this in the man pages or anything. So I google.
I get another NixOS Wiki hit:
https://nixos.wiki/wiki/Binary_Cache
Which tells me I can use curl to check if a package is available in a given binary cache.
But I need to know the package’s hash to do that. How do I find the hash for cabal-install?
I remember path-info. Maybe that’ll do it:
$ nix path-info cabal-install
error: attribute 'cabal-install' in selection path 'cabal-install' not found
Heavy sigh.
$ nix path-info nixpkgs.cabal-install
error: path '/nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-3.4.0.0' is not valid
Okay! That’s the hash, presumably. Which means I can:
$ curl https://cache.nixos.org/y4irz2hw26l467cvc8is503wwwf0whpi.narinfo
404
Neat. Yep. Sure isn’t there.
It also shows a much more interesting command:
$ nix path-info -r /nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-3.4.0.0 --store https://cache.nixos.org/
error: path '/nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-3.4.0.0' is not valid
Okay? Did that… work?
$ nix path-info nixpkgs.cabal-install --store https://cache.nixos.org/
error: path '/nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-3.4.0.0' is not valid
You will be confounded to hear that nix path-info --help does not document a --store argument, but it does use it in an example – one that is similar to the NixOS Wiki example. I would definitely not have thought to look here for information about binary caches, but… okay. Sure.
I suspect maybe --store is a command accepted by all nix commands, so I try nix --help. But it isn’t there either.
$ nix --help | grep -- --store
$ echo $?
1
So… it’s magic? I guess? --store can just be an HTTP URL of an arbitrary remote store somewhere that magically exposes some particular magic path structure or something maybe I have no idea.
$ nix path-info nixpkgs.cabal-install --store https://ianthehenry.com
[0.0 MiB DL]
error: 'https://ianthehenry.com' does not appear to be a binary cache
I check in my Nginx logs and find two requests, one second apart:
"GET /nix-cache-info HTTP/1.1" 404 1004 "-" "curl/7.74.0 Nix/2.3.10"
"PUT /nix-cache-info HTTP/1.1" 405 150 "-" "curl/7.74.0 Nix/2.3.10"
Huuuh. That seems like a pretty weird usage here. “No nix-cache-info? Would you like some?” But I am not curious enough to re-do it and capture the body.
It gives me something to try, though:
$ curl https://cache.nixos.org/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40
Okay? This isn’t anything; I don’t know what I’m doing. I am just playing around now. What I want is to understand the policy and mechanism behind the official – or at least the default – Nix binary cache. It is not something that I can “debug” or “figure out myself.” I need to turn to The Internet.
https://cache.nixos.org/ has some links to “support channels” that I can use, which takes me to https://nixos.org/community/index.html. This lists venues for discussing things. Hmm.
The first one is a forum. I don’t love forums, but the next one is IRC, and I don’t love IRC either, so. I realize looking at the list that I don’t love a single communication medium listed here. Is there an online communication medium that I do love? No. Probably not. I realize that I am a little bit grumpy.
I pick the forum.
I register for an account, verify my email address, and write my first question.
Wait! Right before I write my first question, I remember a term that I can google: “Hydra.” I vaguely recall this being the name of the Nix CI server. If anything is populating the binary cache, it’s probably Hydra, right?
Googling “nix hydra” leads me to… a games studio here in Los Angeles? Called Nix Hydra Games? What are the odds. Small world. I see some pictures of bare-chested men with cat ears. It seems to be kind of a horny games studio.
But googling “nixos hydra” leads me to https://status.nixos.org/, which seems to be a lot less interesting but a lot more relevant.
Build problem! Well I’ll be.1
I look at ~/.nix-defexpr/channels/manifest.nix and see the “name” of my… channel version? Build? Whatever it is. It is this:
"nixpkgs-21.05pre274251.f5f6dc053b1"
I find that build on the Hydra website right here:
https://hydra.nixos.org/build/138335539
And it says it… succeeded. “Duration: 1s.”
Hrm.
I click around this page. I have no idea what I’m doing. Under “constituents” it lists a bunch of packages, which I assume are the “new” packages in this build?
Hmm. Nothing about cabal-install, though. I click around various other recent builds – including many that have “failed” – and it seems like they all have exactly the same list of “constituents.” So I have no idea what that actually means.
Alright, fine. I’ll post something on the forum.
I write a long-winded question:
https://discourse.nixos.org/t/why-am-i-missing-substitutes-for-some-of-my-packages/11878
In the course of doing that I realized that nix-env -u is probably playing me for a fool with that python3 package. That is not the version at nixpkgs.python3, which is probably what I want. That’s just… I dunno, some derivation that exists somewhere in the Nixpkgs expression tree.
Not a good look.
It briefly made me think that I should not use nix-env -u and should just reinstall the latest versions of whatever packages I actually want to have any time they need updating.
But that is… ergonomically very unpleasant.
And also doesn’t seem to apply in general: I’m missing a substitute for the “correct” version of cabal-install:
$ nix-env -iA nixpkgs.cabal-install
COMPILING HASKELL COME BACK IN A FEW HOURS
So… the transition is not going great so far.
Someone replied to my question pretty quickly, but the response was just (and I am paraphrasing here) “Don’t use nix-env -u.” Which does not address my actual question at all.
I can intellectually appreciate that someone is trying to guide a new Nix user in the right direction, but emotionally I am just frustrated by the feeling that this user latched onto the easiest part of my question and completely ignored the meat of it. I sigh loudly and exasperatedly. I decide to wait until morning to respond: never a good idea to correspond on The Internet in a mood like this.
Okay. Rested; relaxed. Once more able to assume the best of people. I write a reply that is hopefully free of any snark, suppressing my natural instinct to write “I’m not actually trying to use Nix I’m just trying to learn Nix to write a weird blog okay.”
In any case, I suppose I should check if a substitute is still unavailable. Maybe cabal-install has been added to the cache since I last updated?
$ nix-env -qas cabal-install
--- cabal-install-3.4.0.0
No. Of course not. That would be too easy.
Well, not this particular version of cabal-install. But I should be able to check if a “newer” cabal-install is present, without even actually updating my channel. Not that updating my channel is dangerous? I’m still not used to thinking rollbacks just being a thing that I can do. But anyway, it’s a chance to try something new:
$ nix-env -qas cabal-install --file https://nixos.org/channels/nixpkgs-unstable
unpacking 'https://nixos.org/channels/nixpkgs-unstable'...
tar: Error opening archive: Unrecognized archive format
error: program 'tar' failed with exit code 1
Er… checks notes…
$ nix-env -qas cabal-install --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz
unpacking 'https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz'...
--- cabal-install-3.4.0.0
Isn’t it kind of weird that I only get one bit of information about the substitute status? Like, what if I have multiple substituters? Can I see which ones a package is present in? Why is querying substitute status weirdly coupled with querying my own local store? It feels like a very different operation. There must be some lower-level command that I should be using here.
Anyway, no, twelve hours later there is still no substitute available for cabal-install. Is the entire Nix world just compiling cabal-install from source right now?
I realize that the intersection of people using a Mac, people writing Haskell, and people using Nix is probably… me. It’s probably just me. I assume that the Linux version is cached just fine. How do I test that again?
$ nix-env -qas cabal-install --argstr system x86_64-linux --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz
--S cabal-install-3.4.0.0
Yeah yeah, alright alright. macOS strikes again. Not an explanation though!
So: my first time using Nix in the wild. It didn’t go so hot. Maybe someone will respond with an explanation of the NixOS cache infrastructure by the time this blog post goes up? Maybe not.
Ten days later: yeah, no one else responded. But in the meantime, two things have happened:
First, I realized that I’m a dummy for using nix-env -qas cabal-install this whole time. I can use nix-env -qasA cabal-install instead. All nix-env -q commands support -A. In my head, -qas was like a single operation that I learned. I didn’t really think about it what it meant.
Second, the package is now there:
$ nix-env -qasA cabal-install --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz
--S cabal-install-3.4.0.0
So this was… whatever. I learned very little. I still do not know what packages get in the cache, or what to do when something is not in the cache (I mean, I know that I can build it myself, but I want to make the ecosystem better for everyone here). I will have to dive more deeply into this at a later date.
I did learn that the Discourse is probably not the right venue for asking questions. I should have just asked on IRC.
I also learned that nix-env -u is dumb and you should never use it – and I haven’t used it since this experience. But more on that in the next post.
- Why is
nix-env -utrying to upgrade mypython3to some weird alpha version? - In the case that I have multiple substituters configured, how can I see which substituters have certain packages?
- How do I see what (package × architecture) pairs are supposed to be cached in the official cache?
-
I forgot to turn off Dark Reader when I took that screenshot I’m sorry I’ll try to do better next time. ↩︎