It’s been several days since I reinstalled a bunch of stuff with Nix instead of Homebrew, and so far everything has been fine. I mean, the software I installed, like, works the same. As far as I can tell.
But I realize that I should probably update my packages, and see if there are any juicy new bytes to keep my hard drive entertained.
I really didn’t expect that this would be worthy of a blog post but, well, here we are: my first upgrade. Attempt.
nix-env -u --dry-run takes a really long time to complete. So long that… yeah, it’d be easier to just run the upgrade and hope everything goes okay.
Second off, spoilers: it didn’t work.
$ nix-env -u upgrading 'cabal-install-188.8.131.52-git' to 'cabal-install-184.108.40.206' upgrading 'python3-3.8.7' to 'python3-3.10.0a5' upgrading 'imagemagick-6.9.11-60' to 'imagemagick-6.9.12-1' upgrading 'git-2.30.0' to 'git-2.30.1' these derivations will be built: /nix/store/9ilafr8whxz231xifls2xzj61lzg9b74-remove-references-to.drv /nix/store/hzpbi7rggv12wjql7y3yqfbli9cvbd6x-tasty-expected-failure-0.12.3.drv /nix/store/04k2sggvcvsscf50cv3fnmvn7v0k7pfw-lukko-0.1.1.3.drv /nix/store/kk10j00ilz03l451dvz6rffgkmfmx9h5-Cabal-220.127.116.11.drv /nix/store/q80n6ps5f70hm7l74d5l8h77wkixszq1-cryptohash-sha256-0.11.102.0.drv /nix/store/0m0m93jfgvqh6zrk2h2zf4vq3yqyg66k-hackage-security-0.6.0.1.drv /nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv /nix/store/6kz156y854kxdh761s8nf9h9gy368w9r-echo-0.1.4.drv /nix/store/qc2p8j26g8r9gqymmr7rfhian8c1bicg-random-1.2.0.drv /nix/store/psqfavhxqnjdldqi80apcyypcrfa9282-edit-distance-0.2.2.1.drv /nix/store/z3mhc4z63b7vp1s8frich1xqhv0vrd1f-resolv-0.1.2.0.drv /nix/store/vqvy1ladrmrm6x21299fjbkypv649b14-cabal-install-18.104.22.168.drv building '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv'... unpacking sources unpacking source archive /nix/store/6qjykfvkri219wk15gjijkfjs9ci0588-Python-3.10.0a5.tar.xz source root is Python-3.10.0a5 setting SOURCE_DATE_EPOCH to timestamp 1612298741 of file Python-3.10.0a5/Misc/NEWS patching sources applying patch /nix/store/345r2zz7pgiyk91j89qlf7mhs95jrv6f-no-ldconfig.patch patching file Lib/ctypes/util.py applying patch /nix/store/9kwzs3pplms8sijf55sdryypzvic4x1s-python-3.x-distutils-C++.patch patching file Lib/_osx_support.py patching file Lib/distutils/cygwinccompiler.py Hunk #1 succeeded at 123 (offset -2 lines). Hunk #2 succeeded at 140 (offset -2 lines). Hunk #3 succeeded at 170 (offset -2 lines). Hunk #4 succeeded at 310 (offset -2 lines). patching file Lib/distutils/sysconfig.py Hunk #1 FAILED at 170. Hunk #2 FAILED at 187. Hunk #3 succeeded at 231 (offset 23 lines). 2 out of 3 hunks FAILED -- saving rejects to file Lib/distutils/sysconfig.py.rej patching file Lib/distutils/unixccompiler.py Hunk #3 FAILED at 183. 1 out of 3 hunks FAILED -- saving rejects to file Lib/distutils/unixccompiler.py.rej patching file Makefile.pre.in Hunk #1 succeeded at 619 (offset 35 lines). builder for '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv' failed with exit code 1 error: build of '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv', '/nix/store/vqvy1ladrmrm6x21299fjbkypv649b14-cabal-install-22.214.171.124.drv' failed
That last line there says that
cabal-install both failed. But the only errors printed seem to be for
python3. That’s weird. What’s the
$ nix-env -u cabal-install upgrading 'cabal-install-126.96.36.199-git' to 'cabal-install-188.8.131.52' these derivations will be built: /nix/store/9ilafr8whxz231xifls2xzj61lzg9b74-remove-references-to.drv /nix/store/hzpbi7rggv12wjql7y3yqfbli9cvbd6x-tasty-expected-failure-0.12.3.drv /nix/store/04k2sggvcvsscf50cv3fnmvn7v0k7pfw-lukko-0.1.1.3.drv /nix/store/kk10j00ilz03l451dvz6rffgkmfmx9h5-Cabal-184.108.40.206.drv /nix/store/q80n6ps5f70hm7l74d5l8h77wkixszq1-cryptohash-sha256-0.11.102.0.drv /nix/store/0m0m93jfgvqh6zrk2h2zf4vq3yqyg66k-hackage-security-0.6.0.1.drv /nix/store/6kz156y854kxdh761s8nf9h9gy368w9r-echo-0.1.4.drv /nix/store/qc2p8j26g8r9gqymmr7rfhian8c1bicg-random-1.2.0.drv /nix/store/psqfavhxqnjdldqi80apcyypcrfa9282-edit-distance-0.2.2.1.drv /nix/store/z3mhc4z63b7vp1s8frich1xqhv0vrd1f-resolv-0.1.2.0.drv /nix/store/vqvy1ladrmrm6x21299fjbkypv649b14-cabal-install-220.127.116.11.drv building '/nix/store/9ilafr8whxz231xifls2xzj61lzg9b74-remove-references-to.drv'... building '/nix/store/kk10j00ilz03l451dvz6rffgkmfmx9h5-Cabal-18.104.22.168.drv'... setupCompilerEnvironmentPhase Build with /nix/store/cwqkic80vn6695kj13cds8640hc07smq-ghc-8.10.4. unpacking sources unpacking source archive /nix/store/89gdr0dkhyxm4bdzgn92mbij5v4ws04n-Cabal-22.214.171.124.tar.gz source root is Cabal-126.96.36.199 setting SOURCE_DATE_EPOCH to timestamp 1000000000 of file Cabal-188.8.131.52/tests/misc/ghc-supported-languages.hs patching sources compileBuildDriverPhase setupCompileFlags: -package-db=/private/tmp/nix-build-Cabal-184.108.40.206.drv-0/setup-package.conf.d -j4 -threaded -rtsopts [ 1 of 241] Compiling Distribution.Compat.Async ( Distribution/Compat/Async.hs, /private/tmp/nix-build-Cabal-220.127.116.11.drv-0/Distribution/Compat/Async.o )
You read that right.
This is attempting to recompile
cabal-install from source.
I interrupt it, because I have compiled enough Haskell to expect that a project of such a size will take several weeks for my laptop to build.
But: what? Why on earth did that happen? I remember that using channels means that I will always have binary caches of packages available. From Chapter 9 of the Nix manual:
The Nixpkgs channel is only updated after all binaries have been uploaded to the cache, so if you stick to the Nixpkgs channel (rather than using a Git checkout of the Nixpkgs tree), you will get binaries for most packages.
I also remember that if a binary cache isn’t available, Nix is supposed to fail – it won’t fallback to doing a (potentially very expensive) build unless I explicitly ask it to.
I double check my understanding in the
--fallback Whenever Nix attempts to build a derivation for which substitutes are known for each output path, but realising the output paths through the substitutes fails, fall back on building the derivation. The most common scenario in which this is useful is when we have registered substitutes in order to perform binary distribution from, say, a network repository. If the repository is down, the realisation of the derivation will fail. When this option is specified, Nix will build the derivation instead. Thus, installation from binaries falls back on installation from source. This option is not the default since it is generally not desirable for a transient failure in obtaining the substitutes to lead to a full build from source (with the related consumption of resources).
Okay. I suppose maybe this does not fall under the “derivation for which substitutes are known for each output path” clause? Perhaps
cabal-install does not have a substitute for “each output path”? Whatever that means?
I remember the incredibly slow
nix-env -qas command, and try it here:
$ time nix-env -qas cabal-install --- cabal-install-18.104.22.168 nix-env -qas cabal-install 29.64s user 7.34s system 60% cpu 1:01.05 total
Sigh. Yes. A full minute to tell me:
---. Obviously I have forgotten the output format of this, so it’s time for another trip to
--status, -s Print the status of the derivation. The status consists of three characters. The first is I or -, indicating whether the derivation is currently installed in the current generation of the active profile. This is by definition the case for --installed, but not for --available. The second is P or -, indicating whether the derivation is present on the system. This indicates whether installation of an available derivation will require the derivation to be built. The third is S or -, indicating whether a substitute is available for the derivation.
So: not installed (because I have an older version installed… wouldn’t it be nice if packages had some kind of unique identifiers so that Nix could tell me that?). Not present (same reason). And no substitute available!
Huh. Well, that’s not great.
I suspect that this is also the reason Python failed.
$ time nix-env -qas python3 --- python3-3.10.0a5 --S python3-3.6.13 --S python3-3.6.13 --S python3-3.7.10 --S python3-3.7.10 -PS python3-3.8.8 --S python3-3.8.8 --S python3-3.8.8 -PS python3-3.8.8 --S python3-3.9.2 --S python3-3.9.2 nix-env -qas python3 9.03s user 1.48s system 97% cpu 10.817 total
Not as slow that time!
I assume that these are sorted by “priority,” and that an
--upgrade will try install that first one – the one without the substitute available. I scroll up, and yep, it was trying to upgrade to
python3-3.10.0a5. Although… it could also just be sorting asciibetically. I have no idea.
So: why do I not have a substitute for these?
Did I just happen to
nix-channel --update before, like, the binary caches were populated? Isn’t that – by virtue of using channels – not supposed to happen?
This makes me sad, because I strongly suspect that I didn’t just happen to try to upgrade on the one day that the
python3 build is broken. It makes me think that I absolutely would not have been able to install
python3 if I hadn’t used a substitute. How many of these builds actually work, and how many of them just magically appear to work because whoever is populating the binary cache has just the right Mac?
I do not know.
I once again acknowledge that I am on “unstable” Nixpkgs but like… I guess I assumed that that meant “bleeding edge.” Like I was gonna get all the latest stuff, and some of the latest stuff might not be stable. Not that the package definitions themselves were unstable.
I would also once again like to point out that another good name for the “unstable” channel would be the “default” channel since it is the channel that every single person who installs Nix is on.
But okay sure whatever: the
--fallback switch is irrelevant here because Nix doesn’t “know” any substitutes –
--fallback only matters for transient issues with substituters. Sure. But why don’t I have a substitute here?
I have no idea how to answer this question. I check https://cache.nixos.org/, but it’s just a cute little landing page. It doesn’t tell me anything about anything.
I feel like I’m not going to find this in the man pages or anything. So I google.
I get another NixOS Wiki hit:
Which tells me I can use
curl to check if a package is available in a given binary cache.
But I need to know the package’s hash to do that. How do I find the hash for
path-info. Maybe that’ll do it:
$ nix path-info cabal-install error: attribute 'cabal-install' in selection path 'cabal-install' not found
$ nix path-info nixpkgs.cabal-install error: path '/nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-22.214.171.124' is not valid
Okay! That’s the hash, presumably. Which means I can:
$ curl https://cache.nixos.org/y4irz2hw26l467cvc8is503wwwf0whpi.narinfo 404
Neat. Yep. Sure isn’t there.
It also shows a much more interesting command:
$ nix path-info -r /nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-126.96.36.199 --store https://cache.nixos.org/ error: path '/nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-188.8.131.52' is not valid
Okay? Did that… work?
$ nix path-info nixpkgs.cabal-install --store https://cache.nixos.org/ error: path '/nix/store/y4irz2hw26l467cvc8is503wwwf0whpi-cabal-install-184.108.40.206' is not valid
You will be confounded to hear that
nix path-info --help does not document a
--store argument, but it does use it in an example – one that is similar to the NixOS Wiki example. I would definitely not have thought to look here for information about binary caches, but… okay. Sure.
I suspect maybe
--store is a command accepted by all
nix commands, so I try
nix --help. But it isn’t there either.
$ nix --help | grep -- --store $ echo $? 1
So… it’s magic? I guess?
--store can just be an HTTP URL of an arbitrary remote store somewhere that magically exposes some particular magic path structure or something maybe I have no idea.
$ nix path-info nixpkgs.cabal-install --store https://ianthehenry.com [0.0 MiB DL] error: 'https://ianthehenry.com' does not appear to be a binary cache
I check in my Nginx logs and find two requests, one second apart:
"GET /nix-cache-info HTTP/1.1" 404 1004 "-" "curl/7.74.0 Nix/2.3.10" "PUT /nix-cache-info HTTP/1.1" 405 150 "-" "curl/7.74.0 Nix/2.3.10"
Huuuh. That seems like a pretty weird usage here. “No
nix-cache-info? Would you like some?” But I am not curious enough to re-do it and capture the body.
It gives me something to try, though:
$ curl https://cache.nixos.org/nix-cache-info StoreDir: /nix/store WantMassQuery: 1 Priority: 40
Okay? This isn’t anything; I don’t know what I’m doing. I am just playing around now. What I want is to understand the policy and mechanism behind the official – or at least the default – Nix binary cache. It is not something that I can “debug” or “figure out myself.” I need to turn to The Internet.
The first one is a forum. I don’t love forums, but the next one is IRC, and I don’t love IRC either, so. I realize looking at the list that I don’t love a single communication medium listed here. Is there an online communication medium that I do love? No. Probably not. I realize that I am a little bit grumpy.
I pick the forum.
I register for an account, verify my email address, and write my first question.
Wait! Right before I write my first question, I remember a term that I can google: “Hydra.” I vaguely recall this being the name of the Nix CI server. If anything is populating the binary cache, it’s probably Hydra, right?
Googling “nix hydra” leads me to… a games studio here in Los Angeles? Called Nix Hydra Games? What are the odds. Small world. I see some pictures of bare-chested men with cat ears. It seems to be kind of a horny games studio.
But googling “nixos hydra” leads me to https://status.nixos.org/, which seems to be a lot less interesting but a lot more relevant.
Build problem! Well I’ll be.1
I look at
~/.nix-defexpr/channels/manifest.nix and see the “name” of my… channel version? Build? Whatever it is. It is this:
I find that build on the Hydra website right here:
And it says it… succeeded. “Duration: 1s.”
I click around this page. I have no idea what I’m doing. Under “constituents” it lists a bunch of packages, which I assume are the “new” packages in this build?
Hmm. Nothing about
cabal-install, though. I click around various other recent builds – including many that have “failed” – and it seems like they all have exactly the same list of “constituents.” So I have no idea what that actually means.
Alright, fine. I’ll post something on the forum.
I write a long-winded question:
In the course of doing that I realized that
nix-env -u is probably playing me for a fool with that
python3 package. That is not the version at
nixpkgs.python3, which is probably what I want. That’s just… I dunno, some derivation that exists somewhere in the Nixpkgs expression tree.
Not a good look.
It briefly made me think that I should not use
nix-env -u and should just reinstall the latest versions of whatever packages I actually want to have any time they need updating.
But that is… ergonomically very unpleasant.
And also doesn’t seem to apply in general: I’m missing a substitute for the “correct” version of
$ nix-env -iA nixpkgs.cabal-install COMPILING HASKELL COME BACK IN A FEW HOURS
So… the transition is not going great so far.
Someone replied to my question pretty quickly, but the response was just (and I am paraphrasing here) “Don’t use
nix-env -u.” Which does not address my actual question at all.
I can intellectually appreciate that someone is trying to guide a new Nix user in the right direction, but emotionally I am just frustrated by the feeling that this user latched onto the easiest part of my question and completely ignored the meat of it. I sigh loudly and exasperatedly. I decide to wait until morning to respond: never a good idea to correspond on The Internet in a mood like this.
Okay. Rested; relaxed. Once more able to assume the best of people. I write a reply that is hopefully free of any snark, suppressing my natural instinct to write “I’m not actually trying to use Nix I’m just trying to learn Nix to write a weird blog okay.”
In any case, I suppose I should check if a substitute is still unavailable. Maybe
cabal-install has been added to the cache since I last updated?
$ nix-env -qas cabal-install --- cabal-install-220.127.116.11
No. Of course not. That would be too easy.
Well, not this particular version of
cabal-install. But I should be able to check if a “newer”
cabal-install is present, without even actually updating my channel. Not that updating my channel is dangerous? I’m still not used to thinking rollbacks just being a thing that I can do. But anyway, it’s a chance to try something new:
$ nix-env -qas cabal-install --file https://nixos.org/channels/nixpkgs-unstable unpacking 'https://nixos.org/channels/nixpkgs-unstable'... tar: Error opening archive: Unrecognized archive format error: program 'tar' failed with exit code 1
Er… checks notes…
$ nix-env -qas cabal-install --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz unpacking 'https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz'... --- cabal-install-18.104.22.168
Isn’t it kind of weird that I only get one bit of information about the substitute status? Like, what if I have multiple substituters? Can I see which ones a package is present in? Why is querying substitute status weirdly coupled with querying my own local store? It feels like a very different operation. There must be some lower-level command that I should be using here.
Anyway, no, twelve hours later there is still no substitute available for
cabal-install. Is the entire Nix world just compiling
cabal-install from source right now?
I realize that the intersection of people using a Mac, people writing Haskell, and people using Nix is probably… me. It’s probably just me. I assume that the Linux version is cached just fine. How do I test that again?
$ nix-env -qas cabal-install --argstr system x86_64-linux --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz --S cabal-install-22.214.171.124
Yeah yeah, alright alright. macOS strikes again. Not an explanation though!
So: my first time using Nix in the wild. It didn’t go so hot. Maybe someone will respond with an explanation of the NixOS cache infrastructure by the time this blog post goes up? Maybe not.
Ten days later: yeah, no one else responded. But in the meantime, two things have happened:
First, I realized that I’m a dummy for using
nix-env -qas cabal-install this whole time. I can use
nix-env -qasA cabal-install instead. All
nix-env -q commands support
-A. In my head,
-qas was like a single operation that I learned. I didn’t really think about it what it meant.
Second, the package is now there:
$ nix-env -qasA cabal-install --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz --S cabal-install-126.96.36.199
So this was… whatever. I learned very little. I still do not know what packages get in the cache, or what to do when something is not in the cache (I mean, I know that I can build it myself, but I want to make the ecosystem better for everyone here). I will have to dive more deeply into this at a later date.
I did learn that the Discourse is probably not the right venue for asking questions. I should have just asked on IRC.
I also learned that
nix-env -u is dumb and you should never use it – and I haven’t used it since this experience. But more on that in the next post.
- Why is
nix-env -utrying to upgrade my
python3to some weird alpha version?
- In the case that I have multiple substituters configured, how can I see which substituters have certain packages?
- How do I see what (package × architecture) pairs are supposed to be cached in the official cache?