Back in Chapter 6, we got a sort of whirlwind introduction to cross-compilation.

No; perhaps “introduction” is the wrong word. It was more of a detailed description of an algorithm used somewhere within the dependency resolution phase of cross-compilation, presented without context or motivation, in a section that mostly left us with more questions than answers.

So hopefully now we’ll get an actual introduction.

Chapter 9. Cross-compilation

One might think that cross-compilation is a fairly niche concern. However, there are significant advantages to rigorously distinguishing between build-time and run-time environments!

So this is an interesting argument. I accept that cross-compilation is valuable – you don’t need to convince me – but this particular line of argumenting is very strange to me. I would assume that cross-compilation is completely orthogonal to the distinction between build-time and runtime: I would assume that Nix would make that distinction regardless.

The fact that they’re presented coupled together here is very interesting: it implies that, to someone who actually knows something about building software, the two ideas go hand in hand. As someone who knows almost nothing about building software, this is very unintuitive but also very intriguing.

Anyway, I learn that the hostPlatform/targetPlatform/buildPlatform terminology comes from autoconf.

I learn a very weird thing, which is that these keys exist as stdenv.buildPlatform (and friends). I don’t know what that means. Like, I can evaluate that in nix repl:

nix-repl> pkgs.stdenv.targetPlatform
{ aesSupport = false; avx2Support = false; avx512Support = false; avxSupport = false; config = "x86_64-apple-darwin"; emulator = «lambda @ /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/systems/default.nix:110:18»; extensions = { ... }; fma4Support = false; fmaSupport = false; gcc = { ... }; is32bit = false; is64bit = true; isAarch32 = false; isAarch64 = false; isAlpha = false; isAndroid = false; isAvr = false; isBSD = false; isBigEndian = false; isCompatible = «lambda @ /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/systems/default.nix:28:22»; isCygwin = false; isDarwin = true; isEfi = true; isFreeBSD = false; isGenode = false; isGhcjs = false; isJavaScript = false; isLinux = false; isLittleEndian = true; isMacOS = true; isMinGW = false; isMips = false; isMmix = false; isMsp430 = false; isMusl = false; isNetBSD = false; isNone = false; isOpenBSD = false; isOr1k = false; isPower = false; isPowerPC = false; isRedox = false; isRiscV = false; isSparc = false; isStatic = false; isSunOS = false; isUClibc = false; isUnix = true; isVc4 = false; isWasi = false; isWasm = false; isWindows = false; isi686 = false; isiOS = false; isx86 = true; isx86_32 = false; isx86_64 = true; libc = "libSystem"; linux-kernel = { ... }; linuxArch = "x86_64"; parsed = { ... }; qemuArch = "x86_64"; rustc = { ... }; sse3Support = false; sse4_1Support = false; sse4_2Support = false; sse4_aSupport = false; ssse3Support = false; system = "x86_64-darwin"; uname = { ... }; useAndroidPrebuilt = false; useiOSPrebuilt = false; }

What does it mean? I don’t know. I would think these would be set on stdenv while I’m in the process of building something… like, it’s not a global quality of stdenv. It’s a quality of a particular derivation at a particular point in time. Right?

Anyway, I already read the overview of what these are to try to understand some of Chapter 6, but to recap:

  • buildPlatform is the thing you expect
  • hostPlatform is the platform on which a derivation will run
  • targetPlatform is the platform for which this derivation can produce executable code, and is a weird hack to accommodate the fact that gcc (and other tools) can only produce code for a single platform, and if you want to compile for ARM on an Intel machine or whatever, you need to have a separate GCC executable that can produce ARM machine code. It’s not just a flag. Which, yes, is crazy, but here we are.

These “platforms” are actually attribute-sets, and we learn a little about what attributes they contain:

system is a sort abbreviation for the entire set, of the format [cpu]-[os]. Alright. We’ve seen that before.

config is also a sort of abbreviation, but a longer abbreviation of the form [cpu]-[vendor]-[os] or the form [cpu]-[vendor]-[os]-[abi]. Apparently the former is from LLVM and is called the “LLVM triple.” Never heard that before. This field ends with the lovable line:

This needs a better name than config!

Indeed.

parsed is config but parsed into a set:

nix-repl> pkgs.buildPlatform.config
"x86_64-apple-darwin"

nix-repl> pkgs.buildPlatform.parsed
{ _type = "system"; abi = { ... }; cpu = { ... }; kernel = { ... }; vendor = { ... }; }

Interesting _type there. Don’t know what that means.

libc is the name of the C standard library.

is*: These predicates are defined in lib.systems.inspect, and slapped onto every platform. They are superior to the ones in stdenv as they force the user to be explicit about which platform they are inspecting. Please use these instead of those.

Huh. I don’t know what “the ones in stdenv” are. But looking for examples, this is like is32bit, isDarwin, isMusl – all kinds of stuff. isJavaScript?? What isJavaScript?? Can you… no… what? I have to dig in.

But I’m not sure… how. There is no, like, list of all platforms, that I can tell. There is pkgs.platforms, but I don’t know what it is – there are only a few entries, and they are not in the format described here. Ah, right, it’s pkgs.lib.platforms. Kinda forgot about that. It’s been a little while.

nix-repl> pkgs.lib.platforms.js
[ "js-ghcjs" ]

I don’t… what? Why is that a platform?

Hmm. I guess that because ghc is one of those compilers that has the stupid targetPlatform thing, if you want to “cross-compile” Haskell to JS you need to use a different version of GHC? I don’t… hmm. This feels strange to me somehow: I would think of something like js_of_ocaml in the same way that I would think of, I dunno, pandoc or something. It produces an output, but I’m not… cross-compiling. I assume this makes some part of the Haskell infrastructure cleaner, or something? I don’t know.

platform: This is, quite frankly, a dumping ground of ad-hoc settings (it’s an attribute set). See lib.systems.platforms for examples–there’s hopefully one in there that will work verbatim for each platform that is working. Please help us triage these flags and give them better homes!

I quoted that one because it’s fun-loving and jovial.

9.2.2. Theory of dependency categorization

Ooookay. We get off to a great start:

Alright. I’m down. I’m into it. I think. Hopefully this will give me the context that I need to understand the whole thing about platforms as integer offsets relative to whatever. Hopefully this will make that make sense.

We start off slow: runtime dependencies need to have matching hosts. Naturally. No further questions.

A build time dependency, however, has a shift in platforms between the depending package and the depended-on package. “build time dependency” means that to build the depending package we need to be able to run the depended-on’s package. The depending package’s build platform is therefore equal to the depended-on package’s host platform.

Right, okay. The abstract terms make it a little hard to read, but this is also kind of a trivial statement.

If both the dependency and depending packages aren’t compilers or other machine-code-producing tools, we’re done.

Oh, good! So all of the complexity with the weird integer-shifting relative offset stuff exists just to accommodate targetPlatform?

That seems surprising to me. But also, maybe? I keep reading.

And indeed buildInputs and nativeBuildInputs have covered these simpler build-time and run-time (respectively) changes for many years.

nativeBuildInputs is a weird term for “runtime dependencies,” but sure.

But if the dependency does produce machine code, we might need to worry about its target platform too. In principle, that target platform might be any of the depending package’s build, host, or target platforms, but we prohibit dependencies from a “later” platform to an earlier platform to limit confusion because we’ve never seen a legitimate use for them.

I am not familiar with the idea of “later” or “earlier” platforms. That hasn’t been explained. But I’m going to guess that buildPlatform is earlier than hostPlatform is earlier than targetPlatform.

So I think what this is saying is: say we want to build a compiler that runs on a Mac but produces code for iOS.

myPhoneCompiler:
  hostPlatform = mac
  targetPlatform = ios

Now say it has dependencies:

myPhoneCompiler:
  hostPlatform = mac
  targetPlatform = ios
  buildPlatform = <whatever you're building it on>
  dependsOn:
    gcc (host = <parent.buildPlatform>, target = mac)
    my-assembler (host = mac, target = ios)

I think that what the above sentence said is that this isn’t allowed. My compiler can’t depend on an iOS assembler because the target platform of the assembler is “later” than the host platform? Even though it’s a runtime dependency?

I feel like I’m probably not understanding this correctly, because this seems perfectly reasonable to me. Maybe it will become more clear if I keep reading…

Finally, if the depending package is a compiler or other machine-code-producing tool, it might need dependencies that run at “emit time”. This is for compilers that (regrettably) insist on being built together with their source langauges' standard libraries. Assuming build != host != target, a run-time dependency of the standard library cannot be run at the compiler’s build time or run time, but only at the run time of code emitted by the compiler.

That’s not… a runtime dependency, then? Man. This would be so easy to understand with concrete examples. I hope we get to that. Because my brain doesn’t know what to think when it hears “a runtime dependency of the standard library.” I can’t think of what that would mean, so I am unable to wrap my head around this.

Putting this all together, that means we have dependencies in the form “host → target”, in at most the following six combinations:

Dependency’s host platform Dependency’s target platform
build build
build host
build target
host host
host target
target target

Okay. Let’s try to make these concrete. Let’s say I’m trying to make a compiler that will run on a Mac for iOS. And I want to build it on my very powerful Windows gaming computer, because my laptop is slow and ancient.

So my compiler can have six different kinds of dependencies:

Dependency’s host platform Dependency’s target platform wat
windows windows some kind of code generator or macro expander or something?
windows mac a normal compiler
windows ios a compiler for a library that my compiler’s executables will link against
mac mac some sort of JIT compiler or something? This is a runtime dependency that for some reason has a target.
mac ios this is like the assembler example above
ios ios a JIT compiler that I will embed inside my iOS apps?

Maybe? I don’t feel good about the mac/mac or ios/ios examples. I don’t know if I can abbreviate “host = target” to “there is no target it’s just a regular dependency.” I feel like “target platform” should really be an optional attribute, and this table doesn’t include the three cases were it’s null or whatever.

Anyway, I now understand the later/earlier distinction, by observing which pairs are left out of this table: my compiler cannot depend on something that runs on a Mac and produces code for Windows, because that makes no sense. Nor something that runs on iOS and produces code for a Mac. Which like, yeah, okay, that doesn’t make sense either. Nor iOS → Windows, which makes the least sense of all.

Oh! I spoke too soon. The very next paragraph:

Some examples will make this table clearer.

Yes! Yes it will. Let’s see:

Suppose there’s some package that is being built with a (build, host, target) platform triple of (foo, bar, baz). If it has a build-time library dependency, that would be a “host → build” dependency with a triple of (foo, foo, *) (the target platform is irrelevant). If it needs a compiler to be built, that would be a “build → host” dependency with a triple of (foo, foo, *) (the target platform is irrelevant). That compiler, would be built with another compiler, also “build → host” dependency, with a triple of (foo, foo, foo).

Gosh crystal clear. Come on. That’s not… that’s not an example. Calling it baz instead of “target platform” really doesn’t help me. Let’s see if I can make sense of this with my concrete platforms:

Suppose there’s some package that is being built with a (build, host, target) platform triple of (windows, mac, ios). If it has a build-time library dependency, that would be a “host → build” dependency with a triple of (windows, windows, *) (the target platform is irrelevant). If it needs a compiler to be built, that would be a “build → host” dependency with a triple of (windows, windows, *) (the target platform is irrelevant). That compiler, would be built with another compiler, also “build → host” dependency, with a triple of (windows, windows, windows).

What does it mean by “host → build” dependency? What does it mean by “build → host”? What does that arrow mean? I think of, like, “x → y” meaning “x depends on y.” Or maybe “x is a dependency of y.” I guess that’s ambiguous. But obviously these arrows mean something else. What do they mean? I have no idea. That didn’t really… explain anything to me.

9.2.3. Cross packaging cookbook

Ideally, the information above is exhaustive, so this section cannot provide any new information

Ideally. Ideally.

So this section is like a Q&A? Of like common errors? Apparently there are three common errors:

What if my package’s build system needs to build a C program to be run under the build environment?

depsBuildBuild = [ buildPackages.stdenv.cc ];

I actually think I could have answered that! So yeah, you need a C compiler that runs on the your build platform and targets your build platform. Makes sense.

My package fails to find ar.

Many packages assume that an unprefixed ar is available, but Nix doesn’t provide one. It only provides a prefixed one, just as it only does for all the other binutils programs. It may be necessary to patch the package to fix the build system to use a prefixed ar.

I am fortunate enough that I have not had to work in the C/C++ world, so I don’t know what this means. Wikipedia tells me:

Today, ar is generally used only to create and update static library files that the link editor or linker uses and for generating .deb packages for the Debian family; it can be used to create archives for any purpose, but has been largely replaced by tar for purposes other than static libraries.

Well, TIL. But what is a “prefixed” ar? Apparently this means, like arm64-ar or something – platform/architecture/whatever prefixed. Makes sense!

My package’s testsuite needs to run host platform code.

doCheck = stdenv.hostPlatform == stdenv.buildPlatfrom;

That seems like kind of an anti-pattern? Wouldn’t that lock you out of cross-compilation? Didn’t this chapter start by stressing how important it was to support cross-compilation? I’m surprised that this is a “cookbook recipe” and not a “please don’t do this but if you really have to recipe.”

Anyway, those are the three common things that can go wrong while cross-compiling. Neat.

9.3. Cross-building packages

Okay, we finally learn how we can actually cross-compile things:

nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/> lib>).systems.examples.fooBarBaz' -A whatever

What on earth is import <nixpkgs/> lib>? Is that… that’s gotta be a typo or an escape problem, right? What… huh?

Let’s try it out! Let’s see if I can build a package on my Linux machine for Mac and then copy the binary over to my Mac.

So… how do I do that? There is no lib.systems.examples.x86_64-darwin. So how do I… huh. It seems like all of the examples are things that you would usually cross-compile to: iOS, Raspberry PI, etc. Maybe I can just pass { config = "x86_64-apple-darwin"; }? Let’s try it.

claudius $ nix-build '<nixpkgs>' --arg crossSystem '{ config = "x86_64-apple-darwin"; }' -A hello
error: Package ‘Xcode.app’ in /nix/store/f40pgpk3q4xyf8v6jps7b2pvyffzi2gz-nixpkgs-21.05pre280331.54c1e44240d/nixpkgs/pkgs/os-specific/darwin/xcode/default.nix:36 has an unfree license (‘unfree’), refusing to evaluate.

a) To temporarily allow unfree packages, you can use an environment variable
   for a single invocation of the nix tools.

     $ export NIXPKGS_ALLOW_UNFREE=1

b) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

Alternatively you can configure a predicate to allow specific packages:
  { nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
      "Xcode.app"
    ];
  }

c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)

Ha! Cool! I mean, yeah, there’s no way that’s gonna work. Can’t install Xcode on a Linux computer. I should just do this the other way around, and build a Linux package on my Mac. I didn’t want to do that because I didn’t want to compile GCC on my laptop, as I’ve done it before and remember it taking about 10 hours. (In addition to being gross and weird, the targetPlatform thing also wastes electricity.)

But sure; who knows; let’s give it a shot:

claudius $ NIXPKGS_ALLOW_UNFREE=1 nix-build '<nixpkgs>' --arg crossSystem '{ config = "x86_64-apple-darwin"; }' -A hello
error: infinite recursion encountered, at undefined position
(use '--show-trace' to show detailed location information)

Unexpected. But alright. Why not?

claudius $ NIXPKGS_ALLOW_UNFREE=1 nix-build '<nixpkgs>' --arg crossSystem '{ config = "x86_64-apple-darwin"; }' -A hello --show-trace
error: while evaluating the attribute 'stdenv' of the derivation 'hello-2.10-x86_64-apple-darwin' at /nix/store/f40pgpk3q4xyf8v6jps7b2pvyffzi2gz-nixpkgs-21.05pre280331.54c1e44240d/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:203:11:
while evaluating the attribute 'defaultNativeBuildInputs' of the derivation 'stdenv-linux' at /nix/store/f40pgpk3q4xyf8v6jps7b2pvyffzi2gz-nixpkgs-21.05pre280331.54c1e44240d/nixpkgs/pkgs/stdenv/generic/default.nix:88:14:
while evaluating the attribute 'depsTargetTargetPropagated' of the derivation 'x86_64-apple-darwin-clang-wrapper-7.1.0' at /nix/store/f40pgpk3q4xyf8v6jps7b2pvyffzi2gz-nixpkgs-21.05pre280331.54c1e44240d/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:197:11:
while evaluating the attribute 'buildInputs' of the derivation 'compiler-rt-7.1.0-x86_64-apple-darwin' at /nix/store/f40pgpk3q4xyf8v6jps7b2pvyffzi2gz-nixpkgs-21.05pre280331.54c1e44240d/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:203:11:
while evaluating the attribute 'stdenv' of the derivation 'libc++abi-7.1.0-x86_64-apple-darwin' at /nix/store/f40pgpk3q4xyf8v6jps7b2pvyffzi2gz-nixpkgs-21.05pre280331.54c1e44240d/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:203:11:
infinite recursion encountered, at undefined position

Shrug. Let’s give my laptop a shot:

$ nix-build '<nixpkgs>' --arg crossSystem '{ config = "x86_64-unknown-linux"; }' -A hello
error: Target specification with 3 components is ambiguous
(use '--show-trace' to show detailed location information)

Fair enough.

$ nix-build '<nixpkgs>' --arg crossSystem '{ config = "x86_64-unknown-linux-gnu"; }' -A hello
these derivations will be built:
  /nix/store/p1cjqks77sbhkcd934an3shabh001pj8-stdenv-darwin.drv
  /nix/store/30z82c835dj5ln8g9992aqpcqp5camxg-linux-headers-5.11.drv
  /nix/store/50lxj3ijcm59mhhcf3r39izjq1zw209g-x86_64-unknown-linux-gnu-binutils-2.35.1.drv
  /nix/store/mkwymmlf1qgwggygszvgzyprwln3nfaz-x86_64-unknown-linux-gnu-binutils-wrapper-2.35.1.drv
  /nix/store/ln9s87mq6qqrvl5a4cj3pik2i0svnx39-x86_64-unknown-linux-gnu-stage-static-gcc-debug-10.2.0.drv
  /nix/store/s8sqjgbzp5hg8kniwrgnxx5w83rgs29k-x86_64-unknown-linux-gnu-stage-static-gcc-debug-wrapper-10.2.0.drv
  /nix/store/fvrqy4wyvk3b6y8g4n85c8imm47h1ynd-stdenv-darwin.drv
  /nix/store/b3zywbj7zxfg5726a320vl83z60phx4n-glibc-2.32-37-x86_64-unknown-linux-gnu.drv
  /nix/store/vi3b33dq1rsj2izjgrrq23738klb931k-x86_64-unknown-linux-gnu-binutils-wrapper-2.35.1.drv
  /nix/store/364fiav2wd8znpz1l0jl9lh7i2nnizhi-x86_64-unknown-linux-gnu-stage-final-gcc-debug-10.2.0.drv
  /nix/store/nyhnb22b6m27bwv3sl9pylxxsg8ammih-x86_64-unknown-linux-gnu-stage-final-gcc-debug-wrapper-10.2.0.drv
  /nix/store/ms6r0lrhsmghn6g82104hz1pkwidm7ny-stdenv-darwin.drv
  /nix/store/dq7ivx09wggf7cvq2d07n96gharpfg18-hello-2.10-x86_64-unknown-linux-gnu.drv
these paths will be fetched (166.17 MiB download, 191.06 MiB unpacked):
  /nix/store/3h286a44qp6ayn239bkc6ad3c5bax852-texinfo-6.7
  /nix/store/3l70d7kcfsh91w6792h4fqs4kjbq17py-glibc-reinstate-prlimit64-fallback.patch?id=eab07e78b691ae7866267fc04d31c7c3ad6b0eeb
  /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
  /nix/store/4vnfkpaj1vwaiahhqgwn7cgn04l79lxz-which-2.21
  /nix/store/58i8n9c7h1fas60yjx57f01bdzkdvqkn-linux-5.11.tar.xz
  /nix/store/5df8r14rlh97pka1zd33jqxsmcpm5fzx-libelf-0.8.13
  /nix/store/73xby7vcy20l0k7m0044ah73sa08kwjp-python3-minimal-3.8.8
  /nix/store/d4lh03b725krh9f398aqqmym17v017jv-binutils-2.35.1.tar.bz2
  /nix/store/fn7kd6112rx1i8fg53r9mblayxsvkqzy-libmpc-1.2.0
  /nix/store/izpqgk9aahvp7v6mn96f3n7q3fimdywy-mpfr-4.1.0
  /nix/store/mxmzc9azimdwzyjbk61zan1ny5nkx8l6-expand-response-params
  /nix/store/nqa9d4yxz1l2cgswzqr1pkm4jfrksm0q-locale-C.diff
  /nix/store/pnidiw0hllhclkkmqsymvgf7bz75w2kx-mpfr-4.1.0-dev
  /nix/store/xpiqxz99cghxbb996i10js3byaifzqdp-patchelf-0.12
  /nix/store/yl0acs9vzng0cd4qfdcgb3zpwjm034nw-glibc-2.32.tar.xz
...

So this is interesting. The packages with their prefixed names are not in the binary cache, so we’ve got to build all of them. And that’s gonna take.. some time. But apparently a macOS version of patchelf is in the cache, so we can just fetch that.

Anyway, it’s compiling GCC now, so I should go get a power cord. In the meantime… let’s keep reading.

Oh! Ha. Okay. The very next part says, basically, “note that it would be nice if you could just say '{ config = "<arch>-<os>-<vendor>-<abi>"; }' and not have to use the examples set, but there are ambiguities sometimes so what are we gonna do.” Well, it seems to be working well for these very common platforms? So far, at least.

Okay, now we get to something interesting.

One would think that localSystem and crossSystem overlap horribly with the three *Platforms (buildPlatform, hostPlatform, and targetPlatform; see stage.nix or the manual).

These seem like very different things to me. The *Platforms exist for individual derivations. localSystem and crossSystem exist for all of Nixpkgs. But sure? They are, like, both platforms. Or systems? Okay, yeah, the terminology is confusing.

Actually, those identifiers are purposefully not used here to draw a subtle but important distinction: While the granularity of having 3 platforms is necessary to properly *build* packages, it is overkill for specifying the user’s *intent* when making a build plan or package set.

Yeah, exactly. Also interesting to see what looks like some un-interpreted markdown? I don’t know what format these docs are written in. But I guess not markdown.

A simple “build vs deploy” dichotomy is adequate: the sliding window principle described in the previous section shows how to interpolate between the these two “end points” to get the 3 platform triple for each bootstrapping stage.

What sliding window principle? I think this is stale. The previous section had no such description. None of the previous sections did.

9.4. Cross-compilation infrastructure

The categorizes of dependencies developed in Section 9.2.2, “Theory of dependency categorization” are specified as lists of derivations given to mkDerivation, as documented in Section 6.3, “Specifying dependencies”. In short, each list of dependencies for “host → target” of “foo → bar” is called depsFooBar, with exceptions for backwards compatibility that depsBuildHost is instead called nativeBuildInputs and depsHostTarget is instead called buildInputs. Nixpkgs is now structured so that each depsFooBar is automatically taken from pkgsFooBar. (These pkgsFooBars are quite new, so there is no special case for nativeBuildInputs and buildInputs.) For example, pkgsBuildHost.gcc should be used at build-time, while pkgsHostTarget.gcc should be used at run-time.

This is the first I’m hearing of pkgs*. I think of pkgs as an abbreviation for the expression import <nixpkgs> {}. Are these… also attributes? On a derivation? Let’s keep going.

Now, for most of Nixpkgs’s history, there were no pkgsFooBar attributes, and most packages have not been refactored to use it explicitly. Prior to those, there were just buildPackages, pkgs, and targetPackages. Those are now redefined as aliases to pkgsBuildHost, pkgsHostTarget, and pkgsTargetTarget. It is acceptable, even recommended, to use them for libraries to show that the host platform is irrelevant.

Again, what are we talking about here? What is pkgsFooBar? This is written as the manual had already discussed these attributes, but it hasn’t.

But before that, there was just pkgs, even though both buildInputs and nativeBuildInputs existed. [Cross barely worked, and those were implemented with some hacks on mkDerivation to override dependencies.] What this means is the vast majority of packages do not use any explicit package set to populate their dependencies, just using whatever callPackage gives them even if they do correctly sort their dependencies into the multiple lists described above. And indeed, asking that users both sort their dependencies, and take them from the right attribute set, is both too onerous and redundant, so the recommended approach (for now) is to continue just categorizing by list and not using an explicit package set.

Ahhhh. I see. So this is describing an alternative way to specify dependencies: callPackage is apparently going to pass my derivation an argument called pkgs, and it’s also going to pass an argument called pkgsBuildHost, and I can reference packages in those sets instead of listing them as depsBuildHost (or in addition??). I don’t know.

To make this work, we “splice” together the six pkgsFooBar package sets and have callPackage actually take its arguments from that. This is currently implemented in pkgs/top-level/splice.nix. mkDerivation then, for each dependency attribute, pulls the right derivation out from the splice. This splicing can be skipped when not cross-compiling as the package sets are the same, but still is a bit slow for cross-compiling. We’d like to do something better, but haven’t come up with anything yet.

Huh. Okay. Hmm.

9.4.2. Bootstrapping

Each of the package sets described above come from a single bootstrapping stage. While pkgs/top-level/default.nix, coordinates the composition of stages at a high level, pkgs/top-level/stage.nix “ties the knot” (creates the fixed point) of each stage. The package sets are defined per-stage however, so they can be thought of as edges between stages (the nodes) in a graph. Compositions like pkgsBuildTarget.targetPackages can be thought of as paths to this graph.

Umm.

While there are many package sets, and thus many edges, the stages can also be arranged in a linear chain. In other words, many of the edges are redundant as far as connectivity is concerned. This hinges on the type of bootstrapping we do. Currently for cross it is:

  1. (native, native, native)
  2. (native, native, foreign)
  3. (native, foreign, foreign)

I don’t know, friends. I don’t know what this is trying to tell me. I am reading the words, but no information is coming out.

In each stage, pkgsBuildHost refers to the previous stage, pkgsBuildBuild refers to the one before that, and pkgsHostTarget refers to the current one, and pkgsTargetTarget refers to the next one. When there is no previous or next stage, they instead refer to the current stage. Note how all the invariants regarding the mapping between dependency and depending packages' build host and target platforms are preserved. pkgsBuildTarget and pkgsHostHost are more complex in that the stage fitting the requirements isn’t always a fixed chain of “prevs” and “nexts” away (modulo the “saturating” self-references at the ends). We just special case each instead. All the primary edges are implemented is in pkgs/stdenv/booter.nix, and secondarily aliases in pkgs/top-level/stage.nix.

Look, I can appreciate that this is a very difficult problem to solve and a lot of effort was put into solving it. This feels like one of those cases where someone spent weeks getting really deep into something and when they finally came up for air they sort of forgot how to talk to people who did not spend weeks implementing cross-compilation. Does the casual Nix user understand this description? Should they?

If one looks at the 3 platform triples, one can see that they overlap such that one could put them together into a chain like:

(native, native, native, foreign, foreign)

If one imagines the saturating self references at the end being replaced with infinite stages, and then overlays those platform triples, one ends up with the infinite tuple:

(native..., native, native, native, foreign, foreign, foreign...)

On can then imagine any sequence of platforms such that there are bootstrap stages with their 3 platforms determined by “sliding a window” that is the 3 tuple through the sequence. This was the original model for bootstrapping. Without a target platform (assume a better world where all compilers are multi-target and all standard libraries are built in their own derivation), this is sufficient. Conversely if one wishes to cross compile “faster”, with a “Canadian Cross” bootstraping stage where build != host != target, more bootstrapping stages are needed since no sliding window providess the pesky pkgsBuildTarget package set since it skips the Canadian cross stage’s “host”.

Ah yes. The Canadian Cross. A normal term that the average person is familiar with.

Apparently this is a normal term in compiler terms, but as someone who has not worked in compilers, I had never heard of it. I looked it up. Yeah, it’s the three-platform case.

Anyway – I guess these sections got re-ordered? Such that the sliding window thing is now described after the section that references it. Okay. And the paragraph that references the “saturation” is also before the paragraph introducing that concept. Okay.

I still have no idea what’s going on; I have completely lost sight of the problem we’re trying to solve and the solution that apparently solves it.

There is a large, multi-paragraph “note” that follows this. It begins this way:

It is much better to refer to buildPackages than targetPackages, or more broadly package sets that do not mention “target”. There are three reasons for this.

And then it describes the reasons. And look: I don’t understand the reasons. I can add no valuable commentary here. The reasons talk about bootstrapping stages and infinite reference cycles; one of the reasons appears to be truncated – the sentence just ends, without punctuation, and without, as far as I can tell, a complete thought. I cannot make sense of this note.

I will not try too hard.

I feel like this is written… not for the consumption of the average Nix user. Either that or the average Nix user knows a lot more about cross-compilation than I do.

I really wish that the manual spent more time explaining, like, the small subset of all of this that I need to know in order to write packages. Explain what I should be doing, and explain why. And then save the dependency resolution algorithm in a separate, like, I dunno, design document or something.

It occurs to me that all of this explanation might be confusing because Nix has not found the right way to model this problem. But I think that’s probably wrong: I expect that this is actually just an incredibly hard problem and there is no solution that is going to make it natural to someone who does not understand the problem and who has not spent a lot of time thinking about the problem.

But like, the whole targetPlatform thing feels like such a weird hack and I feel like there’s got to be some kind of representation of that triple that makes sense for compilers and for not-compilers that doesn’t require us to specify a field for every package that is only meaningful for some of them? A definition of targetPlatform that makes sense regardless of whether or not you are generating code?

I don’t know. I’m rambling. The cross-compilation chapter is over. I don’t really feel like I understand cross-compilation any better. But now I know about crossSystem? That’s kind of nice? I’m happy to know about that.

Oh hey! My build just finished. Great timing.

$ file /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello
/nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/c3bkb04jbbwm983xl0r8r6ixl171ldk9-glibc-2.32-37-x86_64-unknown-linux-gnu/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

Alright. So can I copy that executable? Probably not. It is dynamically linked. But I dunno. Maybe?

Where can I get ldd on macOS? It’s not in coretuils or binutils or gcc or anything I can think of. I have to copy it to my NixOS box just to see what’s happening:

claudius $ ldd hello
linux-vdso.so.1 (0x00007ffe1d3ad000)
libc.so.6 => /nix/store/z0b60y0khix9jb74ka56gw7b7n9s8awx-glibc-2.26-131/lib/libc.so.6 (0x00007f08f1a70000)
/nix/store/c3bkb04jbbwm983xl0r8r6ixl171ldk9-glibc-2.32-37-x86_64-unknown-linux-gnu/lib/ld-linux-x86-64.so.2 => /nix/store/z0b60y0khix9jb74ka56gw7b7n9s8awx-glibc-2.26-131/lib64/ld-linux-x86-64.so.2 (0x00007f08f1e22000)

I assume I can’t just run this. But I’ll try?

claudius $ ./hello
zsh: no such file or directory: ./hello

claudius $ file ./hello
./hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/c3bkb04jbbwm983xl0r8r6ixl171ldk9-glibc-2.32-37-x86_64-unknown-linux-gnu/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

Very weird. Very weird. It most certainly is a file. It is also most certainly executable. Why is zsh giving me an error here?

claudius $ strace -f ./hello
execve("./hello", ["./hello"], 0x7ffedf6142f8 /* 44 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

What?

claudius $ strace -f ~/scratch/hello
execve("/home/ian/scratch/hello", ["/home/ian/scratch/hello"], 0x7fffd6f9b788 /* 44 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

So I’m guessing that this is just a horrible Linux error, because of course the file does exist, but it’s missing some dependency or something and it’s manifesting with this terrible error because, you know, all it can do is give me an error number. It can’t tell me what file is actually missing. So zsh assumes it’s the only file it knows about. Ugh.

Anyway, I didn’t really expect to be able to do this. I believe I should have to use nix copy to transfer the file with all of its dependencies. Let’s try it?

$ nix copy --to ssh://claudius /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello
[1/0/2 copied (1.9/28.4 MiB)] copying path '/nix/store/c3bkb04jbbwm983xl0r8r6ierror:
cannot add path '/nix/store/c3bkb04jbbwm983xl0r8r6ixl171ldk9-glibc-2.32-37-x86_64-unknown-linux-gnu'
because it lacks a valid signature
[0 copied (2.0 MiB)]
error: writing to file: Broken pipe

Sigh. Same error from nix-copy-closure.

$ nix-store --export /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello | ssh claudius 'nix-store --import'
error: cannot add path '/nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu'
because it lacks a valid signature

There’s no fooling them.

I don’t really remember anything about… signatures. But I do remember something about generating signatures? Or something?

I search through my diary. I find, in part 15, a single unexplained snippet that I quoted from the manual:

nix sign-paths --key-file /etc/nix/key.private $OUT_PATHS

And part 16 contained a passing reference to nix-store --generate-binary-cache-key. It implies this is for, you know, binary caches. But maybe it’s the same mechanism? I would expect that.

But I don’t know how to specify valid public keys on my remote server.

I look up man nix.conf. Here’s the relevant stuff:

require-sigs
  If set to true (the default), any non-content-addressed path added or copied
  to the Nix store (e.g. when substituting from a binary cache) must have a
  valid signature, that is, be signed using one of the keys listed in
  trusted-public-keys or secret-key-files. Set to false to disable signature
  checking.

secret-key-files
  A whitespace-separated list of files containing secret (private) keys. These are used to sign locally-built paths. They can be generated using nix-store --generate-binary-cache-key. The corresponding public key can be distributed to other users, who can add it to trusted-public-keys in their nix.conf.

trusted-public-keys
  A whitespace-separated list of public keys. When paths are copied from
  another Nix store (such as a binary cache), they must be signed with one of
  these keys. For example:
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= 
  hydra.nixos.org-1:CNHJZBh9K4tP3EKF6FkkgeVYsS3ohTl+oS0Qa8bezVs=.

Okay. So theoretically I can add trusted-public-keys to my server. Except…

My server is NixOS. My /etc/nix/nix.conf begins with the lines:

# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!

Okay, sure. So I can just add something to my configuration.nix:

nix.trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" "mbp:XOc406wjeYvKOOBZ7oCtyPHBPOgNhDIRvx5GU47WLC0=" ];

But this doesn’t work:

claudius $ sudo nixos-rebuild switch
error: The option `nix.trusted-public-keys' defined in `/etc/nixos/configuration.nix' does not exist.

Hmmm. I can’t find any documentation that describes what the translation from Nix attributes to config names is. I searched. I googled. There is no documentation in the NixOS manual for how to configure your nix.conf. Which seems crazy to me.

I’m going to guess that it expects a camelCase key, for some reason? Just to be confusing?

claudius $ sudo nixos-rebuild switch
error: The option `nix.trustedPublicKeys' defined in `/etc/nixos/configuration.nix' does not exist.

I guess I would have been more annoyed if that had worked.

Ah, it’s in man configuration.nix. I should’ve looked there sooner. And sadly, it’s this:

nix.binaryCachePublicKeys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" "mbp:XOc406wjeYvKOOBZ7oCtyPHBPOgNhDIRvx5GU47WLC0=" ];

Sigh. I want to emphasize this excerpt from man nix.conf:

binary-cache-public-keys
  Deprecated: binary-cache-public-keys is now an alias to trusted-public-keys.

But NixOS, it seems, did not get the same treatment. But it generated the key trusted-public-keys in nix.conf – so someone updated the value without updating the attribute name. Heavy sigh.

Anyway, I finally managed to add a single line to my nix.conf. And now, on my laptop:

$ nix sign-paths --key-file secret /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello 

$ nix copy --to ssh://claudius /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello
[1/0/2 copied (2.1/28.4 MiB)] copying path '/nix/store/c3bkb04jbbwm983xl0r8r6ierror: cannot add path '/nix/store/c3bkb04jbbwm983xl0r8r6ixl171ldk9-glibc-2.32-37-x86_64-unknown-linux-gnu' because it lacks a valid signature
[0 copied (2.1 MiB)]
error: writing to file: Broken pipe

Womp womp. It seems nix sign-paths does not sign the closure of the path?

$ nix sign-paths --recursive --key-file secret /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello

$ nix copy --to ssh://claudius /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello
[2 copied (28.4 MiB)]

I guess that kinda makes sense.

Anyway. Now I can finally run my cross-compiled package:

claudius $ /nix/store/w4y584y6iam5j2la6br1qkk48vn0d453-hello-2.10-x86_64-unknown-linux-gnu/bin/hello
Inconsistency detected by ld.so: get-dynamic-info.h: 146: elf_get_dynamic_info: Assertion `info[DT_RUNPATH] == NULL' failed!

Hahaha. Well, we tried. We tried cross-compiling. It did not go great. In keeping with the theme of the chapter, I suppose.

Maybe I didn’t cross-compile properly? I did just say { config = "x86_64-unknown-linux-gnu"; }. I could be more specific…

$ nix-build '<nixpkgs>' --arg crossSystem '{config = "x86_64-unknown-linux-gnu"; libc = "glibc"; linux-kernel = { autoModules = true; baseConfig = "defconfig"; name = "pc"; target = "bzImage"; }; linuxArch = "x86_64"; parsed = { _type = "system"; abi = { _type = "abi"; name = "gnu"; }; cpu = { _type = "cpu-type"; arch = "x86-64"; bits = 64; family = "x86"; name = "x86_64"; significantByte = { _type = "significant-byte"; name = "littleEndian"; }; }; kernel = { _type = "kernel"; execFormat = { _type = "exec-format"; name = "elf"; }; name = "linux"; }; vendor = { _type = "vendor"; name = "unknown"; }; }; qemuArch = "x86_64"; sse3Support = false; sse4_1Support = false; sse4_2Support = false; sse4_aSupport = false; ssse3Support = false; system = "x86_64-linux"; uname = { processor = "x86_64"; release = null; system = "Linux"; }; }' -A hello
these derivations will be built:
...

Let’s try that? It is recompiling a lot of stuff. Which kind of surprises me. But it seems like a good sign? I don’t know what in that set it was not defaulting to, though. Did it not assume libc = "glibc"? I don’t know.

Hmm. I thought this would be done a lot quicker. Like, I already have the right gcc, don’t I? It’s not rebuilding everything? I hope?

(It seems to be rebuilding everything.)

The next morning: it failed?

cycle detected in the references of '/nix/store/fxpb3ia4hd0qx51qqmw9kgpb3sw36q2m-x86_64-unknown-linux-gnu-stage-final-gcc-debug-10.2.0' from '/nix/store/rs9v4cryx7j8lv8fj6ym0da637kp5s1g-x86_64-unknown-linux-gnu-stage-final-gcc-debug-10.2.0-lib'
cannot build derivation '/nix/store/mrgaa3w6nvwy8c8jhsbm30fzw61rqz2c-x86_64-unknown-linux-gnu-stage-final-gcc-debug-wrapper-10.2.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zaw6nq0lyl9cmc68sdm7kpn2cljb3c0w-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wfld8hma7a2cr8h6v9i57qrz1zsnh0kw-hello-2.10-x86_64-unknown-linux-gnu.drv': 1 dependencies couldn't be built

Umm hmm. I check the cookbook: no; this is not one of the three things that is supposed to go wrong.

Well. Okay. I guess… I guess that this doesn’t work. Not even a little bit.

I should give up now, but I press on, and find a relevant issue:

https://github.com/NixOS/nixpkgs/issues/88213

It claims to be fixed months ago. It seems I’ve hit both of the issues described in that thread: it’s possible that my original attempt at cross-compilation was valid, modulo the CoreFoundation issue, and that my second attempt for some reason brought back the cyclic reference issue that is supposed to be fixed.

So I’m not sure.

I see this (currently) open PR:

https://github.com/NixOS/nixpkgs/pull/103517

Which seems to fix the info[DT_RUNPATH] == NULL assertion error. But it has been open for five months, with no activity in the last three.

Anyway my overall impression here is that cross-compilation is not… quite… ready for prime time. I get the sense that this is a brave new world and I have inadvertently stepped into a work in progress. I will now gracefully step out, and try again in a few months.

  • Why is there a stdenv.targetPlatform? What does that mean?
  • How can I get ldd on my macOS box?
  • Why did Nix try to install Xcode to build hello for macOS?