Yes, I took a long break from reading Nix documentation.
But I did not give up on Nix, just as Nix has not given up on me. I’ve simply been busy, and finding myself less intrinsically motivated to read the Nixpkgs manual than I was when I was reading the Nix manual.
I know you were worried. The number of messages I got encouraging me to keep going, not to give up when I’m so close to the home stretch – it was almost nonzero.
So here we go. Let’s get right back into it.
Chapter 11. Fetchers
When using Nix, you will frequently need to download source code and other files from the internet. Nixpkgs comes with a few helper functions that allow you to fetch fixed-output derivations in a structured way.
I want to highlight the terminology here: “helper functions that allow you to fetch fixed-output derivations.”
I’m still trying to get a handle on the term “derivation,” and I do not like the way that it’s used here.
My understanding so far is that helpers like fetchurl don’t “fetch” derivations. They return derivations. Or to be persnickety about the language, they evaluate to fixed-output derivations.
You aren’t fetching the derivation: you’re creating a derivation which itself is responsible for fetching a source file, or a tarball, or whatever.
Now, I could update my understanding of the term derivation based on this new information, but I’m not going to. I am confident enough in my understanding from all the way back in part 11 that the language here is just incorrect.
Which is unfortunate, because I don’t think that anywhere in the preceding thousands of words of manual has this actually been spelled out. I only know that fetchurl returns a derivation because I read the source of fetchurl. If I hadn’t, I might still be very confused. This would the perfect place to explain how these fetchers work, though, so let’s see if the manual does that.
Alright, returning to the manual:
We learn the difference between fetchurl and fetchzip: one stores a file exactly; the other decompresses it.
Despite the name,
fetchzipis not limited to .zip files and can also be used with any tarball.
Fair enough.
fetchpatchworks very similarly tofetchurlwith the same arguments expected. It expects patch files as a source and and performs normalization on them before computing the checksum. For example it will remove comments or other unstable parts that are sometimes added by version control systems and can change over time.
Ah, neat. I remember seeing fetchpatch before and vaguely wondering what its deal was.
We learn about some other fetchers: fetchsvn, fetchgit, fetchfossil, fetchcvs, and finally fetchhg. All pretty straightforward.
And then some helpers for common Internet services: fetchFromGitHub, fetchFromGitLab, fetchFromGitiles, fetchFromBitbucket, fetchFromSavannah, and fetchFromRepoOrCz. I’ve never heard of Savannah before. I think my Google turned up the right Savannah. Also never heard of repo.or.cz.
rg tells me that exactly two packages use fetchFromRepoOrCz:
$ sd nix info tinycc
tcc-0.9.27 Small, fast, and embeddable C compiler and interpreter
$ sd nix info cdimgtools
cdimgtools-0.3 Tools to inspect and manipulate CD/DVD optical disc images
And fetchFromSavannah is used in exactly one place: in config.nix for the ruby package. It isn’t used to fetch the Ruby source: its used to fetch a package called config. I have never heard of this package before, but it seems to be present in Nixpkgs already as nixpkgs.gnu-config, where its source is fetched using fetchgit and a path to the Savannah repository.
So it seems like nixpkgs.ruby should just depend on that, and that maybe fetchFromSavannah is not 100% pulling its weight here.
Chapter 12. Trivial Builders
Nixpkgs provides a couple of functions that help with building derivations.
See, but the “fetchers” are also functions that help with building derivations, but you never said that.
Also, it’s sort of confusing to use the term “building derivations.” I take it here to mean, like, constructing derivations. Creating derivations. Not building in the traditional sense of compilation and linking and whatnot.
Anyway.
First up is runCommand: it takes a string, and that string is run as a bash script. I believe. It doesn’t actually say bash, but I’m assuming here. I am also assuming that it’s run with the same PATH that source $stdenv/setup would give you.
But let’s take a look. It would be a trivial function to write ourselves, I think – take the string, pass it as an argument to bash -c. Or write it to a file or whatever. Let’s see what the official version looks like:
$ cat ~/src/nixpkgs/pkgs/build-support/trivial-builders.nix
let
runCommand' = runLocal: stdenv: name: env: buildCommand:
stdenv.mkDerivation ({
name = lib.strings.sanitizeDerivationName name;
inherit buildCommand;
passAsFile = [ "buildCommand" ];
}
// (lib.optionalAttrs runLocal {
preferLocalBuild = true;
allowSubstitutes = false;
})
// env);
in
rec {
...
runCommand = runCommandNoCC;
...
runCommandNoCC = runCommand' false stdenvNoCC;
...
Okay! Yeah. Very simple. I recall passAsFile from part 13 – so the builder will execute with an environment variable called $buildCommand which will be the path to a temporary file.
But wait… this is a little too simple, isn’t it? How does it know… to execute that file? I don’t recall mkDerivation treating buildCommand as some special argument. Or the stdenv builders making use of such a magic variable. Certainly that was not mentioned in the discussion of the “generic builder”.
But if we take a look at the source:
$ grep 'genericBuild' -A8 ~/src/nixpkgs/pkgs/stdenv/generic/setup.sh
genericBuild() {
if [ -f "${buildCommandPath:-}" ]; then
source "$buildCommandPath"
return
fi
if [ -n "${buildCommand:-}" ]; then
eval "$buildCommand"
return
fi
It clearly is special. Huh.
Then we have runCommandCC, which puts cc in your environment. This is interesting: the bare runCommand references stdenvNoCC, which seems like something that I should have heard of by now. It seems like something I should use, when writing my own derivations, when I do not actually depend on cc (which I almost always will not). But the manual has never mentioned it before. Huh.
Then we have runCommandLocal, which will always execute the command, and never download a cached version from the wilderness.
Note: This sets
allowSubstitutestofalse, so only userunCommandLocalif you are certain the user will always have a builder for thesystemof the derivation. This should be true for most trivial use cases (e.g. just copying some files to a different location or adding symlinks), because there thesystemis usually the same asbuiltins.currentSystem.
I don’t really understand what this comment means. How can I be “certain the user will always have a builder for the current system?” Is system == builtins.currentSystem sufficient? The comment implies that, but I have no idea what it actually means by “have a builder for the system.” What’s the builder? bash? I don’t… what?
There’s a comment in the source:
# `runCommandCCLocal` left out on purpose.
# We shouldn’t force the user to have a cc in scope.
Which doesn’t really explain anything.
It makes me think that setting allowSubstitutes = false… means that nothing is going to be downloaded? It doesn’t just disable downloading the output of this derivation – as I would have expected – but also all of its dependencies? Is that a correct reading of this?
That seems… crazy to me. I don’t believe that. But then I have no idea what the note is talking about.
Next up, a grab bag: writeTextFile, writeText, writeTextDir, writeScript, writeScriptBin.
writeTextFile is the only one that’s actually described. The manual says that it writes a text file. Presumably it takes a string as input, and writes that string as a text file. But the manual doesn’t say that. The manual hardly says anything, actually. The only other thing it says is “You can also set executable to true to make this file have the executable bit set.” So maybe this takes a map?
The others are just “wrappers over writeTextFile.” But there’s nothing about what they do, or how they work, or why I would use them.
So… yeah we’re gonna go to the source.
/* Writes a text file to the nix store.
* The contents of text is added to the file in the store.
*
* Examples:
* # Writes my-file to /nix/store/<store path>
* writeTextFile {
* name = "my-file";
* text = ''
* Contents of File
* '';
* }
* # See also the `writeText` helper function below.
*
* # Writes executable my-file to /nix/store/<store path>/bin/my-file
* writeTextFile {
* name = "my-file";
* text = ''
* Contents of File
* '';
* executable = true;
* destination = "/bin/my-file";
* }
*/
writeTextFile =
{ name # the name of the derivation
, text
, executable ? false # run chmod +x ?
, destination ? "" # relative path appended to $out eg "/bin/foo"
, checkPhase ? "" # syntax checks, e.g. for scripts
}:
runCommand name
{ inherit text executable;
passAsFile = [ "text" ];
# Pointless to do this on a remote machine.
preferLocalBuild = true;
allowSubstitutes = false;
}
''
n=$out${destination}
mkdir -p "$(dirname "$n")"
if [ -e "$textPath" ]; then
mv "$textPath" "$n"
else
echo -n "$text" > "$n"
fi
${checkPhase}
(test -n "$executable" && chmod +x "$n") || true
'';
Look at that. Much better documentation than the manual.
It’s weird that we don’t just… it’s weird that we passAsFile and then mv. We can’t just like… passAsTheRightThing? I dunno. There is some mechanism by which passAsFile already knows how to create files somewhere. It seems weird to do it… this way. Anyway. It probably doesn’t really matter.
/*
* Writes a text file to nix store with no optional parameters available.
*
* Example:
* # Writes contents of file to /nix/store/<store path>
* writeText "my-file"
* ''
* Contents of File
* '';
*
*/
writeText = name: text: writeTextFile {inherit name text;};
Ah. So there’s the trivial one.
/*
* Writes a text file to nix store in a specific directory with no
* optional parameters available.
*
* Example:
* # Writes contents of file to /nix/store/<store path>/share/my-file
* writeTextDir "share/my-file"
* ''
* Contents of File
* '';
*
*/
writeTextDir = path: text: writeTextFile {
inherit text;
name = builtins.baseNameOf path;
destination = "/${path}";
};
This one feels pretty unnecessary to me, but rg tells me that it’s used a ton in the nixos/ subdirectory, and in a couple places elsewhere. A random excerpt:
nixos/tests/haproxy.nix
27: documentRoot = pkgs.writeTextDir "index.txt" "We are all good!";
Aw.
writeScript is just writeText with executable = true. writeScriptBin is just writeScript with a destination of bin. I can see how all of these are useful, in their own ways.
/*
* Similar to writeScript. Writes a Shell script and checks its syntax.
* Automatically includes interpreter above the contents passed.
*
* Example:
* # Writes my-file to /nix/store/<store path> and makes executable.
* writeShellScript "my-file"
* ''
* Contents of File
* '';
*
*/
writeShellScript = name: text:
writeTextFile {
inherit name;
executable = true;
text = ''
#!${runtimeShell}
${text}
'';
checkPhase = ''
${stdenv.shell} -n $out
'';
};
Okay, cute.
This one isn’t documented in the manual, but it seems pretty useful. We also have writeShellScriptBin, which does what you’d expect.
I have never heard of bash -n, but presumably it “checks its syntax.” Obligatory mention of shellcheck, one of the nobler things a human hand has wrought.
man bash does not document -n as a flag, because it seems -n is short for set -n before the script. So if you’re curious what it does, you need to find the documentation for the set builtin.
How do you do that? I honestly don’t know the right answer. I do it by running man bash and then / set. Searching for “set” gives thousands of results, but searching for “ set” only gives a few irrelevant results before you find the one you want.
I should really learn how to use man better. That can’t be the right way to find documentation, but it’s what I’ve always done.
Also weird that we have two different shells mentioned in that script: runtimeShell and stdenv.shell. What’s runtimeShell? I am guessing this is a “host platform” vs “build platform” distinction, but I’ve never seen such a thing before.
It is not documented anywhere in the nixpkgs manual – the only occurrence of the string runtimeShell is in a couple of code examples.
From our old friend all-packages.nix:
### SHELLS
runtimeShell = "${runtimeShellPackage}${runtimeShellPackage.shellPath}";
runtimeShellPackage = bash;
No explanation. No notes on when to use it. Just… SHELLS.
Okay.
Anyway, next up we have symlinkJoin:
This can be used to put many derivations into the same directory structure. It works by creating a new derivation and adding symlinks to each of the paths listed. It expects two arguments,
name, andpaths.nameis the name used in the Nix store path for the created derivation.pathsis a list of paths that will be symlinked. These paths can be to Nix store derivations or any other subdirectory contained within.
Huh! Okay. Kinda neat. So I could have used that to declaratively create my own user environment, if fate had not presented me the much simpler way.
Alright! I like these. These are good functions. But these are not the only functions in trivial-builders.nix. I already showed writeShellScript; here’s another fun one:
# Create a C binary
writeCBin = name: code:
runCommandCC name
{
inherit name code;
executable = true;
passAsFile = ["code"];
# Pointless to do this on a remote machine.
preferLocalBuild = true;
allowSubstitutes = false;
}
''
n=$out/bin/$name
mkdir -p "$(dirname "$n")"
mv "$codePath" code.c
$CC -x c code.c -o "$n"
'';
There’s another function called linkFarm, undocumented in the manual, which operates like symlinkJoin, but provides a different output structure.
/*
* symlinkJoin is used to create a derivation with a familiar directory
* structure (top-level bin/, share/, etc), but with all actual files being symlinks to
* the files in the input derivations.
*
* symlinkJoin is used many places in nixpkgs to create a single derivation
* that appears to contain binaries, libraries, documentation, etc from
* multiple input derivations.
*
* linkFarm is instead used to create a simple derivation with symlinks to
* other derivations. A derivation created with linkFarm is often used in CI
* as a easy way to build multiple derivations at once.
*/
In other words, linkFarm does the trivial thing:
/nix/store/qc5728m4sa344mbks99r3q05mymwm4rw-myexample
|-- foobar -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1
`-- hello-test -> /nix/store/qy93dp4a3rqyn2mz63fbxjg228hffwyw-hello-2.10
And symlinkJoin does a merge:
/nix/store/sglsr5g079a5235hy29da3mq3hv8sjmm-myexample
|-- bin
| |-- hello -> /nix/store/qy93dp4a3rqyn2mz63fbxjg228hffwyw-hello-2.10/bin/hello
| `-- stack -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1/bin/stack
`-- share
|-- bash-completion
| `-- completions
| `-- stack -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1/share/bash-completion/completions/stack
|-- fish
| `-- vendor_completions.d
| `-- stack.fish -> /nix/store/6lzdpxshx78281vy056lbk553ijsdr44-stack-2.1.3.1/share/fish/vendor_completions.d/stack.fish
...
Neat. There are a few other functions in trivial-builders – applyPatches, which is very rarely used, on account of the standard builder having first-class support for patching.
requireFile:
/* Print an error message if the file with the specified name and
* hash doesn't exist in the Nix store. This function should only
* be used by non-redistributable software with an unfree license
* that we need to require the user to download manually. It produces
* packages that cannot be built automatically.
These weirdos:
# Copy a path to the Nix store.
# Nix automatically copies files to the store before stringifying paths.
# If you need the store path of a file, ${copyPathToStore <path>} can be
# shortened to ${<path>}.
copyPathToStore = builtins.filterSource (p: t: true);
# Copy a list of paths to the Nix store.
copyPathsToStore = builtins.map copyPathToStore;
copyPathsToStore is used in exactly one place, the fetchRepoProject function. The last chapter didn’t mention this, but it seems to exist to support git-repo projects, which I have never heard of. fetchRepoProject is used in exactly one place: the amdvlk package:
$ sd nix info amdvlk
amdvlk-2021.Q2.2 AMD Open Source Driver For Vulkan
So, okay. Let’s stop talking about trivial builders now.
Chapter 13. Special builders
There are two special builders.
The first is buildFHSUserEnv. If you are not familiar with the acronym FHS, it stands for Filesystem Hierarchy Standard. This is a convention that some Linuxes adhere to, and some packages expect. But NixOS does not adhere to this convention, and thus packages that expect it will probably be disappointed.
It uses Linux namespaces feature to create temporary lightweight environments which are destroyed after all child processes exit, without root user rights requirement.
Neat. And this is used in a lot of places! But I assume, based on that note, that it’s not going to work on my mac.
Looking through packages that use it, they all feel pretty… heavy. android-studio. steam. dropbox. A bunch of stuff I’ve never heard of. foldingathome! That seems like the sort of thing I would actually try to install with Nix, and that I know (because I just googled it) works in theory on macOS. Let’s give it a shot.
$ nix-shell -p foldingathome
error: Package ‘fahclient-7.6.13’ in /nix/store/4rvsrjbd2f351zgdh38as0xzwrlmvzkm-nixpkgs-21.05pre287374.1c16013bd6e/nixpkgs/pkgs/applications/science/misc/foldingathome/client.nix:53 is not supported on ‘x86_64-darwin’, refusing to evaluate.
So, exactly what I expected.
I’m not going to worry too much about buildFHSUserEnv. It’s neat to know it exists, but it’s not something I can really imagine myself using, so I will not bother to read through the usage in detail.
The next “special builder” is far more interesting and useful. It’s called mkShell.
Now, I have not written anything about mkShell before. This is the first time it’s come up in the manual.
But I actually know about the function already.
I know about the function because, in addition to writing this incredibly long series of posts about the Nix documentation, I have actually been using Nix for the last couple of months as well.
But I haven’t been writing a blog post every time I did anything with Nix – although I do mean to. I’ve been keeping notes. And I’ve tried to learn as little as possible “outside” this series. But I learned about mkShell, and I didn’t tell you, and I am sorry.
So here I will summarize what I know. In fact, I’ll just quote my notes at you:
On March 17 I tried to make a
nix-shellfor my Drinking with Datalog post.I tried to write this:
with import <nixpkgs> {}; souffleBut of course that would give me the dependencies of
souffle, notsouffleitself. And I don’t know how to write ashell.nixequivalent ofnix-shell -p souffle, so I just run that command every time and feel weird about it.But eventually I google and find this page:
https://nixos.wiki/wiki/Development_environment_with_nix-shell
Which got me this:
with import <nixpkgs> {}; mkShell { nativeBuildInputs = [ souffle ]; }
So now you’re caught up. You know as much as I do. I wanted to write a shell.nix file, and that’s how I did it. I didn’t read that whole wiki page, just skimmed and found the mkShell function.
Anyway. Now the manual teaches it to me for the first time:
pkgs.mkShellis a special kind of derivation that is only useful when using it combined withnix-shell. It will in fact fail to instantiate when invoked withnix-build.
I think this is such a basic thing that it should be a part of the Nix quick start guide. I think as soon as you introduce nix-shell -p, you gotta introduce mkShell right after it. It’s only right. Instead, this incredibly useful thing is hidden in this weird “Special Builders” chapter of the Nixpkgs manual. And I would bet pretty good money that no regular Nix user (as opposed to an aspiring Nixpkgs maintainer) has ever actually read this manual in such detail before.
Anyway; home stretch now.
Chapter 14. Images
The manual describes support in Nixpkgs for AppImage, Docker, OCI, and Snap images.
This is rather a long chapter, going into some detail about these different formats.
I have no interest in any of these formats, so I am going to quickly skim for the big picture.
So, AppImage: basically, if you want to put something in Nixpkgs, but it’s published as an AppImage container, you can just use some nice helper functions to wrap it up and turn it into a derivation.
Okay.
Docker: these appear to be helpers for creating Docker images, possibly out of other existing Docker images.
I consider myself very lucky that, at this point in my life, I have never had to use Docker or create a Docker image or really learn much at all about Docker, and I’m certainly not going to start now.
But it seems that Nixpkgs has quite a lot of functions if you are inclined to use them. I’m kind of surprised by the depth of support here: I would expect the intersection between people who like Nix and people who like Docker would be rather small. But I suppose many people are forced to use Docker through no fault of their own, and if you have to make a Docker image you may as well do it in a sort of declarative way.
Searching through Nixpkgs reveals that these functions are basically only used by the kubernetes package, but I imagine they are mostly used by people using Nix for private enterprises outside of Nixpkgs.
Anyway.
There is exactly one use of ociTools in Nixpkgs, in something called Railcar. Never heard of it. But again, I would expect this is more for Nix users to use for themselves.
Lastly Snap. Seems like functions to create Snapcraft… snaps. Not Snapps? Some sort of cutesy GNU-like “Snapp’s Not APPS”?
I pay even less attention to this section, because I am very very confident that whatever the future has in store for me, it does not involve publishing software on Canonical’s “app store for Linux.”
Alright! And that’s the end.
We learned about the “builders.”
Sort of. There are actually two more chapters in this section. But the next chapter is language-specific infrastructure, which is definitely its own post, and the one after that is about configuring specific complicated packages, which feels like a different concern than the simple “builders” we’ve seen so far.
Nowhere in any of those chapters did the manual say something like “Hey, fetchurl returns a derivation,” which is an insight that I find very important. Although I admit this is not really the right place to say that. It’s important enough information that it should be in the Nix manual, not buried in the Nixpkgs manual.
- Do we need
fetchFromSavannahhere? - What does it mean to “have a builder for the
system” ofrunCommandLocal? - Does
allowSubstitues = falsedisable substitutes for all dependencies of a derivation as well? - What is
runtimeShell, and when should I prefer it over$SHELL?