Okay. We aren’t actually done with Chapter 5 yet. We got through the bulk of it, but we still have a few short sections to go.

5.2. Generators

Generators are functions that create file formats from nix data structures, e. g. for configuration files. There are generators available for: INI, JSON and YAML

Well, as a matter of fact, looking at lib.generators, it seems that there are generators for:

toGitINI
toINI
toJSON
toKeyValue
toPlist
toPretty
toYAML

Some of those are self-explanatory… but not all of them. What is GitINI? I have to turn to the source:

/*
 * Generate a git-config file from an attrset.
 *
 * It has two major differences from the regular INI format:
 *
 * 1. values are indented with tabs
 * 2. sections can have sub-sections

Okay, sure.

I don’t really know what a “key-value” is either. I guess = separated? Or it’s configurable? I dunno. Doesn’t matter.

What is toPretty?

/* Pretty print a value, akin to `builtins.trace`.
  * Should probably be a builtin as well.
  */

A lot of these functions have those weird misaligned asterisks. But who cares? We got pretty-printing! Can it work??

nix-repl> lib.generators.toPretty { x = 1; }
error: 'toPretty' at /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/generators.nix:202:14
called with unexpected argument 'x', at (string):1:1

Umm okay hmm. The documentation doesn’t say anything about… how to actually use this function. But reading the source, it seems…

nix-repl> lib.generators.toPretty { multiline = true; } { x = 1; }
"{\n  x = 1;\n}"

So close. It, you know, it did pretty print the value, I guess. In a way that is completely useless to me.

Sigh. So… still searching, then.

Returning to the manual:

All generators follow a similar call interface: generatorName configFunctions data, where configFunctions is an attrset of user-defined functions that format nested parts of the content.

Okay; it did immediately describe how to call generator functions.

Nothing super interesting here, and the section is over.

5.3. Debugging Nix Expressions

This section just says “there are functions in lib.debug for debugging.” That’s not… we already saw that. The manual already said that. This is not a useful section.

5.4. prefer-remote-fetch overlay

prefer-remote-fetch is an overlay that download sources on remote builder. This is useful when the evaluating machine has a slow upload while the builder can fetch faster directly from the source. To use it, put the following snippet as a new overlay:

self: super:
  (super.prefer-remote-fetch self super)

Okay… I haven’t actually used overlays yet, so I have to stop and remind myself how they work. So this overlay is a function that takes the entire Nixpkgs set, and returns a new Nixpkgs set. Right? Which means prefer-remote-fetch is the name of a function in the top-level of Nixpkgs that takes two arguments: the “final” package set, and the “current” package set. In other words, there is an overlay-shaped function in the top-level Nixpkgs set.

Okay. So it looks kind of goofy, but the point-free equivalent would be much more hairy, so we just write it out. Fair enough.

And in fact yes; I remembered how this worked correctly:

nix-repl> pkgs.prefer-remote-fetch
«lambda @ /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/pkgs/build-support/prefer-remote-fetch/default.nix:13:1»

I don’t think I’ve seen a kebab-cased attribute before. I didn’t even realize that was a valid identifier. I love kebab-case. It’s a shame that Nix seems to use camelCase for everything else.

Okay, so I am curious how this works. I have not actually used a remote builder (yet), but this feels… magic? Not in a good way? Like I have no idea how this would be implemented. Let’s take a peek.

$ cat ~/src/nixpkgs/pkgs/build-support/prefer-remote-fetch/default.nix

# An overlay that download sources on remote builder.
# This is useful when the evaluating machine has a slow
# upload while the builder can fetch faster directly from the source.
# Usage: Put the following snippet in your usual overlay definition:
#
#   self: super:
#     (super.prefer-remote-fetch self super)
# Full configuration example for your own account:
#
# $ mkdir ~/.config/nixpkgs/overlays/
# $ echo 'self: super: super.prefer-remote-fetch self super' > ~/.config/nixpkgs/overlays/prefer-remote-fetch.nix
#
self: super: {
  fetchurl = args: super.fetchurl (args // { preferLocalBuild = false; });
  fetchgit = args: super.fetchgit (args // { preferLocalBuild = false; });
  fetchhg = args: super.fetchhg (args // { preferLocalBuild = false; });
  fetchsvn = args: super.fetchsvn (args // { preferLocalBuild = false; });
  fetchipfs = args: super.fetchipfs (args // { preferLocalBuild = false; });
}

Aha. So it just monkeypatches all the fetch* functions to insist preferLocalBuild = false. Okay. Not so magic, then.

I used to write a bunch of Objective-C code, and in Objective-C monkeypatching is called swizzling, which is a much more whimsical term and I wish it had become standardized but here we are.

I realize that we’re just modifying a set here and these are functions not methods so neither term actually applies but whatever you get what I’m saying.

Short section.

5.5. pkgs.nix-gitignore

pkgs.nix-gitignore is a function that acts similarly to builtins.filterSource but also allows filtering with the help of the gitignore format.

More kebab-case!

There are two subsections here that describe how to use it.

The intro said – and I am quoting – “pkgs.nix-gitignore is a function,” but it seems that it’s not, it’s a set that contains a bunch of functions.

Okay. Seems useful, for, you know, packaging stuff.

Nothing for hgignore! But since we still can’t actually install Mercurial with Nix I am not surprised by that. No idea when or if 2.3.11 is ever going to come out.

okay we really did it this time

That’s the end! We read Chapter 5.

But… there are still a lot more functions that we didn’t talk about.

Like, if I just look at lib, and filter all its values just for the sets, and exclude the ones we’ve already talked about…

cli = { ... };
customisation = { ... };
fetchers = { ... };
filesystem = { ... };
fixedPoints = { ... };
kernel = { ... };
licenses = { ... };
maintainers = { ... };
mergeAttrBy = { ... };
meta = { ... };
misc = { ... };
modules = { ... };
platforms = { ... };

There are more! There are a lot more.

Of course, not all of those are actually function namespaces. licenses, maintainers, platforms, etc… these are just collections of values. mergeAttrBy seems to be a cutesy sort of collection of specific merge functions? What is this?

Ah, okay – there’s a function mergeAttrByFunc which allows you to merge two sets by calling a specific function for each attribute. So if I’m reading this right:

nix-repl> myMerge = lib.mergeAttrByFunc {
            small = lib.trivial.min;
            large = lib.trivial.max;
          }

nix-repl> myMerge { small = 10; large = 10; }
                  { small = 20; large = 20; }
error: attempt to call something which is not a function but a set, at (string):1:1

Okay, no; it’s a much weirder API:

nix-repl> lib.mergeAttrByFunc {
            mergeAttrBy = {
              small = lib.trivial.min;
              large = lib.trivial.max;
            };
            small = 10;
            large = 10;
          }
          { small = 20; large = 20; }
{ large = 20; mergeAttrBy = { ... }; small = 10; }

That’s just… why is it intrusive like that. That’s so gross. Anyway mergeAttrBy is a function of the mergeAttrBy shape that provides sane default merge values for derivation attributes – concatenates the buildInputs, tries to concatenate the shell scripts as strings – although I feel like there might be some weird edge case here; I don’t know if you can just conjoin two lines of shell like that safely in all cases. It’s shell. There’s gotta be a case where that’s not okay, right?

This is not important.

Anyway, these functions are defined in a file called deprecated.nix, so… yeah. I dunno what the non-deprecated way to do this is.

Oh, look!

 /* deprecated:

    For historical reasons, imap has an index starting at 1.

    But for consistency with the rest of the library we want an index
    starting at zero.
 */
 imap = imap1;

Well that explains why Nix’s imap is explicitly called imap0. Huh!

But anyway, about a non-deprecated mergeAttrByFunc… I dunno. It’s a pretty simple helper to define yourself. The fact that it doesn’t exist in lib.attrsets makes me think it is not really useful in practice. And lib.attrsets.zipWithFunc is a more general version (merges an arbitrary number of sets, not just two; merge function is determined by a function call instead of a set lookup). I dunno. Whatever. Question answered.

Which means all of these:

cli = { ... };
customisation = { ... };
fetchers = { ... };
filesystem = { ... };
fixedPoints = { ... };
kernel = { ... };
meta = { ... };
misc = { ... };
modules = { ... };

Are lib function namespace-alikes that are not documented in the manual.

So… let’s dive in, I guess.

The source actually seems really well commented, so I’m going to just start there.

Starting with cli, there are only two functions – toGNUCommandLine and toGNUCommandLineShell. There’s an example in the source:

cli.toGNUCommandLine {} {
  data = builtins.toJSON { id = 0; };
  X = "PUT";
  retry = 3;
  retry-delay = null;
  url = [ "https://example.com/foo" "https://example.com/bar" ];
  silent = false;
  verbose = true;
}
=> [
  "-X" "PUT"
  "--data" "{\"id\":0}"
  "--retry" "3"
  "--url" "https://example.com/foo"
  "--url" "https://example.com/bar"
  "--verbose"
]

And toGNUCommandLineShell just calls lib.strings.escapeShellArgs on the return value. Easy.

lib.customization is the home of:

callPackageWith
callPackagesWith
extendDerivation
hydraJob
makeOverridable
makeScope
makeScopeWithSplicing
overrideDerivation

What’s hydraJob?

/* Strip a derivation of all non-essential attributes, returning
   only those needed by hydra-eval-jobs. Also strictly evaluate the
   result to ensure that there are no thunks kept alive to prevent
   garbage collection. */

Huh. I don’t know what hydra-eval-jobs is. I hope to find out more.

What are scopes? On makeScope:

/* Make a set of packages with a common scope. All packages called
   with the provided `callPackage' will be evaluated with the same
   arguments. Any package in the set may depend on any other. The
   `overrideScope'` function allows subsequent modification of the package
   set in a consistent way, i.e. all packages in the set will be
   called with the overridden packages. The package sets may be
   hierarchical: the packages in the set are called with the scope
   provided by `newScope' and the set provides a `newScope' attribute
   which can form the parent scope for later package sets. */

So this is like… a way to group packages that have common dependencies, that all need to share the same dependencies? I think I need to see a concrete example of this.

I see it used in such files as php-packages.nix, emacs-packages.nix, ocaml-packages.nix, etc, as well as files like gnome-3/default.nix and kde/default.nix. And qt stuff… so like basically what you’d expect. Packages that exist in a particular “ecosystem.”

makeScopeWithSplicing has this to say for itself:

/* Like [makeScope], but aims to support cross compilation. It's still ugly, 
   but hopefully it helps a little bit. */

A humble comment.

Man, we still have a lot to get through.

nix-repl> :p lib.fetchers
{ proxyImpureEnvVars = [ "http_proxy" "https_proxy" "ftp_proxy" "all_proxy" "no_proxy" ]; }

That one was easy. I assume this is a top-level thing like this so that it can be easily modified in an overlay or whatever.

nix-repl> :p lib.filesystem
{
  haskellPathsInDir = «lambda @ .../nixpkgs/lib/filesystem.nix:6:23»;
  listFilesRecursive = «lambda @ .../nixpkgs/lib/filesystem.nix:50:24»;
  locateDominatingFile = «lambda @ .../nixpkgs/lib/filesystem.nix:28:26»; 
}

Well that’s a little weird.

# haskellPathsInDir : Path -> Map String Path
# A map of all haskell packages defined in the given path,
# identified by having a cabal file with the same name as the
# directory itself.

# locateDominatingFile :  RegExp
#                      -> Path
#                      -> Nullable { path : Path;
#                                    matches : [ MatchResults ];
#                                  }
# Find the first directory containing a file matching 'pattern'
# upward from a given 'file'.
# Returns 'null' if no directories contain a file matching 'pattern'.

# listFilesRecursive: Path -> [ Path ]
#
# Given a directory, return a flattened list of all files within it recursively.

Weird. haskellPathsInDir is used in exactly one place: pkgs/build-support/emacs/buffer.nix:

# nix-buffer function for a project with a bunch of haskell packages
# in one directory
haskellMonoRepo = { ... }:

Shrug emoticon.

locateDominatingFile is used in exactly zero places. And listFilesRecursive is… also used in exactly zero places. Huh.

What’s the policy for deprecating or removing dead code? Like presumably people could be depending on these functions, in their own private package repositories. But surely… surely Nixpkgs does remove dead code? Sometimes? It’s not an append-only thing?

I don’t know.

nix-repl> :p lib.fixedPoints
{
  composeExtensions = «lambda @ .../nixpkgs/lib/fixed-points.nix:75:5»;
  composeManyExtensions = «lambda @ .../nixpkgs/lib/lists.nix:52:20»;
  converge = «lambda @ .../nixpkgs/lib/fixed-points.nix:32:14»;
  extends = «lambda @ .../nixpkgs/lib/fixed-points.nix:69:13»;
  fix = «lambda @ .../nixpkgs/lib/fixed-points.nix:19:9»;
  fix' = «lambda @ .../nixpkgs/lib/fixed-points.nix:25:10»;
  makeExtensible = «lambda @ .../nixpkgs/lib/fixed-points.nix:109:48»;
  makeExtensibleWithCustomName = «lambda @ .../nixpkgs/lib/fixed-points.nix:109:34»;
}

The implementation of fix matches that of Haskell:

fix = f: let x = f x; in x;

So if I’m following correctly, this won’t “recurse” until a fixed-point is reached; this just calls the function with the output of itself, which will resolve any self-references to the argument.

But uhh how does that work if the argument is a plain old value? I haven’t really thought about this. I have never actually used fix, just been vaguely mystified by it in the past.

To dig in a little bit, I try to write:

nix-repl> lib.fixedPoints.fix (x: if x % 2 == 0 then x / 2 else x + 1)
error: syntax error, unexpected $undefined, expecting THEN, at (string):1:30

But that does not work with a very mysterious error message. Is… is % not defined?

nix-repl> 1 % 2
error: syntax error, unexpected $undefined, expecting $end, at (string):1:3

Huh. Okay. Fine.

nix-repl> builtins.mod
error: attribute 'mod' missing, at (string):1:1

nix-repl> builtins.modulo
error: attribute 'modulo' missing, at (string):1:1

Ugh okay I have to pull up the documentation to find that it’s lib.trivial.mod:

nix-repl> lib.fixedPoints.fix (x: if lib.trivial.mod x 2 == 0 then x / 2 else x + 1)
error: infinite recursion encountered, at /nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1/nixpkgs/lib/trivial.nix:225:20

Okay. I take a step back and think about this. It makes sense; I was not reading this correctly at all.

I was thinking of the function Nix calls converge: gimme a function, gimme an input, apply until it returns a value. That’s what I was trying to write.

But that’s not fix. fix actually pulls a value out of a function from nothing. There is no “seed” that it’s starting with.

So how can it call the function in the first place?

Well, because laziness: it can call the function with like… a “placeholder,” sort of – a thunk. And that value will not actually be evaluated until you ask for it. So you just can’t ask for it until the function itself has returned: you have to ensure that the argument is not evaluated eagerly. I think, anyway. Who knows if I’m getting this right. So I think I can say:

nix-repl> lib.fixedPoints.fix (x: [ x ])
[ [ ... ] ]

And that’s just fine. It’s the list that contains itself. nix repl is even smart enough to let me print this “infinite” value without blowing up my screen.

nix-repl> :p lib.fixedPoints.fix (x: [ x ])
[ [ «repeated» ] ]

Cool.

But I can’t say:

nix-repl> lib.fixedPoints.fix (x: x)
error: infinite recursion encountered, at (string):1:25

Because the function I passed to fix tries to evaluate its argument before it returns.

Or, no; that’s not quite right: the function doesn’t try to evaluate its argument; nix repl tries to evaluate it. All the evaluation is “pull-based.” And when nix repl tries to pull the value out, it finds an infinite loop.

nix-repl> infiniteValue = lib.fixedPoints.fix (x: x)

nix-repl> infiniteValue
error: infinite recursion encountered, at (string):1:26

But similarly, I’m sure I can’t say:

nix-repl> lib.fixedPoints.fix (x: builtins.seq x [ x ])
error: infinite recursion encountered, at undefined position

It’s pretty nice that Nix can detect this instead of just hanging! Thanks, Nix.

Okay; this makes sense to me. Does this make sense to you? I feel like this is maybe weird. I wish the Nix manual had spent some time more talking about laziness. I feel like most people are probably not coming to Nix with much experience with this concept.

Anyway, moving past fix: there’s something called fix' that I don’t really understand.

# A variant of `fix` that records the original recursive attribute set in the
# result. This is useful in combination with the `extends` function to
# implement deep overriding. See pkgs/development/haskell-modules/default.nix
# for a concrete example.
fix' = f: let x = f x // { __unfix__ = f; }; in x;

I do not think my brain can handle the concrete example, so I do no such thing, but make a note so that maybe I will come back to this later.

We see extends, as previously mentioned, which appears to be the machinery that “overlays” are implemented in terms of – a similar “self: super:” interface.

composeExtensions composes functions of the self: super: shape. makeExtensible just adds an extend function to a set, which is the correct partial application of exend.

Okay. Makes sense. Machinery for overriding, basically.

nix-repl> :p lib.kernel
{
  freeform = «lambda @ .../nixpkgs/lib/kernel.nix:14:14»;
  module = {
    optional = false;
    tristate = "m";
  };
  no = {
    optional = false;
    tristate = "n";
  };
  option = «lambda @ .../nixpkgs/lib/kernel.nix:8:12»;
  whenHelpers = «lambda @ .../nixpkgs/lib/kernel.nix:19:17»;
  yes = {
    optional = false;
    tristate = "y";
  };
}

Your guess is as good as mine.

The first comment in kernel.nix is:

# Keeping these around in case we decide to change this horrible implementation :)

And whenHelpers is described as:

/*
  Common patterns/legacy used in common-config/hardened/config.nix
 */

???

Look at uses, it seems like this is actually used for configuring the Linux kernel? Interesting. Sort of feels like lib is this giant monolithic catch-all instead of individual things having their own helpers. At least in some cases. I’m sure that’s not a fair generalization to make, considering that I am only looking at lib.

nix-repl> :p lib.meta
{
  addMetaAttrs = «lambda @ .../nixpkgs/lib/meta.nix:15:18»;
  appendToName = «lambda @ .../nixpkgs/lib/meta.nix:40:18»;
  dontDistribute = «lambda @ .../nixpkgs/lib/meta.nix:21:20»;
  hiPrio = «lambda @ .../nixpkgs/lib/meta.nix:15:28»;
  hiPrioSet = «lambda @ .../nixpkgs/lib/meta.nix:69:15»;
  lowPrio = «lambda @ .../nixpkgs/lib/meta.nix:15:28»;
  lowPrioSet = «lambda @ .../nixpkgs/lib/meta.nix:59:16»;
  mapDerivationAttrset = «lambda @ .../nixpkgs/lib/meta.nix:46:26»;
  platformMatch = «lambda @ .../nixpkgs/lib/meta.nix:84:19»;
  setName = «lambda @ .../nixpkgs/lib/meta.nix:27:13»;
  setPrio = «lambda @ .../nixpkgs/lib/meta.nix:50:13»;
  updateName = «lambda @ .../nixpkgs/lib/meta.nix:35:16»;
}

Okay; I assume these all do the trivial things, calling, like, derivation.overrideAttrs the way I’d expect.

No! In fact, looking at the source:

addMetaAttrs = newAttrs: drv:
  drv // { meta = (drv.meta or {}) // newAttrs; };

They’re just doing the even-more-trivial thing. Modifying it as a set. Huh! So there’d be a disagreement between, like, derivation.meta and derivation.drvAttrs.meta. Which I guess never matters? Hopefully?

Okay the next one is pretty fun.

We have lib.misc, and it contains a bunch of stuff, and I’m not going to talk about all the stuff.

I assumed this would be a different set of “miscellaneous functions” than the “miscellaneous functions” in lib.trivial. But it’s not. lib.misc contains all the functions defined in lib/deprecated.nix. Which really makes me think that it’s all the deprecated functions. But it’s not called lib.deprecated: that would be crazy. It’s called misc. Oooookay. I’m not going to dig into those because I am assuming they’re all deprecated.

Lastly we have lib.modules, which is massive.

There is no real description or high-level summary in modules.nix of what these functions are or what they do, but I think that I’m gathering from various comments hidden deep in the file that this is a NixOS thing?

I google “nixos modules” and get a hit on the wiki:

Modules are files combined by NixOS to produce the full system configuration. A module contains a Nix expression. It declares options for other modules to define (give a value). It processes them and defines options declared in other modules.

lib.modules seems to contain the machinery for doing that. So I’m just going to skip all of that.

And we’re done! We read Chapter 5; we did all the extra credit. Next time it looks like we’ll get to learn about stdenv. I’m excited.

• Why is prefer-remote-fetch kebab-cased?
• Is it safe to get rid of locateDominatingFile (and other unused lib functions)?
• What’s with fix' and the __unfix__ attribute? Should I care about this?
• What is hydra-eval-jobs?
• Where is lib.warn defined?
• Why is lib.misc not called lib.deprecated?
• Why is lib.trivial not called lib.misc?