Okay, we’re going to talk about Chapter 15 now.

Except that, by one crude estimate, Chapter 15 accounts for more than one third of the entire Nixpkgs manual.

It’s really 26 chapters in one, although they aren’t called chapters, they’re just called sections.

It’s a very, very long chapter.

So… we’re only actually going to talk about parts of Chapter 15.

Chapter 15. Languages and frameworks

The chapter is arranged alphabetically, and alphabetically the first language is Agda.

It might surprise you to hear this, but I do not know Agda.

But I read some words about how to build Agda project using Nix, and it seems… fine? It seems like it’s a thing that you can do, if it’s a thing that you want to do. But I don’t exactly get very much out of it.

And that’s going to be the case for most of these languages (and frameworks). So I’m, well, I’m going to skip almost all of them.

It will keep me sane; it will keep this blog post to a somewhat readable length.

Here’s the full list of things:

Agda
Android
BEAM Languages (Erlang, Elixir & LFE)
Bower
Coq
Crystal
Emscripten
GNOME
Go
Haskell
Idris
iOS
Java
User’s Guide to Lua Infrastructure
Node.js
OCaml
Perl
PHP
Python
Qt
R
Ruby
Rust
TeX Live
Titanium
Vim

And here’s the much smaller list of things that I could say anything useful or intelligent about:

Bower
Go
Haskell
iOS
Node.js
OCaml
Python
Rust

So let’s give it a shot.

15.4. Bower

I included Bower in this list mostly because it seems like a really simple place to start. I have used Bower before, some decades ago, but I’m genuinely surprised that it still exists. The Bower homepage even says:

While Bower is maintained, we recommend using Yarn and Webpack or Parcel for front-end projects

Which is… yeah.

But I understand that not every project can switch away; software is a burden that we must live with forever. And anyway, Bower is such a trivial “platform” that I feel like it’s a good place to start.

In case you’re not familiar: this is basically a way to specify your web frontend dependencies by writing a JSON file (like package.json) that will be downloaded into a bower_components directory (like node_modules). And then… nothing else. There’s nothing else. It’s just a package registry. It’s up to you what you do with the bower_components; how you actually incorporate the source files it downloads.

I happen to have two projects that use Bower sitting in my ~/src. One is a Purescript project, and I seem to recall that (at the time) Purescript used Bower as their semi-official package registry of record. I imagine that’s changed, but here we are. This project is sort of complicated, though, and not a very good test subject.

The other is a project from 2014: a simple chat client I wrote as an experiment to try out Ember.js.¹ Seems like that oughtta work.

$ cat bower.json

{
  "name": "Tocky",
  "dependencies": {
    "ember": "=1.5.0-beta.2",
    "socket.io-client": "~0.9.16",
    "momentjs": "=2.5.1"
  },
  "overrides": {
    "socket.io-client": {
      "main": "dist/socket.io.js"
    }
  }
}

Now I don’t have Bower anymore, because, you know, it is 2021. And it doesn’t appear to be present in Nixpkgs – nix search bower gives one result, and it is not in any way related.

But the manual tells me I can find it in nodePackages.bower.

But how could I have found that out for myself? How can I search through nodePackages?

nix search --help is no help. I suppose I will need to use nix-env -qa for this.

$ nix-env -qaA nixpkgs.nodePackages
(tens of thousands of lines of output)

Hmm.

$ nix-env -qaA nixpkgs.nodePackages bower
error: selector 'bower' matches no derivations

$ nix-env -qaA nixpkgs.nodePackages '*bower*'
error: An empty regex is not allowed in the POSIX grammar.

$ nix-env -qaA nixpkgs.nodePackages '.*bower.*'
node_bower-1.8.12
node_bower2nix-3.2.0

Yes; very intuitive.

Also very annoying that this gives me the names of these derivations, when what I want is the attribute path. How do I see the path? man nix-env teaches me -P:

$ nix-env -qaA nixpkgs.nodePackages '.*bower.*' -P
nixpkgs.nodePackages.bower      node_bower-1.8.12
nixpkgs.nodePackages.bower2nix  node_bower2nix-3.2.0

Which seems like always more useful than the “name.” Derivation names seem like such a huge design mistake.

I also still wish I could just nix search this.

Allllright. Anyway. Back to it.

The manual tells me I can run bower2nix and get an equivalent Nix expression to my simple little bower.json:

$ nix-shell -p nodePackages.{bower,bower2nix}
(downloading hundreds of millions of derivations)
...
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: typescript@~1.8.9 (node_modules/typescript):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: request to https://registry.npmjs.org/typescript failed: cache mode is 'only-if-cached' but no cached response available.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: tsd@~0.6.5 (node_modules/tsd):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: request to https://registry.npmjs.org/tsd failed: cache mode is 'only-if-cached' but no cached response available.

audited 95 packages in 0.824s \ postinstall:bower2nix: WARN optional SKIPPING OPTIONAL D[0m

4 packages are looking for funding
  run `npm fund` for details

found 11 vulnerabilities (5 low, 6 high)
  run `npm audit fix` to fix them, or `npm audit` for details
@nix { "action": "setPhase", "phase": "fixupPhase" }wer2nix: timing audit body Completed in 1m
post-installation fixup
rewriting symlink /nix/store/50lf045gfws0nk86gyxmnf9yksrc07ih-node_bower2nix-3.2.0/bin to be relative to /nix/store/50lf045gfws0nk86gyxmnf9yksrc07ih-node_bower2nix-3.2.0
patching script interpreter paths in /nix/store/50lf045gfws0nk86gyxmnf9yksrc07ih-node_bower2nix-3.2.0

Whole lotta noise at the end there. Including what looks like some improper ANSI escapes? I don’t know. I don’t care.

I am comforted by the warning about how many vulnerabilities this has. I dunno if that’s from bower or bower2nix or whether or not I should care in the slightest.

Anyway, let’s run it:

[nix-shell:~/src/old/tocky]$ bower2nix
# Generated by bower2nix v3.2.0 (https://github.com/rvl/bower2nix)
{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
Parsing /Users/ian/src/old/tocky/bower.json failed: ReferenceError: primordials is not defined

Huh. I don’t know what that means.

I think maybe the format of a bower.json has changed since 2014.

No: Googling the error tells me this is some sort of Node version incompatibility thing?

Which I would not think possible, given that, you know, this is supposed to be Nix country. Packages can specify which Node they depend on, can’t they?

$ nix-env -qa nodejs
nodejs-10.24.1
nodejs-12.22.1
nodejs-14.16.1
nodejs-14.16.1
nodejs-15.14.0
nodejs-16.1.0

Or is this some deficiency of the Nix nodePackages infrastructure that they all assume latest Node?

Let’s suspect it’s the fault of my bower.json. Let’s try to use the one from the example:

[nix-shell:~/scratch/bowering]$ cat bower.json

{
  "name": "my-web-app",
  "dependencies": {
    "angular": "~1.5.0",
    "bootstrap": "~3.3.6"
  }
}

[nix-shell:~/scratch/bowering]$ bower2nix
# Generated by bower2nix v3.2.0 (https://github.com/rvl/bower2nix)
{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
Parsing /Users/ian/scratch/bowering/bower.json failed: ReferenceError: primordials is not defined

Well, we tried. I go to file an issue, but someone has beaten me to it:

https://github.com/rvl/bower2nix/issues/20

Well, theoretically, if this worked, then I could use this to create a bower_components directory. Presumably in the Nix store though, right?

How does that work?

Let’s see if we can piece a little bit more together. The manual tells me that, had bower2nix worked, it would have given me output like this:

{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
  (fetchbower "angular" "1.5.3" "~1.5.0" "1749xb0firxdra4rzadm4q9x90v6pzkbd7xmcyjk6qfza09ykk9y")
  (fetchbower "bootstrap" "3.3.6" "~3.3.6" "1vvqlpbfcy0k5pncfjaiskj3y6scwifxygfqnw393sjfxiviwmbv")
  (fetchbower "jquery" "2.2.2" "1.9.1 - 2" "10sp5h98sqwk90y4k6hbdviwqzvzwqf47r3r51pakch5ii2y7js1")
]; }

I save that as bower-components.nix. Now the manual tells me that to actually create my bower_components directory, I need to call the Nix function buildBowerComponents.

Okaaaay. I don’t know how to just “call” that function and build it. But I know how to make a default.nix that does just that:

[nix-shell:~/scratch/bowering]$ cat default.nix

{ pkgs ? import <nixpkgs> {} }:
pkgs.buildBowerComponents {
  name = "bowering";
  generated = ./bower-components.nix;
  src = ./.;
}

[nix-shell:~/scratch/bowering]$ nix-build
these derivations will be built:
  /nix/store/adzqz5nz7yzzvsipfbrsba0h42phwgd3-angular-1.5.3.drv
  /nix/store/hl0l67smi2cy3j6i0v5lx84qrhbfcskh-jquery-2.2.2.drv
  /nix/store/q0pdb64p59qlzzzaxccfzrk7yfy77px2-builder.pl.drv
  /nix/store/v4gvynz3spyykqwqi6bpzbwdhqbi7lnq-bootstrap-3.3.6.drv
  /nix/store/cddah8by5f3dnfjs3rmd64yg6mcfdb98-bower-env.drv
  /nix/store/5vil9g9x000kwxcbg0bg7dab9j9i6iv8-bower_components-bowering.drv
these paths will be fetched (0.10 MiB download, 0.19 MiB unpacked):
  /nix/store/mxgzq97yax2zzgl52gql8i1kzkvpi6dv-nss-cacert-3.63
building '/nix/store/q0pdb64p59qlzzzaxccfzrk7yfy77px2-builder.pl.drv'...
copying path '/nix/store/mxgzq97yax2zzgl52gql8i1kzkvpi6dv-nss-cacert-3.63' from 'https://cache.nixos.org'...
building '/nix/store/adzqz5nz7yzzvsipfbrsba0h42phwgd3-angular-1.5.3.drv'...
hash mismatch in fixed-output derivation '/nix/store/1isbljl2cfqknmacnv0iq81jx8bay3qx-angular-1.5.3':
  wanted: sha256:1749xb0firxdra4rzadm4q9x90v6pzkbd7xmcyjk6qfza09ykk9y
  got:    sha256:0z50hlpj3f5nfnxcaxr5qn07jnavfksqy1xcwz0nmnh2rff0dljf
cannot build derivation '/nix/store/cddah8by5f3dnfjs3rmd64yg6mcfdb98-bower-env.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5vil9g9x000kwxcbg0bg7dab9j9i6iv8-bower_components-bowering.drv': 1 dependencies couldn't be built
error: build of '/nix/store/5vil9g9x000kwxcbg0bg7dab9j9i6iv8-bower_components-bowering.drv' failed

Okay. So the example output was just… wrong. It had all the wrong hashes for the packages. That’s weird. I update them all:

[nix-shell:~/scratch/bowering]$ cat bower-components.nix

{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
  (fetchbower "angular" "1.5.3" "~1.5.0" "0z50hlpj3f5nfnxcaxr5qn07jnavfksqy1xcwz0nmnh2rff0dljf")
  (fetchbower "bootstrap" "3.3.6" "~3.3.6" "1pq8av43bm0il6ml90yqh6vdfvdfzbx7adb8swbr8fp6zdzhv8bw")
  (fetchbower "jquery" "2.2.2" "1.9.1 - 2" "1a05yawxryww5agsmib3slvqsg1f9mb4dr7spy7zb2ag7agmp8s7")
]; }

And try again:

[nix-shell:~/scratch/bowering]$ nix-build
these derivations will be built:
  /nix/store/x9lckg0v6x87vx7n3b1lmqp7g0yy2vgf-bower_components-bowering.drv
building '/nix/store/x9lckg0v6x87vx7n3b1lmqp7g0yy2vgf-bower_components-bowering.drv'...
bower cached        https://github.com/twbs/bootstrap.git#3.3.6
bower cached        https://github.com/angular/bower-angular.git#1.5.3
bower cached        https://github.com/jquery/jquery-dist.git#2.2.2
bower install       bootstrap#3.3.6
bower install       angular#1.5.3
bower install       jquery#2.2.2

bootstrap#3.3.6 bower_components/bootstrap
└── jquery#2.2.2

angular#1.5.3 bower_components/angular

jquery#2.2.2 bower_components/jquery
/nix/store/yvkngx82xwc0zzm7pdawj8rd8vhdp5vb-bower_components-bowering

Okay! Now we’re getting somewhere.

[nix-shell:~/scratch/bowering]$ tree result | head
result
└── bower_components
    ├── angular
    │   ├── README.md
    │   ├── angular-csp.css
    │   ├── angular.js
    │   ├── angular.min.js
    │   ├── angular.min.js.gzip
    │   ├── angular.min.js.map
    │   ├── bower.json

So the result wasn’t the bower_components directory, as I would have expected. Instead it was a directory with a subdirectory called bower_components, for some reason. I am not very interested in the actual details here, but sure. I made the bower_components directory with Nix, sort of, eventually.

Now, I still have a bower.json file, so this would still be compatible (at least in theory) with non-Nix users. I like that: it doesn’t replace bower altogether, it merely augments it with Nix caching and determinism and such.

But bower.json isn’t the source of truth to Nix; I have to manually run bower2nix in order to keep it in sync. That’s gross. I don’t really like that. I can definitely imagine a fuzzy version specification causing bower to silently resolve to a newer version of a dependency, and then the Nix build will diverge from the non-Nix builds without my realizing.

But that’s sort of a fault of Bower, for not producing a lock file: the same thing could happen with two different people not using Nix by just running bower install at different points in time. So we’ll see how Nix handles this problem with “real” platforms.

A bigger issue here is that I can reference this Nix store bower_components directory from the build instructions of my default.nix. But if I’m actually developing iteratively, I will be in a nix-shell, and then… how am I supposed to use this? I’m not, am I? Am I just supposed to run bower in my nix-shell?

This feels weird. Is there a way to say that I depend on bower, but only in nix-shell, not in a normal build?

Are nix-shell environments necessarily identical to the nix-build environments?

I don’t know. Let’s try some more, and see if they explain this.

15.9. Go

I said I could say something interesting about Go, but I lied. I’ve never really used Go, and I don’t know much about its module system, except that at one point it involved referencing GitHub repository paths directly with no sense of version constraints except “well it doesn’t build anymore.”

So I thought I may as well learn how Go has changed while I was here.

The function buildGoModule builds Go programs managed with Go modules. It builds a Go modules through a two phase build:

An intermediate fetcher derivation. This derivation will be used to fetch all of the dependencies of the Go module.

A final derivation will use the output of the intermediate derivation to build the binaries and produce the final output.

Okay. So it seems like Go projects have a special vendor/ directory, that holds all of the dependencies, and the Nix function buildGoModule will… regenerate that vendor directory. If you want it to. It doesn’t have to. You can also have it trust the contents of vendor, in which case… this is just shorthand for running go build?

I look through the source in ~/src/nixpkgs/pkgs/developement/go-modules/generic/default.nix.

And basically, there are two derivations here: one called go-modules, which is just the vendor/ directory, and one that is the actual project. It’s quite a bit more complicated than go build, however, and involves quite a lot of GOPATH arithmetic and directory detection (??) and apparent scraping of stdout in order to set a consistently useful exit code (???). And at least some small amount of cross-compilation complication.

Anyway, okay, I dunno; seems kinda reasonable. No magical go2nix situation here. Operates with normal go mod vendor commands, and gives you some decent build defaults.

There’s also support for the “legacy” non-module Go dependencies, which requires writing out dependencies as Nix expressions manually (?). And writing out dependencies is annoying, because you need, you know, the hash.

Where do you get the hash?

Do you put in an empty string and run it and copy the “correct” hash from the error message you get?

That’s what I was doing for the Bower example. It did not feel good. I did not feel smart while I was doing that.

I remember finding – somewhat hidden, in the end of the Command Reference – a command called nix-prefetch-url. man nix-prefetch-url says:

This command is just a convenience for Nix expression writers. Often a Nix expression fetches some source distribution from the network using the fetchurl expression contained in Nixpkgs. However, fetchurl requires a cryptographic hash. If you don’t know the hash, you would have to download the file first, and then fetchurl would download it again when you build your Nix expression. Since fetchurl uses the same name for the downloaded file as nix-prefetch-url, the redundant download can be avoided.

Which, okay. I am less concerned with the redundant download – although that is great – and more concerned with the ergonomics. Does this work with a Git repo?

The manual gives this Go example:

[ 
  {
    goPackagePath = "gopkg.in/yaml.v2"; 

    fetch = {
      type = "git"; 
      url = "https://gopkg.in/yaml.v2";
      rev = "a83829b6f1293c91addabc89d0571c246397bbf4";
      sha256 = "1m4dsmk90sbi17571h6pld44zxz7jc4lrnl4f27dpd1l8g5xvjhh";
    };
  }
  {
    goPackagePath = "github.com/docopt/docopt-go";

    fetch = {
      type = "git";
      url = "https://github.com/docopt/docopt-go";
      rev = "784ddc588536785e7299f7272f39101f7faccc3f";
      sha256 = "0wwz48jl9fvl1iknvn9dqr4gfy1qs03gxaikrxxp9gry6773v3sj";
    };
  }
]

Anyone want to take bets on how magical nix-prefetch-url is?

$ nix-prefetch-url https://github.com/docopt/docopt-go
[0.2 MiB DL]
path is '/nix/store/fqq189a0wcavhl0rkmw9w20kkcqhnc4g-docopt-go'
0lb30s8i7q30rrv5jzlssrva9rj99rkaxnn3yam6ll2my8wzwnpi

$ cat /nix/store/fqq189a0wcavhl0rkmw9w20kkcqhnc4g-docopt-go
<!DOCTYPE html>
<html lang="en" data-color-mode="auto" data-light-theme="light" data-dark-theme="dark">
  <head>
    <meta charset="utf-8">
...

About what I expected.

In any case there’s no way to specify the rev, so this command isn’t going to work regardless.

Let’s see if there’s a modern, nix command equivalent. There are only two plausible-sounding subcommands…

$ nix hash-file --help
Usage: nix hash-file <FLAGS>... <PATHS>...

Summary: print cryptographic hash of a regular file.

Flags:
      --base16       print hash in base-16
      --base32       print hash in base-32 (Nix-specific)
      --base64       print hash in base-64
      --sri          print hash in SRI format
      --type <TYPE>  hash algorithm ('md5', 'sha1', 'sha256', or 'sha512')

Note: this program is EXPERIMENTAL and subject to change.

$ nix hash-path --help
Usage: nix hash-path <FLAGS>... <PATHS>...

Summary: print cryptographic hash of the NAR serialisation of a path.

Flags:
      --base16       print hash in base-16
      --base32       print hash in base-32 (Nix-specific)
      --base64       print hash in base-64
      --sri          print hash in SRI format
      --type <TYPE>  hash algorithm ('md5', 'sha1', 'sha256', or 'sha512')

Note: this program is EXPERIMENTAL and subject to change.

Neither of which appear to be anything.

Well, open question I guess.

15.10. Haskell

The documentation for the Haskell infrastructure is published at https://haskell4nix.readthedocs.io/. The source code for that site lives in the doc/ sub-directory of the cabal2nix Git repository and changes can be submitted there.

That’s it! That’s the whole chapter. That was easy reading.

Sigh, no; I suppose I have to follow that first link there.

But this makes me think that it’s going to be very complicated, and I am afraid.

I remember, five years ago, when I was first doing Nix, there being a mirror of every single Hackage package in Nixpkgs itself, as a Nix expression.

But that makes so little sense to me now that it feels like it must be a false memory. There are too many packages, and in order to do any useful dependency resolution you would need to have every single version of every single package.

But let’s see. Let’s take a look at the Nixpkgs User’s Guide.

Nixpkgs distributes build instructions for all Haskell packages registered on Hackage, but strangely enough normal Nix package lookups don’t seem to discover any of them, except for the default version of ghc, cabal-install, and stack:
$ nix-env -i alex
error: selector ‘alex’ matches no derivations
$ nix-env -qa ghc
ghc-8.10.2

That’s… what? I assume by “strangely enough” it means “to most users of Nix” or something.

The Haskell package set is not registered in the top-level namespace because it is huge. If all Haskell packages were visible to these commands, then name-based search/install operations would be much slower than they are now. We avoided that by keeping all Haskell-related packages in a separate attribute set called haskellPackages, which the following command will list:
$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages
haskellPackages.a50              a50-0.5
haskellPackages.AAI              AAI-0.2.0.1
haskellPackages.abacate          abacate-0.0.0.0
haskellPackages.abc-puzzle       abc-puzzle-0.2.1
haskellPackages.abcBridge        abcBridge-0.15
haskellPackages.abcnotation      abcnotation-1.9.0
haskellPackages.abeson           abeson-0.1.0.1
[... some 14000 entries omitted  ...]

Okay, so this is basically what we already saw with nodePackages above. It goes on to show how to install them. They just use the exact name that the package has on Hackage, with this fun caveat:

(Actually, this convention causes trouble with packages like 3dmodels and 4Blocks, because these names are invalid identifiers in the Nix language. The issue of how to deal with these rare corner cases is currently unresolved.)

Can’t you just quote them? Aren’t arbitrary strings allowed as attributes?

nix-repl> { "3blocks" = 1; }
{ "3blocks" = 1; }

Shrug. Not important.

Our current default compiler is GHC 8.10.x and the haskellPackages set contains packages built with that particular version. Nixpkgs contains the last three major releases of GHC and there is a whole family of package sets available that defines Hackage packages built with each of those compilers, too:
nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc8104
nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc901

Okay, neat. Don’t love smashing numbers together like that, but alright.

It goes on to describe normal cabal stuff and Nix stuff… nix-shell -p, etc. Nothing too interesting.

It describes a lot about making a compiler with certain packages available. I assume this means, like, built-in to ghci or something? I don’t really understand. Then getting a GHC with Hoogle and doc support and stuff. On the ghcWithHoogle:

This environment generator not only produces an environment with GHC and all the specified libraries, but also generates a hoogle and haddock indexes for all the packages, and provides a wrapper script around hoogle binary that uses all those things. A precise name for this thing would be “ghcWithPackagesAndHoogleAndDocumentationIndexes”, which is, regrettably, too long and scary.

Love it.

And that looks pretty cool? Like, I wanna hoogle from the command-line and have it print URLs to local Haddock documentation. That seems great.

Next we see haskell-language-server… I am not really going to set up a full Haskell dev environment via Nix right now, but it looks pretty tame. It recommends putting this in a shell.nix and not installing it globally, which is reassuring; I was wondering about that. But then do I need to open a new instance of Emacs from within that nix-shell in order for it to “see” the hls on my PATH? Or is there some Emacs wizardry by which it can detect that I am in a Nix project, and automatically use the correct binary?

Who am I kidding. It’s Emacs. I’m sure it can do that.

I’ll have to come back to that later, though. That sounds like something that deserves a post of its own.

Finally we get to stack.

I have actually used Nix to build stack project before, because I host some Haskell projects on my NixOS server, so I had to figure out how to do that. Here’s what I did (this is a very old project that I should maybe go in and update…)

claudius $ cat stack.yaml

flags: {}

packages:
- '.'

extra-deps:
- suspend-0.2.0.0
- timers-0.2.0.3

resolver: lts-10.4

nix:
  enable: true
  packages:
  - sqlite
  - zlib
  - binutils

But I was sort of using Nix “inside out.” I was still running regular stack build commands; stack just knows how to transparently create a Nix environment. Which is great, but also meant that I didn’t really know anything about Nix while I was doing this.

Anyway, I learn that I can also set an arbitrary custom shell.nix expression and use that to customize the build environment, instead of using the simple declarative packages list that stack provides me. Neat.

Next up I learn about “ad-hoc environments for nix-shell.” And I see this very cool example:

nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [mtl pandoc])"

I didn’t know you could do that! I’ve only ever seen attribute names passed to nix-shell -p. I didn’t know you could also pass expressions! Today I learned.

Anyway, let’s try that. Do you think it’ll make me build GHC from source?

It did not. It peacefully downloaded most of Hackage, while I nervously watched my free disk space dwindle to nothing, and then… with 4.7 gibibytes to spare, it dropped me into a shell.

Okay. Neat. It’s cool that that works. And now I have pandoc.

Of course, I already had pandoc, but now I have… a different pandoc, I guess. A newer pandoc? An older pandoc? pandoc --version output is identical. I don’t know.

Anyway. That’s neat, and it teaches me how to write the equivalent shell.nix – although, oddly, it does not use the mkShell function. It just makes a derivation and supplies a buildInputs manually, and a shellhook to set the right environment variables (a weird GHC quirk that it explained and I did not mention).

Next up, something weird:

If your own Haskell packages have build instructions for Cabal, then you can convert those automatically into build instructions for Nix using the cabal2nix utility, which you can install into your profile by running nix-env -i cabal2nix.

So my impression from that statement is that Nix does this automatically for everything in Hackage, but if I want to do the equivalent thing myself (because caching, consistency, whatever), then this is how. Okay. I test it on that same simple Haskell project:

[nix-shell:~/src/basilica]$ cabal2nix .

{ mkDerivation, aeson, base, base16-bytestring, bytestring
, case-insensitive, classy-prelude, configurator, containers
, cryptohash, DRBG, esqueleto, filepath, http-client, http-types
, io-streams, lens, lib, lifted-base, monad-control, monad-logger
, mtl, persistent, persistent-sqlite, persistent-template, random
, scotty, suspend, text, time, timers, transformers, unix-time, wai
, wai-websockets, warp, websockets, wreq
}:
mkDerivation {
  pname = "basilica";
  version = "0.1.0.0";
  src = ./.;
  isLibrary = false;
  isExecutable = true;
  executableHaskellDepends = [
    aeson base base16-bytestring bytestring case-insensitive
    classy-prelude configurator containers cryptohash DRBG esqueleto
    filepath http-client http-types io-streams lens lifted-base
    monad-control monad-logger mtl persistent persistent-sqlite
    persistent-template random scotty suspend text time timers
    transformers unix-time wai wai-websockets warp websockets wreq
  ];
  license = lib.licenses.mit;
}

Very tame! Okay. Presumably this will be supplied some fancy wrapper called mkDerivation.

Now it shows me how to write an override that will allow me to use my own custom packages with the Nix resolution logic.

Hmm. This is a little weird, since that feels like a global property – at least a user-wide property – and not a per-project property, as I would want. It feels like I’m unlikely to have a conflict among my own private Haskell packages where this would ever cause a problem, so I do not worry about it further.

And… okay! That’s it. All pretty tame.

15.12. iOS

This component is basically a wrapper/workaround that makes it possible to expose an Xcode installation as a Nix package by means of symlinking to the relevant executables on the host system.

Since Xcode can’t be packaged with Nix, nor we can publish it as a Nix package (because of its license) this is basically the only integration strategy making it possible to do iOS application builds that integrate with other components of the Nix ecosystem

Interesting.

I… I suppose that some people like Nix quite a bit, and they are willing to go through some weird trouble to get iOS projects building under Nix. This sounds crazy to me, on the surface – I can’t imagine as a developer that this experience is ergonomic enough to use – but I do understand wanting to use Nix as your CI environment, I guess.

I don’t actually have Xcode installed on this computer, on account of it requiring more than 40GB of free space to install and this laptop only having a 128GB SSD. It’s very, very old.

So I can’t really test this, or see if it actually works.

15.15. Node.js

Okay. So we already saw the nodePackages thing. But apparently it’s not a complete mirror of the NPM registry. I assumed that the haskellPackages was a complete mirror of Hackage, because it didn’t say otherwise, but now we see:

As a rule of thumb, the package set should only provide end user software packages, such as command-line utilities. Libraries should only be added to the package set if there is a non-NPM package that requires it.

When it is desired to use NPM libraries in a development project, use the node2nix generator directly on the package.json configuration file of the project.

I try running node2nix on a simple JavaScript project that has a shockingly large number of dev dependencies for being just a set of helper functions.

This created some Nix files in the current directory:

[nix-shell:~/src/effing]$ wc -l *.nix
   17 default.nix
  567 node-env.nix
   40 node-packages.nix
  624 total

[nix-shell:~/src/effing]$ cat default.nix

# This file has been generated by node2nix 1.9.0. Do not edit!

{pkgs ? import <nixpkgs> {
    inherit system;
  }, system ? builtins.currentSystem, nodejs ? pkgs."nodejs-12_x"}:

let
  nodeEnv = import ./node-env.nix {
    inherit (pkgs) stdenv lib python2 runCommand writeTextFile;
    inherit pkgs nodejs;
    libtool = if pkgs.stdenv.isDarwin then pkgs.darwin.cctools else null;
  };
in
import ./node-packages.nix {
  inherit (pkgs) fetchurl nix-gitignore stdenv lib fetchgit;
  inherit nodeEnv;
}

[nix-shell:~/src/effing]$ cat node-packages.nix

# This file has been generated by node2nix 1.9.0. Do not edit!

{nodeEnv, fetchurl, fetchgit, nix-gitignore, stdenv, lib, globalBuildInputs ? []}:

let
  sources = {};
  args = {
    name = "effing";
    packageName = "effing";
    version = "2.0.1";
    src = ./.;
    buildInputs = globalBuildInputs;
    meta = {
      description = "Function functions.";
      license = "MIT";
    };
    production = true;
    bypassCache = true;
    reconstructLock = true;
  };
in
{
  args = args;
  sources = sources;
  tarball = nodeEnv.buildNodeSourceDist args;
  package = nodeEnv.buildNodePackage args;
  shell = nodeEnv.buildNodeShell args;
  nodeDependencies = nodeEnv.buildNodeDependencies (lib.overrideExisting args {
    src = stdenv.mkDerivation {
      name = args.name + "-package-json";
      src = nix-gitignore.gitignoreSourcePure [
        "*"
        "!package.json"
        "!package-lock.json"
      ] args.src;
      dontBuild = true;
      installPhase = "mkdir -p $out; cp -r ./* $out;";
    };
  });
}

Interestingly, all of this seems, just… generic? It didn’t seem to… read my dependencies. Or maybe it doesn’t handle devDependencies? There’s no reference to, like, a pinned mocha or something. I dunno. Let’s see what happens if we try to use it:

[nix-shell:~/src/effing]$ nix-build
(doing stuff)

It worked! And it produced a bunch of results. What are they? The manual doesn’t say anything about it. Let’s take a look:

[nix-shell:~/src/effing]$ tree result*
result
├── bin -> lib/node_modules/.bin
└── lib
    ├── package-lock.json
    └── package.json
result-2
└── lib
    └── node_modules
        └── effing
            ├── CHANGELOG.md
            ├── LICENSE
            ├── README.md 
            ⋮
result-3
└── bin
    └── shell
result-4
├── nix-support
│   └── hydra-build-products
└── tarballs
    └── effing-2.0.1.tgz

So result is… I don’t know. It contains a broken symlink. And a copy of package.json, with slightly different whitespace. And a basically empty package-lock.json that contains no actual locks.

result-2 is… just a literal copy of my source? But in a lib/node_modules subdirectory. Okay.

result-3 is just this one script:

[nix-shell:~/src/effing]$ cat result-3/bin/shell
#! /nix/store/30njb8l701pwnm5ya749fh2cgyc2d70m-bash-4.4-p23/bin/bash -e

exec /nix/store/30njb8l701pwnm5ya749fh2cgyc2d70m-bash-4.4-p23/bin/bash

Okay.

And result-4 is, I guess, the actual build result? But zipped up?

[nix-shell:~/src/effing]$ tar tvf result-4/tarballs/effing-2.0.1.tgz
-rw-r--r-- 0/0            1054 1985-10-26 01:15 package/LICENSE
-rw-r--r-- 0/0             953 1985-10-26 01:15 package/package.json
-rw-r--r-- 0/0             111 1985-10-26 01:15 package/CHANGELOG.md
-rw-r--r-- 0/0            9401 1985-10-26 01:15 package/README.md
-rw-r--r-- 0/0             502 1985-10-26 01:15 package/default.nix
-rw-r--r-- 0/0           20335 1985-10-26 01:15 package/node-env.nix
-rw-r--r-- 0/0            1017 1985-10-26 01:15 package/node-packages.nix

Nope. Okay. I have no idea. And the manual doesn’t really have any more words to say.

15.16. OCaml

Alright, we’re sort of getting into a rhythm. There’s a function called buildDunePackage. It does the thing. Except, does it? Does it do the thing?

The manual doesn’t say where to find the buildDunePackage function, but convention has me guess to look in ocamlPackages, which turns out to be correct.

I try to make a trivial default.nix for one of my Dune projects:

$ cat default.nix

let pkgs = import <nixpkgs> {}; in
pkgs.ocamlPackages.buildDunePackage rec {
  pname = "hexagain";
  version = "0.0.0";
  minimumOCamlVersion = "4.03";
  src = ./.;
}

$ nix-build
error: dune is not available for OCaml 4.12.0
(use '--show-trace' to show detailed location information)

Hmm. Umm. Unexpected. I look through the nixpkgs source and find this:

if !lib.versionAtLeast ocaml.version "4.02"
then throw "dune is not available for OCaml ${ocaml.version}"

But… that can’t be right. Because. 4.12.0 is “at least” 4.02. Isn’t it? If you tell me otherwise, I’m going to throw a fit.

nix-repl> pkgs.lib.versionAtLeast "4.12.0" "4.02"
true

Okay, good. So it must be coming from the other throw…

if lib.versionOlder ocaml.version "4.08"
then throw "dune is not available for OCaml ${ocaml.version}"

nix-repl> pkgs.lib.versionOlder "4.12.0" "4.08"
false

Okay, what the heck. I’m very confused. I think my local ~/src/nixpkgs has diverged from my channel? I think I’m looking at code that is not the code I’m running. Because when I run --show-trace it points me to a file that doesn’t exist.

I have been very hesitant to update my channel, because I have been bit by it in the past, and lost access to the binary cache.

So I carefully update my local clone to the exact version of my channel, and…

Find the actual source of my grief:

if !lib.versionAtLeast ocaml.version "4.02"
|| lib.versionAtLeast ocaml.version "4.12"
then throw "dune is not available for OCaml ${ocaml.version}"

Aha. Well. I have no idea… why it isn’t compatible. With the default OCaml. Seems… well, you know how it seems.

I think when I was writing this package, I was doing so against 4.07.1. But there’s no ocamlPackages_4_07_1. So… let’s see if it builds with something way older?

I have to cancel the first attempt so I can collect garbage, and get rid of all the Haskell/Node stuff that I’ve built up, because my disk is dangerously close to full. I try again…

$ nix-build
warning: dumping very large path (> 256 MiB); this may run out of memory

I’ve never seen this before, and I don’t know what it means. It printed that and then just hung, for a long time, before the usual “these derivations will be built” message.

One of the derivations it claims will be built is:

/nix/store/70rhm4bbwrpnbiqsg88zlv1wfkpmj7lh-x11env.drv

I don’t know what that’s about.

And now it’s compiling the OCaml compiler from source.

I guess that there’s no substitute available for 4.03? So the fallback is to build from source?

I have another idea: install it with something other than Nix, where substitutes are still available for old versions of things.

I don’t want to spend a week compiling OCaml, so I interrupt it and move on. Maybe one day Nix’s Dune infrastructure will support the default OCaml.

15.19. Python

Okay, we’re still going, huh. There’s still more. Only two more! Let’s get it over with.

So: Python. Python has this crazy versioning mess unseen anywhere else in the programming world. So there are multiple Pythons in Nixpkgs.

So far this seems pretty similar to the Haskell stuff.

$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])'

Sure sure.

A couple weird formatting problems with this section of the manual. Huh.

This describes how to use the nix-shell -i shebang. And then a shell.nix:

with import <nixpkgs> {};
(python38.withPackages (ps: [ps.numpy ps.toolz])).env

Interesting to see that .env? What is that? I get that it needs to be different than the nix-shell -p invocation, but I don’t know in general how to go from a nix-shell -p to a shell.nix. Which… is sort of a crazy statement. Maybe mkShell works in all cases? I don’t know.

This section seems to be nicely written for people who are not super experienced with Nix. It explains this expression in detail, but not the .env part. Hmm.

Oh! It does explain how to combine this with mkShell:

with import <nixpkgs> {};
let
  pythonEnv = python38.withPackages (ps: [
    ps.numpy
    ps.toolz
  ]);
in mkShell {
  buildInputs = [
    pythonEnv

    black
    mypy

    libffi
    openssl
  ];
}

And yeah, it looks like the trivial thing works. Nice.

Next it describes how to do this with an overlay. Neat.

And then we see the obligatory buildPythonPackage.

Alright. We have a function called fetchPypi, which can grab dependencies.

Anyway, we see some code examples, which are functions that take a whole bunch of other Python derivations as arguments. It doesn’t really explain… how these work, or how to call these, or what this means.

Like, these are functions that need to be called with something like callPackage, presumably. Except not callPackage; they need to be called with specifically python38.pkgs or whatever.

I can see that there is a function called python38.pkgs.callPackage that presumably does this, but there’s no mention of it so far.

Let’s keep going.

Checks. More stuff. Wow. This is a really long section. I kind of skim it. I’m losing steam. This is like a full reference. Which is great! But not something I really need to read right now. I don’t expect to ever actually write Python again.

We get to pypi2nix. python2nix. A whole FAQ. Seems very impressive. But I do not read it. I admire the care that the Nix Python people have put into this documentation, though.

15.23. Rust

rustPlatform.buildRustPackage. It has this to say:

buildRustPackage requires a cargoSha256 attribute which is computed over all crate sources of this package. Currently it is obtained by inserting a fake checksum into the expression and building the package once. The correct checksum can be then take from the failed build.

Seems chill? I dunno. Seems fine. All of this seems reasonable. Looks easy, compared to what we’ve seen so far.

Tests can’t depend on the file structure in target/. Fair enough.

Nixpkgs contains a tool called carnix (nix-env -iA nixos.carnix), which can be used to turn a Cargo.lock into a Nix expression.

That Nix expression calls rustc directly (hence bypassing Cargo), and can be used to compile a crate and all its dependencies.

I don’t know… why I would want this. I like cargo. The example looks terrifying.

The manual goes on to describe how I can configure the build. Seems nice. But then:

Oftentimes you want to develop code from within nix-shell. Unfortunately buildRustCrate does not support common nix-shell operations directly (see this issue) so we will use stdenv.mkDerivation instead.

And then it just shows, like, a trivial shell.nix that just adds cargo and rustc as dependencies. Nothing fancy here.

It shows how to use nightly Rust. It involves… downloading Nix expressions from the nixpkgs-mozilla repo? Okay. Neat? I haven’t had occasion to use anything but the stable Rust yet myself, but I have not really written any Rust worth noting at this point in my life.

15.24. TeX Live

No no I am kidding. I am done. We are done.

So, I think… in general, I like the idea of having a nix-shell with stuff in it. I like using that to specify project dependencies. It’s a good thing to do. I have, in fact, been doing a lot of things with nix-shell -p because it’s such a nice way to get an environment set up very quickly.

But I don’t really understand the advantage yet of wanting to nix-build all my things. Like, to distribute software. Nor do I really understand the value add of something like cabal2nix. Most languages I write already have a quite nice way to do dependency resolution, and dependencies are generally small and easy enough that the /nix/store sharing isn’t really meaningful.

But anyway.

I suppose it’s something that I will need to see in practice. Who knows how it actually feels to use it. I will reserve judgment for now.

Well, that was quite a chapter.

And it’s also… very nearly the end of the manual. That was the last really substantial chapter, I think. I can see the light at the end of the tunnel.

How can I nix search through nixpkgs.nodePackages?
How am I supposed to use e.g. bower2nix while iterating on a package from nix-shell?
Can I have a dependency that only shows up in nix-shell, without being a build-time dependency of my derivation?
How can I see the hash for a git repository, ala nix-prefetch-url?
What the heck is up with the Node.js stuff?

Look, it was 2014. React was still slick with vernix; we didn’t know any better back then. Even though seven years have passed, I can still remember the creeping horror of Ember.js’s “Conflation over Composition” design philosophy… ↩︎