Ian Henry

How to Learn Nix, Part 49: nix-direnv is a huge quality of life improvement

Sun, 28 Jan 2024 00:00:00 +0000

The reason I discovered an ancient blog post the other day was that I had something new to say about Nix for the first time in over two years.

The thing I want to say is this: nix-direnv is great. It fixes roughly every problem that I’ve had with nix-shell, and does so in a much nicer way than my previous ad-hoc solutions.

This is important because I mostly just use Nix to document and install per-project native dependencies. I do use it to install “global” tools as well, but that is rarely very interesting, and most of my interaction with Nix these days consists of editing small shell.nix files.

But it took a bit of doing to get to the point that I felt good about using Nix for this. For one thing, shells don’t register GC roots, which means that every time you collect garbage you have to re-download all the dependencies for the project you were working on. We overcame that hurdle in part 37, by making a custom wrapper around nix-shell that sets up GC roots correctly, but it was surprisingly difficult.

For another thing, Nix is pretty insistent that you use bash as your interactive shell. I figured out a workaround for that in Nix classic, but essentially failed to make nix develop similarly usable.

nix-direnv solves both of these problems. Instead of spawning a new shell, it just adds environment variables to your existing shell. And when it evaluates shell.nix, it automatically registers the result as a GC root.

It also only re-evaluates shell.nix when it actually changes, which means that it in the typical case there’s no startup time. In contrast, my GC-root-installing wrapper takes about 750ms to open a typical shell (raw nix-shell, without the GC root evaluation dance, takes only 400ms). This doesn’t sound very long, because it’s not – I’m running Nix on what I can only characterize as a supercomputer. But I originally installed Nix on a laptop that pre-dated germ theory, and its startup latency was a lot more annoying.¹

nix-direnv also automatically updates the environment when shell.nix changes, so you don’t have to close and re-open your nix-shell whenever you add a dependency. Not only is this ergonomically better, but it also means that you don’t mess up your shell history every time you add a dependency or exit a project.

I had never used direnv before, and to this date the only thing I’ve used it for is managing my Nix shells. But it’s a general tool for managing per-directory environment variables, which is essentially all that nix-shell is. nix-shell can also register bash functions – if you’re using bash – which is useful if you want to use it to debug a derivation. But for my purposes, environment variables are all I really need.

direnv has some built-in support for Nix, but it isn’t great; direnv publishes a table outlining some of the advantages of using nix-direnv. nix-direnv is some sort of plugin(?) that replaces the native Nix support with something much better. And it’s great. It makes the “reproducible developer environment” aspect of Nix just work™. And it’s pretty easy to use:

First off, install nixpkgs.direnv and nixpkgs.nix-direnv.

I installed them with nix-env, using the same declarative wrapper that I wrote in part 22. If you install nix-direnv in a different way, the following will be different.

Installing nix-direnv doesn’t “enable” the plugin; you have to separately tell direnv about it:²

mkdir -p ~/.config/direnv
echo 'source ~/.nix-profile/share/nix-direnv/direnvrc' > ~/.config/direnv/direnvrc
echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc

Once you do that, you have to run the following commands in every directory that you want to nix-shellify:

echo 'use nix' > .envrc
direnv allow

And you’re done. That’s it! Now every time you navigate to that directory, you’ll have…

$ cd ~src/project
direnv: loading ~/src/project/.envrc
direnv: using nix
direnv: nix-direnv: using cached dev shell
direnv: export +CONFIG_SHELL +HOST_PATH +IN_NIX_SHELL +MACOSX_DEPLOYMENT_TARGET +NIX_BUILD_CORES +NIX_CFLAGS_COMPILE +NIX_COREFOUNDATION_RPATH +NIX_DONT_SET_RPATH +NIX_DONT_SET_RPATH_FOR_BUILD +NIX_ENFORCE_NO_NATIVE +NIX_IGNORE_LD_THROUGH_GCC +NIX_INDENT_MAKE +NIX_NO_SELF_RPATH +NIX_STORE +PATH_LOCALE +SOURCE_DATE_EPOCH +XDG_DATA_DIRS +__darwinAllowLocalNetworking +__impureHostDeps +__propagatedImpureHostDeps +__propagatedSandboxProfile +__sandboxProfile +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +gl_cv_func_getcwd_abort_bug +name +nativeBuildInputs +nobuildPhase +out +outputs +patches +phases +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system -PS2 ~PATH

Oh. Well that’s not great.

By default direnv prints every environment variable that it adds, removes, or changes. Which makes sense if you’re using it for, like, credentials or something, but for Nix shells it’s just a waste of scrollback.

There’s not really a simple way to suppress printing that giant export line, but you can hack it away by adding something like this to your .zshrc:

export DIRENV_LOG_FORMAT="$(printf "\033[2mdirenv: %%s\033[0m")"
eval "$(direnv hook zsh)"
_direnv_hook() {
eval "$(direnv export zsh 2> >(egrep -v -e '^....direnv: export' >&2))"
};

(The .s in the regex exclude the “dim text” control characters at the beginning of the line.)

That removes the giant export line without removing the rest of the input. And now:

$ cd ~src/project
direnv: loading ~/src/project/.envrc
direnv: using nix
direnv: nix-direnv: using cached dev shell

Ahhh. That’s better.

I’ve been using nix-direnv for a few months now, and I must say: I wish that I had installed it sooner. It’s a much nicer experience than the default nix-shell, and I’m happy that I can get rid of the bespoke hacks that I’ve accrued over the years.

…almost. The one thing this does not help with is nix-shell -p. nix-shell -p is a useful way to “temporarily” install packages without actually putting them on your PATH, and I still use my zsh hack so that nix-shell -p doesn’t drop me into a bash session. Although I do this rarely enough that I could probably just suffer through it.

To be fair I use tmux and just always have sessions open for the projects I’m working on, so it’s not like it was annoying very often. ↩︎
I think this is the sort of thing that home-manager does for you automatically, but I don’t use home-manager. ↩︎

How to Learn Nix, Part 48: Installing (single-user) Nix on macOS

Fri, 26 Jan 2024 00:00:00 +0000

Note: I wrote the vast majority of this post on August 24, 2022, but I decided that it was a whiny rant that I didn’t want to publish. I came across the draft in January 2024, remembered that this entire series is a whiny rant, and decided to publish it anyway.

I’ve been a happy Nix user for about 18 months now, and– well, not happy happy, but satisfied… no… not really satisfied either; perhaps it’s more of a resigned disgruntlement; a feeling that despite its many flaws, it’s still better than anything else out there, and I’ve invested so much time into it already that it would be a shame to give up now, so… am I describing Stockholm syndrome?

I’ve been imprisoned in Nix’s castle for about 18 months now, and I recently came into a new laptop – one of those shiny blue numbers – so naturally one of the first things I did was install Nix on it.

It would be nice if this blog post were only like a paragraph long, wouldn’t it? Step one: run the installer; there is no step two.

But I am afraid to report: there is very much a step two.

a tale of two nixen

There are two flavors of Nix, and you must choose at installation time which variety you will run.

You can install “single-user” Nix, in which Nix is a binary that you run as an interactive command that does things and forks subprocesses and works like any other package manager you’ve ever used. Or you can install “multi-user” Nix, in which Nix is a daemon, running as root, and the nix command merely makes requests to that daemon, which in turn coordinates a set of special nixbld users that exist so that you don’t build all of your software as root.

If you’re using NixOS, you have to use multi-user Nix, which makes sense to me: Nix controls the entire operating system, and there needs to be some central puppeteer to coordinate changes to the kernel or whatever.

But if you don’t use NixOS, then you have a choice. You can choose to set up a persistent daemon, create a dozen nixbld users, and have every command you run proxy through this centralized service. Or you can choose… not to do that.

I don’t know why you’d choose a multi-user installation when the single-user option is available. There might be a very good reason to prefer multi-user Nix, even when you are installing Nix on a laptop with a single user, but I do not know what it is. Obviously there is no documentation to explain the distinction or persuade the user one way or the other, so I can only speculate.

And needless to say, when I first tasted the forbidden fruit of the Nix tree all those year ago, I chose a single-user install, and I have never regretted that.

But then something upsetting happened.

macOS has always been an unloved corner of Nix, and installing Nix on macOS has been rather fraught with peril ever since macOS Catalina came out.

The problem seems so trivial: Nix requires a /nix directory to exist, but macOS does not let you create new top-level root directories, so you have to actually create a separate volume and mount it as /nix. Which might not sound difficult, but getting that to work with FileVault and the various hardware encryption configurations of different generations of Apple hardware and… I dunno, probably some other complicating factors, I don’t know how computers work… there was a heroic effort to keep macOS Nix working at all, at a time when some Nix maintainers were arguing that it would be easier to deprecate support for macOS entirely.

But there was an unfortunate casualty of this struggle: the single-user installation option was removed.

If you want to install Nix on macOS today, you must use a multi-user installation. That means running nix-daemon, that means messing with launchctl, creating a bunch of build users…

Which brings things back to me.

I had a new laptop. I wanted to install Nix on it. And the only option facing me…

Was to get in there and hack up an unofficial, unsupported single-user install, because I just can’t abide the idea of a multi-user Nix install.

And it’s not only a philosophical objection! Although I admit that I have a visceral aversion to the idea of installing a daemon running as root for something that absolutely does not need to be a daemon or running as root, I can understand that many people do not care about that.

But there is a real, practical reason to avoid the multi-user configuration: if you are using a multi-user Nix install on macOS, then every time you upgrade macOS, Nix will break, and you will have to go in and do some surgery to fix it.

This was true when I wrote this, in 2022. I don’t know if this is still true, but an alternate Nix installer – which also only supports multi-user installs on macOS – puts “survives macOS upgrades” at the top of its advantages over the official installer. Which implies to me that this is still a problem.

This is because the multi-user installation edits some of the system-provided config files – like /etc/bashrc – to insert some Nix setup things. And every time (every time?) you upgrade macOS, it will overwrite these files and restore them to the defaults. Is it hard to fix? No. Is it annoying? Yes. Is it sufficiently annoying to try to figure out a single-user install? To each their own.

In any case: the only difficult thing about installing Nix on macOS is creating that darn /nix directory. And clearly the multi-user install step knows how to do that. So I should be able to steal that part of it, create the directory, and then manually perform a regular single-user install. Right?

Right.

right

Well it worked great, actually! It wasn’t hard at all. The steps are:

First, download the Nix install script.

The official Nix installation instructions are of the curl | bash variety. But if you actually look at the script that you’re curling, it just turns around and curls another script¹ and runs that.

Actually a directory full of them. So the first thing to do is to get that directory of actual install instructions.

You can find them on GitHub, although I just modified the original script to download the installer, extract it, and then quit before doing anything else (er, and also to not delete the extracted artifacts on EXIT). The really interesting one is create-darwin-volume.sh, which, as you can probably guess, creates /nix.

But! You cannot just invoke this script by itself. This script is invoked by the larger installation script, which sets some environment variables that the create-darwin-volume expects to be able to read. Specifically it expects to be able to read…

You know what? I don’t think you care. Or rather, if you do care, I’m sure that you can figure it out from reading the script. It wasn’t very hard, and by the time you’re reading this the precise interplay between the two scripts probably changed anyway, and you would have to dive into the code to figure out what the modern version requires anyway.

Once you’ve created your /nix, modify the install script to remove the lines that force a multi-user install on macOS:

case "$(uname -s)" in
"Darwin")
INSTALL_MODE=daemon;;
*)
INSTALL_MODE=no-daemon;;
esac

Then run the install script, and you’re done.

or just, like, just make a multi-user install

It’s the future now – I wrote this section in 2024, which is part of why this segue is so abrupt.

So, I didn’t do that. I am still running the single-user nix install that I hacked together a year and a half ago. I was kind of worried initially that I was overlooking something that made this more complicated and it would break the moment I upgraded Nix, but no: it has held up perfectly. I don’t know why the official install script forbids single-user installs on macOS, because they seem to work better than multi-user installs – I have never once had to think about Nix during a macOS upgrade.

But I feel that I should correct something that I wrote previously: there are actual reasons why you might prefer a multi-user install, even on macOS.

It’s actually kind of nice to run Nix builds as special nixbld users. Those users only have permission to read and write to the Nix store; they can’t accidentally stick config files in your home directory during install time, for example, or reference a source file that you didn’t actually copy to the store. Whereas, with a single-user install, it’s possible to write Nix expressions that – accidentally or not – depend on files anywhere on your computer.

So the multi-user install has two advantages:

You get some level of sandboxing/protection from malicious packages
You get an error when you write a non-hermetic nix expression, instead of silently succeeding

Those advantage don’t seem worth any added complexity to me, let alone having to deal with upgrade problems, but not everyone uses Nix exactly the way that I do.

(1) is interesting from a security perspective in theory, but since I usually download pre-built derivations from the binary cache and then run it as myself, I am already vulnerable to malicious nixpkgs. Not as vulnerable, perhaps, if the binary cache did some kind of fancy virus scanning of uploaded executables… but we’re in the same ballpark.

I don’t really care about (2) at all, although I could see this being useful to people who regularly share Nix expressions with other human people. The cost of making a mistake here just seems so low to me, and the fix is so simple, but I guess if you make this mistake very often it would be nice to hear about it right away.

There might be even more advantages than these! But we have once again reached the limits of my knowledge.

why do you care so much

It’s 2022 again.

I can’t really explain why I’m so averse to a multi-user Nix installation. But I am.

And I know that if I had met Nix just a little bit later in life – after 2.4 was released, when the single-user install option was deprecated – I never would have come back to Nix in the first place.

Which is a shame, because I really do like Nix, and as much as I like to razz it, I still prefer it – philosophically and practically – to Homebrew.

So I am writing this post to say: you can still install Nix on macOS like any other package manager. You don’t need to take one look at it and go “whoa what the heck I’m not setting up some root-user daemon just to download packages from the internet” and close the tab. You can still use it like you would use any other package manager. You just… have to get your hands dirty first.

Which might actually be the perfect introduction to Nix, come to think of it.

Amusingly, even though the first step included no checksum (though you can find one if you care), this script does checksum the downloaded file. Maybe because it’s served from a different subdomain? I don’t know. ↩︎

The Fibonacci Matrix

Sun, 30 Jul 2023 00:00:00 +0000

When you think about the Fibonacci sequence, you probably imagine a swirling vortex of oscillating points stretching outwards to infinity:

Okay, no, obviously you don’t. Yet.

When you think about the Fibonacci sequence, you probably flush with a latent rage when you remember that it is, more often than not, the way that we introduce the concept of “recursive functions” to new programmers, in some sort of cruel hazing intended to make it harder for them to ever appreciate how recursion can help them write better programs. Sometimes we even add memoization, and call it “dynamic programming,” in order to impress upon them that even the most trivial problems deserve complex, inefficient solutions.

Er, okay, you probably don’t think about the Fibonacci sequence much at all. It doesn’t, you know, come up very often.

But I hope that you will spend some time thinking about it with me today, because I think that the Fibonacci sequence – despite being a terrible showcase for recursion – is a really interesting vector for discussing some techniques from linear algebra.

how to fibonacci	space complexity	time complexity
insane recursion	exponential	exponential
memoized insane recursion	linear	linear
trivial iteration	constant	linear
exponentiation-by-squaring	constant	logarithmic
eigendecomposition	let's talk

We will spend no time on the recursive Fibonaccis; I’m sure that you’ve seen them before. Instead, let’s skip right to the “obvious” way to calculate Fibonacci numbers:

function fib(n) {
if (n == 0) {
return 0;
}
let current = 1;
let previous = 0;
for (let i = 0; i < n - 1; i++) {
const next = current + previous;
previous = current;
current = next;
}
return current;
}

No recursion, no memoization. We have two pieces of state: the “current number” and the “previous” number, and at every step of the iteration we advance both of these to new values.

But there’s something very interesting about this function: the new values for our state are a linear combination of the old values.

current' = current + previous
previous' = current

Using x' to mean “the next value for x.”

And you might recognize this as a “system of linear equations.” I think it’s more obvious when we write it like this:

current' = 1 * current + 1 * previous
previous' = 1 * current + 0 * previous

And you might remember that there’s another, more cryptic way to write down a system of linear equations:


current'
previous'


1	1
1	0


current
previous

This is exactly the same thing! This is just another way of writing the equation – it’s just a shorthand notation.

Here, let’s test it out to make sure of that:


1	1
1	0


8
5


1 • 8 + 1 • 5
1 • 8 + 0 • 5


13
8

Well that’s exactly what we expected – 13 is the next Fibonacci number in the sequence, and 8 was the previous one.

We can, of course, repeat this process, by applying the system of linear equations again:


1	1
1	0


13
8


21
13

Or, to put that another way:


1	1
1	0


1	1
1	0


8
5


21
13

And here’s why we care: matrix multiplication is associative, so we can actually think of that like this:


1	1
1	0


1	1
1	0


8
5


21
13

Or:


2	1
1	1


8
5


21
13

In other words: given a system of linear equations to find the next state of our iteration, we can square the matrix-of-coefficients of the system to find a new system of linear equations that represents “two states from now.”

Of course we don’t need matrices to do this. We can compute a formula for “two steps” of our iteration using term substitution:

current' = current + previous
previous' = current
current'' = current' + previous'
previous'' = current'
current'' = (current + previous) + current
previous'' = (current + previous)
current'' = 2 * current + previous
previous'' = current + previous

Which is a new system of linear equations – which we can represent as a matrix as well.


current''
previous''


2	1
1	1


current
previous

We got the same result, because of course we did: multiplying by this matrix really means “advance to the next state.” Multiplying twice means “advance to the next state and then advance to the next state after that.”

And we can keep going. What’s the state three steps from now?


1	1
1	0


1	1
1	0


1	1
1	0


3	2
2	1

Or, more concisely:


1	1
1	0


3	2
2	1

If we do this repeatedly, you might notice a familiar pattern start to emerge:


1	1
1	0


5	3
3	2


1	1
1	0


8	5
5	3


1	1
1	0


13	8
8	5

Which makes sense, doesn’t it? Because if we multiply this matrix with the matrix [1 0] – our starting values – then it’s going to advance forward through six steps of the Fibonacci sequence in a single leap. So naturally we have to be encoding something about the sequence itself in the matrix – otherwise we wouldn’t be able to advance by N steps in constant time.

Now, the insight that takes this from linear to logarithmic is that we don’t have to do this multiplication one step at a time. We can multiply in leaps and bounds.

Let’s call our original starting matrix F, for Fibonacci.


1	1
1	0

We’ve already calculated F²:

F²


2	1
1	1

And now it’s only one more matrix multiplication to calculate F⁴:

F⁴

F²F²


2	1
1	1


2	1
1	1

We can use this fact to calculate arbitrary matrix powers, by breaking the problem up into products of powers of two:

F²¹ = F¹⁶F⁴F¹

And by doing that, we can calculate the nth Fibonacci number in only log₂(n) steps.

Okay, so that’s fun and all, but that’s not really what this blog post is about.

I don’t know about you, but if I came across this matrix in the wild, I would not think “Oh, that’s the Fibonacci sequence”:


1	1
1	0

I would probably think “huh, I dunno, it’s like, a reflection, sort of, or maybe a shear; what’s a shear again, hang on, I need to see a picture.”

That is, I am used to thinking of matrices as transformations of points in space – scales and rotations and things like that. I’m not really used to thinking of matrices as “state machines.”

But this duality is the beauty of linear algebra! Matrices are transformations of points in space and graphs and state machines all at the same time.

So let’s take a look at the Fibonacci transformation, applied to arbitrary points in R²:

That animation is progressively applying and removing the transformation, so we can get some intuition for how it deforms a square. But we’re really more interested in repeated applications of the transformation. So let’s start with the same points, but multiply by that same matrix over and over:

Interesting. Over time, they have a tendency to stretch out along the long diagonals of this rhombus. Let’s zoom out:

Every time a point reflects over that diagonal, it reflects at a slightly different angle, slowly converging towards this straight line.

You might already have an idea of what that straight line means. You might know that, if you look at the ratio between subsequent Fibonacci numbers, they approximate the golden ratio:¹

1 / 1 = 1
2 / 1 = 2
3 / 2 = 1.5
5 / 3 = 1.666...
8 / 5 = 1.6
13 / 8 = 1.625
21 / 13 = 1.61538462
34 / 21 = 1.61904762

The golden ratio is irrational, but every subsequent Fibonacci number is a better and better rational approximation. (The golden ratio is around 1.618033988749 – so we’re already pretty close.)

It’s interesting to see that these estimations don’t “sneak up” on the golden ratio. In fact they alternate between over- and under-estimating it. Which is exactly what we saw in our visualization!

If you return to the “state machine” interpretation of our matrix, remember that the value we’re plotting as x is really “the current Fibonacci number,” and the value we’re plotting as y is “the previous Fibonacci number.” So the ratio between successive numbers – x/y – is just the slope of the lines that our points are traveling along. And we could see points reflecting over that diagonal, over- and under-shooting it, slowly converging… towards the line whose slope is the golden ratio.

Which is, in fact, the “long diagonal” of our rhombus.

And this makes sense, I think – this isn’t some weird coincidence. The golden ratio is all about the ratio between parts and wholes being the same as ratio between parts. And the Fibonacci sequence is all about adding together parts to become wholes that become parts in the next number of the sequence.

Here, our two parts are the “current” and “previous” values, and the whole that they make is the “next” Fibonacci number. Even if we start with two numbers that are completely unrelated to the Fibonacci sequence – say, 8 and 41 – the simple way that we pick the next number will cause us to approximate the golden ratio after only a few iterations:

8 / 41 = 0.1951219
(8 + 41 = 49) / 8 = 6.125
(49 + 8 = 57) / 49 = 1.16326531
(57 + 49 = 106) / 57 = 1.85964912
(106 + 57 = 163) / 106 = 1.53773585

Why is that? Well, because of the definition of the golden ratio.

(A + B) / A = A / B = φ

This is extremely unrigorous, but I can try to sketch out a very informal argument for why this is:

Let’s say the ratio between A and B is some unknown quantity S. It’s not the golden ratio, it might not be anywhere near the golden ratio; we have no idea what it is. In my 8 and 41 example, it wasn’t even in the right ballpark.

A/B = S

(A + B) / A = (1 + B / A) = 1 + (1/S)

So the ratio between the next element in our series and A will be (1 + (1/S)).

We still don’t know what S is! But if we do this again…

A' / B' = 1 + (1/S)

(A' + B') / A' =

(1 + (B' / A')) =

1 + (1 / (1 + (1 / S)))

After each iteration, the original S will become a smaller and smaller component in the final answer, until eventually we’ll just have an expression that looks like this:

1 + (1 / (1 + (1 / (1 + (1 / (1 + (1 / ...)))))))

Whatever our original S was, its contribution to the final result will eventually be negligible. Even after just a few iterations, we can see that the choice of S doesn’t make a huge difference in the outcome:

1 + (1 / (1 + (1 / (1 + (1 / (1 + (1 / -5000))))))) = 1.6667

1 + (1 / (1 + (1 / (1 + (1 / (1 + (1 / 0.001))))))) = 1.5002

And of course even that will fade away after a few more steps.

In fact the version of that expression with an infinite number of steps – where there is no S at all, but just an infinite sequence of divisions – is the “continued fraction” expression of the golden ratio.

Except, well, I’m lying here.

That residue will not fade away for all values of S. First of all, if S is zero, it doesn’t matter how small that term gets – you’re not going to squeeze a number out of it.

But there is another, more interesting value of S that breaks this rule. There is one other number that will not tend towards 1.618 when you repeatedly take its reciprocal and add one. It is the number that is already one plus its own reciprocal:

1 + (1 / 1.61803399) = 1.61803399

Oh, gosh, yes, the golden ratio is one plus its own reciprocal. But I was talking about the other number with that property:

1 + (1 / -0.61803399) = -0.61803399

This number is (1 - φ), and it is also -φ^-1. The golden ratio is weird like that.

That number is a weird number, because if we have two numbers with that ratio – say, -1.236 and 2 – and we applied our transformation, those points would not spread their wings towards the diagonal. What would they do instead?

Aha. Well, that makes sense.

Some points tend towards the top right, some points tend towards the bottom left, but some points get stuck. Sucked into the origin, cursed to forever travel along this one straight line.

Points along the long diagonal also travel in a straight line – they don’t bounce over the diagonal, because they’re already on it. Let’s just focus on these perfectly straight lines:

Not all matrices will produce straight lines like this when you apply them repeatedly. A rotation matrix, for example, will always change the direction of every single line each time you multiply a point by it.

These straight lines are called eigenvectors, which is German² for something like “intrinsic vector” or “characteristic vector."³

Well, to be more precise, any particular point on those straight lines is an “eigenvector.” The vector [φ 1] is an eigenvector, and so is [-2.1φ -2.1]. And the vector [-1/φ 1] is an eigenvector, and so is [-2/φ 2].

But all of the eigenvectors on each line are “similar,” so I’m just going to pick [φ 1] and [(1-φ) 1] as our two representative eigenvectors.

When you multiply an eigenvector of a matrix by the matrix itself, you get back a new eigenvector on “the same line.” That is to say, you get back another eigenvector that is just some scalar multiple of the original eigenvector.

For example, when we multiply our first eigenvector by the Fibonacci matrix:


1	1
1	0


φ
1


φ + 1
φ

Well… it’s not obvious that this is the case, but we actually just scaled the vector by φ. Because φ² = φ + 1. The golden ratio is weird.

Similarly:


1	1
1	0


1 - φ
1


2 - φ
1 - φ

We scaled it by (1 - φ), again somewhat cryptically:

(1 - φ)(1 - φ) =

(1 - 2φ + φ²) =

(1 - 2φ + φ + 1) =

(2 - φ)

So when we multiply our Fibonacci matrix with its eigenvectors, we scale those numbers by φ and (1 - φ). These scaling factors are called “eigenvalues,” and it’s weird that they look so much like the eigenvectors. That’s… that’s a weird Fibonacci coincidence, a weird golden ratio thing, and not a general pattern that holds for eigenvectors and eigenvalues in general.

Okay, so why do we care about this?

Well, once we know the eigenvectors and eigenvalues of the matrix, we can actually perform repeated matrix multiplication in constant time.

…Sort of. You have to imagine a big asterisk after that sentence, which I will explain below.

To explain how, we’re going to need to do a little bit of linear algebra. But first, I just want to restate everything I’ve said so far in explicit notation:

Multiplying F with each eigenvector is the same as multiplying that eigenvector by its corresponding eigenvalue. So:


1	1
1	0


φ
1


φ
1

And:


1	1
1	0


1 - φ
1

(1 - φ)


1 - φ
1

Right. But there’s actually a way to write those two equalities as a single equality:


1	1
1	0


φ	1 - φ
1	1


φ	1 - φ
1	1


φ	0
0	1 - φ

Instead of writing out each eigenvector as a separate column vector, I stuck them into a matrix. And instead of scaling each one by a scalar, I multiplied that matrix by a diagonal matrix.

This is the same statement, though: right-multiplication by a diagonal matrix just means “scale the columns of the left matrix by the corresponding diagonal value.” We can gut check this by performing the multiplication, and seeing that we’re making the exact same statements as before:


1	1
1	0


φ	1 - φ
1	1


φ + 1	2 - φ
φ	1 - φ


φ	1 - φ
1	1


φ	0
0	1 - φ


φ + 1	2 - φ
φ	1 - φ

But now we’re making these statement about both eigenvectors in parallel.

This equality – this statement about how multiplication by the Fibonacci matrix scales eigenvectors – is the secret to computing Fibonacci numbers in “constant time”:


1	1
1	0


φ	1 - φ
1	1


φ	1 - φ
1	1


φ	0
0	1 - φ

The trick here is that we’re going to right-multiply both sides of the equation by the inverse of our eigenvector matrix. This will eliminate it from the left-hand side entirely:


1	1
1	0


φ	1 - φ
1	1


φ	0
0	1 - φ


φ	1 - φ
1	1

-1

And now we have a new way to calculate the “next Fibonacci number.” Previously we knew how to do it by multiplying with the matrix F. Now we can do it by multiplying with, uhh, this inverse eigenvector matrix thing, and then the diagonal matrix of eigenvalues, and then the non-inverse matrix-of-eigenvectors.

Much simpler, right?

This is getting really long and complicated and I’m going to run out of space soon, so let’s give these things names:


1	1
1	0


φ	1 - φ
1	1


φ	0
0	1 - φ

That’s an upper-case lambda, and look, it’s just the convention for the eigenvalue matrix. Eigenvalues are called λ, and when you put them in a diagonal matrix you call it Λ. I don’t make the rules here.

Now that we have some abbreviations, we can write that as the much more palatable:

F = Q Λ Q^-1

Now, the whole reason that we’re doing this is to take advantage of another trick of associativity:

F² = (Q Λ Q^-1)(Q Λ Q^-1)

F² = Q Λ (Q^-1Q) Λ Q^-1

F² = Q Λ Λ Q^-1

F² = Q Λ² Q^-1

That was very abstract, so take a second to think about what this means. F² is the matrix that calculates two steps of our Fibonacci state machine. And we can use this same trick to calculate any power of F, just by calculating powers of Λ.

Fⁿ = Q Λⁿ Q^-1

And this is good, because Λ is a diagonal matrix. And it’s really easy to exponentiate a diagonal matrix! You just exponentiate each element of its diagonal. We don’t even need to use repeated squaring.

This means that we can actually calculate arbitrary powers of F in constant time… if we pretend that exponentiation of a scalar is a constant time operation.

It’s not, though. I mean, yes, exponentiation of an IEEE 754 64-bit floating-point number is constant time, but that’s not what we said. We’re talking about exponentiating an irrational number, and my computer can only represent approximations of that number, and that floating-point error adds up fast. So in order to actually use this to compute large Fibonacci numbers, we would need to use arbitrary-precision floating point, and exponentiating arbitrary precision values is not constant time. It’s… I don’t know, probably logarithmic? But like both to the exponent and the size of the result, and the size of the result is increasing exponentially, so it nets out to linear? I don’t actually know.⁴

But I don’t want to spoil the fun. This is still a very interesting trick, and it’s worth understanding how it works, even if it doesn’t actually give us a way to compute arbitrarily large Fibonacci numbers in constant time.

So: what are we doing.

We moved a bunch of symbols around, and we wound up with this expression:

Fⁿ = Q Λⁿ Q^-1

But I don’t really know what Q^-1 means, and it’s not really clear to me why I should care. Why is multiplying by these three weird matrices the same as multiplying by F? What, intuitively, are we doing here?

At a high level, we’re translating points into a different coordinate system, then doing something to it, and then translating them back into our original coordinate system.

You already know that we can write any point in space as a vector – X and Y coordinates. That’s what we’ve been doing this whole time.

But we can also write a point in space as the sum of two other vectors. Like, [5 3]. We could write that as [1 2] + [4 1] instead. Which, okay, sure. That’s not very interesting.

One “interesting” way to write [5 3] is as the sum of these two vectors: [5 0] + [0 3]. Or, to say that another way:


1
0


0
1


5
3

This is interesting because [1 0] and [0 1] are basically the “X axis” and “Y axis.” And we can think of the point [5 3] as a (trivial!) linear combination of these two axes.

But we could pick different axes. We can pick any vectors we want as our axes,⁵ so let’s pretend for a moment that our axes are [1 1] and [1 -1] instead. Which means that we would write [5 3] as:


1
1


1
-1


5
3

Or, to write that another way:


1	1
1	-1


4
1


5
3

Now we can think of this vector-of-coefficients, [4 1], as another way to identify the point in space x=5 y=3 when we we’re pretending that our axes are [1 1] and [1 -1]. Except in linear algebra we’d call these “basis vectors” instead of “axes.”

But how did we find the coefficients [4 1]? Well, I just found that one by hand; it was pretty easy. But in general, if we want to express some other point using these basis vectors – let’s say [63 -40] – we’ll need to solve an equation that looks like this:


1	1
1	-1


?
?


63
-40

And we can do that by, you know, regular algebra. We “divide” both sides by our matrix-of-basis-vectors, by left-multiplying with the inverse matrix:


1	1
1	-1

-1


1	1
1	-1


?
?


1	1
1	-1

-1


63
-40

And after the inverses cancel, we’re left with the following formula:


?
?


1	1
1	-1

-1


63
-40

And the problem reduces to matrix inversion.

Now, I don’t know about you, but I don’t remember how to invert a matrix. I know there’s a formula in two dimensions, but the only thing I remember about it is that it involves calculating the determinant, and I forgot how to do that too. So let’s just ask a computer to invert it for us:


?
?


0.5	0.5
0.5	-0.5


63
-40

Hmm. I feel like I probably could’ve worked that out myself.

But that lets us solve the equation, and figure out how to write the point [63 -40] as a combination of the vectors [1 1] and [1 -1]:


0.5	0.5
0.5	-0.5


63
-40


51.5
11.5

Great! We did it.

And here’s why we care:

We can use this exact same trick to write down the points in our Fibonacci sequence as a linear combination of our two eigenvectors. Like this:

Click or tap to add points there, to see how we can write each point in space as a combination of the “short diagonal” and “long diagonal” eigenvectors of our matrix.

Normally to identify a point in space we would give its XY coordinates: go this far along the X-axis, then this far along the Y-axis. But here we’re representing points in “φ” and “1 - φ” coordinates: go this far along the short diagonal, then this far along the long diagonal.

But how do we know how far to go along these diagonals? Well, we “divide by” the eigenvectors. In other words, we have to compute the inverse of this matrix:


φ	1 - φ
1	1

-1

√5


1	φ - 1
-1	φ

Now, matrix inversion is boring, so I’m just presenting the answer here. This inverse matrix is how we can convert from “XY coordinates” into “eigenvector coordinates.”

Let’s work through a concrete example to make sure this works.

[8 5] is a point on the Fibonacci sequence. We can express that as a combination of eigenvectors instead:

√5


1	φ - 1
-1	φ


8
5

≈


4.96
0.04

4.96 and 0.04 are the coefficients we will pair with our eigenvectors: we have to travel 4.96 units down the long diagonal, and 0.04 units along the short diagonal to arrive at the point [8 5].

4.96


Φ
1

0.04


1 - Φ
1

≈


8.025
4.96


-0.025
0.04


8
5

Great. It worked!

But that wasn’t very interesting – we just converted our point into the eigenvector basis and then right back into the normal XY basis. It was kind of a pointless transformation.

But we don’t have to do the unconversion immediately. We can keep the point in this “eigenbasis” for a little while, and do stuff to the vector-of-coefficients, and then convert it back.

Specifically, we can scale the coefficients by the eigenvalues of our Fibonacci matrix. We can multiply the “long diagonal” component by Φ², and multiply the short diagonal component by (1 - Φ)², and we’ll have a new point: something close to [12.985 0.015]. And if we convert that back into XY coordinates:

12.985


Φ
1

0.015


1 - Φ
1


21
13

We just advanced our point two more steps along the Fibonacci sequence, with nothing more than scalar exponentiation and a constant number of vector operations.

This is exactly the same as the expression:

F² = Q Λ² Q^-1

But as someone with no background in linear algebra, I find it easy to get lost in the notation, so it’s easier for me to think about this as operations on separate column vectors rather than as operations on matrices. Even though they are the same thing.

Of course, calculating two steps of the Fibonacci sequence in constant time isn’t that impressive. But we can do the same with Φ¹⁰⁰⁰, and use that to calculate the thousandth Fibonacci number in constant time.

…Assuming we could calculate Φ¹⁰⁰⁰ in constant time. Which we can’t, in real life.

Alright.

The post is over; you saw the trick. “Eigendecomposition,” this is called.

I glossed over a few steps – I spent absolutely no time explaining how I knew the eigenvalues and eigenvectors of this matrix, for example. I just asserted that they were related to the golden ratio. But in reality you can solve for them, or ask a computer to do it for you. It’s pretty mechanical, like matrix inversion – it seems linear algebra is best explored with a repl nearby.

In any case, I think that the why of eigendecomposition is more interesting than the how.

As for the Fibonacci sequence… well, this is a pretty terrible way to actually calculate Fibonacci numbers. Even if we pretend that we only care about numbers that can fit in IEEE 754 double-precision floats, we still can’t use this technique to calculate very many Fibonacci numbers, because the floating-point error adds up too quickly.

But if we only care about double-precision floats… well, there is one more Fibonacci implementation to consider. It’s an algorithm that that runs in constant time, and constant space, and covers the full gamut of floating-point numbers without accumulating any error at all…

const fibs = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887, 9227465, 14930352, 24157817, 39088169, 63245986, 102334155, 165580141, 267914296, 433494437, 701408733, 1134903170, 1836311903, 2971215073, 4807526976, 7778742049, 12586269025, 20365011074, 32951280099, 53316291173, 86267571272, 139583862445, 225851433717, 365435296162, 591286729879, 956722026041, 1548008755920, 2504730781961, 4052739537881, 6557470319842, 10610209857723, 17167680177565, 27777890035288, 44945570212853, 72723460248141, 117669030460994, 190392490709135, 308061521170129, 498454011879264, 806515533049393, 1304969544928657, 2111485077978050, 3416454622906707, 5527939700884757, 8944394323791464];
const fib = (n) => fibs[n];

But it’s more fun to overthink it.

In case you are one of today’s lucky 10,000: the golden ratio is also very close to the conversion rate between miles and kilometers, so you can use Fibonacci numbers to approximate conversions between miles and kilometers in your head. For example, 80km ≈ 50mi. This is a weirdly good conversion – the exact answer is 49.7097mi.

It even works when you don’t have a round Fibonacci number to work with. 120kph is probably around 90% of 130kph, and 90% of 80mph is 72mph… the correct answer would be 74.6mph, but we got a decent ballpark with nothing but eyeball math. ↩︎
Er, the German word is Eigenvektor, so it’s like… half a loan word? A loan sub-word? Whatever, the “eigen” part is the relevant bit here. ↩︎
Straight lines are eigenvectors, but eigenvectors are not necessarily straight lines. Rotation matrices also have eigenvectors, but they have complex eigenvectors. Straight lines like this are real eigenvectors. ↩︎
The exact same argument applies to the “logarithmic” exponentiation-by-squaring algorithm as well – squaring arbitrarily large numbers requires arbitrary precision multiplication. It feels different to me, though, because of floating point error: when you’re exponentiating eigenvalues, you need to use arbitrary precision arithmetic even when your final answer could fit into a double. But the integer squaring approach only needs bigints when the Fibonacci numbers themselves become too large to fit into words. ↩︎
As long as the vectors we pick are “linearly independent” – there’s no way to express [5 3] as a combination of [1 0] and [2 0], for example. ↩︎

My Kind of REPL

Wed, 05 Jul 2023 00:00:00 +0000

I want to tell you about an idea that has had a huge influence on the way that I write software. And I mean that in the literal sense: it’s changed the way that I write software; it’s re-shaped my development workflow.

The idea is this: you can write programs that modify themselves.

And I don’t mean macros or metaprogramming or anything fancy like that. I mean that you can write programs that edit their own source code. Like, the files themselves. The actual text files on disk that have your source code in them.

That’s not the whole idea, though. There’s more to it: you write programs that can edit themselves, and then you use that as your REPL.

Instead of typing something into a prompt and hitting enter and seeing the output on stdout, you type something into a file and hit some editor keybinding and the result gets inserted into the file itself. Patched on disk, right next to the original expression, ready to be committed to source control.

This read-eval-patch loop is already a little useful – it’s a REPL with all of the conveniences of your favorite editor – but we’re still not done. There’s one more part to the idea.

Once you have your expression and your “REPL output” saved into the same file, you can repeat that “REPL session” in the future, and you can see if the output ever changes.

And then you use this technique to write all of your tests.

That’s the whole idea. You use your source files as persistent REPLs, and those REPL sessions become living, breathing test cases for the code you were inspecting. They let you know if anything changes about any of the expressions that you’ve run in the past. This kind of REPL can’t tell you if the new results are right or wrong, of course – just that they’re different. It’s still up to you to separate “test failures” from expected divergences, and to curate the expressions-under-observation into a useful set.

This is pretty much the only way that I have written automated tests for the last six years. Except that it isn’t really – this is the way that I haven’t written automated tests. This is the way that I have caused automated tests to appear for free around me, by doing exactly what I have always done: writing code and then running it, and seeing if it did what I wanted.

I’m exaggerating, but not as much as you might think. Of course these tests aren’t free: it takes work to come up with interesting expressions, and I have to write setup code to actually test the thing I care about, and there is judgment and curation and naming-things involved. And of course there is some effort in verifying that the code under test actually does the thing that I wanted it to do.

But the reason that it feels free is that nearly all of that is work that I would be doing already.

Like, forget about automated tests for a second – when you write some non-trivial function, don’t you usually run it a couple times to see if it actually does the thing that you wanted it to do? I do. Usually by typing in interesting expressions at a REPL, and checking the output, and only moving on once I’m convinced that it works.

And those expressions that I run to convince myself that the code works are exactly the expressions that I want to run again in the future to convince myself that the code still works after a refactor. They are exactly the expressions that I want to share with someone else to convince them that the code works. They are, in other words, exactly the test cases that I care about.

And yes, I mean, I’m still exaggerating a bit. There is a difference between expressions that you run once and expressions that you run a thousand times. In particular, you want your “persistent REPL” outputs to be relatively stable – you don’t want tests to fail because the order of keys in a hash table changed. And usually you want to document these cases in some way, to say why a particular case is interesting, so that it’s easier to diagnose a failure in the future.

But we’re getting a little into the weeds there. Let’s take a step back.

You’ve heard the idea, now let’s see how it feels. I want to demonstrate the workflow to you, but it’s going to require a little bit of suspension of disbelief on your part: this is a blog post, and I don’t really have time to show you any particularly interesting tests. I also have to confess that I don’t write very many interesting tests outside of work, so this will be a combination of completely fake contrived examples and trivial unit tests lifted from random side projects.

With that in mind, let’s start with one from the “extremely contrived” camp:

Yes, I know that you haven’t actually implemented a sorting algorithm since the third grade, but I hope that it gives you the gist of the workflow: you write expressions, press a button, and the result appears directly in your source code. It’s like working in a Jupyter notebook, except that you don’t have to work in a Jupyter notebook. You don’t have to leave the comfort of Emacs, or Vim, or VSCode, or BBEdit – as long as your editor can load files off a file system, it’s compatible with this workflow.

In case you skipped the video, the end result looks something like this:

(use judge)
(defn slow-sort [list]
(case (length list)
0 list
1 list
2 (let [[x y] list] [(min x y) (max x y)])
(do
(def pivot (in list (math/floor (/ (length list) 2))))
(def bigs (filter |(> $ pivot) list))
(def smalls (filter |(< $ pivot) list))
(def equals (filter |(= $ pivot) list))
[;(slow-sort smalls) ;equals ;(slow-sort bigs)])))
(test (slow-sort [3 1 2]) [1 2 3])
(test (slow-sort []) [])
(test (slow-sort [1 1 1 1 1]) [1 1 1 1 1])

Those tests at the end might look like regular equality assertions – and, in a way, they are. But I only had to write the “first half” of them – I only wrote the expressions to run in my “REPL” – and my test framework filled in right-hand sides for me.

Now, as you probably noticed, there were a lot of parentheses in that demo. Don’t be alarmed: the parentheses are entirely incidental. I just happen to be spending a lot of time with Janet lately, and I’ve written a test framework that’s heavily based around this workflow.

But there’s nothing parentheses-specific about this technique. I first met this type of programming in OCaml, via the “expect test” framework. The specifics of OCaml’s expect tests are a little bit different, but the overall workflow is the same:

let%expect_test "formatting board coordinates" =
print_endline (to_string { row = 0; col = 0 });
[%expect "A1"];
print_endline (to_string { row = 10; col = 4 });
[%expect "E11"]
;;

Instead of the expression-oriented API we saw in Janet, an OCaml expect test is a series of statements that print to stdout, followed by automatically-updating assertions about what exactly was printed. This makes sense because OCaml doesn’t really have a canonical way to print a representation of any value,¹ so you have to decide exactly what you want your REPL to Print.

And this workflow works in normal languages too! Here’s an example in Rust, using the K9 test framework:²

#[test]
fn indentation() {
 k9::snapshot!(
 tokenize(
 "
a
 b
 c
d
 e
 f
g
"
 ),
 "a ␤ → b ␤ c ␤ ← d ␤ → e ␤ → f ␤ ← ← g ␤"
 );
}

We’re pulling from the “trivial side project” category now. That’s a test I wrote for an indentation-sensitive tokenizer, but again, I just wrote an expression that I wanted to check and then checked that the result looked right. I didn’t have to spend any time crafting the token stream that I expected to see ahead of time – I only had to write down an expression that seemed interesting to me.

The fact that you don’t actually have to write the expected value might seem like a trivial detail – it’s not that hard to write your expectations down up front; people do it all the time – but it’s actually an essential part of this workflow. It’s an implausibly big deal. It is, in fact, the first beautiful thing I want to highlight about this technique:

1. When it’s this easy to write tests, you write more of them

The fact that I don’t have to type the expectation for my tests matters a lot more than it seems like it should. Like, I’m an adult, right? I can walk through that code and say “a, newline, indent, b, newline…” and I can write down the expected result. How hard can it be?

I actually just timed myself doing that: it took almost exactly 30 seconds to read this input and write out the expected output by hand (sans Unicode). 30 seconds doesn’t sound very long, but it’s friction. It was a boring 30 seconds. It’s something that I want to minimize. And what if I have 10 of these tests? That 30 seconds turns into, like, I don’t know; math has never been my strong suit. But I bet it’s a lot.

And that was an easy example! Let’s look at something more complicated: here’s one of dozens of tests for the parser that that tokenizer feeds into:

#[test]
fn test_nested_blocks()
 k9::snapshot!(
 parse(
 "
foo =
 x =
 y = 10
 z = 20
 y + z
 x
"
 ),
 "foo (n) = (let ((x (let ((y 10) (z 20)) (+ y z)))) x)"
 );
}

Of course I could have typed out the exact syntax tree that I expected to see ahead of time, but I know that I won’t do that – I don’t even want to time myself trying it. If I actually had to type the input and output for every single test, I’d just write fewer of them.

And yes, you still have to verify that the output is correct. But verification is almost always easier than constructing the output by hand, and in the rare cases where it isn’t, you still have the option to write down your expectations explicitly.

And sometimes you don’t have to verify at all! Often when I’m very confident that my code already works correctly, but I’m about to make some risky changes to it, I’ll use this technique to generate a bunch of test cases, and just trust that the output is correct. Then after I’m done refactoring or optimizing, those tests will tell me if the new code ever diverges from my reference implementation.

But I don’t just write more tests because it’s easy. I usually write more tests because it makes my job easier. Which is the next beautiful thing that I want to highlight:

2. This workflow makes tests useful immediately

We all know that tests are valuable, but they’re not particularly valuable right now. The point of automated testing isn’t to make sure that your code works, it’s to make sure that your code still works in five years, right?

But you won’t be working on this same codebase in five years. And there’s no way that you would make a change that breaks one of these subtle invariants you’ve come to rely on. The time you spend writing automated tests might help someone, eventually, but that’s time that you could instead be spending on “features” that your “company” needs to “make payroll this month.” Which means that – if you’re already confident that your code is correct – it can be tempting to not spend any time hardening it against the cold ravages of time.

But when tests actually provide immediate value to you – when writing tests becomes a useful part of your workflow, when it becomes a part of how you add new features – suddenly the equation changes. You don’t write tests for some nebulous future benefit, you write tests because it’s the easiest way to run your code. The fact that this happens to make your codebase more robust to future changes is just icing.

But in order to really benefit from this kind of testing, you might have to rethink what a “good test” looks like. Which brings me to…

3. Good tests are good observations of your program’s behavior

There are good tests, and there are bad tests. This is true whether or not you’re writing this particular style of test, but the ceiling for what a “good test” can look like is much, much higher when you’re using this technique.

I’ll try to show you what I mean. Let’s look at a test for a function that finds the winner in a game of Hex:

let%expect_test "find winning path" =
let board = make_board ~size:5 "A4 A2 B4 B3 C3 C4 D2 D3" in
print_s [%sexp (find_winning_path board : Winning_path.t)];
[%expect "None"];
play_at board "E2";
print_s [%sexp (find_winning_path board : Winning_path.t)];
[%expect "(Black (A4 B4 C3 D2 E2))"];

Okay. Sure. I can sort of think for a minute and see that there was no winning path until black played at E2, and then there was one, and it’s probably reporting the correct path. But let’s compare that with an alternative:

let%expect_test "find winning path" =
let board = make_board ~size:5 "A4 A2 B4 B3 C3 C4 D2 D3" in
print_board board (find_winning_path board);
[%expect {|
There is no winning path:
1 2 3 4 5
A . w . b .
B . . w b .
C . . b w .
D . b w . .
E . . . . .
|}];
play_at board "E2";
print_board board (find_winning_path board);
[%expect {|
Black has a winning path:
1 2 3 4 5
A . w . * .
B . . w * .
C . . * w .
D . * w . .
E . * . . .
|}];

Isn’t that better? This test has convinced me in a single glance that the code is working correctly. And if, at some point in the future, the code stops working correctly – if I refactor the find_winning_path function and break something – I claim that this test will give me a decent head start on figuring out where I went wrong.

Of course it took some work to write this visualization in the first place. Not a lot of work – it’s just a nested for loop – but still, it did take longer to write this test than the (Black (A4 B4 C3 D2 E2)) version.

This is some of my favorite kind of work, though. It’s work that will speed up all of my future development, by making it easier for me to understand how my code works – and, even better, by making it easier for other people to understand how my code works. I only have to write the print_board helper once, but I can use it in dozens of tests – and so can my collaborators.

Here’s another test that took a little work to produce, but that I found extremely useful during development:

You can imagine this as taking each of the top shapes and tracing them around the perimeter of the bottom shapes, maintaining contact at all times. Sort of a tricky function to write correctly, and one where visualization really helps me see that I got it right – and, of course, to see what I did wrong when I inevitably break it.

But that’s kind of cheating: that visualization uses Emacs’s iimage-mode to render a literal image inline with my code, which is not something that you can do everywhere. And since part of the appeal of this technique is the universal accessibility, I should really stick to text.

Here, here’s a better example. This is a function that, given a source of uniformly distributed random numbers, gives you a source of normally distributed random numbers:

(defn marsaglia-sample [rng]
(var x 0)
(var y 0)
(var r2 0)
(while (or (= r2 0) (>= r2 1))
(set x (math/rng-uniform rng))
(set y (math/rng-uniform rng))
(set r2 (+ (* x x) (* y y))))
(def mag (math/sqrt (/ (* -2 (math/log r2)) r2)))
[(* x mag) (* y mag)])

Except… does it work? I just copied this from Wikipedia; I don’t really have very good intuition for the algorithm yet. Did I implement it correctly?

It’s not immediately obvious how I would write a test for this. But if I weren’t trying to “test” it – if I just wanted to convince myself that it worked – then I’d probably plot a few thousand points from the distribution and check if they look normally distributed. And if I had any doubts after that, maybe I’d calculate the mean and standard deviation as well and check that they’re close to the values I expect.

So that’s exactly what I want my automated tests to do:

(deftest "marsaglia sample produces a normal distribution"
(def rng (math/rng 1234))
(test-stdout (plot-histogram |(marsaglia-sample rng) 100000) `
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣷⣾⣦⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣷⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣿⣿⣷⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣶⣦⣄⡀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣶⣤⣤⣄
μ = 0.796406
σ = 0.601565
`))

And hark! That’s clearly wrong. And the test tells me exactly how it’s wrong: it’s only producing positive numbers (the histogram is centered around the mean). This helps me pinpoint my mistake, and to fix it:

(defn symmetric-random [rng x]
(- (* 2 x (math/rng-uniform rng)) x))
(defn marsaglia-sample [rng]
(var x 0)
(var y 0)
(var r2 0)
(while (or (= r2 0) (>= r2 1))
(set x (symmetric-random rng 1))
(set y (symmetric-random rng 1))
(set r2 (+ (* x x) (* y y))))
(def mag (math/sqrt (/ (* -2 (math/log r2)) r2)))
[(* x mag) (* y mag)])
(deftest "marsaglia sample produces a normal distribution"
(def rng (math/rng 1234))
(test-stdout (plot-histogram |(marsaglia-sample rng) 100000) `
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣦⣾⣿⣦⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣴⣿⣿⣿⣿⣿⣿⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⢀⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣶⡀⠀⠀⠀⠀⠀⠀
⢀⣀⣀⣤⣶⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣶⣤⣀⣀⡀
μ = 0.001965
σ = 0.998886
`))

And, like, yes: writing this test required me to write a Unicode histogram plotter thingy, like a crazy person. But I’ve already written a Unicode histogram plotter thingy, like a crazy person, so I was able to re-use it here – just as I will be able to re-use it again in the future, the next time I feel like it will be useful.

In fact I’ve accumulated a lot of little visualization helpers like this over the years – helpers that allow me to better observe the behavior of my code. These helpers are useful when I’m writing automated tests, because they help me understand when the behavior of my code changes, but they’re also useful for regular ad-hoc development. It’s useful to be able to whip up a histogram, or a nicely formatted table,³ or a graph of a function, when I’m doing regular old printf debugging.

Which reminds me…

4. You can use printf debugging right in your tests

I’ve shown you two types of these “REPL tests” so far.

There’s a simple expression-oriented API:

(test (+ 1 2) 3)

And there’s the imperative stdout-based API:

let%expect_test "board visualization" =
print_board (make_board ~size:5 "A4 A2 B4 B3 C3 C4 D2 D3");
[%expect {|
1 2 3 4 5
A . w . b .
B . . w b .
C . . b w .
D . b w . .
E . . . . .
|}];

When I first starting writing tests like this, the stdout-based API seemed a little weird to me. Most of my tests just printed simple little expressions, and the fact that I had to spread that out across multiple lines was annoying. And if I did want to create a more interesting visualization, like the one above, couldn’t I just write an expression to return a formatted string?

Yes, but: print is quick, easy, and familiar. It’s nice that rendering that board was just a nested for loop where I printed out one character at a time. Of course I could have allocated some kind of string builder, and appended characters to that, but that’s a little bit more friction than just calling print. I’d rather let the test framework do it for me automatically: the easier it is to write visualizations like this, the more likely I am to do it.

But when I actually spent time writing tests using this API, I came to really appreciate a subtle point that I had initially overlooked: stdout is dynamically scoped.

Which means that if I call print in a helper of a helper of a helper, it’s going to appear in the output of my test. I don’t have to thread a string builder through the call stack; I can just printf, and get printf-debugging right in my tests.

For example: our function to make normally distributed random numbers relies on “rejection sampling” – we have to generate uniformly distributed random numbers until we find a pair that lies inside the unit circle.

I’m curious how often we actually have to re-roll the dice. It’s not an observable property of our function, but we can just print it out in the implementation:

(defn marsaglia-sample [rng]
(var x 0)
(var y 0)
(var r2 0)
(var iterations 0)
(while (or (= r2 0) (>= r2 1))
(set x (symmetric-random rng 1))
(set y (symmetric-random rng 1))
(set r2 (+ (* x x) (* y y)))
(++ iterations))
(def rejections (- iterations 1))
(when (> rejections 0)
(printf "rejected %d values" rejections))
(def mag (math/sqrt (/ (* -2 (math/log r2)) r2)))
[(* x mag) (* y mag)])

And then re-run our test, exactly like we did before:

(deftest "marsaglia sample produces a normal distribution"
(def rng (math/rng 1234))
(test-stdout (plot-histogram |(marsaglia-sample rng) 100) `
rejected 1 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
rejected 2 values
rejected 1 values
rejected 1 values
rejected 1 values
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⠀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⡀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⡇⠀⡇⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡄⢠⠀⢸⣤⡄⡇⢠⣧⡇⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⢠⡇⢸⡄⣼⣿⡇⡇⢸⣿⡇⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡆⡆⠀⢸⢸⡇⢸⡇⣿⣿⡇⡇⣾⣿⡇⡇⠀⠀⠀⢰⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⡆⠀⠀⠀⡇⣷⣶⢸⣾⣷⣾⣷⣿⣿⣷⣷⣿⣿⣷⡇⢰⡆⠀⣾⡇⡆⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⢰⠀⠀⡇⠀⡆⡆⣷⣿⣿⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⢸⣷⡆⣿⡇⣷⢰⡆⠀⠀⠀⠀⠀⠀
⠀⠀⠀⢸⢸⡇⡇⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡇⣿⣿⣿⣿⡇⣿⠀⠀⠀⠀⠀
μ = -0.054361
σ = 1.088199
`))

Alright, so we rejected 33 points, which means that we generated 133 – a rejection rate of around 25%. The area of the unit circle is π, and the area of its bounding square is 4, so I’d expect us to reject around (1 - π/4) ≈ 21.5% of points. Pretty close!

Now, this is the sort of thing that I wouldn’t keep around in an automated test. This was exploratory; this was a gut check. That output is not in a very useful format, and I don’t think it’s really worth ensuring that this rate remains stable over time – I can’t imagine that I’ll screw that up without breaking the function entirely. But the important takeaway is that it was really, really easy to sprinkle printfs into code, and to see their output right next to the code itself, using the exact same workflow that I’m already using to test my code.

And while this output was temporary, I actually do wind up committing many of the exploratory expressions that I run as I’m developing. Because…

5. Good tests make good documentation

Compare:

sep-when is a function that takes a list and a predicate, and breaks the input list into a list of sub-lists every time the predicate returns true for an element.

With:

(test (sep-when [1 2 3 5 3 6 7 8 10] even?)
[[1] [2 3 5 3] [6 7] [8] [10]])

My brain can intuitively understand the behavior of this function from the example code more easily than it can parse and understand the (ambiguous!) English description. The description is still helpful, sure, but the example conveys more information more quickly. To me, anyway.

I think that this is mostly because my brain is very comfortable reading REPL sessions, which might not be true for everyone. And I’m not saying that good English-language documentation isn’t important – ideally, of course, you have both. But there are two important advantages to the second version:

Tests don’t lie; they can’t become stale.
I didn’t have to write it.

I spent a couple minutes trying to come up with a pithy English description of this function, and I don’t even think I did a very good job. Writing good documentation is hard, but generating examples is easy – especially when my test framework is doing half the work.

And since it’s easy to generate examples, I do it in cases where I wouldn’t have bothered to write documentation in the first place. This sep-when example was a real actual helper function that I pulled from a random side project. A private, trivial helper, not part of any public API, that I just happened to write as a standalone function – in other words, not the sort of function that I would ever think to document.

It’s also a trivial function, far from the actual domain of the problem I was working on, so it’s not the sort of thing I would usually bother to test either. And I might regret that – bugs in “trivial” helpers can turn into bugs in actual domain logic that can be difficult to track down, and I probably only have a 50/50 shot of getting a trivial function right on the first try in a dynamically-typed language.

But when the ergonomics of writing tests is this nice, I actually do bother to test trivial helpers like this. Testing just means writing down an expression to try in my “REPL,” and I can do that right next to the implementation of the function itself:

(defn sep-when [list f]
(var current-chunk nil)
(def chunks @[])
(eachp [i el] list
(when (or (= i 0) (f el))
(set current-chunk @[])
(array/push chunks current-chunk))
(array/push current-chunk el))
chunks)
(test (sep-when [1 2 3 5 3 6 7 8 10] even?)
[[1] [2 3 5 3] [6 7] [8] [10]])

Which means that, not only will I catch dumb mistakes early, but I also automatically generate examples for myself as I code. If I come back to this code a year from now and can’t remember what this function was supposed to do, I only have to glance at the test to remind myself.

And on, and on…

I could keep going, but I think that my returns are starting to diminish here. If you still aren’t convinced, James Somers wrote an excellent article about this workflow that I would recommend if you want to see more examples of what good tests can look like in a production setting.

But I think that it’s difficult to convey just how good this workflow actually is in any piece of writing. I think that it’s something you have to try for yourself.

And you probably can! This technique is usually called “inline snapshot testing,” or sometimes “golden master testing,” and if you google those phrases you will probably find a library that lets you try it out in your language of choice. I have personally only used this technique in OCaml, Janet, and Rust, but in the spirit of inclusivity, I also managed to get this working working in JavaScript, in VSCode, using a test framework called Jest:

I will be the first to admit that that workflow is not ideal; Jest is absolutely not designed for this “read-eval-patch-loop” style of development. Inline snapshots are a bolted-on extra feature on top of a traditional “fluent” assertion-based testing API, and it might take some work to twist Jest into something useful if you actually wanted to adopt this workflow in JavaScript. But it’s possible! The hard part is done at least; the source rewriter is written. The rest is just ergonomics.

There is a cheat code, though, if you cannot find a test framework that works in your language: Cram.

Cram is basically “inline snapshot tests for bash,” but since you can write a bash command to run a program written in any other language, you can use Cram as a language-agnostic snapshot testing tool. It isn’t as nice as a native testing framework, but it’s still far, far nicer than writing assertion-based tests. And it works anywhere – I use Cram to test the test framework that I wrote in Janet, for example, since I can’t exactly use it to test itself.

Okay. I really am almost done, but I feel that I should say one last thing about this type of testing, in case you are still hesitant: this is a mechanic. This is not a philosophy or a dogma. This is just an ergonomic way to write tests – any tests.

You can still argue as much as you want about unit tests vs. integration tests and mocks vs. stubs. You can test for narrow, individually significant properties, or just broadly observe entire data structures. You can still debate the merits of code coverage, and you can even combine this technique with quickcheck-style automatic property tests. Here, look:

module Vec2 = struct
type t =
{ x : float
; y : float
}
[@@deriving quickcheck, sexp_of]
let length { x; y } =
sqrt (x *. x +. y *. y)
;;
let normalize point =
let length = length point in
{ x = point.x /. length
; y = point.y /. length
}
;;
end

Pretty simple code, but does it work for all inputs? Well, let’s try a property test:

let assert_nearly_equal here actual expected ~epsilon =
[%test_pred: float]
~here:[here]
(fun actual -> Float.( < ) (Float.abs (actual -. expected)) epsilon)
actual
let%expect_test "normalize returns a vector of length 1" =
Quickcheck.test [%quickcheck.generator: Vec2.t]
~sexp_of:[%sexp_of: Vec2.t]
~f:(fun point ->
assert_nearly_equal [%here] (Vec2.length (Vec2.normalize point)) 1.0 ~epsilon:0.0001)
[@@expect.uncaught_exn {|
(* CR expect_test_collector: This test expectation appears to contain a backtrace.
 This is strongly discouraged as backtraces are fragile.
 Please change this test to not include a backtrace. *)
("Base_quickcheck.Test.run: test failed"
(input ((x -3.3810849992682576E+272) (y 440.10383674821787)))
(error
((runtime-lib/runtime.ml.E "predicate failed"
((Value 0) (Loc test/demo.ml:20:15) (Stack (test/demo.ml:29:26))))
"Raised at Ppx_assert_lib__Runtime.test_pred in file \"runtime-lib/runtime.ml\", line 58, characters 4-58\
 \nCalled from Base__Or_error.try_with in file \"src/or_error.ml\", line 84, characters 9-15\
 \n")))
Raised at Base__Error.raise in file "src/error.ml" (inlined), line 9, characters 14-30
Called from Base__Or_error.ok_exn in file "src/or_error.ml", line 92, characters 17-32
Called from Expect_test_collector.Make.Instance_io.exec in file "collector/expect_test_collector.ml", line 262, characters 12-19 |}]
;;

This is not, you know, a good property test, but just focus on the workflow – even though it’s just an assertion failure, it’s nice to see that error in context. I don’t need to manually correlate line numbers with my source files, even if I’m working in a text editor that can’t jump to errors automatically.

So the mechanic of source patching is still somewhat useful regardless of what sort of tests you’re writing. This post has focused on pretty simple unit tests because, you know, it’s a blog post, and we just don’t have time to page in anything complicated together. And I’ve especially emphasized testing-as-observation because that happens to be my favorite way to write most tests, and it’s a unique superpower that you can’t really replicate in a traditional test framework.

But that doesn’t mean that read-eval-patch loops are only useful in these cases, or that you can only adopt this technique with a radical change to the way you think about testing.

Although it might not hurt…

The convention where I work is to render all data structures as s-expressions, but I didn’t want to upset you any further. ↩︎
Rust also has expect_test, which I have not tried, and Insta, but I had trouble getting it to work when I tried it a few years ago. ↩︎
I couldn’t find room for this in the post – it isn’t very flashy – but in real life I think that tables are the most useful tool in my testing lunchbox. You can see a few examples here of tests with tables in them, although the tables are not really the stars there. ↩︎

Generalized Macros

Tue, 18 Apr 2023 00:00:00 +0000

I’ve been writing a lot of Janet lately, and I’ve been especially enjoying my time with the macro system.

Janet macros are Common Lisp-flavored unhygienic gensym-style macros. They are extremely powerful, and very easy to write, but they can be pretty tricky to get right. It’s easy to make mistakes that lead to unwanted variable capture, or to write macros that only work if they’re expanded in particular contexts, and it can be pretty difficult to detect these problems ahead of time.

So people have spent a lot of time thinking about ways to write macros more safely – sometimes at the cost of expressiveness or simplicity – and almost all recent languages use some sort of hygienic macro system that defaults to doing the right thing.

But as far as I know, no one has approached macro systems from the other direction. No one looked at Common Lisp’s macros and said “What if these macros aren’t dangerous enough? What if we could make them even harder to write correctly, in order to marginally increase their power and expressiveness?”

So welcome to my blog post.

I want to show you an idea for a new kind of macro. A macro that can not only rewrite itself, but actually rewrite any form in your entire program.

I think that “what, no, why on earth” is an entirely reasonable response to that statement, so let’s take a look at a motivating example together.

Lots of languages have something like defer that will run an expression at the end of a block, whether or not the rest of the code raises an exception. defer is just like wrapping the rest of the function in a try-finally block, except that, well, you don’t actually have to do any wrapping. Which means no indentation increase, and no extra nested parentheses.

Here’s an example of what defer might look in Janet:

(do
(def f (file/open "foo.txt"))
(defer (file/close f))
(def contents (file/read f))
(do-something-dangerous-with contents))

Now, we can’t implement defer as a traditional macro, because a traditional macro can only rewrite itself. But what we really want to do is rewrite the parent form that the defer appears in, to wrap its siblings in a finally expression:

(do
(def f (file/open "foo.txt"))
(finally
(do
(def contents (file/read f))
(do-something-dangerous-with contents))
(file/close f)))

So generalized macros let us write exactly this.¹

I want to start at the punchline and work backwards, because my favorite part is how simple it is to write this. So although I don’t actually expect this to make any sense yet, let’s go ahead and look at the implementation of this defer macro:

(defmacro defer [] [expr]
(macro lefts rights
~(,;lefts (finally (do ,;rights) ,expr))))

Pretty tame, right?

So in order to do understand how this works, we’ll have to change three things about macros:

Macros no longer have to appear at the head of the form; they can appear anywhere within a form.
Macros now have two argument lists: the forms “to the left” of the macro, and the forms “to the right” of the macro.
Macros can either return new abstract syntax trees – like a traditional macro – or they can return new, anonymous macros as first-class values.

Let’s go through these ideas one at a time in slightly more detail, and then we’ll circle back to the definition of defer.

Macros can appear anywhere

Scheme already has something called “identifier macros,” which can appear anywhere within a form. You can use them to say that foo is a macro, and it can appear in any expression context, and then make (+ 1 foo) expand to something like (+ 1 (some complicated expression)).

But identifier macros can still only rewrite themselves. In order to do anything interesting with this, we need to add…

Macros can see forms “to the left” and “to the right”

On the face of it this might sound like I’m trying to introduce “infix macros,” so that you could write something like (1 + 2) and rewrite that to the traditional (+ 1 2) syntax. And, to be clear, that is a thing you can do:

(defmacro + [left] [right]
[+ left right])

I think that infix macros could be very useful – we’ll talk more about that in a bit – even if infix math is not particularly compelling to someone practiced in the prefixual arts.

But the real reason to support “infix” macros is for cases like defer, where the (defer ...) expression occurs in the middle of a form, and acts as sort of an “infix” expansion point. But in order for that to work, we need to add…

First-class macros

I think this is the trickiest part to wrap your head around, but it’s the most important. This is the trick that allows macros to rewrite not only themselves, but also the forms around themselves – their parents, their grandparents, their… cousins? I guess? Any other form in your program, actually.

So the idea is that we can create first-class anonymous macros, and return them from our macro implementations. And then those macros will get expanded in the context of the parent form that they now appear in.

This is a lot like returning an anonymous function, except that functions are perfectly reasonable values to put in your abstract syntax trees… so it’s like returning a special function, a function with a little tag attached that says “hey, I’m not a real runtime value, I’m a macro, so you should call me before you finish macro expansion.”

And just to be super explicit: this is different from a macro returning a syntax tree that contains another macro invocation. You can already write “recursive macros,” or macros that return invocations of other macros. But by creating actual new first-class macros at expansion time, you can close over macro arguments and reference them during the next phase of expansion.

So with these changes in mind, let’s come back to the implementation of defer:

(defmacro defer [] [expr]
(macro lefts rights
~(,;lefts (finally (do ,;rights) ,expr))))

Our defer macro takes two binding forms: [] and [expr]. So it expects no arguments to the left – the word defer has to appear at the beginning of its form – and it expects exactly one argument to its right. In other words, it looks like a normal, traditional prefix macro of one argument.

But then it returns an anonymous macro that closes over its expr argument. So if we just look at one step of the expansion, we’ll see an abstract syntax tree that looks like this:

(do
(def f (file/open "foo.txt"))
<macro>
(def contents (file/read f))
(do-something-dangerous-with contents))

But macro expansion isn’t over. After expanding the (defer ...) form, the macro expander will notice that it expanded to another macro, so it will expand that. Which winds up invoking our anonymous macro, passing it ['do '(def f ...)] as its “left” arguments and ['(def contents ...) '(do-something...)] as its “right” arguments.

And then that will return a replacement for the entire (do ...) form, giving us our final result:

(do
(def f (file/open "foo.txt"))
(finally
(do
(def contents (file/read f))
(do-something-dangerous-with contents))
(file/close f)))

Neat, right?

This type of macro gives us a lot more freedom to decide how we want our code to look. I’m honestly not really sure how much more freedom, because I haven’t spent very much time with the idea yet. But I’ve been thinking about it for a while, and I’ve come up with a few examples of things that we can do with this – some much dumber than others.

Let’s take a look at a few of them.

Nest less

I think that reducing the number of nested parentheses and general indentation might be the most compelling use case for this sort of macro. I mean, really this is all defer does: it lets you write finally with a little less nesting, and with the expressions in a slightly different order.

I spend most of my programming time writing OCaml. OCaml doesn’t have “block scope” or “function scope” like most languages – it has expression scope. You introduce new “variables” (they can’t actually vary; all OCaml bindings are “const") using let ... in, and the binding only exists on the right-hand side of that particular expression.

If you think about the nesting of the OCaml abstract syntax tree, it looks something like this:

(let x = 10 in
(let y = 20 in
(x + y)))

But of course you don’t write OCaml like that. For one thing, the parentheses are redundant:

let x = 10 in
let y = 20 in
x + y

For another thing, this triangular indentation is really annoying. So you actually write it like this:

let x = 10 in
let y = 20 in
x + y

The parse tree for that expression is still nested, but it doesn’t look nested – you always format your code linearly.

So lisps also have let, but lisps don’t have the luxury of leaving off the parentheses, so we’re back to the first example:

(let [x 10]
(let [y 20]
(+ x y)))

And if we tried to write that without indentation, then…

(let [x 10]
(let [y 20]
(+ x y)))

Immediate aneurysm.

Fortunately, every lisp dialect that I know of mitigates this problem substantially by allowing let to introduce multiple bindings in a single form:

(let [x 10
y 20]
(+ x y))

Which means that we only have to increase the nesting by one level in the very common case that we have a series of let expressions. But by using generalized macros instead, we can write a version of let that doesn’t increase indentation at all. I’ll call it def:

(defmacro def [] [name value]
(macro lefts rights
~(,;lefts (let [,name ,value] ,;rights))))

def lets us write code like this:

(def x 10)
(def y 20)
(+ x y)

Which gets transformed into code like this:

(let [x 10]
(let [y 20]
(+ x y)))

Of course Janet – and most lisps – have a “linear assignment” form like this built into the language. In Janet it’s called – coincidentally enough – def.

In fact in Janet, def is actually the primitive way to create new bindings, and let is a macro that just desugars to do + def, which is very reasonable and pragmatic, but feels weird to me.

It feels weird to me because, in my mind, let should be syntax sugar for fn – Janet’s word for lambda. Because, after all, these two expressions are equivalent:²

(let [x 10] (+ x 1))
((fn [x] (+ x 1)) 10)

let allows us to write the expression in a much more natural order, but we can introduce new bindings without any lets at all.

This might sound like weird mathematical lambda calculus trivia, but it’s not: it’s important to understand introducing new variables as a special-case of function application, even if this particular function application happens to be trivial.

Because we can apply the same technique that we just used – rewriting def to let, and rewriting let to fn – to do something much more interesting.

Generalized function application

So Haskell has something called do notation. You’ve probably seen something like this before:

addAll :: [Int] -> [Int] -> [Int]
addAll xs ys = do
x <- xs
y <- ys
return (x + y)

ghci> addAll [1, 2] [10, 20]
[11,21,12,22]

This is equivalent to the following Janet code:

(defn add-all [xs ys]
(mapcat (fn [x]
(mapcat (fn [y]
[(+ x y)])
ys))
xs))

But I think the Haskell code is easier to read. Partly that’s because the argument order to Janet’s mapcat function makes the values we’re traversing appear in reverse order in our source code, and we could fix this by redefining mapcat with a argument different order:

(defn add-all [xs ys]
(mapcat xs (fn [x]
(mapcat ys (fn [y]
[(+ x y)])))))

This reminds me of the transformation we did when we changed ((fn [x] (+ x 1)) 10) into (let [x 10] (+ x 1)). So what if we take it one step further, and do the same thing we did to get def?

(defn add-all [xs ys]
(as x mapcat xs)
(as y mapcat ys)
[(+ x y)])

It’s not quite as concise as Haskell’s do notation: Haskell is able to use the type of the expression to determine what <- means, so there’s no need to specify the mapcat bit: it’s implied from the fact that we gave it a list.

Janet doesn’t have an analog for type classes, so we have to be a little more explicit, but this means that we can do more than just “bind” with the as macro. We can also map:

(defn add-all [xs ys]
(as x mapcat xs)
(as y map ys)
(+ x y))

Implementing as is just as easy as implementing def:

(defmacro as [] [name f arg]
(macro lefts rights
~(,;lefts (,f (fn [,name] ,;rights) ,arg))))

If you haven’t programmed in a language like Haskell, this particular syntax sugar might seem a little odd at first. But a specialized notation for generalized function application is extremely useful – we have it in OCaml too, through a syntax extension called ppx_let:

let%bind x = xs in
let%bind y = ys in
return (x + y)

I think OCaml’s notation is actually more clear than Haskell’s – it highlights the symmetry between “ordinary” let bindings and “fancy” let bindings like these. And because it can do more than just bind, we can also avoid the explicit return in OCaml:

let%bind x = xs in
let%map y = ys in
x + y

map and bind aren’t the only functions of this variety, either. Although monads are ubiquitous in OCaml, I spend a lot of my time working with arrows as well. And arrows have yet another notation: let%sub and let%arr.

All of these are generalizations of regular function application. Without worrying about what any of this means, just look at how similar the shape of these different type signatures are:

val (@@) : 'a -> ('a -> 'b) -> 'b
val map : 'a f -> ('a -> 'b) -> 'b f
val bind : 'a f -> ('a -> 'b f) -> 'b f
val sub : 'a s -> ('a r -> 'b s) -> 'b s
val arr : 'a r -> ('a -> 'b) -> 'b s

Okay, I know; this isn’t supposed to be a blog post about monads or arrows. Let’s get back to macros.

Infix operators

So Janet has some very useful “threading” macros that allow you to write code in a more “linear” fashion than you could without them. They’re useful when you’re performing a series of transformations to a value:

(filter |(string/has-prefix? "a" $)
(map string/ascii-lower
(map |($ :name)
people)))

With the power of threading macros, you could write that like this instead:

(->> people
(map |($ :name))
(map string/ascii-lower)
(filter |(string/has-prefix? "a" $)))

This is a lot like “method chaining” in other languages:

people
.map(person => person.name)
.map(name => name.toLowerCase())
.filter(name => name.startsWith("a"))

It’s easier for me to read the linear notation that uses the threading macro. But it’s not actually any easier to write it.

I like that method chaining allows me to write the code in the order of the operations: “Start with people, get the name, lowercase it, filter it to names that start with ‘a.'”

When writing the threading macro, though, the way you type this is “start with people, okay wait, go back, surround it in parentheses, add a ->> at the beginning, now move the cursor to the end of the form, and then get the name…”

I don’t like that. And I know that there are fancy editors that allow you to easily wrap expressions in threading macros or any other without repositioning the cursor, but I’d rather use a syntax that doesn’t require a structural editor to work with comfortably.

So here’s another way to write this:

(people
@ (map |($ :name))
@ (map string/ascii-lower)
@ (filter |(string/has-prefix? "a" name)))

This uses @ as an infix function application macro. I prefer | myself, and that’s the notation that I chose for Bauble, but | is how you create short anonymous functions in Janet, so I don’t want to step on that.

The main reason I prefer this notation is that it’s easier for me to type. I don’t know that it’s any easier to read than ->, but it allows me to write code in the order that I think it, and I like being able to choose a syntax that maps neatly onto my brain.

Another infix macro that I like is .. . is a convenient macro for looking up a keyword in a struct or table, so that you can write struct.key instead of (get struct :key).

In Janet struct.key parses as a single symbol, so we can’t actually implement this as a generalized macro without a separate preprocessing step to split it into three symbols. But we can use it as struct . key, which parses as three separate symbols:

(defmacro . lefts [key & rights]
~(,;(drop-last lefts)
(get ,(last lefts) ,(keyword key))
,;rights))

Which we can then use like this:

(print foo . bar)

(print (get foo :bar))

This is very goofy looking, but you could imagine a language where . always parsed as its own symbol, so that we could just write foo.bar and have that expand to (get foo :bar) automatically.

Another infix macro that I think could be interesting is :, to create a pair.

Janet already uses : as a leader character for declaring keywords, so this won’t work in Janet. But you could imagine, again, a different language where : is used as a short way to create a pair of two elements. So:

foo:bar

Would become:

(foo bar)

This might be useful in languages that use wrapped lets, where you have to write:

(let ((x 10)
(y 20))
(+ x y))

Instead, you could write that as:

(let (x:10 y:20)
(+ x y))

But have it parse in exactly the same way.

You could imagine it in cond as well:

(cond
(> x 0): "positive"
(< x 0): "negative"
(= x 0): "zero"
true: "nan")

Of course Janet doesn’t require wrapping each entry in a cond expression in parentheses, so this isn’t as compelling in Janet.

For a slight variation on this, imagine a macro called ::. It’s just like : – it creates a pair – but the pair appears in reverse order from how you write it. We’re well off the Janet path now, but we could use this as a concise notation for adding type annotations without an explosion of parentheses.

Let’s say we have a – function? macro? – something called Int that provides a type annotation to our compiler. We’d normally write the type of an expression like this:

(def x (Int (compute-thing)))

But look all those close parens! I don’t want to balance those. So instead, we could write:

(def x (compute-thing) :: Int)

Which is equivalent to the less intuitive (to me):

(def x Int:(compute-thing))

:: doesn’t mean “type annotation,” though, it just means “wrap in parentheses.” We could use it to do dumber things:

"hello" :: print

Which would expand, of course, to (print "hello"). But… I don’t know why you would want to do that.

Comment

Something that I occasionally wish for is a (comment ...) macro that lets me ignore code.

You can’t actually write such a macro in Janet. Janet has a macro called comment in the standard library, but comment always expands to nil, and nil is not nothing. This means there are lots of places you can’t use (comment ...):

(defn foo [a (comment this is a comment)]
(print a))

If you tried to compile that, you’d get an error:

compile error: unexpected type in destruction, got nil

Because after macro expansion, the compiler actually sees:

(defn foo [a nil]
(print a))

Which is not valid.

With generalized macros, though, you can write a comment that actually disappears:

(defmacro comment [] [&]
(macro lefts rights [;lefts ;rights]))
(defn foo [a (comment this is a comment)]
(print a))

After expansion, that will just be:

(defn foo [a]
(print a))

Like it never happened.

`if`/`else`

One thing that sometimes trips me up when I’m writing Janet is if.

if takes three forms: a boolean expression, an expression to evaluate if it’s truthy, and an expression to evaluate if it’s falsy. Which is nice and concise, but it’s different enough from other languages that I use – languages with explicit elses – that sometimes I’ll write code like this by mistake:

(if should-do-something-important
(print "okay, performing important work:")
(perform-important-work))

That actually doesn’t perform important work – (perform-important-work) is the “else” section of that conditional. In order to do more than one thing in the “then” branch, we have to wrap all of the statements in do.

And of course Janet has a when macro that does exactly what I want:

(when should-do-something-important
(print "okay, performing important work:")
(perform-important-work))

Which doesn’t have an else branch, and usually when I’m writing an if without an else I should just use when in the first place.

But.

Generalized macros actually let us write if with an explicit else. I’m not saying this is a good idea, but they let us write something like:

(if (empty? name)
(print "you must provide a valid name"))
(else
(print "okay i think it checks out"))

This example is a little weird because, in order for this to work nicely, we’ll have to rename the built-in if. I’ll call the ternary version if-then-else for this example, and say that if now means the same thing as Janet’s when.

(defmacro else [] [else-exprs]
(macro lefts rights
(match (last lefts)
['if & then-exprs]
~(,;(drop-last lefts)
(if-then-else (do ,;then-exprs) (do ,;else-exprs))
,;rights)
(error "else must come immediately after if"))))

This is interesting because the else macro actually rewrites the form before itself, changing the if to an if-then-else and then fussing with its arguments.

Weirder, more exotic things

I could keep going, but, as you have probably noticed, this is already a very long blog post. There’s a lot more that you can do with generalized macros – some of it useful, some of it unhinged, and we don’t have time to talk about all of it.

So far we’ve only seen macros that rewrite their parents or immediate siblings, but you can write macros that return macros that return macros, and use them to rewrite arbitrary forms anywhere in your program. You could write a macro that rewrites “the nearest enclosing function definition,” recursively accumulating first-class macros until finally expanding all of them.

You can write actual left- and right-associative infix operators, and I think that if you tried hard enough, you could even use dynamic variables and controlled macro expansion to implement infix operator precedence (although I don’t think you should).

You could implement (a weaker version of) Janet’s splice built-in as a generalized macro. You could implement “identifier macros” that look around themselves and expand to something different when they appear as the first argument to a (set ...) form. You could, you could…

You could do a lot of things, but I’m going to have to leave these as exercises to the reader, because it’s time to switch gears and talk about why you shouldn’t do this.

Problem one: the repl

The main reason that this seems like a bad idea is that macros like this don’t work at the top level.

If you’re just using the repl, and you type (defer (file/close f)), what happens? One of the arguments to that macro is “everything to the right.” But there isn’t anything to the right! At least, not yet. And it won’t be able to supply everything to the right until you stop typing altogether.

This might not seem like a big deal for defer – just don’t use defer at the repl – but it is a big deal for, say, def. And I don’t know an elegant way to solve this problem: in the general case, macros could look arbitrarily far ahead, so we’d have to wait until we closed the repl session to be able to expand them. And that’s kinda gross.

Problem two: generalized macros don’t always compose

All of the examples that we’ve seen so far play nicely together, but it’s possible to write generalized macros that don’t compose with one another.

The problem is that macro behavior can depend on expansion order. Regular macros always get a chance to run before their arguments get expanded, which is very convenient. Generalized macros don’t have that luxury – because macros can see the forms around them, the order that you expand those forms matters.

In my implementation I chose to expand macros in a depth-first, left-to-right order. So macros always see the arguments “to the left” of themselves fully expanded, and the arguments “to the right” completely unexpanded.

And this can be problematic. For example, let’s say we make an infix alias for set, called :=:

(var x 0)
(x := 1)

Which expands to:

(var x 0)
(set x 1)

This is a trivial generalized macro to write.

Now let’s say we have another macro, which looks at the form to its left to see if it immediately follows set. When it does, it rewrites that set to something else. We could use this to implement some kind of custom associative data structure:

(defmacro at [] [dict key]
(macro lefts rights
(if (= lefts ['set])
~(assign ,dict ,key ,;rights)
~(,;lefts (lookup ,dict ,key) ,;rights))))

So that macro lets us write:

(set (at dict key) 10)

And have that expand to:

(assign dict key 10)

Meanwhile, if we write (print (at dict key)), that will expand to (print (lookup dict key)).

Each of these generalized macros make sense on their own. But if we try to use them together, they just don’t work:

((at dict key) := 1)

Because (at dict key) expands first. It looks at the forms to its left, sees that there aren’t any, so after one step of expansion we have:

((lookup dict key) := 1)

Then we expand :=, and finish with:

(set (lookup dict key) 1)

Which of course was not what we wanted.

I think that using generalized macros safely requires really understanding the effect that they have on the syntax tree of your program. They’re more like ->> and friends – explicit syntax re-arrangers – than they are like other kinds of macros.

Problem three: you have created in your code a work of madness that no other human being can possibly hope to understand

ugh not again

Prior art

I feel like this approach is so simple that it must have been done before, but I can’t find any references to it. That said, I have no idea how to search for it effectively! So if you’ve seen this technique before, or if you’ve heard of it being used in the past, I’d love to hear about it.

Proof of concept

I implemented this macro system in Janet, in order to play around with it and test out my macro implementations.

It was pretty easy to write! It really is a very modest generalization of a traditional macro system. I didn’t actually write a custom module loader that would let you use this as your default macro system in Janet code, but I wrote (the equivalent of) macex, and adding the custom module loader would be pretty easy if you wanted to use it “for real.”

You can look at the code here: https://github.com/ianthehenry/macaroni

Or take a peek at some of the tests, to see the examples in this post in action, as well as some weirder things that didn’t make the cut.

Of course we could teach do to scan through all of its arguments and look for defers, and implement this that way, but then this would only work in do expressions. What if we want to do this inside a (fn [] ...) block? Or an if? Or a while? By making defer itself do the transformation, we can use it anywhere – and make it easy to add new macros that behave like defer, without having to teach do about them. ↩︎
Back in the Old Days, JavaScript only had function-scoped variables, so the only way to create new bindings in JavaScript code was to create and invoke an anonymous function like this. The pattern was so common that we called them “immediately invoked function expressions,” and it was a real thing that you had to do in order to, for example, close over distinct values in a loop. ↩︎

Why Janet?

Wed, 12 Apr 2023 00:00:00 +0000

I never thought it could happen to me. I mean, parentheses? In this day and age? But for the past couple years, my go-to programming language for fun side projects has been a little Lisp dialect called Janet.

(print "hey janet")

I like Janet so much that I wrote an entire book about it, and put it on The Internet for free, in the hopes of attracting more Janetors to the language.

I think you should read it, but I know that you don’t believe me, so I’m going to try to convince you. Here’s my attempt at a sales pitch: here is why you – you of all people – should give Janet a chance.

Janet is simple

Janet is an imperative language with first-class functions, a single namespace for identifiers, and lexical block scoping. The core of the language is very small, consisting of only eight instructions: do, def, var, set, if, while, break, fn. But thanks to macros, there are lots of high-level wrappers that give you more powerful or convenient control flow.

You can “learn” Janet in an afternoon, because the runtime semantics are extremely familiar: think JavaScript, plus value types, minus all the wats. And the rest of the language is small: the entire standard library fits on one page. It was this ease of getting started that got me hooked in the first place.

Janet is distributable

It’s easy to compile Janet programs into native executables that statically link the Janet runtime. And you can share those programs with other people, without asking them to install Janet first – or your project’s dependencies, or anything else for that matter. You don’t even have to tell them it’s written in Janet!

The way that Janet pulls this off is very elegant: Janet compiles itself to bytecode, and then writes that bytecode into a .c file that also starts up the Janet runtime. Then it compiles that C file with your system’s C compiler. Since Janet is designed to be easy to embed, this makes perfect sense: it is, essentially, embedding itself into a trivial C executable.

A simple Janet “hello world” compiled to a native binary weighs under a megabyte (784K for Janet 1.27.0 on aarch64 macOS, but your mileage may vary). This includes the full Janet runtime, garbage collector, and even the bytecode compiler – so you can write programs that evaluate Janet code at runtime, if you want to.

This makes Janet an excellent choice for writing little command-line apps. Which is especially true when you consider that…

Janet is unrealistically good at parsing text

Instead of regular expressions, Janet’s text wrangling is based around parsing expression grammars. Parsing expression grammars are simpler, more powerful, and more predictable than regular expressions. They aren’t line-oriented, so they can parse multi-line text without a problem. They can also parse HTML, or JSON, or any other non-regular language. They can also parse binary file formats – they have no problems with arbitrary null bytes.

They really are parsers: structured, composable, first-class parsers. And they’re pretty easy to learn!

Janet has the best subprocess DSL of any high-level language

There is a third-party library called sh that provides a shell scripting DSL that allows you to express pipes and redirects directly in Janet. Like this:

($ find . -name *.janet | say)

It’s pretty incredible. It’s such a nice DSL that I dedicated a whole chapter of Janet for Mortals to it – and the things that you can do with it. It elevates Janet from a reasonable alternative to Perl to a reasonable alternative to Bash for a surprisingly large range of programs.

Janet is embeddable

Lua has become the de facto “embedded language,” which is a shame, because… well, this isn’t a post about Lua. You might not care about this very much, but there’s a chance that it’s just because you haven’t tried it yet: being able to write progams that expose scripting interfaces is a pretty fun superpower.

Embedding Janet is very easy: the Janet runtime is a small C library, and all you have to do is link it in and then call regular C functions to manipulate Janet values. You can even embed it into websites, and write static sites with custom programmable DSLs!

Janet has mutable and immutable collections

Janet’s collection types come in mutable and immutable flavors. Immutable collections have value semantics: the immutable vector [1 2] is indistinguishable from (take 2 [1 2 3]), despite the fact that they have different memory addresses. Mutable collections, on the other hand, have reference semantics: the hash table @{:x 1 :y 2} is only equal to itself. Another hash table with the same keys and values is a distinct object.

Not every language has immutable composite values built right into the standard library!

Macros, macros, macros

I think this is the real reason you should learn Janet, but I didn’t want to lead with it because I didn’t want to scare you off.

You can write Janet just fine without ever learning how to write macros. But you should learn how, because writing macros is fun. It feels different than any sort of programming that I’ve done before.

Writing macros requires thinking twice at once: you’re writing code to write code, so you have to keep two threads of execution straight in your mind: the code that is running now, at compile time, manipulating values and abstract syntax trees, and the code that you are manipulating, the application code that you produce, the code that will run in the future.

Janet’s macros are not hygienic, and Janet does not have a separate namespace for functions. But by allowing you to unquote literal functions, Janet makes it possible to write macros that are completely referentially transparent. It’s an incredibly simple and elegant solution to an otherwise very delicate problem. And the fact that it is possible to do this in Janet highlights my next favorite feature…

Janet lets you pass values from compile-time to run-time

This is the most interesting thing about Janet, in my opinion. But it might not sound very interesting at first – really all it means is that any Janet value can be serialized to disk and read back in later.

But this serialization is implicit: when you compile a Janet program, it runs all of the top-level instructions – regular statements, function declarations, whatever – and then, once it’s executed all of the top-level values, Janet writes down a snapshot of your program’s state to disk.

And it’s a full snapshot of your program’s state: shared references are preserved, so mutable values can still be mutated after you “resume” the snapshot. Generators remember exactly what instruction they need to run the next time you resume them. Closures gonna close.

Macros are a special-case of compile-time code execution – manipulating abstract syntax trees to create new functions – but this is a superpower that you can enjoy without any macros at all. Making a game? Reticulate your splines ahead of time! Or embed assets in your final binary by reading files at compile time – you can perform arbitrary side effects!

Janet for Mortals has an example of using this to autogenerate database bindings based on a SQL schema file – a bit of a silly example, but something that would be quite difficult to do in most languages.

Janet feels good in the hand

This is completely subjective, but I love Janet’s syntax. It strikes a perfect balance of simplicity, uniformity, and variety.

It uses pervasive parentheses, but breaks them up with [] for lists and {} for tables.

Mutable literals are always prefixed with @: @"mutable string", {:immutable hash-table}, etc.

Anonymous functions are written (fn [x] (+ 1 x)), but there’s a shorthand notation for lifting any expression into a function with |: |(+ 1 $).

Janet supports “splats” or “spreads” with ;: (+ ;args).

String literals can be written with any number of backticks, and closed with the same number of backticks. Escape sequences like \n don’t apply in backtick-quoted strings, so you can create strings with any contents without ever thinking about how to escape them – all you have to do is wrap them in a sufficient number of backticks.

Rest parameters use & instead of .: (defn foo [first & rest] ...).

Janet doesn’t support reader macros, so the syntax itself is fixed. If you know how to read Janet, you can read all Janet programs. Which is not to say you can make sense of them…

Janet prefers comfort to tradition

Janet does not adhere to the ancient customs. CAR is called first. PROGN is called do. LAMBDA is fn, and SETQ is def. nil is not the empty list; it is its own type, and there are first-class Booleans in the language. It eschews EQ, EQL, EQUAL, and EQUALP. There is nary a linked list in sight.

This isn’t really good or bad, but I thought it was worth calling out: if you saw the parentheses and assumed FORMAT was not far behind, maybe give Janet a second look.

Janet for Mortals

Tue, 04 Apr 2023 00:00:00 +0000

I wrote a book.

It’s called Janet for Mortals, and it’s free, and it’s on the internet, and you can read it right now.

And you should read it right now, instead of reading this blog post, because this blog post is not very interesting if you haven’t read the book. Heck, this blog post is not very interesting even if you have read the book. This blog post is a thinly-veiled promotion for my book to slip into my newsletter and RSS feed, with just enough additional content to pad it out to the length of a real post.

The book is about Janet, a programming language that I have written about before. I’ve been using Janet a lot lately, and I’ve been having a lot of fun with it, and I think that more people should know about it so that they can have fun with it too. People like you.

I’m not really going to talk much about why you should read the book, or even why you should care about Janet in the first place – that will come in a later post. Instead, this is going to be a short retrospective of what it was like to write my first technical book.

We’ll start with some numbers.

It took me twenty weeks to write the book, working in my spare time. I had originally estimated twelve weeks, which turned out to be a really good guess for how long I spent writing the book, but I didn’t account for how much time I would spend working on book-adjacent coding side quests.

The final book is pretty short: 44k words of English prose, if you don’t count any of the code snippets. I tried to find an example of a famous book with a similar word count to put that number in perspective, and the best I can do is The Great Gatsby, which clocks in at 47k words. It’s right on the border between novella- and novel-length, but it’s less than half as long as my series of posts about Nix, which is sort of terrifying.

But writing English words was only a fraction of the work. Over the course of those five months, I also spent a lot of time on:

the website itself
jimmy, bindings to a C++ library of persistent data structures
Toodle.Studio, an interactive turtle graphics playground
cmd, a command-line argument parsing library
judge, an inline snapshot testing framework
to do, a command-line todo list manager

These things are not very interesting by themselves, but this blog post just exists to promote the book, so I’m going to reflect on them now. You are welcome to stop reading at any point and go read the actual book instead.

The Website (2 weeks)

The most interesting thing about Janet for Mortals is that it has a built-in repl. At any point you can press escape and pull it up, and it’s docked to the bottom of the page, just out of the way of the text. I’m sure that it’s not the first programming book to include a repl like this, but I’ve never actually seen it done before.

The editor portion of the repl is CodeMirror, which I had used previously in Bauble. CodeMirror doesn’t know anything about Janet out of the box, but I had already implemented some basic language support when I wrote Bauble.

But I skipped a lot of the Janet language when I was writing the grammar for Bauble, like ``multi`backtick`quoted`` strings, because they didn’t really matter in Bauble’s constrained DSL. But they mattered for the book, so I had to spend time figuring out how to implement them.

But fleshing out the CodeMirror grammar (or more precisely, the Lezer grammar that CodeMirror uses) had an unexpected side effect: it let me re-use the grammar to do syntax highlighting for the code snippets in the book itself.

Janet for Mortals is just a static site, but no static site generators that I know of know how to highlight Janet code. For a long time the book was entirely black-and-white, which I don’t mind, but I knew I needed to add some color before I released it upon an unsuspecting public – I think Janet’s syntax is pretty unfamiliar to most people reading the book, and anything to make it look friendlier helps.

So I knew that I’d have to roll my own syntax highlighter of some kind, and I ended up just writing a simple static site generator in redo, which was not a very good fit, but… well, least bad choice that I know of. The meat of the generator is written in JavaScript, so that I could plug in the Lezer grammar for free, but it’s all tied together by a fragile web of shell.

I used Remark to implement the parsing of the book’s source, and it was nice how much control I had over the generated output. I even added a simple extension to label code blocks, which was pretty easy.

I also re-used Remark in the client itself, as part of the repl. The docstrings in the Janet standard library are written in Markdown, and I’m actually parsing and rendering them to HTML on the fly as part of the repl autocomplete.

I spent a long time getting the repl autocomplete working well – because this book was written for newcomers to the Janet language, I thought that the help-as-you-type would be really useful for people trying to follow along in the repl.

Autocomplete works by dynamically querying the Janet environment via WebAssembly at repl startup. This means if you define a new symbol with a docstring, it won’t actually appear in the autocomplete output, but I think that’s… fine. I could re-generate the autocompletions after every command is run, but… I don’t think there’s much value in that.

The most interesting part of the repl is probably the (report) function, which takes a string and POSTs it to a simple web server that sticks it into a SQLite database for me to peruse later. It’s not really any different than a comment box, but I feel like there’s something fun about doing it from the repl. I’m really glad that I added it – it’s been fun reading people’s feedback, and I’ve fixed quite a few errors because of it. I’m sad that I didn’t implement any way to respond, though!

The backend for reports is not written in Janet; it’s a tiny Haskell application that just listens for POST requests and sticks them into a SQLite database.

There are people using Janet to make websites, but I am not one of them: the primary thing I want out of a web server is security, and I just don’t think Janet or its HTTP libraries are “battle-tested” enough for me to connect them to the internet.

I also just think the idea of using a dynamically-typed interpreted language to build a web service is crazy, when there are optimizing compilers right there, but that’s a whole other conversation.

`jimmy` (1 week)

I spent a little bit of time writing bindings to immer, a library of persistent data structures. I never finished them, and probably won’t, at least not until I have a use for them. But as a demonstration of how to interop with C++ code from Janet, I think it was successful.

https://toodle.studio (2 weeks)

Last year I wrote a little art playground called Bauble. It was my first time embedding Janet in the browser, and I had a pretty tough time figuring out how to do that.

There weren’t a lot of resources back then about embedding Janet period, and doing it in the browser added an extra layer of difficulty. I’d never used WebAssembly or Emscripten before, or even TypeScript, and it turns out there are no tutorials on how to write TypeScript Emscripten WebAssembly Janet bindings, so I spent a while figuring out how all the pieces worked together.

And I’m glad I did, because I think the final product is really neat: it’s a website that is not written in JavaScript. I mean, a lot of it is. The UI is, still. But the actual application logic is all Janet.

I thought that that was a really useful superpower of Janet, and I wanted to make the technique more accessible. In fact this was a big motivation for writing this book about Janet – I wanted people to know that this was possible in the first place, and I wanted to make it easier to get started with it.

But Janet for Mortals doesn’t talk about Bauble at all. Bauble is actually not very interesting from an interop perspective: Bauble is completely stateless, and basically uses Janet to implement a pure function from strings to strings (they’re… pretty complicated strings; Bauble is a Janet-to-GLSL compiler, but they are strings nonetheless). I didn’t think it was a very good showcase for everything you can do with Janet, so I briefly considered talking about how I implemented the repl in the book, but I decided that that was far too boring. So I wrote Toodle.Studio – an obvious fork of Bauble – instead.

Toodle.Studio seems a lot simpler than Bauble, but the interop with JavaScript is much more involved. Toodle.Studio has to execute long-running Janet programs asynchronously over time. It has to think about memory management, as the JavaScript code retains multiple references to the same Janet values. It has to pass complex nested data structures to and from Janet, going through C++ as an intermediary. It does a very simple version of all of these things, but it’s a pretty good showcase for the techniques.

But the most interesting part of Toodle.Studio isn’t the interop or the memory management. The most interesting part of Toodle.Studio is the logo.

I wasn’t really planning on making a logo – this is a demo project for a book, after all – but sadly I had no choice. When I was getting ready to release the website, I showed it to my partner, because it’s rare that I work on something comprehensible to normal human beings. I thought she’d like it, but she was aghast.

“You said you were working on turtle graphics,” she said. “Where are the turtles?”

I tried to explain that the turtles aren’t really turtles, that it’s like a flea circus, and the turtles are metaphors – but she was having none of it. The lack of turtles was a base betrayal, so I had to spend a day or so modeling a cute animated turtle in Bauble to act as the logo. And making its eyes follow the mouse, of course.

Relationship repaired. The logo wound up being my favorite part of the site, and it was fun to get a chance to use Bauble to make something “real.”

`cmd` (2 weeks)

One of the things that I spent the most time on, oddly enough, was a command-line argument parsing library. The library itself only gets, like, three paragraphs of screen time in the book, but it was very important to me that it exist before the book came out, so that I could unambiguously claim that “Janet is an excellent scripting language.” Before cmd, that was still true, but the phrasing was more “Janet is a great choice for scripting and writing CLI tools, except that the argument parsing is kind of janky, sorry, but hey at least it’s better than Bash.”

cmd was heavily inspired by Core.Command, which is the best command-line argument parsing library that I have ever used. I’m extremely spoiled by how easy it makes writing CLIs, and I wanted to replicate that experience in Janet. cmd is definitely not as good as Core.Command – types, my goodness, types make everything so much easier – but it has 95% of the features I care about, and the concise notation makes it more pleasant to use in ad-hoc scripts.

One thing that I miss, though, is that Core.Command autogenerates Bash completion functions. I want to add that to cmd one day – the API is designed so that that will be possible to do. But… so many projects, so little time.

`judge` (1 week)

judge was one of the first things that I wrote in Janet, all the way back in 2021. I think that it worked pretty well considering that I didn’t know anything about Janet when I wrote it, but now I do, so I rewrote it from scratch. Not only is the API much nicer to use now, but the implementation is way simpler – and easier to make changes to.

The main differences between Judge v1 and Judge v2 are that tests can now appear inside regular source files, not just the test/ directory, and I added the test-macro and test-stdout helpers, which are extremely useful. The OCaml equivalent of test-stdout is pretty much the only way that I write tests professionally, because OCaml doesn’t really have a way to embed arbitrary data in source code, so we turn everything into a string.

After publishing the book – which has a whole chapter on testing with Judge – I had a chance to spend a little more time improving Judge, and I finally added an --interactive mode, which I’ve been wanting for a long time. And since I’m not spending all my time working on this book anymore, I’ve actually had a few opportunities to use the new Judge, and I gotta say: it’s nice. It’s really nice. I know I can’t impress upon you just how nice it is in this post – it really needs a demo, and I’m too lazy to record one right now – but I’m very happy with how it feels to use it to write Janet.

`to do` (2 hours)

I picked this project to highlight for the scripting chapter, because it’s a non-trivial thing that I had done in Bash before, and actually found it pretty painful. Parsing multi-line text with Sed is not fun, and trying to do date manipulation with date in a way that works the same on macOS and Linux is… basically impossible, as far as I can tell. I quickly ran into the limits of my patience, and gave up on the idea some years ago.

It was really fun to return to this with the full power of PEGs and sh and cmd at my disposal. I immediately surpassed all of features of my original Bash todo list, and was able to add quite a few more (like fzf multi-select – good luck constructing null-terminated strings in Bash).

The book covers a very simplified version of the app – it can’t schedule tasks for the future, and there’s no concept of “skipping” tasks. Those features are important for my todo list workflow, but they are probably not important for your todo list workflow, so the book only discusses the core functionality of adding things to a list and crossing them off. I think that it makes a good starting point to run with and make your own – paired with zsh-autoquoter, it’s actually a surprisingly useful app!

If you put all of these projects together, I was writing code for almost half the time that I spent working on the book. Eight out of twenty weeks, plus some periods where I was doing both at once.

I didn’t really budget for that going into this. I thought that I’d improve Judge, and I thought that I’d write an argument parser. But I thought that writing an argument parser would be way easier than it actually was. And I thought that I’d just talk about Bauble – it never occurred to me that I’d write another art playground just because Bauble was too easy.

So that’s the story of writing the book. Or really, everything but writing the book. All the other things. The writing itself isn’t that interesting. I wrote it in Markdown, in Sublime Text, which is my favorite editor for writing long-form prose. I have nothing interesting to say about that part.

Two new versions of Janet came out while I was writing the book, and I did have to go back and update the chapters on debugging and native modules to keep up with changes to the language. I plan on keeping the book up to date with the latest Janet release – we’ll see how long I can keep that up.

I haven’t done much to promote the book yet. I submitted it to Hacker News, and I submitted it to Lobsters, and I wrote a very half-hearted tweet about it. The reception was pretty much as good as I could have hoped for: it was on the Hacker News front page all day, and even held the number one spot for a while.

What does that actually mean? Well, according my Nginx access logs, I got:

30,025 unique visitors on Friday
9,568 unique visitors on Saturday
3,777 unique visitors on Sunday

Those numbers don’t mean very much, though. Those are just people who clicked on a link – the number of people who actually read the book is much, much smaller.

I don’t actually know how much smaller exactly, because I don’t have any client-side behavior-tracking analytics on the site. But I can sort of try to guess, by looking at my access logs. It seems like retention is not great:

Chapter One had 22% as many visitors as the home page.
Chapter Two had 20% as many visitors as Chapter One.
Chapter Three had 69% as many visitors as Chapter Two.
Chapters Four and Nine, “Pegular Expressions” and “Xenofunctions,” had more visitors than Chapter Three.

I’m guessing that last bit is because people clicked on those chapters to see what they were about, which just goes to show that unique visitor count is not a very dependable metric.

My best attempt at answering the question “how many people are actually reading the book” is 387, as of the end of the launch weekend. So far 387 unique IP addresses have loaded five or more distinct chapters, which is probably a decent proxy for the metric I care about.

I really had no expectations for what these numbers would be before I launched the book. It’s a big time commitment to read a weird book about a programming language you’ve barely heard of, and 387 seems simultaneously low (compared to, say, any blog post) and high (I don’t think I’ve ever read a book off a HN link). But it’s more than zero!

Alright, I think that’s enough. I’ll close with some fun facts:

The Janet language is named after an immortal being in The Good Place who helps mortals navigate the afterlife, hence the title.
The chapter with the fewest visits is currently “Testing and Debugging,” despite being the third-to-last chapter. This does not surprise me at all, but I think it’s a shame: the last three chapters are by far the most interesting in the book, and the style of testing described in that chapter is one of the biggest productivity upgrades that I have personally experienced in my engineering career.
So far I’ve received 494 reports from the built-in repl reporting feature. Most of these were of the “hey nice book” or “testing” variety, but I’ve gotten several dozen typo reports, clarification requests, or otherwise useful comments through it as well.
The most interesting report was just “you should listen to this song: https://www.youtube.com/watch?v=46i3LbIbbhI.” No context, no explanation, and I have no way to reply for clarification. But… thanks! It’s a good song. I’m into it.
A few people asked me questions without including any kind of contact info, so I have no way to answer them. I hope that they found peace, wherever they are. I’m not ignoring you. I just… I only implemented an extremely primitive one-way feedback function.
I’m going to plug the book one last time.

With feeling: Janet for Mortals! Out now! The first infinity visitors get their copy for free!

Visualizing Delaunay Triangulation

Mon, 11 Jul 2022 00:00:00 +0000

Take a look at this:

This is a triangulation of a set of random points, such that all the points are connected to one another, all of the faces are triangles, and the edges include the convex hull of the points.

I would like to claim that this is not a very “good” triangulation. This algorithm tends to produce lots of long, slivery triangles, and a really uneven distribution of edge counts across different vertices – some vertices have way more edges than they need to.

Here’s a different triangulation. These are the exact same points, but I triangulated them smarter:

Isn’t that better? (Click to compare!)

Depending on your dice rolls you might still have a few slivers, usually around the outer perimeter, but I’m willing to bet that it’s a noticeable improvement over the first attempt.

This triangulation is called the Delaunay triangulation, and I would like to claim that it is a very good triangulation. It is, in fact, the best possible triangulation, for some definitions of “best.”

Why should I, a busy person with important responsibilities, care about triangles?

Okay, right, good question. You’ve made it this far in life without triangulating anything; why should you start now?

There are some obvious applications of triangulation algorithms in computer graphics, where everything is made out of triangles, but you knew that already and you are still unmoved.

Let’s try something else.

One textbook example of a non-graphics application of the Delaunay triangulation is interpolating spatial data.

Pretend, for a moment, that you’re a surveyor, and you’ve just finished measuring the elevation of a bunch of discrete points in some area. You sling your weird tripod thingy over your shoulder, and get ready to leave for the first vacation you’ve been able to take with your son since Terence left – when your boss appears at your door.

“What’s the elevation three clicks south of the north ridge, or however surveyors talk?” she asks.

You glance down at the oversized roll of mapping paper (?) on your desk, but you already know what you’ll find: you didn’t take a measurement three clicks south of the north ridge. But you did measure a few points nearby. Perhaps, you think, I can construct a triangulation of these points, and then map it to a three-dimensional mesh, and use that to interpolate the elevations between nearby points.

You call your son to tell him that you’ll be a little late – it’s 1998 in this situation; texting isn’t a thing yet – and get to work.

But after the montage ends, you look upon your work, and find… oh.

Well that’s not going to work.

You pick up the phone to make another difficult call, but as your finger hovers over the rotary dial, you suddenly remember an incredibly long blog post that you read once – or that you will read; it’s 1998 still; this blog post hasn’t been published yet; it’s too late to change it now; just go with it – and you put the receiver back on the cradle.

And then you construct the Delaunay triangulation of the exact same points:

(Click to add sheep, obviously.)

You shoot your boss a quick fax and get ready to hit the road. But as you stand up to leave, you see her standing in your doorway yet again.

“I just got off the corded phone with President Clinton,” she says. “Your triangles are so good that he’s appointing you Surveyor General. When you get back from vacation, I’ll be reporting to you.”

So that’s the first reason to learn about Delaunay triangulation.

Here’s another one: did this idea of interpolating “nearby” points remind you of anything else? Perhaps… the Voronoi diagram? If so, you are smart and attractive: one easy way to construct the Voronoi diagram is to calculate the Delaunay triangulation, because the Delaunay triangulation is the dual of the Voronoi diagram.¹ Which is neat if you know what “dual” means. If you don’t know what dual means, keep reading! We’re going to talk about duals a lot later on.

So those are like… some good reasons to learn about Delaunay triangulations.

But I did not learn about Delaunay triangulations for a good reason.

I learned about Delaunay triangulations for the dumbest reason you can possibly imagine.

Here, it’s easier if I just show you.

I heard the phrase “smash that subscribe button” once, and after a brief moment’s existential crisis as I realized that I am now old enough that I recoil from the parlance of youth, I thought it would be funny to make a subscribe button that you could smash.²

In case you already smashed that one, here’s another:

It’s kinda fun, right?

I’ve been meaning to add an email subscription button to my website for a while, and the thought of implementing this visual gag was enough of an incentive for me to actually get around to it.

I thought that a decent way to randomly fracture a shape into polygons would be to compute the Gabriel graph of a set of random points inside the button.³ And the easiest way to compute the Gabriel graph is to compute the Delaunay triangulation and then remove a few edges from it. So that’s why I started looking into this in the first place.

But it was pretty hard! Although it’s easy to find explanations of Delaunay triangulation algorithms in vague, mathematical terms, I couldn’t find very many resources describing how to actually implement them. Like, I’m trying to construct an infinitely large triangle, but my computer keeps catching on fire – what do I do now, Professor Triangle?

So this is going to be an explanation for the “working programmer.” I’m going to describe a deceptively simple algorithm for Delaunay triangulation, and then the actual edge-cases that you have to think about when you try to implement it. And I’m going to do it without ever using the word “manifold.”

Starting… now.

What is a Delaunay triangulation?

The Delaunay triangulation is the triangulation that “maximizes the minimum angle in all of the triangles.”

Here’s, it’s easier if we look at an example:

Obviously all the angles of a triangle add up to 180°, but the triangles on the left get there with two very acute angles and one obtuse angle, while the triangles on the right have only acute angles – but they are not as acute as the acute angles on the left.

These are the only two triangulations of these four points, so the one on the right is the optimal triangulation according to our criteria – in other words, it is the Delaunay triangulation of this point set.

Here’s an equivalent, but more useful definition: the Delaunay triangulation is the triangulation such that, if you circumscribe a circle around every triangle, none of those circles will contain any other points.

(Click or tap to pause that, if the flashing is distracting.)

I’m going to make a potentially surprising claim: this is the only such triangulation of these points that satisfies that property. If you were to change any single edge – rearrange any of these triangles in any way – you would have at least one point inside another triangle’s circle territory.⁴

It’s worth noting that this is a global property of the triangulation – it’s a statement about every triangle and every point at the same time. Global properties are expensive to check and expensive to change, so we’d much rather focus on a local property during our algorithm.

So let’s consider a single edge on the triangulation. Specifically an interior edge, not an edge around the perimeter.

There are two triangles on either side of this edge. Pick one at random, and draw a circle around it.

Now check if the tip of the other triangle adjacent to this edge lies inside or outside of this circle. If it lies outside the circle, then this edge is “locally Delaunay.” If it lies inside the circle, then the edge is not “locally Delaunay.”

In this case, the point is very much inside the circle, so we know that this is a bad edge. But if we then flip the edge so that it connects the other two points of our quadrangle, then this flipped edge is guaranteed to be locally Delaunay:

Notice that there might be other points that lie inside this circle that we didn’t check.

Because we only checked local Delaunayness, which is only concerned with this one particular point on the other side of the edge.

But here’s the trick: as long as every edge of the triangulation is locally Delaunay, the entire triangulation is a Delaunay triangulation. If this were a computational geometry class we’d spend the next ten minutes proving that statement, but for now I’ll just ask you to trust me on it.

Okay. So now that we know these things, we can start to imagine an algorithm. We want to make a triangulation such that all edges are “locally Delaunay.” If we started with any triangulation, we could just loop over all the edges, flipping any edges that are not locally Delaunay, and repeat until we don’t find any edges that need to be flipped.

Would that work? Or is there a chance that we’d get into an infinite loop, repeatedly flipping and un-flipping edges due to some weird complicated cycle? It’s not obvious to me that this is the case, but apparently it does work – but only in two dimensions! – and will eventually terminate after O(n²) flips.

But we can do better.

Guibas & Stolfi’s incremental triangulation algorithm

There are several “better” algorithms, but the one I’m going to describe comes from Guibas & Stolfi’s seminal 1985 page-turner “Primitives for the Manipulation of General Subdivisions and the Computation of Voronoi Diagrams."

It’s an incremental algorithm, which means that you can start triangulating something even if you don’t know all of the points up-front, or you can use it to add new points to an existing Delaunay triangulation without having to re-triangulate it from scratch.

It’s the most popular algorithm that I came across, probably because it’s very easy to understand it conceptually.⁵ But for all the explanations of it that exist on The Internet, I could not find a single resource that actually explains it in sufficient detail that you could go write the code for it! I had to cobble an understanding together from the original paper, lecture notes, actual lectures on YouTube – and ultimately by sitting down and working out some details myself.

So I’m going to try to correct that here.

But before I describe the algorithm, let me show it to you. You already know about “locally Delaunay” edges, which is the only real trick here – see if that’s sufficient to work out what the algorithm is doing.

Click to start/stop the animation, or drag the scrubber to move frame-by-frame.

Okay, so at a high level, the algorithm is deceptively simple:

Start with a single “infinitely large” triangle – although note that, in tonight’s performance, infinitely large triangle will be portrayed by appropriately-sized finite triangle, in order to make the visualization easier to understand.

Next, loop through your points, and add them into the triangulation one-by-one. Notice that our single giant triangle is, trivially, a valid Delaunay triangulation, which is a precondition that will hold at the beginning of every iteration.

Every time you add a point, split the triangle that contains it into three new triangles, and then check the locally Delaunay condition. You know that you started with a valid Delaunay triangulation, so you only have to look at the edges immediately surrounding the point that you added – the rest of the edges are already good. Flip any edges that are not locally Delaunay, and then check the edges adjacent to those flipped edges, because the act of flipping might have made them invalid. Then once you’re done checking and potentially flipping all of the “dirty” edges, you’re done! You have a valid Delaunay triangulation once again, and you’re ready for the next point.

Once you run out of points, just remove the infinitely large outer triangle – and the edges connected to it – and you have your final triangulation.

Easy, right?

no; you’ve used the phrase “deceptively simple” twice already

Right. So that explanation might be sufficient if you’re a mathematician, but if you actually tried to write the code for this, you’ll quickly run into questions like the following:

I can’t actually construct an infinitely large bounding triangle – is it okay if I just construct a triangle large enough that it contains all of my points?
How am I supposed to find the triangle that contains my new point?
What if the new point lies on the edge of an existing triangle?
How am I supposed to do this “circle test” in the first place?
What edges do I actually need to check, once I’ve added my new edges?
Wait, how do I “flip” edges?
Wait, how do I even model edges? How do you even represent triangulations on a computer? I thought I could just model this as a graph, with vertices and edges and pointers, but it seems like I need to keep track of faces as well, and somehow this is a fundamentally different sort of thing than I’ve ever encountered before?

Well, those were the questions I had, anyway.

And now I have the answers.

Let’s start with (3) because (3) is easy: if a point falls on an existing edge, you want to delete the edge, then create four new triangles instead of the normal three. The odds of this happening in a randomly generated point set is astronomically low, so here’s a hand-picked example that demonstrates what this looks like:⁶

This prompts a new question – “how do I delete an edge?” – but we’ll answer that at the same time we figure out how to flip edges.

The rest of the questions have… slightly more involved answers.

So let’s start with a fun one.

How to check if a point is inside a circle

It’s very easy to check if a point is in a circle if you know the circle’s radius and center – you compare the distance from the point to the center of the circle against the circle’s radius.⁷ But we don’t know the circle’s center or radius; we only know these three points on its circumference.

Now, we could solve for the circle’s center and radius. There are many techniques for doing this, but it’s relatively expensive – we don’t want to be taking square roots or doing any floating-point division here, because we’re going to be doing this point-in-circle test a lot.

Instead, Guibas & Stolfi describe a really interesting technique that, when I first encountered it, made absolutely no sense to me.

Their technique is to calculate the determinant of this 4x4 matrix:


A_x	A_y	A_x² + A_y²	1
B_x	B_y	B_x² + B_y²	1
C_x	C_y	C_x² + C_y²	1
P_x	P_y	P_x² + P_y²	1

(Where (A, B, C) are the points that define the circle, and P is the point to test.)

If the determinant is positive, the point is inside the circle. If it’s negative, the point is outside of the circle. If the determinant is zero, the point also lies on the circle.

Okay what.

So I don’t expect this to make sense to you. It didn’t make sense to me when I first heard it. But I’m going to try to explain it, in a little more detail than Guibas & Stolfi explain it.

First off: we have a bunch of 2D points. We’re going to transform them into 3D points, by saying:

z = x² + y²

This gives us a bunch of points in 3D. Specifically, it projects these points upwards onto this weird parabaloid shape:

(Click and drag the points to move them around.)

Okay. So that’s… something.

We know that three points in 3D space uniquely determine a plane (unless the points are co-linear). Planes seem simpler than circles. So you can maybe see how this is a step in the right direction. Let’s draw the plane:

Now, there’s something very interesting about this plane: if we look at the points where it intersects our parabaloid, we can see that it forms some kind of ellipse shape.

But if we look at it straight down from the top, we can see that the ellipse projects a perfect circle onto the XY plane.

Specifically, the circle that passes through all of our original points. So if we project the intersection of this plane with the parabaloid back down onto the XY plane, we get the circle that we actually care about:

Okaaaaay. So we’ve established a correspondence between this weird parabaloid shape and the circle in question. But let’s not forget why we’re doing this: we want to check if a point is inside this circle.

So let’s add another point.

Drag it around a little bit, and hopefully you can see where we’re going with this.

When we project that point upwards to the parabaloid, the point where it hits is either going to lie “under” or “over” our plane. And you can see that it’s going to hit “under” the plane exactly when the point is in the circle.

Okay! So this is interesting. This already suggests a very simple circle test algorithm: instead of finding the equation for the circle, find the equation for this plane. Then project our point upwards to both the plane and the parabaloid. If the projection up to the parabaloid is lower than the projection to the plane, then it’s inside the circle.

It’s easy to find the equation for a plane. Given three points (A, B, C), all we have to do is find any vector orthogonal to the plane, which we can do by taking the cross product of any two vectors on the plane:

N = (B - A) ⨯ (C - A)

And that, combined with any one of our original points, gives us an equation for the plane:

N_x(x - A_x) + N_y(y - A_y) + N_z(z - A_z) = 0

Which we can use to solve for z, giving us a function of x and y:

plane(x, y) = -(N_x(x - A_x) + N_y(y - A_y)) / N_z + A_z

Okay, neat. Now if you recall our parabaloid equation:

parabaloid(x, y) = x² + y²

Then all we have to do is compare the value of these two functions, and we have our circle test:

contains(x, y) = parabaloid(x, y) < plane(x, y)

Okay. That makes sense, right?

But that’s not what Guibas & Stolfi did.

Notice how there are no weird complicated matrices involved in that calculation. Yes, I did take a cross product, but that’s a pretty small amount of linear algebra to stomach compared to the determinant of a 4x4 matrix.

So.

Let’s return to Guibas & Stolfi’s solution. Remember that they propose calculating the determinant of this matrix:


A_x	A_y	A_x² + A_y²	1
B_x	B_y	B_x² + B_y²	1
C_x	C_y	C_x² + C_y²	1
P_x	P_y	P_x² + P_y²	1

Hmm.

I can understand 3x3 matrices. I’ve made my peace with them. I can think of them as transformations of 3D points to other 3D points. I’ve even almost, sort of, reluctantly made peace with determinants: I can understand the determinant of a 3x3 matrix as the rate by which it scales volumes when you transform shapes with it.

4x4 matrices… I have no intuition for them. I would expect the determinant of a 4x4 matrix to be some kind of scaler of hypervolumes. But since it’s only barely four-dimensional – all values on the “W axis” are 1 – we can intuit that we’re probably just dealing with plain volumes again.

And in fact that’s correct: this determinant is actually the (oriented!) volume of the parallelepiped that has these four points as corners.

Whoa, that’s… a lot. That’s kind of hard for me to parse. But in the same way that a parallelogram consists of (2! = 2) identical triangles, a parallelepiped consists of (3! = 6) identical tetrahedra. Let’s focus on just one of them:

So I have two things to say about this.

First off: notice that sometimes the tetrahedron is “inside out:” it has a negative volume, which I represent by changing the color of the wireframe. The crux of the point-in-circle test is checking the sign of that volume, and you can see if you move the free point around that it does invert when the free point passes into or out of the circle.

But! If you move the other points around, you can change whether positive or negative means “point in circle” or “point outside of circle.” Try it: swap the location of any two points on the circle, and watch as the sign flips, even though the circle remains the same.

This is because the orientation of the tetrahedron depends not only on the position of the free point relative to the three circle points, but also the orientation of the three circle points relative to one another. So this test only works if you make sure that you list the points (A, B, C) in a clockwise order. If you list them in counterclockwise order, the circle will be “inside out,” and the test will give you the wrong answer.

Which is pretty annoying.

So… I don’t like this test as much as the plane test in general. It seems more expensive⁸ to calculate the determinant than to calculate the equation of the plane, and it seems really annoying that you have to be careful that you provide the points in the right order or you’ll get the wrong result.

This isn’t actually a problem for us, because we’ll be use a fancy data structure for storing edges that ensures we can only provide points in the correct order.

Anyway.

Second thing:

A more obvious way to calculate the volume of a parallelepiped is to pick one point as the origin, and then to write the other three points relative to this origin. Then we have three points in three dimensions, which we can express as a square matrix:


A_x - P_x	A_y - P_y	(A_x² + A_y²) - (P_x² + P_y²)
B_x - P_x	B_y - P_y	(B_x² + B_y²) - (P_x² + P_y²)
C_x - P_x	C_y - P_y	(C_x² + C_y²) - (P_x² + P_y²)

And the determinant of this matrix gives us the volume of the same parallelepiped as before, but in the simple straightforward way that we remember from that one linear algebra class we almost passed.

Now, the determinant of this matrix is exactly equal to the determinant of the 4x4 matrix that you get by padding each point with a 1. They’re just two ways of saying the same thing. You could verify that algebraically, but I feel like there’s probably some intuitive explanation of this fact, and understanding why this is would lead to a better understanding of determinants in general and maybe teach me how to use higher dimensional matrices to more easily perform calculations in lower dimensions. Like in some way the fourth point we added sort of acts as a new origin? At least in this particular case?

But look: I have to stop somewhere, or I’m never going to finish this blog post. And this is the place I decided to stop exploring Guibas & Stolfi’s point-in-circle test.

For now.

Back to the triangulation

Where were we?

I can’t actually construct an infinitely large bounding triangle – is it okay if I just construct a triangle large enough to contain all of my points?
How am I supposed to find the triangle that contains my new point?
~~What if the new point lies on the edge of an existing triangle?~~
~~How am I supposed to do this circle test in the first place?~~
What edges do I actually need to check, once I’ve added my new edges?
Wait, how do I “flip” edges?
Wait, how do I even model edges? How do you even represent triangulations on a computer? I thought I could just model this as a graph, with vertices and edges and pointers, but it seems like I need to keep track of faces as well, and somehow this is a fundamentally different sort of thing than I’ve ever encountered before?

Okay. Let’s talk about (7) next, because you’re losing steam and it’s very interesting. The other questions are pretty specific to this one algorithm that you don’t actually care about – although I appreciate how polite you are being – but this one is broadly applicable to lots of graph-related, mesh-related, and computational geometry-related problems.

The quad-edge data structure

Okay, so you know how to represent a graph, right? You have some Node or Vertex objects, and you have some Edge objects, and vertices have pointers to their edges and edges have pointers to their vertices. Or you store an adjacency matrix; that’s a thing I’ve heard of.

This works great when you’re dealing with graphs in the abstract – if all you’re doing is Dijkstra’s algorithm or finding biconnected components or whatever, this simple representation works fine.

But in this case we’re also concerned with faces – when we flip an edge, we need to know the specific faces we’re flipping it on, so we don’t do something illegal:

The obvious-to-me way to do this is to add another object to the mix: model Faces explicitly, and have edges keep track of the faces they’re a part of, and every time you change anything about the graph you have to make sure to keep the faces and edges and vertices all in sync. I actually tried this, during my first forays into triangulation. It was awful. Just a case study on how denormalization will ruin your day. We’re not going to do that.

Fortunately, Guibas & Stolfi describe a very clever alternate representation called the “quad-edge” data structure.

To understand the quad-edge structure, you first have to understand the idea of the dual of a graph. Very loosely, if you take a drawing of a graph⁹ and put a vertex in the middle of every face, and then connect all of the adjacent “face-vertices” with edges, you’ll have the dual graph. It’s easier to understand with a picture:

That’s actually a simplification – I’m missing one face from all of those graphs: the outer face, the inside-out face bounded by the outside of the graphs. Here are the actual duals:

But since that is a mess to look at, I will instead adopt the following shorthand, and trust that you can imagine the outer face:

Okay, so duals. Neat. Who cares?

Well, the high level idea of the “quad-edge” structure is that you break every undirected edge A <-> B of your mesh into four separate directed edges: one that represents A -> B, one that represents B -> A, and two others that represent the directed dual edges of the faces on either side of the edge. So for each undirected edge on our graph, we would have four directed edges like so:

So the quad-edge structure represents the structure of a graph and its dual at the same time: a quad-edge is just a list of these four directed edges.

I was very confused about this when I was reading Guibas & Stolfi.¹⁰ They never give a type definition for quad-edges, and they describe operations on quad-edges that don’t actually apply to quad-edges – they apply to the directed edges that comprise quad-edges. Guibas & Stolfi call these “quad-edge refs,” but because that is confusing and clumsy I will call them “quarter-edges” from here on out.

The paper is hard to follow sometimes because it uses the term “edge” a lot, which could mean one of three different things:

A logical edge on a graph – that is, the normal sense of the word.
A “quad-edge,” which represents a single logical edge, and consists of four individual parts.
A “quarter-edge” (my term) or “quad-edge ref” (their term), which is one of the individual parts of a quad-edge.

Now here’s the trick: whenever they say “edge” or “quad-edge,” they actually mean “quarter-edge.” For example:

The second operator is denoted by Splice[a, b] and takes as parameters two edges a and b, returning no value.

Splice is actually an operation on quarter-edges. If you try to make sense of it as an operation on quad-edges, you will have a bad time and be very confused.

Anyway.

Digression over.

Guibas & Stolfi don’t prescribe a concrete representation, but here’s a straightforward way¹¹ to represent the quad-edge as they describe it:

typedef struct {
quarter_edge refs[4];
} quad_edge;
typedef struct {
quad_edge *my_quad_edge;
int my_index;
quad_edge *next_quad_edge;
int next_index;
void *data;
} quarter_edge;
quarter_edge *rot(quarter_edge *edge) {
return edge->my_quad_edge->refs[(edge->my_index + 1) % 4];
}
quarter_edge *sym(quarter_edge *edge) {
return edge->my_quad_edge->refs[(edge->my_index + 2) % 4];
}
quarter_edge *tor(quarter_edge *edge) {
return edge->my_quad_edge->refs[(edge->my_index + 3) % 4];
}
quarter_edge *next(quarter_edge *edge) {
return edge->next_quad_edge->refs[edge->next_index];
}

rot (“rotate”), sym (“symmetric edge”), and tor (“rotate the other way”) are functions that bring you to the other three quarter-edges on the same QuadEdge. next is a pointer that allows you to navigate around to adjacent QuadEdges in the structure. (We’ll come back to this in a minute.)

So you can see that a quad-edge isn’t really a thing; it’s just a bundle of four other things. And in fact I think it’s much simpler if we just pretend that quad-edges don’t exist, and define everything in terms of quarter edges:

typedef struct {
void *data;
quarter_edge *next;
quarter_edge *rot;
} quarter_edge;
quarter_edge *rot(quarter_edge *edge) {
return edge->rot;
}
quarter_edge *sym(quarter_edge *edge) {
return edge->rot->rot;
}
quarter_edge *tor(quarter_edge *edge) {
return edge->rot->rot->rot;
}

Whenever we create these, we can just create four of them at a time, and set their rot pointers in a cycle such that edge == edge->rot->rot->rot->rot. This representation is slightly less efficient, but it’s much simpler, especially when we start manipulating it.

Okay, with that out of the way: what is next?

next points to a quarter-edge that:

Has the same starting point (be it a vertex or a face) as this quarter-edge.
Lies immediately counter-clockwise of this quarter-edge.

This is all a little abstract, so let’s make it concrete. This is a little demo that lets you play with the quad-edge structure: the “current” edge is highlighted in yellow, and you can traverse the structure by pressing WASD (if you have a keyboard).

W sym
A tor
S next
D rot

Notice that the WAD keys will only move you between each of the four quarter-edges of the current quad-edge, while S will allow you to move to other edges.

(Notice that, because the outer face is “inside out,” pressing S while on the outer face will actually move clockwise around the perimeter.)

You can compose these operations predictably to navigate around. For example, to go to the “previous” edge that shares the same point – to navigate clockwise, in other words – press DSD. To navigate around the triangle to the left of the current edge, press ASD repeatedly. To navigate around the triangle to the right of the edge, press WS.

We’ll be doing some of these a lot, so we’ll give them names:

prev = rot.next.rot
lnext = tor.next.rot

The last thing that I didn’t explain is the data pointer.

The data pointer stores arbitrary information about the vertex or face at the beginning of each directed quarter-edge. For our purposes, we’ll use data to store the actual location in space of each vertex, and we won’t store anything on faces. So note that, since there are multiple directed edges out of each vertex, all quarter-edges that you reach by traversing the next pointers – by pressing S, in the demo – will have identical data pointers.

One thing that’s a little weird about this representation is that each “directed edge” doesn’t actually store anything about its destination. Its data pointer only contains its origin, and the destination of the edge is implicit from its sibling quarter-edges. Specifically, each quarter-edge represents a directed edge from edge.data to edge.sym.data.

We’ll refer to edge.sym.data a lot, so I’m going to give it a shorthand as well:

dest = sym.data

So: that’s it! That’s the structure.

How do we… use it?

Well first we need to be able to create quarter-edges – which we always do in groups of four:

function makeQuadEdge(start, end) {
const startEnd = new QuarterEdge();
const leftRight = new QuarterEdge();
const endStart = new QuarterEdge();
const rightLeft = new QuarterEdge();
startEnd.data = start;
endStart.data = end;
startEnd.rot = leftRight;
leftRight.rot = endStart;
endStart.rot = rightLeft;
rightLeft.rot = startEnd;
// normal edges are on different vertices,
 // and initially they are the only edges out
 // of each vertex
 startEnd.next = startEnd;
endStart.next = endStart;
// but dual edges share the same face,
 // so they point to one another
 leftRight.next = rightLeft;
rightLeft.next = leftRight;
return startEnd;
}

That represents a single edge that cuts through a single face – the face on the left and right of it are the same, and it exists on its own, connected to nothing. Guibas & Stolfi have a nice illustration of what this means topologically:

© 1985 ACM 0730-0301/85/0400-0074 — ACM Transactions on Graphics, Vol. 4, No. 2, April 1985
Reprinted by permission of the Association for Computing Machinery

Simple.

The next operation lets us connect edges together, or tear them apart. It’s called splice, and it’s probably just as simple. Let’s see how Guibas & Stolfi explain it:

© 1985 ACM 0730-0301/85/0400-0074 — ACM Transactions on Graphics, Vol. 4, No. 2, April 1985
Reprinted by permission of the Association for Computing Machinery

Aaaah! Oh dear. Topology is happening to us, and it doesn’t feel good.

But the implementation is so simple:

function splice(a, b) {
swapNexts(a.next.rot, b.next.rot);
swapNexts(a, b);
}
function swapNexts(a, b) {
const anext = a.next;
a.next = b.next;
b.next = anext;
}

We’re just swapping a couple of pointers – it seems like it should be simple to understand. But it’s very difficult to intuit the effect that this actually has on our mesh.

Let’s use it in a sentence. splice is a low-level primitive operation, and we’ll never actually call it directly in our algorithm. Instead, we’ll use it to build higher-level operations, and seeing how these operations use splice under the hood is probably the best way to gain intuition for how it works.

So the first thing we’ll need to do is construct a bounding triangle:

function makeTriangle(a, b, c) {
const ab = makeQuadEdge(a, b);
const bc = makeQuadEdge(b, c);
const ca = makeQuadEdge(c, a);
splice(ab.sym, bc);
splice(bc.sym, ca);
splice(ca.sym, ab);
return ab;
}

Then we’ll need to know how to add new edges:

function connect(a, b) {
const newEdge = makeQuadEdge(a.dest, b.data);
splice(newEdge, a.lnext);
splice(newEdge.sym, b);
return newEdge;
}

And how to remove edges:

function sever(edge) {
splice(edge, edge.prev);
splice(edge.sym, edge.sym.prev);
}

And lastly we’ll need to “flip” the diagonal edge of a quadrangle.

function flip(edge) {
const a = edge.prev;
const b = edge.sym.prev;
splice(edge, a);
splice(edge.sym, b);
splice(edge, a.lnext);
splice(edge.sym, b.lnext);
edge.data = a.dest;
edge.dest = b.dest;
}

And that’s it! Those are the only operations we need.

I think the easiest way to understand splice is to work through these high-level operations and convince yourself that this series of pointer updates gives you the QuarterEdge structure that you want. But also: you can just trust me. You aren’t actually trying to implement this.

are we done yet

Almost!

I can’t actually construct an infinitely large bounding triangle – is it okay if I just construct a triangle large enough to contain all of my points?
How am I supposed to find the triangle that contains my new point?
~~What if the new point lies on the edge of an existing triangle?~~
~~How am I supposed to do this circle test in the first place?~~
What edges do I actually need to check, once I’ve added my new edges?
~~Wait, how do I “flip” edges?~~
Wait, how do I even model edges? How do you even represent triangulations on a computer? I thought I could just model this as a graph, with vertices and edges and pointers, but it seems like I need to keep track of faces as well, and somehow this is a fundamentally different sort of thing than I’ve ever encountered before?

Now that you understand the quad-edge/quarter-edge structure, we can talk about (2). It’s clever! It’s not what you’re expecting!

triangle where

So in order to add a new point to your mesh, you first need to locate the triangle that contains that point.

There’s an easy, inefficient way to do this: loop through every triangle, check if your point is inside this triangle, and return as soon as you find one. We’re not going to do that, but it’s useful to talk about how to do it. Specifically: how do you determine if a point lies inside a triangle?

The trick we’re going to use is another linear algebra thing. If you could follow the point-in-circle test, this is going to be easy.

Given a triangle ABC, and another point P that you want to test, construct three new triangles: ABP, BCP, and CAP. Then calculate the sign of the oriented area of each of these new triangles. You can find the oriented area of a triangle ABC by taking the determinant of this matrix:


A_x	A_y	1
B_x	B_y	1
C_x	C_y	1

(That actually gives you the area of the parallelogram consisting of two copies of the triangle, but since we only care about the sign, we won’t bother to divide it by two.)

The sign of the oriented area tells you whether your triangle’s vertices are listed in clockwise (negative) or counter-clockwise (positive) order. If all three of ABP, BCP, and CAP are counter-clockwise, then the point is inside the triangle.¹²

Another, simpler interpretation: given a directed edge AB of a triangle, this test tells you if point P lies “to the left” of the line crossing AB (counter-clockwise) or “to the right” of it (clockwise).

If the point is “to the left of” every (directed, counter-clockwise) edge, then the point is inside the triangle.

but triangle where

So: now we know how to check if a point is inside a triangle. But! We actually get more than one bit of information when we do that test – we get three bits of information. Instead of ANDing them down to a single boolean value, we can use all of the information at our disposal to do something smarter.

So Guibas & Stolfi describe a technique that they credit to Green & Sibson:¹³ traverse the mesh one triangle at a time, always moving “towards” the target point, using the information we learned during the triangle test to decide what “towards” means.

If we find that the point is not “to the left of” one particular edge, then we know that it’s to the right of that edge. And since we know the whole mesh, we know exactly which triangle lies to the right of that edge, and we can check that triangle next.

The triangle immediately to the right of that edge may not contain our point either, but it’s going to be closer to the triangle that does, and if we repeat this enough we will eventually find a home for our point.

Here, let’s take a look at this in action. I omitted the locating step from my first visualization of the triangulation algorithm, but we can add it back in to see the full story:

This algorithm is nice because we probably won’t need to traverse the entire graph, but in the worst case we’ll still have to check every single triangle. Guibas & Stolfi claim that for a set of “reasonably uniform” random points, this search will only take O(√n) tests on average, and you can reduce that to roughly constant time if your newly inserted points are “close” to the previously inserted point. So it’s a bit data-dependent, but worst-case linear.¹⁴

is that it

We’re so close!

I can’t actually construct an infinitely large bounding triangle – is it okay if I just construct a triangle large enough to contain all of my points?
~~How am I supposed to find the triangle that contains my new point?~~
~~What if the new point lies on the edge of an existing triangle?~~
~~How am I supposed to do this circle test in the first place?~~
What edges do I actually need to check, once I’ve added my new edges?
~~Wait, how do I “flip” edges?~~
Wait, how do I even model edges? How do you even represent triangulations on a computer? I thought I could just model this as a graph, with vertices and edges and pointers, but it seems like I need to keep track of faces as well, and somehow this is a fundamentally different sort of thing than I’ve ever encountered before?

The last two questions are very specific to this particular algorithm, and the answers won’t really make you a smarter person. So if you aren’t actually trying to implement this algorithm, you can safely skip to the end.

Whenever you add a new point, and connect it to the points of its enclosing triangle or quadrangle, you’ll have a point at the middle of a spoke. Like this:

All of those newly created edges are already fine. We don’t have to check them. But we do have to check the edges adjacent to them – the edges that form a ring around the newly inserted point.

Whenever you flip an edge, you make this ring bigger, effectively adding two new segments to the ring.

You then have to check those new edges. But notice that they’re still a part of the same ring! And in fact as you flip these edges, they will add new segments to the ring as well.

So you don’t need to store a list of “dirty edges” or recurse to clean up all of these edges. You can just store a specific edge on the ring, check it, and then either move “forward” (if the edge passed) or “backwards” (if you had to flip it).

So the number of edges that you have to check depends on the number of triangles that have the newly inserted point as one of their corners. In the worst case this means means doing O(n) work, but on average across every point insertion this will net out to constant time, apparently, although I can’t find a proof of that.¹⁵ But we’re already doing worst-case O(n) work to find the triangle, so the total runtime of the algorithm remains O(n²).

Easy! Okay. Which brings us to the last question: what do we do about the “infinite bounding triangle?”

There is no triangle you can construct that will be “big enough” to pass for infinite when we do the point-in-circle test – as three points get closer to being colinear, the radius of the circle they define will approach infinity – so we just won’t do the point-in-circle test. Whenever a point on our boundary triangle is involved, we’ll instead check a series of conditions:

If the edge in question is a boundary edge, do not flip it. This one is easy so I’m not going to draw it for you.
If the edge contains a boundary vertex, flip it, unless doing so would create an inside-out triangle.

If an edge separates our newly inserted point from a boundary vertex, do not flip it.

That’s it! Those conditions will give you answers consistent with the answers you would get if your boundary vertices really were infinitely far from the rest of your points. Now all you need to do is to construct a triangle large enough to contain all of your points so that your “point in triangle” test works correctly, and you’re done. (If you don’t know all your points ahead of time, you can make any triangle and resize it as you learn points.)

are we there yet

Yes! We did it. That’s the algorithm. And the data structure. Two for the price of one.

Now that you understand how it works, in far more detail than you ever wanted to, let’s watch the visualization again. I’m excluding the boundary triangle this time so we can zoom in a bit and just bask in the triangular glow.

Well, that’s it. From here it’s a short, linear-time jaunt to computing the Voronoi diagram, the Gabriel graph, and probably some other interesting stuff. But that’ll have to wait for the next post.

Speaking of next posts, if you enjoyed this one, don’t forget to…

No. I can’t actually say it.

You might be wondering why you would want to construct a Voronoi diagram in the first place, to which I can only respond: why wouldn’t you? They’re so cool! ↩︎
I googled this phrase before I wrote this post to make sure that this actually is real slang. It seems that “smash that like button” is a more common phrase, but I don’t have a like button. Should I have a like button? Is that a thing that people would smash? What is… what are we doing here?

Here, if you really want to smash a like button, you can smash this one:

Did that feel good? Did you enjoy that? ↩︎
Actually, what you’re seeing here is not exactly the Gabriel graph – it’s the union of the Gabriel graph and the convex hull, so that the outline of the button is always solid. ↩︎
The Delaunay triangulation is not always unique: imagine four points arranged in a square. There are two ways to cut the diagonal, and either choice gives you a valid Delaunay triangulation. But it is only not unique in the case of co-circular points like that, and this randomly generated example doesn’t have any. ↩︎
In the same paper, Guibas & Stolfi describe another algorithm – a divide-and-conquer algorithm, kinda like merge-sort – that seems more complex, but is actually a lot simpler to code. Originally I wanted to describe both of these algorithms in this blog post, but, well, come on. This is so long already. ↩︎
Fun fact: this pattern of points is called a quincunx. Say it. Quincunx. say it out loud ↩︎
Since finding the distance between two points requires taking a square root, you actually compare the square of the distance against the square of the radius. But you knew that. ↩︎
Citation needed. The test as I presented it requires a division operation, while the determinant calculation only requires multiplication and addition. We can remove the division by subtracting A_z multiplying both sides of the inequality by N_z, but then we need to invert the logic when N_z is negative (which we can do without branching). But I mean… I neither profiled it nor tallied the individual operations, so I won’t try to argue that the plane test is actually faster. Also, like, numerical precision is a thing that computation geometry people worry about? Is the determinant solution more robust in the face of floating-point errors? I don’t know. ↩︎
More precisely, the dual is only defined for a particular planar embedding of a graph. There are lots of ways to draw a picture of a graph, and depending on how you draw it, you might end up with a different dual. ↩︎
The paper actually describes two versions of the data structure, first a general version, and then a simplified version that only works for orientable manifolds. What does that mean? I have no idea. I’ve never studied topology. Manifolds are a topology thing, right? Also sorry I know I said I wouldn’t say manifold and now I’ve said it a bunch. ↩︎
For a less straightforward representation, I was able to find some code on Stolfi’s university homepage that uses a more space-efficient representation. Paraphrasing slightly:
```
typedef char * quad_edge_ref;
typedef struct {
quad_edge_ref next[4];
void *data[4];
} quad_edge;
```
The idea here is that a quad_edge_ref is really a pointer to the interior of a quad_edge – anywhere from 0 to 3 bytes from the struct’s base address – and you use bitmasking operations on the pointer to identify my_index and to compute the actual base address:
```
// 32 bits ought to be enough for anybody
quad_edge_ref *rot(quad_edge_ref edge) {
return (edge & 0xfffffffcu) + ((edge + 1) & 3u);
}
quad_edge_ref next(quad_edge_ref edge) {
return ((quad_edge *) (edge & 0xfffffffcu))->next[edge & 3u];
}
```
So it only works if you know how your struct is aligned, and the size of your pointers, and… look maybe this sort of thing could fly in 1985 but I will speak no more of it here. ↩︎
Assuming the points of your triangle are defined in counter-clockwise order in the first place. If you think of triangles as being clockwise to begin with, then everything is backwards. Gosh winding order is confusing. ↩︎
“Computing Dirichlet tesselation in the plane,” 1977. ↩︎
There is an even more clever but much more complicated triangle traversal algorithm that works in logarithmic time by building up something called the “Delaunay tree.” It was first described by Boissonnat & Teillaud in their 1989 paper “On the randomized construction of the Delaunay tree.” It’s conceptually very simple, but I’m not going to explain it because just look at how long this blog post is already. There is a decent overview of it starting on page 46 of this PDF. ↩︎
I assume because you can calculate the average number of edges incident to each vertex and that is going to be some small finite number that doesn’t depend on the number of vertices. Like some kind of Euler’s formula for Delaunay triangulations. But I don’t know. I’m sure the proof is out there, somewhere, but I am too tired to look any further. ↩︎

zsh-autoquoter makes shell quoting slightly less annoying

Sun, 08 May 2022 00:00:00 +0000

Earlier this year I wrote zsh-autoquoter, a little zsh extension that automatically quotes arguments to commands of my choice.

(asciinema demo)

I’ve been using it for a few months now, and I’ve gotta say: it’s pretty nice! I wish I’d written this a long time ago. Here are a few use cases I’ve found for it so far:

1. commit messages

git has a neat feature where you can specify a commit message directly on the command line, with the -m flag:

$ git commit -m 'fix typo'

It’s really useful for repositories like this blog, where almost every commit is a short little memo like that. But sometimes I want to write something a little more complicated:

$ git commit -m "\"fire\" wasn't clear; rename to \"fire_the_missiles\""

Now, I like to commit “early and often,” using a patch queue workflow, and then rebase entire branches in order to get them ready for “publication” to the outside world. So I do this a lot. And it’s not that annoying to open up a vim buffer to write this, but it’s slightly more friction than just typing it directly.

2. `ssh` commands

Sometimes I want to run complicated shell commands on remote servers, with globs and pipelines and quoted arguments and all sorts of things, without actually starting up an interactive session on that server.

ssh makes this… possible. But I find it really difficult to wrangle all the nested quoting that this entails.

Here’s a simple example of what I mean:

$ ssh user@host "awk '{print $1}' file.txt"

See the problem? $1 is going to be expanded as a shell variable, and fortunately syntax highlighting makes the mistake obvious. (You do have syntax highlighting in your shell, right?)

To pass a literal $1 to awk, you actually have to write:

ssh user@host 'awk "{print $1}" file.txt'

Except, of course, that’s still wrong. You’ve just moved the problem to the remote server. You actually actually have to write something like:

$ ssh user@host "awk '{print \$1}' file.txt"

(Or is it \\$1?)

But with zsh-autoquoter, you don’t have to do any of that. You just type the obvious thing; the same thing you would type if you were running a local command:

$ ssh user@host awk '{print $1}' file.txt

And it will do the thing you want.

3. `todo` lists

Once upon a time I wrote a little command-line todo list app to help keep track of my stack at work.

todo 'write a blog post about zsh-autoquoter'

I liked the app; I liked the interface; I liked how simple it was. But it suffered from a simple ergonomic problem: English strings are really annoying to type on the command line.

todo 'rework dackery'\''s speech about the "hex machine"'

But now I can add a zsh-autoquoter rule for it, and give it another try.

4. `sd --new`

A neat feature of sd is the ability to quickly stash useful oneliners:

sd video poster --new 'ffmpeg -i "$1" -vframes 1 "${basename $1}-poster.png" -y'

That particular example happened to fit into single quotes. But I would really like to be able to stash arbitrary commands without worrying about how to escape them. Like this one, for example:

nix-env -qaA "nixpkgs.$1" --json | jq -r '.[] | .name + " " + .meta.description, "", (.meta.longDescription | rtrimstr("\n"))'

How am I supposed to quote that monstrosity? Well, with zsh-autoquoter, of course.

5. `note`s to self

For many years I have been taking notes in a simple text file, using a Mac GUI application called Alfred. I press a shortcut, Alfred pops up, and I type my note:

Using Alfred for this is a little silly, and I would like to replace this “workflow” with this tiny shell script:

$ cat ~/bin/note

#!/usr/bin/env bash

set -euo pipefail
printf '%s %s\n' $(date '+%Y-%m-%d') "$1" >> ~/Dropbox/notes.txt

But these notes can be long, free-from things. It would have been a nightmare to write something like this, for example:

2022-01-08 TIL about SSH "key types" and "signature types" being different. if
i want to use this key i have to configure sshd to accept ssh-rsa-sha2-256, not
just ssh-rsa. the error message says that ssh-rsa isn't supported, but it
actually means this particular signature type

But not anymore!

those are all of the examples

Tempted? Give it a spin! The extension should work with any zsh plugin manager, and the readme explains how to configure it and enable the (optional) zsh-syntax-highlighting integration.

Apparently you’re supposed to end posts like this with a prominent call to action, so go check out this crazy moth I found on Wikipedia.

A cozy nest for your scripts

Fri, 11 Mar 2022 00:00:00 +0000

sd is a tool for running scripts in your script directory.

$ tree ~/sd
/Users/ian/sd
├── blog
│ ├── edit
│ ├── preview
│ └── publish
├── book
│ ├── open
│ ├── words
│ └── typeset
├── nix
│ ├── diff
│ ├── info
│ ├── install
│ ├── shell
│ └── sync
└── video
└── fix

Directories become command groups; executables become subcommands. Instead of a flat ~/bin, you have a cozy nest for your scripts. And you can type:

$ sd blog publish --deploy

To run:

$ ~/sd/blog/publish --deploy

Which, okay, sure. That’s not that exciting.

The exciting thing is that sd supports fancy autocompletion. Take a look:

$ sd nix <TAB>
diff -- prints what will happen if you run sync
info -- <package> prints package description
install -- <package> use --latest to install from nixpkgs-unstable
shell -- add gcroots for shell.nix
sync -- make user environment match ~/dotfiles/user.nix

sd pulls those descriptions from comments in the scripts themselves:

$ sd nix info --cat

#!/usr/bin/env bash

# <package> prints package description
set -euo pipefail
nix-env -qaA "nixpkgs.$1" --json \
| jq -r '.[] | .name + " " + .meta.description,
 "",
 (.meta.longDescription | rtrimstr("\n"))'

So they’re very easy to write.

sd also makes it easy to create new scripts:

$ sd blog laud --new 'echo "good blog ian"'
$ sd blog laud
good blog ian

So you can easily stash oneliners, and you don’t have to remember to chmod +x. (You can also call --new without providing an initial script, to go straight into your text editor.)

And if you use zsh-autoquoter, you can add a rule that lets you just write:

sd plugin laud --new echo "good plugin ian"
^^^^^^^^^^^^^^^^^^^^^^

Which can be helpful for stashing complex commands. Here’s the rule I use:

ZAQ_PREFIXES+=('sd [^ ]* --new')

You can also modify the template for new scripts, on a per-directory basis. Although new scripts default to a fairly reasonable bash:

$ sd blog laud --edit

 1 #!/usr/bin/env bash
2
3 set -euo pipefail
4
5 echo 'good blog ian'
~
~
"~/sd/blog/laud" 5L, 61B

who cares

Take a peek at your ~/bin. How many scripts there have names like process or run or deploy? How many of them have you forgotten ever writing?

If the answer is “none of them,” sd might not be for you. It sounds like you’ve got this figured out already. Good work.

But sd is for script hoarders, who want to rage against the chaos of their ~/bin. And sd is for people who spend a lot of time combing through ctrl-r, looking for that oneliner that they never thought to make into a script, because it was too much of a hassle and who knows if they’ll ever run it again.

Give it a name! Give it a meaningful command path. Hoard more. The cozy nest is never crowded.

`sd`: a history

A year ago I published my first post about sd.

At the time, sd was a bundle of assorted shell scripts that I cobbled together one afternoon in 2018. I only “publicized” it all because I wanted to refer to it from another blog post I was writing – I didn’t even really launch it in any usable form; I just described the idea and linked to my dotfiles. I didn’t even bother giving it its own repo!

But even in that barely-baked form, the project got a small amount of traction. People reached out to me about it – people who actually wanted to use sd. Which gave me a very good excuse to buckle down and rewrite it, and I’m happy to announce that sd recently reached “1.0” status. It is officially a thing that exists that you can use, instead of “an idea I had that you can implement.”

And the thing that exists is much better than the thing I described a year ago. In a few ways:

First off, autocomplete is much faster than the original version. It used to run find, but now it uses some fancy shell globbing instead. There used to be the tinest delay after pressing tab, but now autocomplete feels instantenous. Which is how autocomplete should feel.

Second off, sd now works as a shell function in addition to working as a standalone executable – mostly because it’s very easy to install it and set up completion and everything with a shell plugin manager. sd is not officially packaged anywhere that I’m aware of, so this is a pretty convenient way to install it, if you already use a zsh plugin manager.

But the biggest change is an ergonomic one.

The original sd didn’t support any flags: the only thing it could do was run scripts in the ~/sd directory. It came with some “built-in” commands, like sd edit, but those were just scripts like any other: ~/sd/edit. This was elegant, I suppose, but I really just did it this way because it was easy to implement and I was lazy.

There were a lot of downsides to that approach. It made installation nontrivial, because you had to bootstrap your ~/sd directory with these “built-in” commands. There was no code sharing between these scripts – except by assuming that certain scripts existed in ~/sd, and failing horribly if they didn’t. There was no autocomplete for the built-in commands, so if you ran sd edit some command you had to type some command yourself. And when you were autocompleting a command at the top-level, you had to tab past all of these “built-in” scripts before you got to the command group that you wanted.

The right answer was clearly to support sd some command --edit, but I hesitated adding argument parsing for a while, because I didn’t like the idea of hijacking arguments. What if you wrote a script that expected a --help?

I got over this, eventually. Mostly because I have never written a script that used --help in my ~/sd. Writing a script that parsed its arguments would imply that I was writing something other than a quick bash script, which would imply that sd might not be the best place for it.

But I added an escape hatch just in case: you can run sd some command --help --really, and sd will execute ~/sd/some/command --help. (And yes, you can also run sd some command --help --really --really if your script expects a --really flag. Have you ever written a script that expects a --really flag? I want to hear about that. That sounds like a fun script.)

the future of `sd`

I dunno, it’s just a cute little program? It doesn’t have much of a future. Check it out. Nourish my fragile ego by lavishing stars upon the repo.