So we’re supposed to be writing a game, right? But in order to make progress, we have to fix a bug. And in order to fix the bug, we have to write a test. And in order to write a test, we have to write a test framework. And in order to write a test framework, we have to understand a thing or two about macros.

Okay.

So, caveat: I am not a lisp person. I didn’t know very much about macros going into all of this – and I still don’t. But when I started writing Judge I knew, like, only the absolute beginner basics. I knew when I needed to (gensym); I knew how to shoot myself in the foot.

And going into all of this, I thought macros were kinda dumb.

Macros are not dumb – let me get that out of the way up front. I have seen the light; I have been convinced. I misunderstood something very important about macros, and I was basing my whole mental dismissal on this one misunderstanding.

This is the story of writing Judge, so I’m going to tell you the story of this misunderstanding. So let’s start with the wrong thing that I believed, and then we’ll talk about a very simple macro that exemplifies this problem.

The wrong thing was this: macros are functions that transform code into other code.

This is not true; this is an incomplete statement. But – for the purposes of this story – we’re going to pretend that we believe it, and try to write a very simple macro operating under this misconception.

on raylib

So Raylib has this very imperative API. If you want to draw something to a texture, you write code like this:

(use jaylib)

(defn main-game-loop []
  (begin-texture-mode my-texture)
  (draw-a-circle)
  (draw-a-rectangle)
  (end-texture-mode))

But that’s weird and gross and hard to read. We would rather write something like:

(use jaylib)

(defn main-game-loop []
  (do-texture my-texture
    (draw-a-circle)
    (draw-a-rectangle)))

That way our syntax tree reflects the degree of nesting in our minds, and we can’t forget to call end-texture-mode or anything like that.

Okay, no problem. This is a very easy macro to write:

(defmacro do-texture [texture & forms]
  ~(do
    (begin-texture-mode ,texture)
    ,;forms
    (end-texture-mode)))

Remember that Janet has its own weird quasiquoting syntax; this is equivalent to the traditional:

(defmacro do-texture [texture & forms]
  `(do
    (begin-texture-mode ,texture)
    ,@forms
    (end-texture-mode)))

The problem is: that macro is wrong.

Or, more precisely, that macro is fragile. It is possible to use that macro in places where it works as intended, and it is possible to use that macro in places where it does not work at all.

The issue of course is begin-texture-mode and end-texture-mode: those functions will be looked up wherever they happen to be expanded. If you use this macro in an environment that redefines them, or that doesn’t have them in scope at all, the macro will not work as intended.

Now, I thought that this was just… par for the course. That unhygienic macros were sort of not referentially transparent; that you needed to understand the precise expansion, and understand the implicit contract of what-needs-to-be-in-scope in order to use them correctly, and there were various hacks such as package-qualifying symbols or having a separate function namespace or whatever in order to make them approximate referential transparency.

In this case, I wrote the do-texture macro, and I know what’s in scope when I use it, and I know how to use it correctly. It’s just a private helper; it’s allowed to make assumptions about how it’s used because it’s only used in the very file that it’s defined.

But even in my own code, this isn’t really going to cut it. do-texture is very useful, and I want to break it into its own module and use it from multiple places.

But there are two ways to import modules in Janet. You can either:

(use jaylib)

Or you can:

(import jaylib)

The former brings unqualified names into scope, like begin-texture-mode. The latter gives you qualified names, like jaylib/begin-texture-mode.

But what if I want to use my do-texture macro from a module in which I decided to (import jaylib)? Well, then you will see my folly laid bare, because the macro expands to refer to functions by their unqualified names.

But is there something I could do to make the macro work in either of these contexts?

functions from code to code

What I’d like to write is something like this:

(use jaylib)

(defmacro do-texture [texture & forms]
  ~(do
    (,begin-texture-mode ,texture)
    ,;forms
    (,end-texture-mode)))

That is, to unquote the functions themselves. So that instead of the macro expanding to the name of the function, the macro expands to the actual function looked up in the lexical scope of the macro definition at compile time, rather than looked up in the environment of its expansion at runtime.

But, of course, macros have to return code; macros are merely syntactic transformations; there’s no way we can refer by reference to this function, because–

Oh.

Oh, I see.

That just…

That just works.

What??

how does that work

So this wouldn’t be mind-blowing at all in an interpreted language, right? Your macro gets expanded, you returned the function, you execute the function. Everything takes place in the same process; you can refer to the function as a pointer and everything is fine.

But Janet is a compiled language. The macro is expanded, bytecode is generated, and then a few days later someone comes up and runs your binary. But the macro expansion refers to the function that existed at compile time, right? How does this new process know how to call that function that was defined during the compilation phase?

I mean, it’s not magic, obviously, but it’s still pretty neat. This reminds me of another programming concept that I’ve heard of but have no real experience with: the Smalltalk concept of “images.” It kinda feels like Janet is constructing an image of a process at compile time and then “resuming” it at runtime. But I’m not going to tug on that particular thread right now; I can already tell that this is going to be a long post.

So clearly my mental model of macros was woefully incomplete. Lisp macros are not just syntactic transformations. They’re transformations of this living, breathing thing, of this abstract syntax/value hybrid tree that somehow gets marshaled to and from our actual executable.

or are they

This got me thinking a lot about lisp macros, and about this problem of scoping. Because I’ve sort of thought about this problem before – I’ve thought about this problem in the abstract – but I never knew it had such a simple and intuitive solution.

In fact, I thought that the problem had complicated and unintuitive solutions. I even thought that the whole point of the Lisp-2 separate-function-namespace thing was to paper over this problem, to make it less likely, by saying “well sure there might be a local variable binding shadowing begin-texture-mode but there’s no way anyone would ever shadow it with another function binding.”

Why did I think this? I don’t know. It seems completely wrong, given what we just learned. But I can walk you through my logic:

Early lisps only had dynamic scoping. With dynamic scoping, macros and functions behave exactly the same.
But then people realized lexical scoping was a really good idea, so newer lisps implemented it – for functions.
And now there’s this weird split between macros (which essentially still use dynamic scoping) and functions (which use lexical scoping).

And I never followed through any further on that. I just… kept making assumptions, which made less and less sense:

It would be really annoying if your macro couldn’t call any functions – like list, for example – because they might be shadowed at the point they were expanded.
Let’s make that way less likely by inventing a second namespace so you can probably have all the functions you want still in scope.
This makes these function-scoping problems very unlikely to cause problems in practice so that’s good enough.

As far as I can tell this is not where Lisp-2s came from – it seems like it was actually a performance hack in order to make a dynamically typed language performant on the hardware of the early 1890s – but I now believe that this was a big reason why the separate function namespace stuck around into Common Lisp. But we’ll get to why I think that later.

Because at this point – returning to the story – I don’t know anything about macros.

So I set to read a bit more about them, fully expecting to find examples of unquoted functions, so I could learn a little bit more about this “more than just syntax” distinction, and see what other cool things I’ve been missing by dismissing macros all these years.

But it was very hard to find any references to this technique!

Most of the macro treatments I could find were written for, you know, complete beginners. They explained (gensym) and not much else. I could find plenty of examples of macros that exhibited the function-name problems – macros that I would call buggy macros – in introductory material. But it took a while to find any that addressed this specific function name problem.

In Let Over Lambda, Doug Hoyte briefly discusses the Lisp-1/Lisp-2 debate, and seems to relate it to macros in some way:

Common Lisp’s wise design decision to separate the variable namespace from the function namespace eliminates an entire dimension of unwanted variable capture problems.

But does it actually eliminate it, or just make it easier to ignore? He goes on to say:

More so than any other property except possibly its incomplete standard, it is this defect of a single namespace that makes Scheme, an otherwise excellent language, unfit for serious macro construction.

I don’t really understand these claims. And I want to add another strong caveat that I don’t know anything about lisp and might be completely misunderstanding this discussion.

Because you have the exact same problem of function capture in a Lisp-2 – it’s just more annoying to deal with it. The Lisp-2 version of this macro seems much worse to me:

(defmacro do-texture [texture & forms]
  `(do
    (funcall ,#'begin-texture-mode ,texture)
    ,@forms
    (funcall ,#'end-texture-mode)))

But that is a small quote from a long chapter; maybe Hoyte is talking about something entirely different. There are, unfortunately, no concrete examples in that section of the book, so I might be trying to project my own problems onto it.

So I kept reading, and looking around, mostly blindly. The first place where I actually, unambiguously, found a discussion about function name capture – as opposed to “variable” name capture – was in Paul Graham’s On Lisp. He devotes an entire section to the problem – section 9.8 on page 143 of the PDF (which is page 130 of the actual book).

He gives an example of function name capture, and then writes (emphasis mine):

What to do about this case? When the symbol at risk of capture is the name of a built-in function or macro, then it’s reasonable to do nothing. In [Common Lisp the Language, 2nd Edition] (p. 260) if the name of anything built-in is given a local function or macro binding, “the consequences are undefined.” So it wouldn’t matter what your macro did – anyone who rebinds built-in functions is going to have problems with more than just your macros.

Otherwise, you can protect function names against macro argument capture the same way you would protect variable names: by using gensyms as names for any functions given local definitions by the macro skeleton. Avoiding free symbol capture, as in the case above, is a bit more difficult. The way to protect variables against free symbol capture was to give them distinctly global names: e.g. *warnings* instead of w. This solution is not practical for functions, because there is no convention for distinguishing the names of global functions – most functions are global. If you’re concerned about a macro being called in an environment where a function it needs might be locally redefined, the best solution is probably to put your code in a distinct package.

And he discusses the “distinct package” approach elsewhere in the chapter, and talks about how annoying it is to do that. I am also amused by the caveat there: “If you’re concerned about a macro being called in an environment where a function it needs might be locally redefined” – how could I not be concerned about that?

But note that there is no mention of the “obvious” solution in this chapter; there is no example of just unquoting the function, as we saw above. Why not?

I had already, at this point, verified that this is something you can do in Common Lisp. This isn’t some Janet-specific wizardry. Yeah, you have to do the ugly funcall ,#' dance, but if you’re living in Common Lisp country you barely even notice that.

But maybe… maybe you couldn’t do this back then? On Lisp was published in 1993. Maybe… maybe Common Lisp macros were entirely syntactic in 1993? Is there a chance that this is some, like, recent development in the lisp world?

I realize I am grasping at straws here; I am looking for some sort of reasonable explanation for this complete misunderstanding, something that will help me understand Hoyte’s claim. Maybe when I first formed these opinions about Lisp, you couldn’t do this unquote-function thing?

Maybe! A lot has changed in the past few decades. I don’t know. Let’s try to learn more.

In “Quasiquotation in Lisp” (1999), Alan Bawden writes:

Any practical system for staged computation will probably have the property Taha and Sheard call “cross-stage persistence.” A system has cross-stage persistence if variables bound in earlier stages can still be referenced in later stages. […] One of the reasons our first version of make-list-proc failed was because Lisp’s quasiquote lacks cross-stage-persistence. […] With cross-stage persistence, variables in constructed code can reference variables from the lexically enclosing environment. In such a system the code objects returned by a quasiquotation must be more than a simple S-expression-like data structure; in order to capture the lexical environment where the quasiquote expression was written, code objects must be closures.

This is cool, but sadly we’ve overshot a little. This is one level above unquoting a function – this is actually referring to a deeper, weirder thing, of unquoting variables themselves. This might be a really cool thing and another useful superpower… but let’s try to just figure out unquoting values first, and then we’ll worry about closing over the compilation environment.

I spent some time reading various things about macros, but it was a while before I came across anything particularly relevant.

But I did learn some interesting things about Common Lisp during this lull. For example, I learned that it’s illegal to shadow certain functions, while I was trying to make a minimal demonstration of this problem:

$ cat example.lisp

(defun twice-function (x)
  (list x x))

(defmacro twice-macro (x)
  (let (($x (gensym)))
    `(let ((,$x ,x))
      (list ,$x ,$x))))

(print (twice-function "hey"))
(print (twice-macro "hey"))

So this works, right, but you can see the fragility. And it prints:

$ sbcl --script example.lisp
("hey" "hey")
("hey" "hey")

As you’d expect. But if you try to pull this:

(defun twice-function (x)
  (list x x))

(defmacro twice-macro (x)
  (let (($x (gensym)))
    `(let ((,$x ,x))
      (list ,$x ,$x))))

(print (twice-function "hey"))
(flet ((list (x) x))
  (print (twice-macro "hey")))

It has some words to say about that:

$ sbcl --script example.lisp
; file: /Users/ian/src/lisp/example.lisp
; in: FLET ((LIST (X) X))
;     (LIST (X) X)
;
; caught WARNING:
;   Compile-time package lock violation:
;     Lock on package COMMON-LISP violated when binding LIST as a local function
;     while in package COMMON-LISP-USER.
;   See also:
;     The SBCL Manual, Node "Package Locks"
;     The ANSI Standard, Section 11.1.2.1.2
;
; caught ERROR:
;   Lock on package COMMON-LISP violated when binding LIST as a local function
;   while in package COMMON-LISP-USER.
;   See also:
;     The SBCL Manual, Node "Package Locks"
;     The ANSI Standard, Section 11.1.2.1.2
;
; compilation unit finished
;   caught 1 ERROR condition
;   caught 1 WARNING condition
Unhandled SB-INT:COMPILED-PROGRAM-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                     {1001550143}>:
  Execution of a form compiled with errors.
Form:
  (FLET ((LIST (X)
         X))
  (PRINT (TWICE-MACRO "hey")))
Compile-time error:
  Lock on package COMMON-LISP violated when binding LIST as a local function
while in package COMMON-LISP-USER.
See also:
  The SBCL Manual, Node "Package Locks"
  The ANSI Standard, Section 11.1.2.1.2

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1001550143}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SB-INT:COMPILED-PROGRAM-ERROR {1001603BF3}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<SB-INT:COMPILED-PROGRAM-ERROR {1001603BF3}>)
2: (INVOKE-DEBUGGER #<SB-INT:COMPILED-PROGRAM-ERROR {1001603BF3}>)
3: (ERROR SB-INT:COMPILED-PROGRAM-ERROR :MESSAGE "Lock on package COMMON-LISP violated when binding LIST as a local function
while in package COMMON-LISP-USER.
See also:
  The SBCL Manual, Node \"Package Locks\"
  The ANSI Standard, Section 11.1.2.1.2" :SOURCE "(FLET ((LIST (X)
         X))
  (PRINT (TWICE-MACRO \"hey\")))")
4: ((LAMBDA NIL :IN "/Users/ian/src/lisp/example.lisp"))
5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (FLET ((LIST (X) X)) (PRINT (TWICE-MACRO "hey"))) #<NULL-LEXENV>)
6: (EVAL-TLF (FLET ((LIST (X) X)) (PRINT (TWICE-MACRO "hey"))) 3 NIL)
7: ((LABELS SB-FASL::EVAL-FORM :IN SB-INT:LOAD-AS-SOURCE) (FLET ((LIST (X) X)) (PRINT (TWICE-MACRO "hey"))) 3)
8: ((LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) (FLET ((LIST (X) X)) (PRINT (TWICE-MACRO "hey"))) :CURRENT-INDEX 3)
9: (SB-C::%DO-FORMS-FROM-INFO #<FUNCTION (LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) {10015603AB}> #<SB-C::SOURCE-INFO {1001560373}> SB-C::INPUT-ERROR-IN-LOAD)
10: (SB-INT:LOAD-AS-SOURCE #<SB-SYS:FD-STREAM for "file /Users/ian/src/lisp/example.lisp" {10008A0D13}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
11: ((FLET SB-FASL::THUNK :IN LOAD))
12: (SB-FASL::CALL-WITH-LOAD-BINDINGS #<FUNCTION (FLET SB-FASL::THUNK :IN LOAD) {B1C774B}> #<SB-SYS:FD-STREAM for "file /Users/ian/src/lisp/example.lisp" {10008A0D13}>)
13: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) #<SB-SYS:FD-STREAM for "file /Users/ian/src/lisp/example.lisp" {10008A0D13}> NIL)
14: (LOAD #<SB-SYS:FD-STREAM for "file /Users/ian/src/lisp/example.lisp" {10008A0D13}> :VERBOSE NIL :PRINT NIL :IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
15: ((FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) #<SB-SYS:FD-STREAM for "file /Users/ian/src/lisp/example.lisp" {10008A0D13}>)
16: ((FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT))
17: ((FLET "WITHOUT-INTERRUPTS-BODY-11" :IN SB-IMPL::PROCESS-SCRIPT))
18: (SB-IMPL::PROCESS-SCRIPT "example.lisp")
19: (SB-IMPL::TOPLEVEL-INIT)
20: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
21: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-IMPL::START-LISP))
22: (SB-IMPL::START-LISP)

unhandled condition in --disable-debugger mode, quitting
("hey" "hey")

Which is fantastic; very useful backtrace; no notes.

And why is this? Why can’t I shadow this function? My assumption would be: because then macro-writers would have to think about that, and worry about unquoting their functions. But that’s such a weird hack! And I would think it would make it more likely for macro-writers to slip up. If you only have to unquote some functions, it’s easy to forget – or not even think about this. If you have to unquote every function you use, then you think about it; function symbols stand out; you aren’t as likely to write fragile macros.

So I might be wrong; that is not a generous reading. I am seeing everything through the lens of this particular problem at the moment; perhaps there is a better reason to prevent programmers from using the names they want to use.

Anyway, you can demonstrate the issue like this instead:

$ cat example.lisp

(defun my-list (&rest args)
  (apply #'list args))

(defun twice-function (x)
  (my-list x x))

(defmacro twice-macro (x)
  (let (($x (gensym)))
    `(let ((,$x ,x))
      (my-list ,$x ,$x))))

(print (twice-function "hey"))
(flet ((my-list (&rest args) (car args)))
  (print (twice-macro "hey")))

$ sbcl --script example.lisp
("hey" "hey")
"hey"

And I suppose there is a pragmatic response to this, which is that I had to try to demonstrate this so-called issue, and I ended up writing kind of weird unnatural code to do so. And that’s fair. I’m not saying that macros can’t work or that there aren’t solutions to make this work. I’m just saying that the unquoting-a-function thing seems like a really nice solution, to the extent that I don’t see why you’d ever want another one.

I mean… maybe because you’re writing in a Lisp-2?

$ cat example.lisp

(defun my-list (&rest args)
  (apply #'list args))

(defun twice-function (x)
  (my-list x x))

(defmacro twice-macro (x)
  (let (($x (gensym)))
    `(let ((,$x ,x))
      (funcall ,#'my-list ,$x ,$x))))

(print (twice-function "hey"))
(flet ((my-list (&rest args) (car args)))
  (print (twice-macro "hey")))

$ sbcl --script example.lisp
("hey" "hey")
("hey" "hey")

That is gross, yeah. But so is marking some identifiers as “too important to shadow.”

Anyway, at this point I was wondering if maybe my assumption that Lisp-2s had something to do with macro writing was just plain wrong, in 2021. But maybe, historically, they were connected? Maybe by looking at this from the other direction, by trying to understand why Common Lisp is a Lisp-2, I could see once and for all if it had anything to do with that?

Fortunately, it turns out there is a paper just about this. It’s called “Technical Issues of Separation in Function Cells and Value Cells”, and it was written by Richard P. Gabriel and Kent M. Pitman (members of the committee to standardize Common Lisp) in order to document the reasoning and rationale behind the decision to make Common Lisp a Lisp-2.

It’s a fascinating read. And there is a section in this paper that talks about this exact problem.

13. Macros and Name Collisions.

Some contend that macros as they exist in Common Lisp have severe semantic difficulties. Macros expand into an expression that is composed of symbols that have no attached semantics. When substituted back into the program, a macro expansion could conceivably take on quite surprising meaning depending on the local environment.

Some symbols that ultimately appear in the expansion of a macro are obtained during macro definition through its parameter list from the macro consumer. It is possible to use those symbols safely. However, writers of macros often work on the hypothesis that additional functional variables may be referenced in macros as if they were globally constant. Consider the following macro definition for Lisp₁ or Lisp₂:
(DEFMACRO MAKE-FOO (THINGS) `(LIST 'FOO ,THINGS))
Here FOO is quoted, THINGS is taken from the parameter list for the Macro, but LIST is free. The writer of this macro definition is almost certainly assuming either that LIST is locally bound in the calling environment and is trying to refer to that locally bound name or that list is to be treated as constant and that the author of the code will not locally bind LIST. In practice, the latter assumption is almost always made.

Yes! Thank you. We’re here. We did it. Someone is going to explain this to us, once and for all.

They show an example of what could go wrong here; you get it. They show an example of a similar problem that affects a Lisp-2, which is similar to what I showed above, but written with that delightful all-caps flair:

(DEFMACRO FOO (X Y) `(CONS 'FOO (CONS ,X (CONS ,Y NIL))))
(DEFUN BAZ (X Y)
  (FLET ((CONS (X Y) (CONS Y X)))
    (FOO X Y)))

And then – drumroll please –

Although few implementations support its full generality in file compilation, a strict reading of the Common Lisp specification seems to imply that writing the following should be acceptable:
(DEFMACRO FOO (X Y) ;take deep breath
  `(FUNCALL ',#'CONS 'FOO (FUNCALL ',#'CONS ,X (FUNCALL ',#'CONS ,Y NIL))))
(DEFUN BAZ (X Y)
  (FLET ((CONS (X Y) (CONS Y X)))
    (FOO X Y)))
Here (BAZ 1 2) should evaluate to (FOO 1 2) just as everyone expected. Of course, FUNCALL is subject to the same problems as CONS was; either FUNCALL must be considered a constant function or a construct such as
('#<COMPILED-CODE CONS>  ...)
Should be specially understood by the compiler and interpreter.

Yes. We did it. I love it. I love the take deep breath comment; I love how he mentions that being a Lisp-2 has the added difficulty of “you’re totally screwed if someone rebinds funcall,” which I admit I assumed was just impossible. It would be like rebinding let; I think of it like a keyword, not a function. But of course it’s important to be precise here.

This paper was published in 1988, five years before On Lisp. But that opening sentence, emphasis mine – “Although few implementations support its full generality in file compilation, a strict reading of the Common Lisp specification seems to imply that writing the following should be acceptable” – makes me think that this is maybe cutting edge stuff, a theoretical thing, not something that lisp compilers at the time actually supported. But, like, it’s standardized now, right? Right??

The very next line is the hilarious:

Given all of this, the thoughtful reader might ask: Why do macros appear to work as often as they do?

I would have thought because that funcall sharp quote thing does work, right? But no; not back then. Their answer is more nuanced, and finally gives me some closure:

The answer seems to be based in history and statistics rather than in some theoretical foundation. In dialects preceding Common Lisp, such as MacLisp, it was fortunate that FLET, LABELS, MACROLET did not exist. Thus in these dialects there was an extremely high likelihood that the function bindings of identifiers in the macro expander’s environment would be compatible with the function bindings of the same identifiers in the program environment. This coupled with the fact that the only free references that most macro expansions tend to make are functional meant that writers of macros could guess enough information about how the expansion would be understood and could develop fairly reliable macro packages.

With the advent of FLET, LABELS, and MACROLET, the risk of conflict is considerably higher. The Scheme community, which has long had constructs with power equivalent to that of these forms, has not adopted a macro facility. This is partly because macros have generally seemed like a semantically empty concept to many of the Scheme designers.

Common Lisp programmers probably have little trouble because they are still programming in a MacLisp programming style, using forms like FLET and LABELS in limited ways. People who use FLET and LABELS heavily may well have learned Lisp with Scheme and do not use macros heavily, or they understand the issues outlined here and write macros carefully.

As time goes on, it should not be surprising to find users of Common Lisp macros reporting more name collision problems. A change from Lisp₂ to Lisp₁ semantics for identifiers would probably speed up the increase in these problems.

Okay. We did it. We’re done. We have our answer, sort of.

Most programmers simply do not experience the name collision problems mentioned in the previous section with the frequency that seems to be appropriate to the depth of the problem, largely because of the existence of a function namespace. FLET and LABELS define functions, and programmers treat function names more carefully than nonfunction names. Therefore, letting function names be freely referenced in a macro definition is not a big problem.

There are two ways to look at the arguments regarding macros and namespaces. The first is that a single namespace is of fundamental importance, and therefore macros are problematic. The second is that macros are fundamental, and therefore a single namespace is problematic.

Okay.

The decision to split out the function namespace pre-dates the ability to unquote functions like this and look them up lexically. And the ramifications that that decision had on ease of macro-writing were well understood. There were people making the argument that Lisp-2ness was important to writing macros at the time of the Common Lisp standardization. There was a time when macros actually were syntactic transformations.

There are other arguments in favor of a Lisp-2 explained in the paper – this is not the only reason you might want to have a separate function namespace – but my assumption that they were related was not entirely fabricated. The separate function namespace did – at the time! – make it easier to write macros.

We could continue following this thread. We could read about Scheme’s eventual decision to support “hygienic” macros; we might even be able to find direct historical evidence for my original lexical scope macro divide theory.

But I’m just trying to make a little game here.

And I am satisfied with this explanation.

Let’s come up for air.

making a game in janet, part 51: a brief overview of maclisp’s 1989 flet implementation

No no no that is a joke.

Armed with this newfound knowledge of how to write semantically meaningful macros, we’re going to return to Judge now, and talk about implementing the macros that make up Judge.

Which is quite trivial in comparison to the journey we just went on. That was the hard part. Any difficulty I had implementing Judge stemmed from this misconception, from not really understanding how macros worked, from not really getting how powerful it was to be able to pass values from compile time to runtime.

Because I actually wrote all of Judge before I understood any this.

Really! Look at this commit. What… what was I thinking?

Remember that example I showed in the very beginning? That simple, four line macro?

(defmacro do-texture [texture & forms]
  ~(do
    (begin-texture-mode ,texture)
    ,;forms
    (end-texture-mode)))

That wasn’t a dumbed down example for the sake of this blog post. That was the actual, real-live macro that made me realize how macros worked. After writing hundreds of lines of macros over the course of multiple days – after writing macros that write other macros, after writing macros that do custom macro expansions of their arguments – it was that one little macro that pushed me over the edge.

So what was I doing that whole time? Well, it’s pretty funny, and pretty embarrassing, so let’s take a look.

In order to implement “private functions” that my macros could call, I actually defined public functions with gensym’d names.

(def- register-test-sym (gensym))
(eval ~(defn ,register-test-sym [filename test-type name f expect-results]
  (array/push all-tests [filename test-type name f expect-results])))

And then I referred to those from my macros.

But what about the use/import ambiguity? I even tried to handle that, by adjusting the symbol names with a prefix:

(defn- get-symbol-prefix [sym]
  (def components (string/split "/" sym))
  (if (= (length components) 1)
    nil
    (in components 0)))

(defn- prefix-symbol [sym prefix]
  (if prefix
    (symbol prefix "/" sym)
    sym))

(defn- prefix-macro [macro-sym]
  (def symbol-prefix (get-symbol-prefix (in (dyn :macro-form) 0)))
  (prefix-symbol macro-sym symbol-prefix))

This is all hilarious in hindsight.

At the time I was doing this, I was of course thinking: this can’t be right. This is awful. There must be a better way. But I was so ingrained in this “macros are functions from syntax to syntax” mentality that it never even occurred to me that you could unquote functions.

I even thought: I could just not define helper functions here. I could just define giant, gnarly, horrible macro expansions that do everything inline, that I could barely write or debug. For some reason I thought this was a common approach to the problem, but I did not think it was appropriate: (test) expands to call a register-test function, and I thought it was gross to inline the body of that (fairly long) function every single time you define a test. (In retrospect – now that I actually understand more about the whole compile time vs. runtime distinction – there is no reason to expand to these function calls in the first place, but that’s a separate issue.)

So: don’t repeat my mistake. That was my main takeaway from all of this; that was the main reason I wanted to tell you about Judge. Macros are neat. Don’t define functions with gensym’d names. It’s completely unnecessary, and you will feel very silly about it afterwards.

There is more to say about Judge; there are more things that I learned, and more mistakes that I made. But I think that’s enough for now – we’ll talk about them in the next post.