Zachary W. Huang

Home Projects Blog Guides Resume

Why Common Lisp?

August 15, 2022

I think Common Lisp (CL for short) is a cool language. In my opinion, CL is probably the closest we’ll ever get to the “one true programming language”. Here are some of my thoughts.

Preface

I know, this post is basically a huge wall of text. Feel free to skim it, especially if you’re not familiar with a Lisp. However, if you think Common Lisp is something you might be interested in, then I would recommend jumping right in and trying to learn it. If you scroll all the way to the bottom, I’ve put together some resources which may be helpful in getting you started.

Disclaimer: I am not an expert.

Table of Contents

Macros

Note: If you are currently thinking about C/C++ macros, please forget about those as you read this section.

I would say that the “killer feature” of Common Lisp is macros. A Common Lisp macro allows you to programmatically generate code at compile time. What makes macros fit so well in with CL is the fact that it is extremely easy to write Lisp code in Lisp.

Common Lisp provides something called “quoting” which basically delays evaluation. Here’s an example of how quoting works:

(print (+ 2 2)) ;; prints out "4"

;; You can quote a term by putting a single quote in front of it
'(print (+ 2 2))  ;; represents lisp code that will print out "4"

(eval '(print (+ 2 2))) ;; prints out "4"

However, it is usually more useful to use a backquote instead, which allows for something called quasiquotation. A backquote works the same as a single quote, except you can “unquote” certain terms. Essentially, it is like a template string, except it’s “template code”.

(setf x 4)

`(print x)  ;; represents (print x)

`(print ,x) ;; represents (print 4)

The backquote forms the basis of almost all Common Lisp macros. Now, we can write code like:

(defmacro print3 (x)
  `(progn
    (print ,x)
    (print ,x)
    (print ,x)))

(print3 (+ 2 2)) ;; will be expanded into the following at compile time:

(progn
  (print (+ 2 2))
  (print (+ 2 2))
  (print (+ 2 2)))


;; We can also run arbitrary lisp code in macros
(defmacro do-reverse (list)
  (reverse list))

(do-reverse ("hello" print)) ;; prints hello

Of course, these are just simple examples - macros allow us to run arbitrary lisp code at compile time and generate code in any which way.

Macros can be used to control evaluation of certain terms, introduce new syntax, and more. Unlike C-style macros, they can be used to manipulate the abstract syntax tree of a program at compile time in Lisp, which is way more powerful than the C-preprocessor copy-and-paste style of macro.

To be honest, its hard to get across just how powerful macros are. Of course, they can be abused to write absolutely unmaintainable code, but using them in moderation can significantly enhance your programming experience. Macros allow you to create programmatic constructs that best fit your program, which is why some people say Lisp is a programmable programming language. In the most extreme case, you can use macros to implement a completely new programming language within Lisp that you can use in your code.

To really understand how far you can go with macros, I’d recommend Paul Graham’s On Lisp and Doug Hoyte’s Let Over Lambda. (Disclaimer: I’m not an expert on these books, especially not the later chapters, which I’ve only really skimmed)

Programs Within Programs

One of the consequences of macros is that you can write Lisp programs that generate Lisp programs. This has both advantages and disadvantages.

Loop

A macro that is included in the Common Lisp standard is loop.

Loop lets you write code like this:

;; prints "hi" forever
(loop
  (print "hi"))

and this:

;; generates a list of odd numbers less than 50
(loop for i from 0 to 50
  when (oddp i) collect i)

and this:

;; calculates statistics on a list of numbers
(loop for i in (list 1.2 6.3 3.4 8.77 3.2)
  summing i into total
  maximizing i into max
  minimizing i into min
  finally (return (list min max total)))

and this:

;; prints out each key and value in a hash table
(loop for key being the hash-keys in hash-table using (hash-value v)
  do (progn
       (print key)
       (print (gethash key hash-table))))

As you can see, loop can be a very powerful tool - it can iterate through ranges, lists, hash tables, and more. However, the drawback is that it is literally its own language - it has its own grammar and set of features that a Lisp coder will need to learn.

In fact, you can even implement entire algorithms using just loop. Below is an excerpt from MAGICL, a matrix library for Common Lisp. The function implemented is an algorithm designed to “calculate the eigenvalues and eigenvectors of a complex matrix using a routine that only works on real matrices” (source, code).

(defmethod eig-lisp ((m matrix/complex-double-float))
  (assert (square-matrix-p m))
  (multiple-value-bind (evals evecs)
      (eig-lisp (embed-complex m))
    (let ((dis-evecs (mapcar #'disembed-vector
                             (matrix-columns-as-vectors evecs))))
      (assert (cl:= (length evals) (length dis-evecs)))
      (loop :until (null evals)
            :for eval := (pop evals)
            :for evec := (pop dis-evecs)
            ;; Real eigenvalues come in pairs. Just take the first
            ;; one.
            :if (< (abs (imagpart eval)) *junk-tol*)
              :collect eval :into final-evals
              :and :collect evec :into final-evecs
              :and :do (progn (pop evals) (pop dis-evecs))
            :else
              ;; Vectors that "disembed" to zero are not true
              ;; eigenvectors. Skip them. (The conjugate will be one
              ;; though.)
              :if (not (zero-vector-p evec))
                :collect eval :into final-evals
                :and :collect evec :into final-evecs
            :finally (progn
                       ;; Check for the correct number of eigenvalues
                       ;; and eigenvectors.
                       (assert (cl:= (ncols m)
                                     (length final-evals)))
                       (assert (cl:= (ncols m)
                                     (length final-evecs)))
                       ;; Check for the correct dimension of
                       ;; eigenvectors,
                       (dolist (evec final-evecs)
                         (assert (cl:= (nrows m)
                                       (size evec))))
                       ;; Extreme sanity check to verify each
                       ;; eigenvalue and eigenvector are actually such
                       ;; by definition.
                       (loop :for eval :in final-evals
                             :for evec :in final-evecs
                             :for ecol := (vector->column-matrix evec)
                             :for m*v := (magicl:@ m ecol)
                             :for l*v := (scale ecol eval)
                             :for zero := (column-matrix->vector (.- m*v l*v))
                             :do (assert (zero-vector-p zero) () ()
                                         "Got an eigenvalue L and an eigenvector V such that L*V /= M.V"))
                       ;; Finally, return the purchase.
                       (return
                         (values
                          final-evals
                          (hstack (mapcar #'vector->column-matrix final-evecs)))))))))

Some might consider this an abuse of loop. Others might consider this a work of art.

Meanwhile, the format macro is used to generate strings that can be printed out (basically the Lisp equivalent of printf). However, it can also be used to write unreadable and un-lisp-like code.

;; Print something out generically
(format t "~S~%" (list 1 2 3 4 5)) ;; "(1 2 3 4 5)"

;; Float formatting - 3 decimal places
(format t "~3$~%" 3.141592) ;; "3.142"

;; Taken from Practical Common Lisp, chapter 18
(defparameter *english-list*
  "~{~#[~;~a~;~a and ~a~:;~@{~a~#[~;, and ~:;, ~]~}~]~}")

(format t *english-list* '())        ;; ""
(format t *english-list* '(1))       ;; "1"
(format t *english-list* '(1 2))     ;; "1 and 2"
(format t *english-list* '(1 2 3))   ;; "1, 2, and 3"
(format t *english-list* '(1 2 3 4)) ;; "1, 2, 3, and 4"

Like loop, format can be useful for a variety of tasks, but the readability vs power tradeoff is definitely something that has to be considered.

Dynamic

CL is a very dynamic language. This is very nice when it comes to prototyping and quickly iterating when designing a program. You don’t have to worry about giving functions or variables types (similarly to languages like Python and Ruby). In addition, you can get by using lists/pairs as your only “data structure”. Lists can be used as sets (use member, union, set-difference), maps (use push, remove, assoc), structures (use custom setter/getter functions, or let over lambda (aka closures)), and more. You can always replace lists with classes, vectors, hashmaps, etc later on when you need to.

Even though Lisp is very dynamic, this doesn’t mean that the type system is weak. There are many builtins for querying a variable’s type, such as null, numberp, stringp, listp, etc, and you can define your own custom types. In addition, implementations like SBCL have compile-time type checking. You can declare a variable or function’s type using declaim and declare, which can help the compiler optimize code. (You can also go full ML/Haskell route and use Coalton, a functional sub-language for Common Lisp). Not to mention, you can also declare inline, speed/safety, scoping, and a few other options which let the CL compiler produce better/faster code (either globally or locally).

As a result, CL lets you code quickly while also transitioning to a more performant and complex system when needed.

Source Code

But how readable is Common Lisp? Well, it depends. Like I mentioned before, abuse of macros and DSLs (or even excessive utility functions) can result in unreadable code. However, based on the code that I’ve read, I think that CL code is relatively understandable, albeit dense.

In smaller projects, documentation can be lacking, so code may be the only documentation. However, it is typically very easy to enter a Lisp codebase and find the exact function/macro that you are looking for. You can also make changes locally, and the code will automatically get recompiled when you load the library again. This is a fact that I appreciate a lot - I can easily dive into a library and debug within it if I need.

On the other hand, plenty of popular libraries have great documentation. In addition, Common Lisp provides a function called describe, which will show all of the information associated with a symbol, including function documentation. This is super useful, especially when working in the REPL.

Packaging/Building

The package system in Common Lisp is quite nice, and it’s quite different from those you might be used to. And like everything else in the Common Lisp standard, you can really adapt it to best fit your needs.

For now, try to forget what you know about code splitting/packaging (like C headers, Python import, JS require, etc). In Common Lisp, a “package” is just a namespace that contains symbols. You can choose which symbols are exported from the namespace along with what symbols are imported from other packages. Any functions or special variables you define within a package become included in it (though they’re not necessarily exposed to other packages).

A package is defined using the defpackage macro and entered using in-package.

(defpackage :my-package
  (use :cl)        ;; include symbols from standard library
  (export :main))  ;; export a symbol called "main"

;; enter the :my-package namespace
(in-package :my-package)

(defun main ()
  (format t "Hello, world!~%"))

However, you don’t have to define a package in the same file in which you use it. But first, let’s talk about how to handle a multi-file project.

Common Lisp provides the function load which can be used to “run” a lisp file.

;; in file1.lisp
(defun f (x) (* x x))

;; in file2.lisp
(load "file1.lisp")
(print (f 10)) ;; prints out "100"

You can see that load can be used as a sort of extremely basic import tool. However, in modern Common Lisp codebases, you won’t see load used at all (except in very specific edge cases, I would assume). Why? Well, manually managing dependencies between files using load would be an absolute pain.

Instead, people use a build system, the standard being ASDF (yes, this is a name conflict with the package manager asdf). Essentially, ASDF lets you specify dependencies between files and potentially other libraries. ASDF will handle the loading of lisp code in the correct order. That being said, ASDF doesn’t care about package style. The build system and the packaging system are orthogonal, which allows for a lot of flexibility in how you structure your project.

A common pattern is to have a single file called package.lisp which defines all of the packages in the project. Then, you can use in-package at the top of each file in order to place your code into the correct package. If you wish, you can split the code of a single package into multiple files, as long as they all start with a valid in-package declaration. To build properly, you just need to tell ASDF that every other file depends on package.lisp.

;; in package.lisp
(defpackage :my-package
  (use :cl)        ;; include symbols from standard library
  (export :foo :bar))  ;; export two symbols: foo and bar

;; in file1.lisp
(in-package :my-package)

(defun foo ()
  (format t "foo~%"))

;; in file2.lisp
(in-package :my-package)

(defun bar ()
  (format t "bar~%"))

Another more recent packaging style is for every single lisp file to define its own package (including its imports and exports). The name of each package is based on the file’s path, so the lisp file at ./utils/macros.lisp should define a package called <package-name>/utils/macros. This is more limiting, but it is done to strictly enforce a tree-like dependency graph and make sure that related code stays close together. Dependencies are then manually (or automatically) specified for ASDF.

;; in main.lisp
(defpackage :my-package
  (use :cl)
  (import-from :my-package/foo :foo) ;; import foo
  (export :main))  ;; export main
(in-package :my-package)

(defun main ()
  (foo))

;; in foo.lisp
(defpackage :my-package/foo
  (use :cl)
  (export :foo))  ;; export foo
(in-package :my-package/foo)

(defun foo ()
  (format t "foo~%"))

With all of this being said, the Common Lisp package system is flexible and useful. With packages, name conflicts are rare, since you typically don’t just throw everything into one namespace. You can always refer to specific functions/variables using the syntax <package>:<symbol>, which further reduces name conflict.

"Production-Ready"

Common Lisp is probably the only language in which having multiple implementations is a good thing. A popular CL implementation is Steel Bank Common Lisp (SBCL), which is mostly known for generating fast code and having good compile-time type checking. If you instead need smaller or more portable executables, you may want to use Embeddable Common Lisp (ECL) instead, which can compile to C and run on a wider collection of architectures. For JVM compatibility, Armed Bear Common Lisp (ABCL) exists. For seamless C++ interop, CLASP is recommended. CLISP is good for beginners (supposedly, I haven’t tried it). Clozure is also an option (no relation with Clojure). And finally, Allegro and LispWorks are commercial implementations (for your commercial needs, I guess).

As you can see, there are plenty of options that may fit your use case. Most of these have a long history - they aren’t just half-baked implementations written by hobbyists for fun.

Actually, let me go on a tangent real quick: Common Lisp is an old language. The first major implementations of Lisp came about in the 1960s. Yes, more than half a century ago. Languages like Python and Java were invented in the 90s. Lisp has had a 30 year lead in design, development, and standardization. Some widely used Common Lisp libraries (like bordeaux-threads) are literally older than modern programming languages like Rust. It is easy to imagine CL as an old, antiquated language (especially since websites relating to Common Lisp tend to look very outdated), but in reality, Common Lisp encapsulates practical language design elements thought up by some very smart programmers and refined over time.

In addition, Common Lisp compilers nowadays are very good, and it shouldn’t be too hard to produce code with speed within an order of magnitude of C (see benchmarks game). Not to mention, it’s extremely easy to interop with C using CFFI if you really need to.

When it comes to distributing executables, the easiest option (unfortunately) is to take the Electron approach and bundle the entire Lisp image up with the program, including the compiler, runtime, etc. This is definitely not great, but it does allow for cool runtime features - you basically have an entire Lisp implementation within your program. At the current moment, a “Hello World” executable produced with SBCL (without compression) is around 40 MB. But with ECL, I get a 35 KB executable. As with many things in Common Lisp, you’ll just have to choose the best option based on your use case.

Powerful

Especially when compared to Scheme, Common Lisp is seen as a practical, pragmatic option. I agree with this view. Common Lisp is very powerful, and it has an all-hands-on-deck standard library that includes:

  • CLOS - a powerful object-oriented mechanism with generic functions, multiple inheritance, and multiple dispatch.
  • a plethora of numeric types - fixnums (aka native signed/unsigned integers), rationals, floats, and complex numbers
  • a condition system that separates the act of raising errors, handling errors, and recovering from errors
  • arrays, hash tables, streams, pretty printing, file I/O, debugging tools (like trace and inspect), and more
  • a great ecosystem with battle-tested packages for concurrency/parallelism, FFI, web servers, etc

If you need to make something, Common Lisp will not limit you.

Cool Ideas

There are plenty of cool ideas in Common Lisp that can’t always be found in many other languages.

Places

In Common Lisp, there is a concept of a “place” - where a value can be located. For example, the first element of a list is found at (car list). Common Lisp provides a macro setf which can be used to set the value of any place. As a result, the syntax of getting a value is exactly the same as setting the value. For example, you can change the first element of a list by doing (setf (car list) 'something-else)

This isn’t limited to lists - the same idea works for hash tables, classes, and even the local environment! Not to mention, you can define custom places, such as for a new data structure (see: generalized variables, setf expanders).

;; hash tables
(setf (gethash key *hash-table*) 'new-value)

;; set an object's field
(setf (field-name object) 'new-value)

;; vectors/sequences
(setf (elt vector idx) 10)

;; variables
(setf (symbol-value 'x) 10)

;; functions
(setf (symbol-function 'f) (lambda (x) (* x x)))

Of course, this idea is not unique to Lisp, but it’s just something that’s nice to have.

Scoping

Common Lisp has lexical scoping for local variables, but it also has a global-ish scoping behavior called dynamic scope for “special variables”.

Lexical Scope

You’re probably well-versed with lexical scoping. It’s how locals are scoped in languages like Python, C, and JavaScript.

{
  let x = 2;    // begin x scope
  x++;          // end x scope
}
console.log(x)  // error, since x is no longer in scope here

Common Lisp lets you establish local variables using let.

(let ((x 10)
      (y 20))
  (print (list x y))) ;; (10 20)

(print x) ;; error, x is not defined
(print y) ;; error, y is not defined

Lexical scope also allows for closures (in combination with first class functions).

function counter() {
  let count = 0;
  return () => { count++; return count };
}

let c = counter()
console.log(c()) // 1
console.log(c()) // 2
console.log(c()) // 3

Similarly, in Common Lisp:

(defun counter ()
  (let ((count 0))
    (lambda () (incf count) count)))

(let ((c (counter)))
  (print (funcall c))  ;; 1
  (print (funcall c))  ;; 2
  (print (funcall c))) ;; 3

Great. But now, let’s see how Common Lisp differs from other languages.

Dynamic Scope

Common Lisp lets you define “special variables” using defparameter and/or defvar. On a superficial level, these seem like global variables. However, they differ in one important way: they can be rebound dynamically. What does this mean?

(defvar *x* 10) ;; it's standard practice to name special variables with astericks

(defun foo ()
  (print *x*))

(foo) ;; prints out 10, as expected

(let ((*x* 20))
  (foo)) ;; prints out 20, since we gave *x* a new value 

(foo) ;; prints out 10 again

Basically, what is happening is that every time a special variable is (re)bound using let, we are giving it a new value within the context of the code being run within that let block. For special bindings, we are essentially creating new variables on a stack - when we bind a special variable in a let, we are putting a new value on the top of that stack. When the let ends, the top value of the stack is popped. Meanwhile, every time the special variable is accessed, the top value of the stack is returned. So in essence, dynamic variables are just global variables that can be shadowed.

Let’s see how this can be used in practice.

;; The CL standard library exposes *standard-input* and *standard-output*
;; These special variables hold streams that can be read from or written to
;; We can rebind these in order to redirect the output of, for example, #'print

(print 10) ;; prints out 10

;; create a new output stream
(let ((stream (make-string-output-stream)))

  ;; redirect *standard-output* to stream
  (let ((*standard-output* stream))
    (print 10)) ;; prints nothing - "10" is redirected to "stream"

  ;; prints out the contents of "stream" (aka "10")
  (print (get-output-stream-string stream)))

I’ve found special variables to especially useful when used like this. They’re basically global variables but without the spaghetti code (though they can be abused if you really try). In general, special variables let you decide whether you’d rather hold information in the dynamic environment or pass it into code as an explicit argument.

Sidenote: I think the closest analog to dynamic scope in other languages is how this works in JavaScript. You can call any method on any object, letting you change the identity of this at runtime - it’s like doing (let ((*this* caller)) (method)) in Common Lisp every time you do caller.method() in JavaScript. You can also call a method in JavaScript without providing an object, which is how the dreaded this is undefined error can arise.

Multiple Dispatch

Multiple Dispatch is a feature of the Common Lisp Object System (CLOS) which allows you to dispatch generic methods based on multiple types.

In contrast, most OOP languages feature single-dispatch. For example, if you have something like object.method(), languages like Java and Python will dispatch the method call method() based on the type of object. The language will look into the class of object and check for a method with matching name and signature.

But with multiple dispatch, the method lookup can depend on multiple objects. This model makes more sense with generic functions such as add(), which don’t naturally belong in a single class. In a single dispatch system, we would have to arbitrarily say that add is defined on a type, like Integer. But in a multiple dispatch system, functions are “free-floating”, and we can define a function body for any given combination of argument types. For example, we could define a method with the signature add(float, float) to add floats and another method add(complex, complex) in order to add complex numbers. When we call the function add, the language will dispatch to the correct method.

Multiple dispatch is really powerful, and it’s a key feature in the language Julia, which has been gaining popularity in the scientific computing space. I’d recommend this video (made by one of the creators of Julia) which explains why multiple dispatch is so great.

Some Last Words

To be honest, I still think I’ve only really scratched the surface of Common Lisp with this post. There’s a reason why people have written books hundreds of pages long on only specific aspects of Common Lisp, such as CLOS or the condition system.

Basically, I feel like learning Common Lisp is a worthwhile journey. To me, it feels like mastering Common Lisp (if that’s even possible) would let me conquer the world.

In the past, I had a similar feeling with C++. However, learning about advanced C++ features really only ever pushed me away from the language (big disclaimer: I have very little knowledge of C++). I just don’t think that learning about lvalues vs rvalues, copy/move semantics, smart pointers, etc will help me become a better programmer or create new things.

For Common Lisp, the opposite is the case. Writing macros, handling conditions/restarts, using CLOS, etc, all seem practical and useful. Sure, I may never use Common Lisp professionally, but it really is just a different world to program in.

I’ll leave you with a fun quote. Greenspun’s Tenth Rule: “Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.” My interpretation of this quote is that Common Lisp occupies a sort of local maxima in terms of abstraction and expressive power, and its easy to reinvent the complex features that already exist in Common Lisp when working with languages like C++. So in my mind, you might as well just start in Common Lisp and drill down if you need extreme control over memory usage and/or performance.

Common Lisp Resources

Here are Common Lisp resources I've found that I've read or plan to read.

Books:

  • Practical Common Lisp by Peter Seibel: a great introduction to Common Lisp that teaches you how to build some very practical pieces of software
  • On Lisp by Paul Graham: a classic that demonstrates many of the foundational aspects of Lisp and teaches some advanced macro techniques
  • Let Over Lambda by Doug Hoyte: a book that demonstrates just how far you can go with macros.
    It also expresses some slightly controversial opinions with slightly condescending language (but don’t let that put you off - just read it).
  • Paradigms of Artificial Intelligence Programming (PAIP) by Peter Norvig: a great book that teaches you Lisp programming by implementing late-20th-century AI techniques (aka non-ML algorithms). This book (and the code in it) is very well written.
  • Object Oriented Programming in Common Lisp by Sonya Keene: See title.
  • The Common Lisp Condition System by Michal Herda: See title.
  • Lisp in Small Pieces by Christian Queinnec: Teaches you Common Lisp by implementing a Scheme.
    You then iterate on the program to add more and more features, and by the end, you’ll have an entire C compiler for your Scheme

Websites & Articles:

References (as in a language reference, not a bibliography):

  • Common Lisp The Language 2 (CLTL2): a full, readable Common Lisp spec
  • Common Lisp Hyperspec: My reference of choice every time I need to look up how to do something in Common Lisp. I would recommend downloading a copy and bookmarking it so you can access the pages faster (you can get it using Quicklisp or from your favorite package manager).
RSS icon github logo linkedin logo

Zachary W. Huang © 2021-2024