Why I haven't jumped ship from Common Lisp to Racket (just yet)

[fare]

Matthias Felleisen jested "Why are you still using CL when Scrbl/Racket is so much better :-)" ? My response was as follows:

Dear Matthias,

you are right Racket is so much better in so many dimensions. I use Lisp because I just can't bear programming in a language without proper syntactic abstraction, and that is a dimension where Racket is far ahead of Common Lisp (CL), which sadly also remains far ahead of the rest of the competition. Racket also has managed to grow a remarkable way to mix typed and untyped program fragments, which sets it ahead of most. But I am under the impression that there are still many dimensions in which Racket lags behind other languages in general and Common Lisp (CL) in particular.

1. The Common Lisp Object System (CLOS) has multiple-inheritance, multi-methods, method combinations, introspection and extensibility via the MOP, generic functions that work on builtin classes, support for dynamic instance class change (change-class, update-instance-for-changed-class) and class redefinition (defclass, update-instance-for-redefined-class), a semi-decent story for combining parametric polymorphism and ad hoc polymorphism (my own lisp-interface-library), etc. Racket seems to still be playing catch-up with respect to ad hoc polymorphism, and is lacking a set of good data structure libraries that take advantage of both functional and object-oriented programming (a good target is Scala's scalaz or its rival cats).
2. While the ubiquity of global side-effects in CL is very bad, the facts that all objects that matter are addressable by a path from some global namespace and that live redefinition is actively supported makes debugging and maintaining long-lived systems with in-image persistent data more doable (see again CLOS's update-instance-for-redefined-class). This is in contrast with the Racket IDE which (at least by default) drops live data when you recompile the code, which is fine for student exercises, but probably wrong for live systems. CL is one of the few languages that takes long-term data seriously (though not quite as seriously as Erlang).
3. Libraries. CL seems to have much more libraries than Racket, and though the quality varies, these libraries seem to often have more feature coverage and more focus on production quality. From a cursory look, Racket libraries seem to be more ambitious in their concepts, but to often stop at "good enough for demo" in their practice. An effort on curating libraries, homogenizing namespaces, etc., could also help Racket (I consider CL rather bad in this respect, yet Racket seems worse). My recent experience with acmart, my first maintained Racket library, makes me think that writing libraries is even higher overhead in Racket than in CL, which is already mediocre.
4. Speedwise, SBCL still produces code that runs noticeably faster than Racket (as long as you don't need full delimited control, which would requires a much slower CL-to-CL compiler like hu.dwim.delico). This difference may be reduced (or even reversed) as Racket adopts the notoriously fast Chez Scheme as a backend (or then again not). Actually, the announcement of the new Racket backend really makes me eager to jump ship.
5. As for startup latency, Common Lisp is also pretty good with its saved images (they start in tens of milliseconds on my laptop), making it practical to write trivial utilities for interactive use from the shell command-line with an "instantaneous" feel. Racket takes hundreds of milliseconds at startup which puts it (barely) in the "noticeable delay" category (though nowhere near as bad as anything JVM-based).

All these reasons, in addition to inertia (and a non-negligible code-base and mind-base), have made me stick to CL - for now. I think Racket is the future of Lisp (at least for me), I just haven't jumped ship right yet. If and when I do, I'll probably be working on some of these issues.

PS (still 2017-03): Here are ways that Racket is indeed vastly superior to CL, that make me believe it's the future of Lisp:

• First and foremost, Racket keeps evolving, and not just "above" the base language, but importantly below. This alone makes it vastly superior to CL (that has evolved tremendously "above" its base abstractions, but hasn't evolved "below", except for FFI purpose, in the last 20 years), which itself remains superior to most languages (that tend to not evolve much "above", and not at all "below" their base abstractions).
• Racket is by far ahead of the pack in terms of Syntactic abstraction. It is the best language in which to define other languages and experiment with them, bar none.
• Racket has a decent module system, including build and phase separation (even separate phases for testing, cross-compilation or whatever you want), and symbol selection and renaming.
• Racket has typed modules, and a good interface between typed and untyped modules. While types in Racket do not compete with those of say Haskell, just yet, they are still evolving, fast, and that contract interface between typed and untyped is far ahead of anything the competition has.
• Racket has lots of great teaching material.
• Racket has a one-stop-shop for documentation, though it isn't always easy to navigate and often lack examples. That still puts it far ahead of CL and a lot of languages.
• Racket provides purity by default, with a decent set of pure as well as stateful data structures.
• Racket has many primitives for concurrency, virtualization, sandboxing.
• Racket has standard primitives for laziness, pattern-matching, etc.
• Racket has a standard, portable, gui.
• Racket has a lively, healthy, user and developer community.

I probably forget more.

PS (2017-08-23): A few months onward, I've mostly jumped ship from Common Lisp... but not to Racket, and instead to Gerbil Scheme.

As ASDF 3.3.0 gets released (imminently), I don't intend to code much more in Common Lisp, except to minimally maintain my existing code base until it gets replaced by Gerbil programs (if ever). (There's also a syntax-control branch of ASDF I'd like to update and merge someday, but it's been sitting for 3 years already and can wait longer.)

What is Gerbil? Gerbil Scheme started as an actor system that vyzo wrote over 10 years ago at MIT, that once ran on top of PLT Scheme. vyzo was dissatisfied with some aspects of PLT Scheme (now Racket), notably regarding performance for low-level system code and concurrency (at the time at least), but loved the module system (for good reasons), so when he eventually jumped ship to Gambit (that had great performance and was good for system programming, with its C backend), he of course first reimplemented the PLT module system on top of Gambit, or at least the essential features of it. (The two module systems were never fully compatible, and have diverged since, but they remain conceptually close, and I suppose if and when the need arise, Gerbil could be made to converge towards PLT in terms of features and/or semantics.)

Why did I choose Gerbil instead of Racket, like I intended?

1. A big reason why I did is that I have a great rapport with the author, vyzo, a like mind whom I befriended back in those days. A lot of our concerns and sense of aesthetics are very much in synch, and that matters both for what there is and what may come to be. Conversely, the bigger features that Racket has that Gerbil is lacking (e.g. a GUI) are those that matter less to me at this point.
2. What there is, the module system, the actor system, the object system, the libraries, is far from as complete as I could wish, but it is all in good taste, and with the promise that they can be molded to what we both want in the future.
3. While the code base is smaller than in PLT, it is also more consistent and with a coherent sense of aesthetics, being implemented by one man (so far). It also happens to cover the kind of domains for which I'm most in need of libraries, and it also has a bias towards industrial applicability that you can't expect from PLT and its legion of academics and interns (see my discussion of PLT above).
4. Sitting on top of Gambit does not just mean relatively efficient code (as far as Scheme is concerned), but it also means enjoying the portability of its GVM, and some of these properties are especially interesting to me: its observability.

Observability is the property (whose name I coined in my PhD thesis) whereby you can interrupt the execution of the program and observe it at the level of abstraction of your language (in this case, the GVM). This already allows Gambit to migrate processes from one machine to the other, even though the machines may be using completely different backends (C on ia32, C on AA64, JS, PHP, Java, etc.) For my thesis, I want to generalize observability from the GVM to arbitrary virtual machines written on top of it for arbitrary languages, with plenty of cool implications (including e.g. Erlang-style robustness; see said thesis). Working on the GVM will save me having to care about plenty of CPU- or VM- dependent backends that I would have to deal with if I wanted to write a compiler from scratch or reuse an existing one. I notably tried to read the source of Chez Scheme, that PLT Racket is adopting as a new backend (moving from its own horribly complex yet ultimately slow C codebase); but it is largely inscrutable, and with its tens of backends would require much more work before a prototype is achieved.

I therefore have a lot of personal reasons to adopt Gerbil. I understand that Gerbil in its current state, is no rival to either Racket or Common Lisp (or Haskell or OCaml, or even blub languages), for most people in most situations. Yet there are probably other people for whom it is a better fit, as it is for me; and I invite these people to come join forces and enjoy writing in Gerbil. It's still barebones in many ways, yet already quite a pleasure to work with, at least for me.