Erlang-in-Lisp project proposal

[fare]

You have till March 31 to apply for funding for the Google Summer of Code. Here's a project I'd like to mentor. Apply at Google, thanks to the LispNYC. (If you're interested but not eligible for the Google SoC, please contact me directly.)
Erlang-in-Lisp

I am really missing a robust higher-order message-passing process algebra for Common-Lisp. The challenge is: can you do as well as Erlang? And the proof of the pudding is: can you actually run Erlang code?

The previous attempts failed to produce a complete workable solution:

• cl-muproc - full erlang-style programming, not stable on all platforms, intraprocess threads only
• gbbopen - distributed blackboard (linda-style?) rather than erlang-style, works stably on many platforms (requires threads)
• philip-jose - server spawning clients via ssh, server has transactional threads via green continuations (could notionally have one server per system thread).
• NetCLOS - implements actors on top of CLOS, allegro 5 only
• ABCL/R - implements actors, applicability to issue at stake unknown.
• ParGCL - GCL only, uses MPI that allows for fancy message passing in a restricted managed context.
• Erlisp - stalled in early stage.

And then of course, you can find inspiration in Termite, Dreme, Kali, Tube, things by Queinnec & co, etc. (see the readscheme bibliography on Distributed, Parallel, and Concurrent Programming.

As far as I know, all these systems assume a trusted network and are otherwise insecure. A better system would route messages via SSL. Nah. A better system would allow to plug in new methods for exchanging messages onto the IOlib event loop. Thus project would thus be:

• develop a few low-level message-passing handlers on top of the IOlib event loop; to be supported:
  • a simple protocol on top of an arbitrary stream (TCP socket, unix pipe, SSL).
  • unix sockets, with their fd-passing goodness
  • a protocol on top of HTTP/HTTPS requests
  • a protocol on top of UDP (with or without a layer for managing timeouts or message acknowledgment)
• if needed, generalize an API to plug-in message handlers on top the IOlib event loop
• on top of low-level message-passing and/or mixed in with it, implement
  • some trivial protocol for messaging for testing (take or adapt it from philip-jose)
  • some easy text protocol that precisely and safely implements Erlang semantics
  • possibly a compatible implementation of the Erlang wire protocol
• develop robust fork-based concurrency
  • fork and manage children the right way, don't try broken SIGCHLD handling, use waitpid
  • implement a portable run-program with every feature a shell spawner may have, from fd redirection to setsid - use it to spawn local and remote processes
  • implement a protocol for talking to children using point-to-point unix sockets.
• develop protocols for routing messages between processes
  • handle broken connections and (in the abstract case) reopening connection
  • handle brokers, directories, name service (to support the abstract case)
• manage processes
  • know how to probe a process status, kill a process (see philip-jose)
  • learn about load-balancing, process niceness, processor affinity, etc.
• support multiple concurrency models
  • adapt cl-muproc to allow for Erlanging using native threads concurrency.
  • adapt philip-jose to allow for Erlanging using green threads.
  • in either case, take special care for proper handling of asynchronous interrupts, as they actually require clever synchronization under the hood.
• Actually implement Erlang on top of this Lisp system
  • implement enough of the Erlang semantics to run simple examples
  • implement a parser front end for Erlang syntax
  • implement enough of Erlang syntax semantics and basic libraries to run standard libraries, test suites and benchmarks
  • make sure we can interoperate with existing Erlang implementations
• Support additional programming styles on top of the previous framework
  • implement mobility in the style of the Tube
  • port some variant of NetCLOS or ABCL/R2

There's more than enough work for two interns, maybe three. Some of the tasks can be started in parallel, others depend on previous tasks, so three is a stretch and more is probably too much. One intern could focus on the messaging protocol while the other one would focus on the concurrency. Mr Concurrency would start with fork, run-program, and if the other one is late or fails to deliver, would continue implementing a CLSH variant on the SCSH pattern. Mr Messaging would implement various messaging and if the other one is late or fails to deliver, would continue implementing various protocols to interoperate with existing software. A hypothetical third intern would start from the other end, parsing Erlang, transforming it into forms runnable in Termite or some such project, identifying what primitives are needed, maybe trying to build a bridge between C modules for Erlang and Lisp, possibly working on interfacing with Erlang internals. Hopefully, all interns would deliver in a timely fashion and cooperate towards a full-fledged Erlang-in-Lisp implementation.

Obviously there isn't enough time to implement and fully debug everything in the list above; depending on the intern's preferences, more time could be spent perfecting some of the previous features, or they could be left done just well-enough to go on the next features.

Yes, I would eagerly mentor such a project. Actually, I may even fund such a project if there are more good applicants found than are accepted by Google.

Update: Matt Bone did it. At least, a good start. Congrats!