Have you looks at Lisp-Flavored Erlang ("LFE")? - http://lfe.io
It's really quite interesting, IMO. --Scott On Mon, Aug 3, 2015 at 11:12 AM, <shenanig...@sonic.net> wrote: > Creators of Erlang have a Lisp background, and one feature of the Erlang > VM (BEAM) that I'd like back-ported into Common Lisp is their process. > > An Erlang "process" is cheap to create, cheap to destroy, cheap when > blocked, and upon exit performs bulk gc of its allocated memory; e.g., > munmap(). > > Handling tens of thousands of requests per second per node isn't uncommon, > and these often have *several* workers per request or connection: hundreds > of thousands of processes. Under such scenarios, anything less than this > approach to lightweight processes might suffer from stalls during long gc > runs that would be avoided or significantly reduced under Erlang's model. > > > How might we get equivalent cheap ephemeral processes into a contemporary > Common Lisp implementation? > > > For those unfamiliar with it, what Erlang means by "process" stems from an > implementation of Actor Model with caveats. Each process is more > co-routine than thread, and many may run within the same OS thread managed > by the Erlang scheduler. The BEAM VM pre-empts processes via "reduction" > counting, which may be understood as unit of VM time. Their famed tag > line, "Let it crash," may be loosely understood in CL terms as an implicit > HANDLER-CASE. > > The open question here is to address a stated non-goal of CL-MUPROC, "we > rely on the CL implementation's MP system" and "considerably heavier than > Erlang processes". [See presentation link from > https://common-lisp.net/project/cl-muproc/ ] > > Some Erlang-on-CL packages use OS threads or in the case of > Erlang-in-Lisp, fork(). While easier for a library author to implement, > these contradict the definition of cheap here. Other such packages punt > the issue altogether and instead focus only on pattern-matching aspects of > Erlang the language. > > Then there's Lisp-Flavoured-Erlang, and Elixir offers hygienic macros that > manipulate the full AST properly-- all play nice on same Erlang VM at > runtime-- for people simply looking to avoid Erlang syntax or semantics. > But let's focus on cheap Erlang style processes in Common Lisp here, please. > > > 1. Semantics of library elements are straight-forward enough (e.g., SPAWN, > EXIT) and may be borrowed wholesale and safely named within their own > package. > > 2. Memory allocation & garbage collection: > > Erlang BEAM VM doesn't begin to reclaim memory until very late, so an > initial implementation here might assume ephemeral worker processes and > omit gc until exit of a worker. However, some processes are long-lived in > practice. > > One compromise might be acceptable: one nursery per process, but anything > promoted to higher generations gets handled however your gc does it now. > > This states nothing about use of shared versus multiple heaps across OS > threads, so such matters may continue to be implementation-dependent. > > 3. Co-routines: > > For something similar to Erlang's reductions, there would need to be a > measure of runtime complexity per process. > > However, I've used single thread co-routines for near realtime systems in > C and CL with a loop of function pointers and ensuring that each referenced > function executes under some threshold determined through experimentation > and confirmed via test cases. No pre-empting needed. While fragile and > not easily portable across different hardware (including same > architecture), this may be acceptable for a preliminary draft. > > Using CL-MUPROC routines as an example and extending MUPROCN: perhaps its > body becomes this top-level list of functions from which interleaving > across processes may occur. Then add variations for playing well with DO or > LOOP semantics. > > 4. Message-passing: > > SBCL's sb-concurrency extension with Queue and Mailbox (implementation of > "Optimistic FIFO Queue") can be the solution here too. More aligned with > CL principles, allowing for multiple mailboxes-- and therefore priority > messaging-- would be a welcome advancement beyond Erlang. (Erlang allows > only one mailbox per process: sequential but with pattern matching, nothing > out-of-band...) > > An important implementation detail that simplifies gc and increases cache > locality across NUMA nodes: messages get duplicated when delivered-- each > process only references its own copy! > > 5. (Almost) nothing shared: > > Erlang enforces strict prohibition against shared memory, and common > practice is to use an in-memory database (ETS) as key/value store in lieu > of globals. Scala allows but discourages shared memory. > > A CL-inspired take on this might be to use SBCL's approach with thread > creation: upon creating a new process, you get: A) global special values, > but modify at own risk... B) LET bindings are local to each process; C) > threads don't inherit dynamic bindings from parent. i.e., > http://sbcl.org/manual/index.html#Special-Variables > > 6. Scheduling processes on OS threads: > > This is delicate in Erlang, and I've experienced specialized use cases > interfering with their scheduler when under heavy load. Instead for our > purposes, let the programmer handle the mapping of processes to OS > threads. Less is more here. > > 7. Finally, gen_server & OTP fiends may be implemented as their own > packages... or skipped entirely! > > > Thoughts on feasibility? > > Thanks, > -Daniel > >