TL;DR: I expect the main Racket distribution to run on Chez Scheme instead of the current Racket VM sometime in the next couple of years.
Background ---------- The core Racket implementation relies on a lot of C code, and that's a problem for maintenance, for porting it to new platforms (say, JavaScript), and for improving performance. My long-term plan has been to rewrite it all in Racket (except for things like GC). Last summer, I started by rewriting Racket's macro and module system from scratch. Although I was able to get a fresh implementation working, the new implementation's performance was not good enough to drop in as a replacement for our current implementation. After making sure that the new expander is really working with data structures and algorithms that are at least as good as the old ones, the new implementation is still off by up to a factor of two in time and 25% or so in space. I'm convinced that Racket's compiler and runtime system will have to be better to support a reimplementation of the core in Racket. Although I have specific ideas on how the current Racket VM could be improved, there are already other compilers and runtime systems that perform better. Notably, Chez Scheme has recently become available under an open-source license (Apache). As a first experiment, I have been able to use the new Racket expander to compile a new Racket regexp implementation to run on Chez. As expected, Chez runs the matcher about 2 times as fast as Racket, putting it generally on par with the C implementation that's currently built into Racket. So far, so good. After more experiments along these lines, I see no scenario where it's easier or more effective to improve Racket than to build on Chez. Although there are some mismatches between the primitive features that Chez and Racket provide, it will mostly work; in rare cases where there's no other solution, we can contribute pull requests to Chez. A Racket on Chez won't be compatible with the current Racket VM at the level of the C API. For that reason and others, I expect the current Racket VM to stick around for projects that use the C API, that need to embed Racket, or so on. The goal of rewriting Racket in Racket has always been to support multiple VMs, so there's no conflict with keeping the current Racket VM, too. Other VM efforts, like Pycket and RacketScript, should also benefit. Still, I expect mainline Racket development to move to Chez as the primary backend. This shift in direction will affect many projects planned to use or improve Racket. Although I expect to implement most of the conversion myself, I've talked with others who are invested in the current Racket direction. So far, everyone seems to be on board. Rewriting the Racket core has always seemed prohibitively difficult and time-consuming, but I think now's the time, especially with the expander and module system out of the way. The other subsystems look easy compared to the part that is done, but there's plenty of room for surprises. Status ------ I've reimplemented the Racket expander, reader, and regexp matcher in Racket, and I've started on the port and filesystem path subsystems. The new subsystem implementations have been developed in the "linklet" branch here: https://github.com/mflatt/racket/tree/linklet See "expander" and "regexp" in the "pkgs" directory, for example. I expect this branch to eventually move to a branch or repo in the "racket" account on GitHub. Here's my work-in-progress toward making Racket reimplementations run on Chez: https://github.com/mflatt/not-a-box/ For example, you can run the regexp implementation in Chez and compare to Racket's using its built-in regexp implemenation in C or to Racket running the new regexp implementation. Plan ---- My immediate plan is to make Racket's reader + expander + module system run on Chez. It could be two weeks or two months. From there, I'll continue rewriting all of the Racket subsystems in Racket, compiling them down to layers of Chez. See "README.txt" in the "not-a-box" repo for a status report and thoughts on various Racket subsystems. When the current Racket VM becomes the legacy Racket VM, I expect that it will retain most of its C implementation, which means that it will keep its performance for backward compatibility. Even this legacy VM, however, will use the new reader, expander, and module system --- essentially as the linklet branch is now. The expander performs well enough to be practical without affecting the runtime performance of the programs that it expands. Currently, I see no obstacle to getting everything running on Chez by the end of the year, but we'll see how it goes. -- You received this message because you are subscribed to the Google Groups "Racket Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/58a46ea7.833f620a.71e85.e2caSMTPIN_ADDED_MISSING%40gmr-mx.google.com. For more options, visit https://groups.google.com/d/optout.
