Dear all,
One of the hardest problems to solve in Mirage over the years has been the
incredible myriad of mechanisms to manage library compilations for all our
different targets. This is, of course, a success disaster of sorts to have now
— when we started the project, we had almost no libraries and just two hardware
targets and this wasn’t a problem! Today, we have hundreds of packages built
and many more targets, with an increasing number coming from the external
community (since any pure OCaml library will do the trick). You can browse to
http://docs.mirage.io to see some of them.
To manage this growth, we’ve contributed and built lots of core OCaml tooling
along the way as well. Thomas Gazagnaire initially lead the construction of
opam with MirageOS in mind, and we heavily integrated the opam package workflow
into MirageOS 3, along with functoria by Gabriel Radanne to keep track of
things. While packaging has been in good shape as a result of opam, build
systems have historically not been well integrated in OCaml. Over the years in
Mirage, we’ve used every combination from make (in the early days), to omake
(in xen) to ocamlbuild (with an epic myocamlbuild) to oasis (that generated
ocamlbuild files). All of these were steady improvements over the previous,
but none were completely satisfactory.
Over the past eighteen months, we’ve been working on unifying the build
infrastructure in Mirage using the dune build system [4]. Features we need for
Mirage were patiently added to dune over the last year by Rudi Grinberg, and
many of us have ported a ton of libraries to Dune over the past year, and
recently Lucas Pluvinage has completed the very complex integration of the
frontend CLI tool and gotten us to a significant milestone! Unikernels for
multiple targets can now be simultaneously built and incrementally developed,
with very consistent cross-compilation and flags tracking!
I wanted to round up why this is important for Mirage with a summary of the
improvements:
- Speed: Dune builds are fast compared to oasis due to optimal build rules. As
a rough metric, there is a 2-3x speed improvement over using oasis/ocamlbuild
in our libraries.
- Usability: Dune does many things out of the box, such as generation and
maintenance of merlin files.
- Monorepos: Dune supports assembling monorepos by placing packages into a
subdirectory, and building it all in one go with a toplevel `dune build` [1].
I’ve got a simple prototype tool that assembles so-called ‘duniverses’ as a
simple way to track all source code that goes into a unikernel in one git repo
[3]. The workflow with this is amazing for revving mirage-types, since it can
be done in a single go instead of having 10+ opam pins.
- Configuration: Dune has a ‘configurator’ library that is just another build
target, so we can unify many of the hacks around pkg-config in one place
- Variants: The biggest addition to the build system has been to support the
“cmi linking hack” as a first-class feature [2], so that we can track C stubs
cross-compiled for different backends in a principled way. This opens the door
for us to support third-party libraries having C stubs, which to date has been
very difficult.
[1] http://www.dra27.uk/blog/platform/2018/08/15/dune-vendoring.html for more
about duniverses
[2] https://dune.readthedocs.io/en/latest/variants.html
[3] https://github.com/avsm/duniverse
[4] https://dune.build
More broadly, a number of us are actively hacking on opam, dune, odoc,
ocamlformat and other ocaml tools to continue to improve the Mirage developer
experience. This is a good time to start moving towards the Mirage 4 release
with some of these improvements exposed to the user. Some of the active PRs
towards the release that does this (primarily opened by Lucas Pluvinage in his
very productive internship at Tarides in recent months are):
- switch the unikernel build system from ocamlbuild to dune. This has been done
for functoria (https://github.com/mirage/functoria/pull/167) and the mirage cli
(https://github.com/mirage/mirage/issues/969)
- Lucas ported some packages to dune in order to make this work, and now all of
mirage-skeleton builds using dune! Forked packages are available in the
mirage-dev remote (https://github.com/mirage/mirage-dev/tree/dune).
- For packages where upstreams are busy or dont want to change build systems,
there is a https://github.com/dune-universe organisation where we can stage
ports to dune (normally via overlays). These forks have to be managed
carefully, but are very useful for packages such as Zarith that have extremely
subtle build requirements
(https://github.com/ocaml/Zarith/compare/master...dune-universe:duniverse-master)
If you have any questions, concerns or feature requests, now is a good time to
bring them up. The next steps are even more exciting: with the consistent
cross-compilation support now available, we should be able to upstream ESP32
(again, work by Lucas in his internship at Cambridge last year) which gives us
support for interesting embedded microcontrollers and IoT devices. There is
also the forthcoming RISC-V support for open hardware that KC and Malte are
working on at IIT-M, thanks to Nicolas Ojeda Bar’s OCaml patch. All in all,
this should make Mirage 4 a really fun release to hack around with!
While I’ve covered just tooling in this post, there are of course numerous
other improvements in libraries and interfaces (thanks to Hannes Menhert here)
and targets (Martin Lucina, Mindy Preston, Thomas Leonard and many others
working on Solo5 and Xen here). Switching the tooling to write down some of
our technical debt should make life a lot simpler for all this other work. My
primary request is that if there is something wrong with the current tooling
(especially opam or dune), that we make sure we collectively create an upstream
issue for it. Several of the core Mirage developers such as Thomas Gazagnaire
or myself are also maintainers of the upstream tooling, and I’d like to make
sure that we remain as close to upstream as possible. This way, we can focus
our efforts in Mirage on the unikernel end of things where we want to change
the world :-)
regards,
Anil
(PS: if I haven’t namechecked you, it’s not because I’ve forgotten you — I’m
quite staggered and humbled by the number of contributions that have gone into
this Mirage 3->4 release cycle all around. Thank you all!)
_______________________________________________
MirageOS-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/mirageos-devel
