Dear everybody,
last week we had a nice discussion. The next one is in one week (June
30th 10:00 - 12:00 CEST at https://meet.jit.si/MirageOS -- see our
shared pad at https://pad.data.coop/To6IOSeNSOK9kFVlgo7XWw?both# for
notes and agenda (add your talking points there) :))
Below are the notes from last week.
Best,
Hannes
## Meeting June 16th 10:00 - 12:00 CEST
- Participants: Pierre, Reynir, Hannes, Sam
### IO-page without Cstruct, and Cstruct.t without Bigarray
- Pierre: have a branch with no bigarray in mirage-tcpip (see
https://github.com/hannesm/mirage-tcpip/tree/no-bigarray)
```
cstruct.6.2.0 git
git+https://github.com/hannesm/ocaml-cstruct.git#no-bigarray
cstruct-lwt.6.2.0 git
git+https://github.com/hannesm/ocaml-cstruct.git#no-bigarray
io-page.3.0.0 git
git+file:///home/user/mirage/io-page#no-cstruct
mirage-net-xen.2.1.5 git
git+file:///home/user/mirage/mirage-net-xen#io-page-ba
mirage-xen.9.0.0 git
git+file:///home/user/mirage/mirage-xen#new-iopage
shared-memory-ring.3.2.1 git
git+file:///home/user/mirage/shared-memory-ring#new-iopage
shared-memory-ring-lwt.3.2.1 git
git+file:///home/user/mirage/shared-memory-ring#new-iopage
tcpip.9.0.1 (uninstalled) git
git+https://github.com/hannesm/mirage-tcpip.git#no-bigarray
vchan.6.0.2 git
git+file:///home/user/mirage/ocaml-vchan#update-iopage-api
vchan-xen.6.0.2 git
git+file:///home/user/mirage/ocaml-vchan#update-iopage-api
xen-gnt.4.0.2 (uninstalled) git
git+file:///home/user/mirage/ocaml-gnt#new-iopage
```
```
[2025-06-16 10:00:52] | ___|
[2025-06-16 10:00:52] __| _ \ | _ \ __ \
[2025-06-16 10:00:52] \__ \ ( | | ( | ) |
[2025-06-16 10:00:52] ____/\___/ _|\___/____/
[2025-06-16 10:00:52] Solo5: Bindings version v0.9.1
[2025-06-16 10:00:52] Solo5: Memory map: 32 MB addressable:
[2025-06-16 10:00:52] Solo5: reserved @ (0x0 - 0xfffff)
[2025-06-16 10:00:52] Solo5: text @ (0x100000 - 0x498fff)
[2025-06-16 10:00:52] Solo5: rodata @ (0x499000 - 0x55afff)
[2025-06-16 10:00:52] Solo5: data @ (0x55b000 - 0x8e4fff)
[2025-06-16 10:00:52] Solo5: heap >= 0x8e5000 < stack < 0x2000000
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen
frontend] connect 0
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [qubes.db]
connecting to server...
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [qubes.db] connected
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen
frontend] create: id=0 domid=1
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen
frontend] sg:true gso_tcpv4:true rx_copy:true rx_flip:false
smart_poll:false
[2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen
frontend] MAC: 00:16:3e:5e:6c:00
[2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ethernet]
Connected Ethernet interface 00:16:3e:5e:6c:00
[2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ipv6] IP6: Starting
[2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ARP] Sending
gratuitous ARP for 10.137.0.27 (00:16:3e:5e:6c:00)
[2025-06-16 10:00:53] Fatal error: exception
Invalid_argument("String.blit / Bytes.blit_string")
[2025-06-16 10:00:53] Raised at Stdlib.invalid_arg in file "stdlib.ml",
line 30, characters 20-45
[2025-06-16 10:00:53] Called from
Ethernet__Ethernet_packet.Marshal.unsafe_fill in file
"duniverse/ethernet/src/ethernet_packet.ml", line 63, characters 4-67
[2025-06-16 10:00:53] Called from Stdlib__Result.map in file
"result.ml", line 25, characters 32-37
[2025-06-16 10:00:53] Called from Ethernet.Make.write.fill in file
"duniverse/ethernet/src/ethernet.ml", line 107, characters 14-60
[2025-06-16 10:00:53] Called from
Frontend.Make.write_already_locked.(fun) in file
"duniverse/mirage-net-xen/lib/frontend.ml", line 304, characters 18-60
[2025-06-16 10:00:53] Called from Lwt.Sequential_composition.try_bind in
file "duniverse/lwt/src/core/lwt.ml", line 2139, characters 10-14
[2025-06-16 10:00:53] Re-raised at Lwt.Miscellaneous.poll in file
"duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29
[2025-06-16 10:00:53] Called from Xen_os__Main.run.aux in file
"duniverse/mirage-xen/lib/main.ml", line 37, characters 10-20
[2025-06-16 10:00:53] Called from Dune__exe__Main.run in file
"mirage/main.ml", line 4, characters 12-29
[2025-06-16 10:00:53] Called from Dune__exe__Main in file
"mirage/main.ml", line 436, characters 2-7
[2025-06-16 10:00:53] Solo5: solo5_exit(2) called
```
- The issue is likely in mirage-net-xen passing a (too) small buffer to
Ethernet, since the source location is a blit of the destination mac
address (at offset 0 into the buffer)
- Pierre: need to also investigate performance, since with the Cstruct
using bytes does more copying
- Hannes: I'm really happy this is moving forward.
- Hannes: the ethernet implementation calls `Netif.write`, which calls
the (mirage-net-xen) Frontend.write with the `~size` parameter, which
then calls `write_no_retry`, which calls `write_already_locked`
(`Cstruct.sub`) -- and I guess we should printf the `~size` parameter
for further debugging
- Hannes: in (mirage-net-xen) `write_already_locked` we call `fillf
(Cstruct.sub cs_shared_block 0 size)`
- Hannes: in (ethernet): `let size = eth_hdr_size + size`
- Hannes: I'd recommend to printf the `size`
- Pierre: there's some too small buffer passed somewhere
- Hannes: I'm really happy to see this moving forward. The blocker in
Mirleft was clearly the hanging of qubesdb/xen vchan. I'd love to think
further about performance, and it seems the roadblock is lifted now :)
### Unikraft
- Sam: The ocaml-unikraft package family has been released to the
opam-repository
- Virgile told that opam-monorepo is complaining with how the
ocaml-unikraft packages are setup
- Sam: we have patches for the mirage tool, also for net, block, time,
etc. - we try to fix the small issues that are left, and it will be
ready soon!
- Hannes: did you do any further benchmarking?
- Sam: not yet, there was a small bug: there are some extra bytes in
each packet - unclear where they are coming from. we can safely just
ignore them, they are always null bytes.
- Sam: we plan to run again some benchmarks this week, and hope is that
they'll be reasonably good
- Pierre: in the end, it will be a potential replacement for solo5?
- Sam: exactly, it is an alternative backend. for unikraft, you don't
need solo5 tenders - you can run on qemu and firecracker. they have
experimental ports for other backends they are working on.
- Pierre: would it be possible to use with xen and qubes, or not?
- Sam: unikraft has a backend for xen, but we didn't add it to the other
layers - since I don't have a path to test it
- Sam: I didn't write the code to generate the xen backend (in
ocaml-unikraft)
- Pierre: would be interesting, esp. if it delivers better performance
- Hannes: which network stack do you use?
- Sam: we initially used the unikraft one (which uses lwip), but now we
use the OCaml one
- Hannes: any news on multiple CPUs in unikraft?
- Sam: not yet. it is on their roadmap. it is not in any released
version. in some parts of the code I needed to debug, it is clear that
it is monocore
- Hannes: what is unikraft currently working on? do they have users?
- Sam: they created a company to smoothen the deployment experience
(unikraft cloud)
- Hannes: I'm really happy you're pushing this work forward, so we can
piggyback on the unikraft cloud, and at the same time we can evaluate
the performance issues from their website (where solo5 is really slow),
and figure out whether the performance is lost in solo5 or OCaml.
### OCaml compiler and cross-compiling
- Sam: The OCaml 5.5 will include the patches needed for
cross-compiling, so we don't need any further patches \o/ (apart from
setting the number of domains to 1)
- Hannes: For OCaml 5.4, do we have a patchset?
- Sam: there's only a single PR left (to enable freestanding target),
will open a PR on ocaml-solo5 and ocaml-unikraft when 5.4 is released
- Hannes: from the OCaml release schedule, 5.4 is expected this summer
(August)
### OxCaml
- JaneStreet published their OxCaml project (including stack allocation,
etc.) at https://oxcaml.org
- MirageOS could use Oxcaml - it includes as well flambda2
- It would be interesting / nice to test the various compilers:
- OCaml, OCaml with flambda1, OxCaml (with plain opam packages),
OxCaml with annotated packages (to use e.g. stack allocation)
- Sam: it is unclear how to use OxCaml with MirageOS, they seem to be
based on the 5.2.0 compiler - and there would be various cross-compiling
PRs needed to use OxCaml inside of MirageOS
- Sam: unclear whether it is worth to backpatch OxCaml, or wait until
they move to OCaml 5.3.0 or even 5.4.0. Maybe we'll find someone who has
an idea of their roadmap.
- Sam: it shouldn't be too hard to get something working, but it would
be more work to upstream clean patches as merged.
