On Mon, Oct 23, 2023 at 11:01:43AM +0200, Markus Armbruster wrote: > Daniel P. Berrangé <berra...@redhat.com> writes: > > > On Fri, Oct 20, 2023 at 02:37:16PM +0200, Markus Armbruster wrote: > >> Fabiano Rosas <faro...@suse.de> writes: > >> > >> > Markus Armbruster <arm...@redhat.com> writes: > >> > > >> >> Fabiano Rosas <faro...@suse.de> writes: > >> >> > >> >>> Signed-off-by: Fabiano Rosas <faro...@suse.de> > >> > > >> > Side question: are we using valid JSON at all? I threw this in a random > >> > online linter and it complains about the single quotes. We could have a > >> > proper tool doing the validation in CI. > >> > >> You've come a sad, sad place. > >> > >> docs/devel/qapi-code-gen.rst: > >> > >> Schema syntax > >> ------------- > >> > >> Syntax is loosely based on `JSON > >> <http://www.ietf.org/rfc/rfc8259.txt>`_. > >> Differences: > >> > >> * Comments: start with a hash character (``#``) that is not part of a > >> string, and extend to the end of the line. > >> > >> * Strings are enclosed in ``'single quotes'``, not ``"double quotes"``. > >> > >> * Strings are restricted to printable ASCII, and escape sequences to > >> just ``\\``. > >> > >> * Numbers and ``null`` are not supported. > >> > >> If your reaction to item 2 is "this is stupid", you'd be exactly right. > >> > >> Here's the conclusion of a discussion on possible improvements we had in > >> 2020: > >> https://lore.kernel.org/qemu-devel/877dt5ofoi....@dusky.pond.sub.org/ > > > > Looking at those options again I so strongly want to be able to > > say "none of the above". > > > > We have a need to describe data structures, and generate code for > > serializing/deserializing them on the wire. We created a language > > for this and wrote our own C code generator, our own docs generator, > > own our json parser & formatter, and now are also writing our own > > Go code generator (Python, Rust too ?). > > > > IMHO this is way too much NiH for my likely, and creates a maint > > burden for ourselves that we could do without. > > > > <open-can-of-worms> > > At the point in time we invented QAPI this was perhaps justifiable, > > though even back then I thought we should have done a binary format > > and used XDR to describe it, as a standard pre-existing language and > > toolset. > > Path dependence... > > I wasn't involved in the decisions that led to QAPI, nor its initial > design and implementation. > > Its design was substantially constrained by QMP, which predates QAPI by > almost two years. I was involved in QMP's design for a bit. We argued > back and forth, and I eventually stepped aside to let the guys doing the > actual work make the decisions.
I'm somewhat conflating QMP & QAPI in my previous message too. > The maintenance burden is modest, but real. The QAPI generator has > gotten ~1.6 patches per week for the last five years, trending down. In > the last year, it's been 68 patches, 437 insertions, 338 deletions > total, much of it in docs/devel/qapi-code-gen.rst. Meanwhile, the > entire project has had 130 times as many patches, 620 times as many > insertions, and 400 times as many deletions. > > QAPI infrastructure maintenance is dwarved several times over by QAPI > schema maintenance. Chiefly patch review. Yeah, patch review on new additions is primary amount and work and indeed the bit we actually do want to be investing most time in, as API modelling is a hard problem that needs careful review. > To be fair, adding language bindings will take a non-trivial one time > investment plus ongoing maintenance for each language. As well as the developer & reviewer man hours cost from creatnig such language bindings, there's also the negative impact of potential developers being discouraged from interacting with QEMU precisely to avoid taking on this man hours cost. Impossible to quantify, as we can't measure how many projects haven't even been started, due to such avoidance. This hidden cost of discouragement is where I think the value of using standard protocols wins out the most. > > > Today I wouldn't suggest XDR, but would be inclined to suggest > > Protobuf. This already has code & docs generators for every > > programming language we would ever need. > > > > Protobuf has a defined serialization format too though which is > > binary based IIUC, so distinct from our JSON wire format. > > > > The interesting question though is whether it would be feasible to > > transition to this today, despite our historical baggage ? With > > recent interest in accessing QAPI for many more languages, it > > Go, Rust, and what else? The big one is Python, which we should have done years ago, since we use it widely in our own test suites. Over time QEMU is widely used enough though, that it would be conceptually useful for many other non-niche languages to have formal APIs for talking to QEMU, but not compelling enough to justify manually writing such APIs or code generators ourselves. > > is timely to consider if we can adopt a standardized toolset to > > reduce this burden of talking to QEMU from other languages. > > > > A transition would need several big ticket items > > > > * A QAPI visitor impl that can spit out the protobuf document. > > ie to translate all our existing QAPI formats into protobuf, > > as doing this manually would be madness. This is probably > > the easy bit. > > This is about machine-assisted translation of the QAPI schema to > protobuf, isn't it? Yeah, schema conversion. > > Above all it does assume it is possible to define a > > loss-less mapping from QAPI -> Protobuf language. I've not proved > > this is possible, but my inclination is that it probably should be, > > and if not, we could likely enable it by strategically deprecated > > and then deleting troublesome bits prior to a conversion. > > </open-can-of-worms> > > While I share your distaste for the massive NIH QAPI has become, I'm not > sure replacing it now is economical. It requires a massive one-time > investment, offset by saving us one-time investments into QAPI bindings > for other languages of interest. Whether it can actually reduce ongoing > maintenance appreciably after the replacement is unclear. This is the really hard to answer question, as the calculating value of a conversion relies on predicting an unknowable future. It is right to be sceptical about the value of changing, but also worth thinking about it none the less. Its probably the kind of project that someone who doesn't have to justify their time would do as an experiment in a fork of QEMU for a while to see how it actually works out. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
