Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Hi
On Fri, Sep 2, 2022 at 3:16 PM Markus Armbruster wrote:
> Marc-André Lureau writes:
>
> > Hi
> >
> > On Thu, Aug 11, 2022 at 5:35 PM Markus Armbruster
> wrote:
> >
> >> Daniel P. Berrangé writes:
> >>
> >> > On Thu, Aug 11, 2022 at 02:50:01PM +0400, Marc-André Lureau wrote:
> >> >> Hi
> >> >>
> >> >> On Thu, Aug 11, 2022 at 2:22 PM Peter Maydell <
> peter.mayd...@linaro.org
> >> >
> >> >> wrote:
> >>
> >> [...]
> >>
> >> >> > As Markus says, your branch ends up moving most of qobject into
> >> >> > qemu-common/. We are never going to let that out of QEMU proper,
> >> >> > because we are never going to allow ourselves to be tied to API
> >> >> > compatibility with it as an external library. So anything that
> >> >> >
> >> >>
> >> >> Why is that? We do it with a lot of dependencies already, with stable
> >> APIs.
> >> >>
> >> >> Furthermore, we don't "have to" be tied to a specific ABI/API, we can
> >> >> continue to link statically and compile against a specific version.
> like
> >> >> with libvfio-user today.
> >> >>
> >> >> And at this point, I am _not_ proposing to have an extra
> "qemu-common"
> >> >> repository. I don't think there are enough reasons to want that
> either.
> >> >>
> >> >>
> >> >>
> >> >> > needs qobject is never going to leave the QEMU codebase. Which
> >> >> > means that there's not much gain from shoving it into subproject/
> >> >> > IMHO.
> >> >>
> >> >>
> >> >> (just to be extra clear, it's qobject not QOM we are talking about)
> >> >>
> >> >> qobject is fundamental to all the QAPI related generated code. Why
> >> should
> >> >> that remain tight to QEMU proper?
> >> >
> >> > Neither qobject nor QOM have ever been designed to be public APIs.
> >> > Though admittedly qobject is quite a bit simpler as an API, I'm
> >> > not convinced its current design is something we want to consider
> >> > public. As an example, just last month Markus proposed changing
> >> > QDict's implementation
> >> >
> >> > https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00758.html
> >> >
> >> >
> >> > If we want external projects to be able to take advantage of QAPI,
> >> > the bare minimum we need to be public is a QAPI parser, from which
> >> > people can then build any code generators desired.
> >>
> >> Basically scripts/qapi/ less the code generators.
> >>
> >>
> > The generated code is used by qemu-ga & storage daemon, at least. They
> are
> > the first potential consumers, after qemu.
> >
> >
> >> Not sure a subproject would be a good fit.
> >>
> >
> > (I won't repeat the arguments of structuring a project)
> >
> >
> >>
> >> Shot from the hip: could the build process spit out something external
> >> projects could consume? It's how "consumables" are commonly delivered.
> >> E.g. .so + a bunch of headers. Sometimes that gets packaged. Sometimes
> >> it gets copied into the consuming tree ("vendored").
> >>
> >>
> > Not sure I follow, but that's just the "install" step isn't it ?
> >
> > Sure, we could have a "qemu-devel", that ships qapi-gen, I guess. But
> then,
> > you would expect stability/versioning. That's not what I am proposing (at
> > least at this point), which is just about the build system and project
> > structure, so we can build and work on subprojects independently. (for
> ex,
> > in my wip branch, qemu-ga meson.build is 115 lines, doesn't need QEMU
> > configure etc)
>
> I'm afraid I'm still wobbly on the benefits of subprojects, or even how
> to work with them.
>
> How exactly would we "build and work independently" on the subprojects
> involving QAPI? git-clone all of QEMU, but build only a subproject (and
> its dependencies)? Am I confused?
>
Yes, QEMU repository would hold some subprojects (like libvhost-user
today), that you can build/develop independently with just meson / ninja.
>
> >> > We don't neccessarily need the current QAPI C code generator. There
> >> > could be a new C generator that didn't use qobject, but instead used
> >> > some standard GLib types like GHashTable/GList instead of QDict/QList.
> >>
> >> Yes, that should be possible.
> >>
> >>
> > I can't see much benefit from doing that extra work. It would create two
> C
> > APIs, making future bindings efforts more difficult etc.
>
> We need to distinguish client-side and server-side APIs / bindings.
>
Indeed.. although imho, it's best when both are similar, or use similar
types / concepts. (gdbus does a pretty good job, for ex)
>
> The existing C generator targets internal, server-side use. It
> generates everything defined in the schema.
>
> The existing introspection generator generates data for external,
> client-side use (via QMP). It generates just the subset visible in QMP.
>
> The proposed Go generator also targets external, client-side use (client
> bindings for QMP), and should also generate just the subset visible in
> QMP.
>
> So should a future C generator of client bindings for QMP.
>
> And then we'd have two distinct C APIs: a server-side one to help us
> implement QMP,
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Marc-André Lureau writes:
> Hi
>
> On Thu, Aug 11, 2022 at 5:35 PM Markus Armbruster wrote:
>
>> Daniel P. Berrangé writes:
>>
>> > On Thu, Aug 11, 2022 at 02:50:01PM +0400, Marc-André Lureau wrote:
>> >> Hi
>> >>
>> >> On Thu, Aug 11, 2022 at 2:22 PM Peter Maydell > >
>> >> wrote:
>>
>> [...]
>>
>> >> > As Markus says, your branch ends up moving most of qobject into
>> >> > qemu-common/. We are never going to let that out of QEMU proper,
>> >> > because we are never going to allow ourselves to be tied to API
>> >> > compatibility with it as an external library. So anything that
>> >> >
>> >>
>> >> Why is that? We do it with a lot of dependencies already, with stable
>> APIs.
>> >>
>> >> Furthermore, we don't "have to" be tied to a specific ABI/API, we can
>> >> continue to link statically and compile against a specific version. like
>> >> with libvfio-user today.
>> >>
>> >> And at this point, I am _not_ proposing to have an extra "qemu-common"
>> >> repository. I don't think there are enough reasons to want that either.
>> >>
>> >>
>> >>
>> >> > needs qobject is never going to leave the QEMU codebase. Which
>> >> > means that there's not much gain from shoving it into subproject/
>> >> > IMHO.
>> >>
>> >>
>> >> (just to be extra clear, it's qobject not QOM we are talking about)
>> >>
>> >> qobject is fundamental to all the QAPI related generated code. Why
>> should
>> >> that remain tight to QEMU proper?
>> >
>> > Neither qobject nor QOM have ever been designed to be public APIs.
>> > Though admittedly qobject is quite a bit simpler as an API, I'm
>> > not convinced its current design is something we want to consider
>> > public. As an example, just last month Markus proposed changing
>> > QDict's implementation
>> >
>> > https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00758.html
>> >
>> >
>> > If we want external projects to be able to take advantage of QAPI,
>> > the bare minimum we need to be public is a QAPI parser, from which
>> > people can then build any code generators desired.
>>
>> Basically scripts/qapi/ less the code generators.
>>
>>
> The generated code is used by qemu-ga & storage daemon, at least. They are
> the first potential consumers, after qemu.
>
>
>> Not sure a subproject would be a good fit.
>>
>
> (I won't repeat the arguments of structuring a project)
>
>
>>
>> Shot from the hip: could the build process spit out something external
>> projects could consume? It's how "consumables" are commonly delivered.
>> E.g. .so + a bunch of headers. Sometimes that gets packaged. Sometimes
>> it gets copied into the consuming tree ("vendored").
>>
>>
> Not sure I follow, but that's just the "install" step isn't it ?
>
> Sure, we could have a "qemu-devel", that ships qapi-gen, I guess. But then,
> you would expect stability/versioning. That's not what I am proposing (at
> least at this point), which is just about the build system and project
> structure, so we can build and work on subprojects independently. (for ex,
> in my wip branch, qemu-ga meson.build is 115 lines, doesn't need QEMU
> configure etc)
I'm afraid I'm still wobbly on the benefits of subprojects, or even how
to work with them.
How exactly would we "build and work independently" on the subprojects
involving QAPI? git-clone all of QEMU, but build only a subproject (and
its dependencies)? Am I confused?
>> > We don't neccessarily need the current QAPI C code generator. There
>> > could be a new C generator that didn't use qobject, but instead used
>> > some standard GLib types like GHashTable/GList instead of QDict/QList.
>>
>> Yes, that should be possible.
>>
>>
> I can't see much benefit from doing that extra work. It would create two C
> APIs, making future bindings efforts more difficult etc.
We need to distinguish client-side and server-side APIs / bindings.
The existing C generator targets internal, server-side use. It
generates everything defined in the schema.
The existing introspection generator generates data for external,
client-side use (via QMP). It generates just the subset visible in QMP.
The proposed Go generator also targets external, client-side use (client
bindings for QMP), and should also generate just the subset visible in
QMP.
So should a future C generator of client bindings for QMP.
And then we'd have two distinct C APIs: a server-side one to help us
implement QMP, and a client-side one to help use use QMP (QMP bindings).
Whether the two would profit from sharing QObject is not clear to me
at this point.
> QObject is very much like GValue though (the naming is better too :), just
> like the QOM & GObject story.
>
> thanks
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Hi
On Thu, Aug 11, 2022 at 5:35 PM Markus Armbruster wrote:
> Daniel P. Berrangé writes:
>
> > On Thu, Aug 11, 2022 at 02:50:01PM +0400, Marc-André Lureau wrote:
> >> Hi
> >>
> >> On Thu, Aug 11, 2022 at 2:22 PM Peter Maydell >
> >> wrote:
>
> [...]
>
> >> > As Markus says, your branch ends up moving most of qobject into
> >> > qemu-common/. We are never going to let that out of QEMU proper,
> >> > because we are never going to allow ourselves to be tied to API
> >> > compatibility with it as an external library. So anything that
> >> >
> >>
> >> Why is that? We do it with a lot of dependencies already, with stable
> APIs.
> >>
> >> Furthermore, we don't "have to" be tied to a specific ABI/API, we can
> >> continue to link statically and compile against a specific version. like
> >> with libvfio-user today.
> >>
> >> And at this point, I am _not_ proposing to have an extra "qemu-common"
> >> repository. I don't think there are enough reasons to want that either.
> >>
> >>
> >>
> >> > needs qobject is never going to leave the QEMU codebase. Which
> >> > means that there's not much gain from shoving it into subproject/
> >> > IMHO.
> >>
> >>
> >> (just to be extra clear, it's qobject not QOM we are talking about)
> >>
> >> qobject is fundamental to all the QAPI related generated code. Why
> should
> >> that remain tight to QEMU proper?
> >
> > Neither qobject nor QOM have ever been designed to be public APIs.
> > Though admittedly qobject is quite a bit simpler as an API, I'm
> > not convinced its current design is something we want to consider
> > public. As an example, just last month Markus proposed changing
> > QDict's implementation
> >
> > https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00758.html
> >
> >
> > If we want external projects to be able to take advantage of QAPI,
> > the bare minimum we need to be public is a QAPI parser, from which
> > people can then build any code generators desired.
>
> Basically scripts/qapi/ less the code generators.
>
>
The generated code is used by qemu-ga & storage daemon, at least. They are
the first potential consumers, after qemu.
> Not sure a subproject would be a good fit.
>
(I won't repeat the arguments of structuring a project)
>
> Shot from the hip: could the build process spit out something external
> projects could consume? It's how "consumables" are commonly delivered.
> E.g. .so + a bunch of headers. Sometimes that gets packaged. Sometimes
> it gets copied into the consuming tree ("vendored").
>
>
Not sure I follow, but that's just the "install" step isn't it ?
Sure, we could have a "qemu-devel", that ships qapi-gen, I guess. But then,
you would expect stability/versioning. That's not what I am proposing (at
least at this point), which is just about the build system and project
structure, so we can build and work on subprojects independently. (for ex,
in my wip branch, qemu-ga meson.build is 115 lines, doesn't need QEMU
configure etc)
> > We don't neccessarily need the current QAPI C code generator. There
> > could be a new C generator that didn't use qobject, but instead used
> > some standard GLib types like GHashTable/GList instead of QDict/QList.
>
> Yes, that should be possible.
>
>
I can't see much benefit from doing that extra work. It would create two C
APIs, making future bindings efforts more difficult etc.
QObject is very much like GValue though (the naming is better too :), just
like the QOM & GObject story.
thanks
--
Marc-André Lureau
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Daniel P. Berrangé writes:
> On Thu, Aug 11, 2022 at 02:50:01PM +0400, Marc-André Lureau wrote:
>> Hi
>>
>> On Thu, Aug 11, 2022 at 2:22 PM Peter Maydell
>> wrote:
[...]
>> > As Markus says, your branch ends up moving most of qobject into
>> > qemu-common/. We are never going to let that out of QEMU proper,
>> > because we are never going to allow ourselves to be tied to API
>> > compatibility with it as an external library. So anything that
>> >
>>
>> Why is that? We do it with a lot of dependencies already, with stable APIs.
>>
>> Furthermore, we don't "have to" be tied to a specific ABI/API, we can
>> continue to link statically and compile against a specific version. like
>> with libvfio-user today.
>>
>> And at this point, I am _not_ proposing to have an extra "qemu-common"
>> repository. I don't think there are enough reasons to want that either.
>>
>>
>>
>> > needs qobject is never going to leave the QEMU codebase. Which
>> > means that there's not much gain from shoving it into subproject/
>> > IMHO.
>>
>>
>> (just to be extra clear, it's qobject not QOM we are talking about)
>>
>> qobject is fundamental to all the QAPI related generated code. Why should
>> that remain tight to QEMU proper?
>
> Neither qobject nor QOM have ever been designed to be public APIs.
> Though admittedly qobject is quite a bit simpler as an API, I'm
> not convinced its current design is something we want to consider
> public. As an example, just last month Markus proposed changing
> QDict's implementation
>
> https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00758.html
>
>
> If we want external projects to be able to take advantage of QAPI,
> the bare minimum we need to be public is a QAPI parser, from which
> people can then build any code generators desired.
Basically scripts/qapi/ less the code generators.
Not sure a subproject would be a good fit.
Shot from the hip: could the build process spit out something external
projects could consume? It's how "consumables" are commonly delivered.
E.g. .so + a bunch of headers. Sometimes that gets packaged. Sometimes
it gets copied into the consuming tree ("vendored").
> We don't neccessarily need the current QAPI C code generator. There
> could be a new C generator that didn't use qobject, but instead used
> some standard GLib types like GHashTable/GList instead of QDict/QList.
Yes, that should be possible.
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
On Thu, Aug 11, 2022 at 02:50:01PM +0400, Marc-André Lureau wrote: > Hi > > On Thu, Aug 11, 2022 at 2:22 PM Peter Maydell > wrote: > > > On Thu, 11 Aug 2022 at 11:09, Marc-André Lureau > > wrote: > > > On Thu, Aug 11, 2022 at 1:05 PM Markus Armbruster > > wrote: > > >> Your moves tear closely related code apart. This is going to be a drag > > >> for developers in general and maintainers in particular. > > >> > > >> Ergonomics suffer when related code is in multiple places. Having to > > >> switch between directories and remember where is what will a constant > > >> low-level pain. Things that used to be simple & quick, like git-grep > > >> qapi/*.c, become more involved. > > >> > > > > > > It's inevitable when a project grows. QEMU is already a very large > > project. Over the years, we have structured the project, by moving things > > and splitting in subdirectories. Imho, this is actually helpful in many > > ways, and we got used to it easily hopefully. > > > > I agree with this. But generally we've tried to do that by > > having the subdirectory splitting and naming match the > > structure of the codebase. The exception, which I strongly > > dislike, is that contrib/ is a grabbag of random stuff. > > subprojects/ is starting to feel like it is also turning > > into "place where random stuff ends up"... > > > > Yes, most of contrib/* should probably be standalone projects. If only we > had some kind of common library/subproject :) > > subproject/* is a meson *cough* convention (imposed location for > subprojects). If we don't want to depend on external released libraries, > that's just the way it is. > > > > > > > Do you see fundamental reasons why qemu-ga or (qemu-img, qemu-nbd, > > storage-daemon, virtiofsd, vhost-user-*, *helper, ivshmem* etc) need to be > > tight to qemu code, release and management process? I am not saying it's > > time to move them out of qemu project, but I believe it's helpful to have > > code that is structured and can be compiled indepently. > > > > > > And by having "standalone" subprojects, we can more easily evolve in new > > directions, without touching the rest of the projects. > > > > As Markus says, your branch ends up moving most of qobject into > > qemu-common/. We are never going to let that out of QEMU proper, > > because we are never going to allow ourselves to be tied to API > > compatibility with it as an external library. So anything that > > > > Why is that? We do it with a lot of dependencies already, with stable APIs. > > Furthermore, we don't "have to" be tied to a specific ABI/API, we can > continue to link statically and compile against a specific version. like > with libvfio-user today. > > And at this point, I am _not_ proposing to have an extra "qemu-common" > repository. I don't think there are enough reasons to want that either. > > > > > needs qobject is never going to leave the QEMU codebase. Which > > > means that there's not much gain from shoving it into subproject/ > > IMHO. > > > (just to be extra clear, it's qobject not QOM we are talking about) > > qobject is fundamental to all the QAPI related generated code. Why should > that remain tight to QEMU proper? Neither qobject nor QOM have ever been designed to be public APIs. Though admittedly qobject is quite a bit simpler as an API, I'm not convinced its current design is something we want to consider public. As an example, just last month Markus proposed changing QDict's implementation https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg00758.html If we want external projects to be able to take advantage of QAPI, the bare minimum we need to be public is a QAPI parser, from which people can then build any code generators desired. We don't neccessarily need the current QAPI C code generator. There could be a new C generator that didn't use qobject, but instead used some standard GLib types like GHashTable/GList instead of QDict/QList. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Peter Maydell writes: > On Thu, 11 Aug 2022 at 11:09, Marc-André Lureau > wrote: >> On Thu, Aug 11, 2022 at 1:05 PM Markus Armbruster wrote: >>> Your moves tear closely related code apart. This is going to be a drag >>> for developers in general and maintainers in particular. >>> >>> Ergonomics suffer when related code is in multiple places. Having to >>> switch between directories and remember where is what will a constant >>> low-level pain. Things that used to be simple & quick, like git-grep >>> qapi/*.c, become more involved. >>> >> >> It's inevitable when a project grows. QEMU is already a very large project. >> Over the years, we have structured the project, by moving things and >> splitting in subdirectories. Imho, this is actually helpful in many ways, >> and we got used to it easily hopefully. > > I agree with this. But generally we've tried to do that by > having the subdirectory splitting and naming match the > structure of the codebase. Good: move a bunch of closely related files from . (tree root) and include/ to new ./whatever/ and include/whatever/. The improvement in organization outweighs the pain of moving. Not so good: split some files off ./whatever into ./else/where/, even though they are closely related. [...]
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Hi On Thu, Aug 11, 2022 at 2:22 PM Peter Maydell wrote: > On Thu, 11 Aug 2022 at 11:09, Marc-André Lureau > wrote: > > On Thu, Aug 11, 2022 at 1:05 PM Markus Armbruster > wrote: > >> Your moves tear closely related code apart. This is going to be a drag > >> for developers in general and maintainers in particular. > >> > >> Ergonomics suffer when related code is in multiple places. Having to > >> switch between directories and remember where is what will a constant > >> low-level pain. Things that used to be simple & quick, like git-grep > >> qapi/*.c, become more involved. > >> > > > > It's inevitable when a project grows. QEMU is already a very large > project. Over the years, we have structured the project, by moving things > and splitting in subdirectories. Imho, this is actually helpful in many > ways, and we got used to it easily hopefully. > > I agree with this. But generally we've tried to do that by > having the subdirectory splitting and naming match the > structure of the codebase. The exception, which I strongly > dislike, is that contrib/ is a grabbag of random stuff. > subprojects/ is starting to feel like it is also turning > into "place where random stuff ends up"... > Yes, most of contrib/* should probably be standalone projects. If only we had some kind of common library/subproject :) subproject/* is a meson *cough* convention (imposed location for subprojects). If we don't want to depend on external released libraries, that's just the way it is. > > > Do you see fundamental reasons why qemu-ga or (qemu-img, qemu-nbd, > storage-daemon, virtiofsd, vhost-user-*, *helper, ivshmem* etc) need to be > tight to qemu code, release and management process? I am not saying it's > time to move them out of qemu project, but I believe it's helpful to have > code that is structured and can be compiled indepently. > > > > And by having "standalone" subprojects, we can more easily evolve in new > directions, without touching the rest of the projects. > > As Markus says, your branch ends up moving most of qobject into > qemu-common/. We are never going to let that out of QEMU proper, > because we are never going to allow ourselves to be tied to API > compatibility with it as an external library. So anything that > Why is that? We do it with a lot of dependencies already, with stable APIs. Furthermore, we don't "have to" be tied to a specific ABI/API, we can continue to link statically and compile against a specific version. like with libvfio-user today. And at this point, I am _not_ proposing to have an extra "qemu-common" repository. I don't think there are enough reasons to want that either. > needs qobject is never going to leave the QEMU codebase. Which > means that there's not much gain from shoving it into subproject/ > IMHO. (just to be extra clear, it's qobject not QOM we are talking about) qobject is fundamental to all the QAPI related generated code. Why should that remain tight to QEMU proper? thanks -- Marc-André Lureau
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
On Thu, 11 Aug 2022 at 11:09, Marc-André Lureau wrote: > On Thu, Aug 11, 2022 at 1:05 PM Markus Armbruster wrote: >> Your moves tear closely related code apart. This is going to be a drag >> for developers in general and maintainers in particular. >> >> Ergonomics suffer when related code is in multiple places. Having to >> switch between directories and remember where is what will a constant >> low-level pain. Things that used to be simple & quick, like git-grep >> qapi/*.c, become more involved. >> > > It's inevitable when a project grows. QEMU is already a very large project. > Over the years, we have structured the project, by moving things and > splitting in subdirectories. Imho, this is actually helpful in many ways, and > we got used to it easily hopefully. I agree with this. But generally we've tried to do that by having the subdirectory splitting and naming match the structure of the codebase. The exception, which I strongly dislike, is that contrib/ is a grabbag of random stuff. subprojects/ is starting to feel like it is also turning into "place where random stuff ends up"... > Do you see fundamental reasons why qemu-ga or (qemu-img, qemu-nbd, > storage-daemon, virtiofsd, vhost-user-*, *helper, ivshmem* etc) need to be > tight to qemu code, release and management process? I am not saying it's time > to move them out of qemu project, but I believe it's helpful to have code > that is structured and can be compiled indepently. > > And by having "standalone" subprojects, we can more easily evolve in new > directions, without touching the rest of the projects. As Markus says, your branch ends up moving most of qobject into qemu-common/. We are never going to let that out of QEMU proper, because we are never going to allow ourselves to be tied to API compatibility with it as an external library. So anything that needs qobject is never going to leave the QEMU codebase. Which means that there's not much gain from shoving it into subproject/ IMHO. If there's stuff that is reasonably independent then it may be worth disentangling from our build process, but it has to be actually independent, or made that way, first, I think. And, as with libslirp, there needs to be a clear beneficiary who would want to take that independent-sub-thingy and use it entirely distinctly from QEMU. thanks -- PMM
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
On Thu, Aug 11, 2022 at 1:05 PM Markus Armbruster wrote: > Marc-André Lureau writes: > > > Hi > > > > On Thu, Aug 11, 2022 at 10:52 AM Markus Armbruster > > wrote: > > > >> Marc-André Lureau writes: > >> > >> > Hi > >> > > >> > On Fri, Aug 5, 2022 at 12:12 PM Markus Armbruster > >> wrote: > >> > > >> >> marcandre.lur...@redhat.com writes: > >> >> > >> >> > From: Marc-André Lureau > >> >> > > >> >> > This is just moving qapi-gen.py and related subdir to qemu-common, > to > >> >> > ease review and proceed step by step. The following patches will > move > >> >> > related necessary code, tests etc. > >> >> > > >> >> > Signed-off-by: Marc-André Lureau > >> >> > >> >> As moved files tend to become low-level annoyances for a long time, > I'd > >> >> like to understand why you want to move them. The commit message > says > >> >> "to ease review", which I suspect isn't the real reason. Perhaps you > >> >> explained all that elsewhere already, but I missed it. > >> >> > >> >> > >> >> > >> > The end goal is to split some projects, such as qemu-ga, to standalone > >> > meson projects/subprojects. We will be able to build them > independently > >> > from the rest of QEMU, and later on perhaps handle them outside of > QEMU > >> > main repository. To achieve this, I first introduce a qemu-common > >> > subproject, where qapi and common units are provided. You can check > >> > https://gitlab.com/marcandre.lureau/qemu/-/commits/qga for a sneak > peek at > >> > current end result. > >> > >> I worry this move of the QAPI generator code into > >> subjprojects/common/scripts/qapi/ will be followed by a move into its > >> own subproject. > >> > > > > Do you mean: it could be moved again to another smaller subproject? not > > really, see below > > > > > >> Ignorant question: could we turn the QAPI generator into a subproject in > >> place? > >> > > > > If it's just the generator, probably the target would then be a python > > project (not meson), similar to python-qemu-qmp. > > > > But I don't see much point, since it's not really a standalone python > > module, it generates code, and that code needs most of what is in > > qemu-common (see > > > https://gitlab.com/marcandre.lureau/qemu/-/tree/qga/subprojects/qemu-common > ). > > It's best to have it together imho. Maybe we can consider a different > > naming or to be more careful not to add stuff that is not strictly needed > > by qapi? > > I had a look at subjprojects/qemu-common in your qga branch. Contents: > > * Subproject machinery > > * Some common headers (glib-compat.h), but not others (qemu/osdep.h). I > guess it's whatever subjproject code needs. > > Is subprojects/qemu-common/include/block/nvme.h there by accident? > It's a header shared with qemu-ga (the commit message explains) > > * Most of the QObject subsystem > > qobject/block-qdict.c is left behind. > It's qemu block specific. Not needed to move at this point, it drags other stuff iirc. > * Most of the QAPI subsystem > > Some visitors left behind: opts, forward, string input / output. Hmm, > only the .c, the .h are in the subjproject. Accident? > If they are not shared with qemu-ga, I didn't move them yet. They can stay specific to qemu or specific subprojects, or we can decide to move them (if that doesn't drag too much stuff along). > > A bit of HMP support left behind. > Can you be more specific? > > * Parts of util/ and include/qemu/ > > Error reporting, key-value CLI, some C utilities, but not others > (e.g. qemu/atomic.h, but not qemu/atomic128.h). I guess it's again > whatever subjproject code needs. > > * Parts of the QAPI Schema subsystem > > Aside: MAINTAINERS mostly not updated. > > That needs fixing. > Your moves tear closely related code apart. This is going to be a drag > for developers in general and maintainers in particular. > > Ergonomics suffer when related code is in multiple places. Having to > switch between directories and remember where is what will a constant > low-level pain. Things that used to be simple & quick, like git-grep > qapi/*.c, become more involved. > > It's inevitable when a project grows. QEMU is already a very large project. Over the years, we have structured the project, by moving things and splitting in subdirectories. Imho, this is actually helpful in many ways, and we got used to it easily hopefully. Do you see fundamental reasons why qemu-ga or (qemu-img, qemu-nbd, storage-daemon, virtiofsd, vhost-user-*, *helper, ivshmem* etc) need to be tight to qemu code, release and management process? I am not saying it's time to move them out of qemu project, but I believe it's helpful to have code that is structured and can be compiled indepently. And by having "standalone" subprojects, we can more easily evolve in new directions, without touching the rest of the projects. Hurts even when merely consuming the subsystem: when I see #include > "qemu/foo.h", the straightforward include/qemu/foo.h may or may not do. > When it
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Marc-André Lureau writes: > Hi > > On Thu, Aug 11, 2022 at 10:52 AM Markus Armbruster > wrote: > >> Marc-André Lureau writes: >> >> > Hi >> > >> > On Fri, Aug 5, 2022 at 12:12 PM Markus Armbruster >> wrote: >> > >> >> marcandre.lur...@redhat.com writes: >> >> >> >> > From: Marc-André Lureau >> >> > >> >> > This is just moving qapi-gen.py and related subdir to qemu-common, to >> >> > ease review and proceed step by step. The following patches will move >> >> > related necessary code, tests etc. >> >> > >> >> > Signed-off-by: Marc-André Lureau >> >> >> >> As moved files tend to become low-level annoyances for a long time, I'd >> >> like to understand why you want to move them. The commit message says >> >> "to ease review", which I suspect isn't the real reason. Perhaps you >> >> explained all that elsewhere already, but I missed it. >> >> >> >> >> >> >> > The end goal is to split some projects, such as qemu-ga, to standalone >> > meson projects/subprojects. We will be able to build them independently >> > from the rest of QEMU, and later on perhaps handle them outside of QEMU >> > main repository. To achieve this, I first introduce a qemu-common >> > subproject, where qapi and common units are provided. You can check >> > https://gitlab.com/marcandre.lureau/qemu/-/commits/qga for a sneak peek at >> > current end result. >> >> I worry this move of the QAPI generator code into >> subjprojects/common/scripts/qapi/ will be followed by a move into its >> own subproject. >> > > Do you mean: it could be moved again to another smaller subproject? not > really, see below > > >> Ignorant question: could we turn the QAPI generator into a subproject in >> place? >> > > If it's just the generator, probably the target would then be a python > project (not meson), similar to python-qemu-qmp. > > But I don't see much point, since it's not really a standalone python > module, it generates code, and that code needs most of what is in > qemu-common (see > https://gitlab.com/marcandre.lureau/qemu/-/tree/qga/subprojects/qemu-common). > It's best to have it together imho. Maybe we can consider a different > naming or to be more careful not to add stuff that is not strictly needed > by qapi? I had a look at subjprojects/qemu-common in your qga branch. Contents: * Subproject machinery * Some common headers (glib-compat.h), but not others (qemu/osdep.h). I guess it's whatever subjproject code needs. Is subprojects/qemu-common/include/block/nvme.h there by accident? * Most of the QObject subsystem qobject/block-qdict.c is left behind. * Most of the QAPI subsystem Some visitors left behind: opts, forward, string input / output. Hmm, only the .c, the .h are in the subjproject. Accident? A bit of HMP support left behind. * Parts of util/ and include/qemu/ Error reporting, key-value CLI, some C utilities, but not others (e.g. qemu/atomic.h, but not qemu/atomic128.h). I guess it's again whatever subjproject code needs. * Parts of the QAPI Schema subsystem Aside: MAINTAINERS mostly not updated. Your moves tear closely related code apart. This is going to be a drag for developers in general and maintainers in particular. Ergonomics suffer when related code is in multiple places. Having to switch between directories and remember where is what will a constant low-level pain. Things that used to be simple & quick, like git-grep qapi/*.c, become more involved. Hurts even when merely consuming the subsystem: when I see #include "qemu/foo.h", the straightforward include/qemu/foo.h may or may not do. When it doesn't, I need to know where to look instead. subprojects/qemu-common/include/ is a lot to type. Sufficiently powerful editors mitigate, but not completely. When changes need to be applied to every instance of an abstraction, it's easy to miss instances "elsewhere". There's a reason the QAPI visitors are all in one place. The actual split seems somewhat arbitrary in places. I suspect more code will move over time. Invalidating "what is where" knowledge. I believe a serious think about other ways to accomplish your goals is called for. > (fwiw, it's a bit of a shame python-qemu-qmp didn't import git history from > qemu.. we did better with libslirp. If we ever move code in standalone > repositories again, we should be careful to keep history with it) Yes, we should preserve history whenever practical. [...]
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Hi On Thu, Aug 11, 2022 at 10:52 AM Markus Armbruster wrote: > Marc-André Lureau writes: > > > Hi > > > > On Fri, Aug 5, 2022 at 12:12 PM Markus Armbruster > wrote: > > > >> marcandre.lur...@redhat.com writes: > >> > >> > From: Marc-André Lureau > >> > > >> > This is just moving qapi-gen.py and related subdir to qemu-common, to > >> > ease review and proceed step by step. The following patches will move > >> > related necessary code, tests etc. > >> > > >> > Signed-off-by: Marc-André Lureau > >> > >> As moved files tend to become low-level annoyances for a long time, I'd > >> like to understand why you want to move them. The commit message says > >> "to ease review", which I suspect isn't the real reason. Perhaps you > >> explained all that elsewhere already, but I missed it. > >> > >> > >> > > The end goal is to split some projects, such as qemu-ga, to standalone > > meson projects/subprojects. We will be able to build them independently > > from the rest of QEMU, and later on perhaps handle them outside of QEMU > > main repository. To achieve this, I first introduce a qemu-common > > subproject, where qapi and common units are provided. You can check > > https://gitlab.com/marcandre.lureau/qemu/-/commits/qga for a sneak peek > at > > current end result. > > I worry this move of the QAPI generator code into > subjprojects/common/scripts/qapi/ will be followed by a move into its > own subproject. > Do you mean: it could be moved again to another smaller subproject? not really, see below > Ignorant question: could we turn the QAPI generator into a subproject in > place? > If it's just the generator, probably the target would then be a python project (not meson), similar to python-qemu-qmp. But I don't see much point, since it's not really a standalone python module, it generates code, and that code needs most of what is in qemu-common (see https://gitlab.com/marcandre.lureau/qemu/-/tree/qga/subprojects/qemu-common). It's best to have it together imho. Maybe we can consider a different naming or to be more careful not to add stuff that is not strictly needed by qapi? (fwiw, it's a bit of a shame python-qemu-qmp didn't import git history from qemu.. we did better with libslirp. If we ever move code in standalone repositories again, we should be careful to keep history with it) > > I said "to ease review and proceed step by step" simply because there are > > no other changes: I don't move the rest of the qapi code & tests all > > together, it's in the subsequent series. > > I'd recommend to provide a bit more context in the commit message, even > if you copy it to several messages in a row. Our future selves will > likely be grateful. > > -- Marc-André Lureau
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Marc-André Lureau writes: > Hi > > On Fri, Aug 5, 2022 at 12:12 PM Markus Armbruster wrote: > >> marcandre.lur...@redhat.com writes: >> >> > From: Marc-André Lureau >> > >> > This is just moving qapi-gen.py and related subdir to qemu-common, to >> > ease review and proceed step by step. The following patches will move >> > related necessary code, tests etc. >> > >> > Signed-off-by: Marc-André Lureau >> >> As moved files tend to become low-level annoyances for a long time, I'd >> like to understand why you want to move them. The commit message says >> "to ease review", which I suspect isn't the real reason. Perhaps you >> explained all that elsewhere already, but I missed it. >> >> >> > The end goal is to split some projects, such as qemu-ga, to standalone > meson projects/subprojects. We will be able to build them independently > from the rest of QEMU, and later on perhaps handle them outside of QEMU > main repository. To achieve this, I first introduce a qemu-common > subproject, where qapi and common units are provided. You can check > https://gitlab.com/marcandre.lureau/qemu/-/commits/qga for a sneak peek at > current end result. I worry this move of the QAPI generator code into subjprojects/common/scripts/qapi/ will be followed by a move into its own subproject. Ignorant question: could we turn the QAPI generator into a subproject in place? > I said "to ease review and proceed step by step" simply because there are > no other changes: I don't move the rest of the qapi code & tests all > together, it's in the subsequent series. I'd recommend to provide a bit more context in the commit message, even if you copy it to several messages in a row. Our future selves will likely be grateful.
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
Hi On Fri, Aug 5, 2022 at 12:12 PM Markus Armbruster wrote: > marcandre.lur...@redhat.com writes: > > > From: Marc-André Lureau > > > > This is just moving qapi-gen.py and related subdir to qemu-common, to > > ease review and proceed step by step. The following patches will move > > related necessary code, tests etc. > > > > Signed-off-by: Marc-André Lureau > > As moved files tend to become low-level annoyances for a long time, I'd > like to understand why you want to move them. The commit message says > "to ease review", which I suspect isn't the real reason. Perhaps you > explained all that elsewhere already, but I missed it. > > > The end goal is to split some projects, such as qemu-ga, to standalone meson projects/subprojects. We will be able to build them independently from the rest of QEMU, and later on perhaps handle them outside of QEMU main repository. To achieve this, I first introduce a qemu-common subproject, where qapi and common units are provided. You can check https://gitlab.com/marcandre.lureau/qemu/-/commits/qga for a sneak peek at current end result. I said "to ease review and proceed step by step" simply because there are no other changes: I don't move the rest of the qapi code & tests all together, it's in the subsequent series. -- Marc-André Lureau
Re: [PATCH v2 11/15] qemu-common: move scripts/qapi
marcandre.lur...@redhat.com writes: > From: Marc-André Lureau > > This is just moving qapi-gen.py and related subdir to qemu-common, to > ease review and proceed step by step. The following patches will move > related necessary code, tests etc. > > Signed-off-by: Marc-André Lureau As moved files tend to become low-level annoyances for a long time, I'd like to understand why you want to move them. The commit message says "to ease review", which I suspect isn't the real reason. Perhaps you explained all that elsewhere already, but I missed it.
