Re: Summary of Re: Making QEMU easier for management tools and applications
On Wed, Feb 12, 2020 at 01:54:42PM +, Stefan Hajnoczi wrote: > On Mon, Feb 10, 2020 at 05:43:13PM +0100, Markus Armbruster wrote: > > Stefan Hajnoczi writes: > > > > > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster > > > wrote: > > >> = Ways to provide machine-friendly initial configuration = > > >> > > >> Two ways to provide machine-friendly initial configuration on par with > > >> QMP have been proposed: > > >> > > >> 1. Extend QMP > > >> > > >>Machines use the CLI only to configure a QMP socket. The remainder > > >>of the CLI becomes human-only, with much relaxed compatibility rules. > > >> > > >> 2. QAPIfy the CLI > > >> > > >>Provide a machine-friendly CLI based on QAPI and JSON. The current > > >>CLI becomes human-only, with much relaxed compatibility rules. > > > > > > Do we keep the existing CLI around in both cases? I'm concerned that > > > we're still following the HMP/QMP approach, which has left QEMU with > > > the legacy HMP monitor that we still haven't removed. > > > > The "HMP is legacy" idea is relatively recent. > > > > I think having separate interfaces for humans and machines makes sense, > > we just need to give both the attention and care they need and deserve. > > > > I think a human-friendly monitor is has its use, but it should ideally > > be done differently than we do HMP now. > > > > Likewise, human-friendly initial configuration should exist, but it > > should ideally be done differently than we do HMP now. > > > > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > > > change in QEMU 6.0. > > > > I'm convinced the current CLI needs cleanup badly, and that means > > incompatible change. The question is how and when to change it. > > > > Here's how I'd like us to do it: > > > > 1. Create machine-friendly initial configuration interface separate from > >the existing CLI > > > >Doesn't mean it cannot be a CLI. > > > > 2. Develop it step by step to feature parity with existing CLI > > > >If we identify misfeatures we don't want anymore, we should > >immediately deprecate them in the existing CLI instead. > > > > 2. Transition machine users to this new interface > > > > 3. Declare the existing CLI to be like HMP: for humans, may change > >incompatibly > > > > 4. Clean up existing CLI step by step to wrap around the > >machine-friendly interface > > > >Whatever we deprecated in step 2 goes to the bit bucket instead. > > > >I'm open to replacing the existing CLI by a separate wrapper process > >instead. > > > >Capability to translate to the machine-friendly interface is > >desirable, so human users can easily transition to the > >machine-friendly interface when they run into a need to automate. > > > > The risk is of course that we fail at step 4 and remain stuck with the > > CLI mess we've made. > > Yes, QEMU does not have a good track record of successfully converting > to new APIs and then removing old code. > > My worry is that this effort will result in the addition of even more > code but we'll still be stuck with the old cruft (both in the user > visible interface and in the implementation). This is why I think any new CLI ought to be done in a new binary, not qemu-system-. I think it is an easier proposition to sell to people that this is a clean break if we make it a new binary. The mere fact the binary exists will make people curious about it. If we add new stuff to existing binaries, it is essentially invisible unless you look for it. Separate binaries would also make life better for documentation IMHO, as we can clearly distinguish legacy and modern in the docs. Indeed the new binary doc shoudl be completely separate, so when people learn about it, they're not distracted by legacy. This way, even if we don't delete qemu-system- for a long time, the new binary would not be polluted by the legacy cruft, even if it still exists in some internal places. Ideally the goal would be that QemuOpts be entirely missing from any code linked into the new binary. This will be challenging given some of the places QemuOpts embeds itself. Perhaps we can split some of the source files to isolate the QemuOpts usage. The block layer is the biggest challenge here. > But we won't get anywhere if we don't try :). This sounds like a > significant project and I wonder if others would be willing to help if > you can break down the tasks for them. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: Summary of Re: Making QEMU easier for management tools and applications
On Mon, Feb 10, 2020 at 05:43:13PM +0100, Markus Armbruster wrote: > Stefan Hajnoczi writes: > > > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster wrote: > >> = Ways to provide machine-friendly initial configuration = > >> > >> Two ways to provide machine-friendly initial configuration on par with > >> QMP have been proposed: > >> > >> 1. Extend QMP > >> > >>Machines use the CLI only to configure a QMP socket. The remainder > >>of the CLI becomes human-only, with much relaxed compatibility rules. > >> > >> 2. QAPIfy the CLI > >> > >>Provide a machine-friendly CLI based on QAPI and JSON. The current > >>CLI becomes human-only, with much relaxed compatibility rules. > > > > Do we keep the existing CLI around in both cases? I'm concerned that > > we're still following the HMP/QMP approach, which has left QEMU with > > the legacy HMP monitor that we still haven't removed. > > The "HMP is legacy" idea is relatively recent. > > I think having separate interfaces for humans and machines makes sense, > we just need to give both the attention and care they need and deserve. > > I think a human-friendly monitor is has its use, but it should ideally > be done differently than we do HMP now. > > Likewise, human-friendly initial configuration should exist, but it > should ideally be done differently than we do HMP now. > > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > > change in QEMU 6.0. > > I'm convinced the current CLI needs cleanup badly, and that means > incompatible change. The question is how and when to change it. > > Here's how I'd like us to do it: > > 1. Create machine-friendly initial configuration interface separate from >the existing CLI > >Doesn't mean it cannot be a CLI. > > 2. Develop it step by step to feature parity with existing CLI > >If we identify misfeatures we don't want anymore, we should >immediately deprecate them in the existing CLI instead. > > 2. Transition machine users to this new interface > > 3. Declare the existing CLI to be like HMP: for humans, may change >incompatibly > > 4. Clean up existing CLI step by step to wrap around the >machine-friendly interface > >Whatever we deprecated in step 2 goes to the bit bucket instead. > >I'm open to replacing the existing CLI by a separate wrapper process >instead. > >Capability to translate to the machine-friendly interface is >desirable, so human users can easily transition to the >machine-friendly interface when they run into a need to automate. > > The risk is of course that we fail at step 4 and remain stuck with the > CLI mess we've made. Yes, QEMU does not have a good track record of successfully converting to new APIs and then removing old code. My worry is that this effort will result in the addition of even more code but we'll still be stuck with the old cruft (both in the user visible interface and in the implementation). But we won't get anywhere if we don't try :). This sounds like a significant project and I wonder if others would be willing to help if you can break down the tasks for them. Stefan signature.asc Description: PGP signature
Re: Summary of Re: Making QEMU easier for management tools and applications
Stefan Hajnoczi writes: > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster wrote: >> = Ways to provide machine-friendly initial configuration = >> >> Two ways to provide machine-friendly initial configuration on par with >> QMP have been proposed: >> >> 1. Extend QMP >> >>Machines use the CLI only to configure a QMP socket. The remainder >>of the CLI becomes human-only, with much relaxed compatibility rules. >> >> 2. QAPIfy the CLI >> >>Provide a machine-friendly CLI based on QAPI and JSON. The current >>CLI becomes human-only, with much relaxed compatibility rules. > > Do we keep the existing CLI around in both cases? I'm concerned that > we're still following the HMP/QMP approach, which has left QEMU with > the legacy HMP monitor that we still haven't removed. The "HMP is legacy" idea is relatively recent. I think having separate interfaces for humans and machines makes sense, we just need to give both the attention and care they need and deserve. I think a human-friendly monitor is has its use, but it should ideally be done differently than we do HMP now. Likewise, human-friendly initial configuration should exist, but it should ideally be done differently than we do HMP now. > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > change in QEMU 6.0. I'm convinced the current CLI needs cleanup badly, and that means incompatible change. The question is how and when to change it. Here's how I'd like us to do it: 1. Create machine-friendly initial configuration interface separate from the existing CLI Doesn't mean it cannot be a CLI. 2. Develop it step by step to feature parity with existing CLI If we identify misfeatures we don't want anymore, we should immediately deprecate them in the existing CLI instead. 2. Transition machine users to this new interface 3. Declare the existing CLI to be like HMP: for humans, may change incompatibly 4. Clean up existing CLI step by step to wrap around the machine-friendly interface Whatever we deprecated in step 2 goes to the bit bucket instead. I'm open to replacing the existing CLI by a separate wrapper process instead. Capability to translate to the machine-friendly interface is desirable, so human users can easily transition to the machine-friendly interface when they run into a need to automate. The risk is of course that we fail at step 4 and remain stuck with the CLI mess we've made. > A project like this could prototype incompatible CLI changes in a > separate git tree. If it achieves the desired unification (CLI, QMP, > configuration file) and simplification (less code, legacy removal) > then it can be merged for an upcoming QEMU major release. That's effectively a really long-lived feature branch. Painful. If it it what it takes, we do it. I hope the process I just sketched permits more incremental development. We can explore conflict-free in qemu-storage-daemon.
Re: Summary of Re: Making QEMU easier for management tools and applications
On Mon, 10 Feb 2020 at 11:08, Daniel P. Berrangé wrote: > > On Mon, Feb 10, 2020 at 11:01:48AM +, Peter Maydell wrote: > > On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi wrote: > > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > > > change in QEMU 6.0. > > > > If we want to do wholesale incompatible changes to the CLI > > I think we definitely need some kind of tool where a user > > can say "here's my old command line, what's the new style > > equivalent?". Otherwise we're going to have a deluge of > > user issues where their old working setups broke and > > QEMU didn't give them any useful hints about why. > > There is a risk that if we promise to have a fully automated conversion > that it will be alot of work, and could force us to introduce hacks into > the new impl just to satisfy conversion. IMHO we shouldn't be afraid of > declaring that some parts of the old syntax can NOT be directly transformed > into new syntax, simply for the sake of making a new impl more practical > to move forward with. Agreed, but we should at least be able to handle the easy stuff and say "this is the general kind of new option syntax and set of options you want" for most of the rest. > An alternative approach to mitigate the disruption is to *not* make any > incompatible changes to qemu-system-. Instead introduce new binaries > with the new syntax and any other architectural changes we wish to make. > The old binaries can be deprecated but remain around for an extended > period of time, to give people and apps time to migrate. We can provide > rough guidance and perhaps partially automated conversion to help people > move, but not aim for a 100% automated conversion. I think our history of failing to actually complete transitions would predict that we'd end up with both the old and the new binaries essentially forever. thanks -- PMM
Re: Summary of Re: Making QEMU easier for management tools and applications
On Mon, Feb 10, 2020 at 11:01:48AM +, Peter Maydell wrote: > On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi wrote: > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > > change in QEMU 6.0. > > If we want to do wholesale incompatible changes to the CLI > I think we definitely need some kind of tool where a user > can say "here's my old command line, what's the new style > equivalent?". Otherwise we're going to have a deluge of > user issues where their old working setups broke and > QEMU didn't give them any useful hints about why. There is a risk that if we promise to have a fully automated conversion that it will be alot of work, and could force us to introduce hacks into the new impl just to satisfy conversion. IMHO we shouldn't be afraid of declaring that some parts of the old syntax can NOT be directly transformed into new syntax, simply for the sake of making a new impl more practical to move forward with. An alternative approach to mitigate the disruption is to *not* make any incompatible changes to qemu-system-. Instead introduce new binaries with the new syntax and any other architectural changes we wish to make. The old binaries can be deprecated but remain around for an extended period of time, to give people and apps time to migrate. We can provide rough guidance and perhaps partially automated conversion to help people move, but not aim for a 100% automated conversion. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: Summary of Re: Making QEMU easier for management tools and applications
On 10/02/20 11:56, Stefan Hajnoczi wrote: > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster wrote: >> = Ways to provide machine-friendly initial configuration = >> >> Two ways to provide machine-friendly initial configuration on par with >> QMP have been proposed: >> >> 1. Extend QMP >> >>Machines use the CLI only to configure a QMP socket. The remainder >>of the CLI becomes human-only, with much relaxed compatibility rules. >> >> 2. QAPIfy the CLI >> >>Provide a machine-friendly CLI based on QAPI and JSON. The current >>CLI becomes human-only, with much relaxed compatibility rules. > > Do we keep the existing CLI around in both cases? I'm concerned that > we're still following the HMP/QMP approach, which has left QEMU with > the legacy HMP monitor that we still haven't removed. > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > change in QEMU 6.0. > > A project like this could prototype incompatible CLI changes in a > separate git tree. If it achieves the desired unification (CLI, QMP, > configuration file) and simplification (less code, legacy removal) > then it can be merged for an upcoming QEMU major release. I think Daniel had a good point in suggesting a (possibly) throwaway fork for either (1) or (2). Let's see what kind of change is needed to do 100% QMP-based configuration of guests (or at least to QMP-ify configuration of devices and backends---things that can already have an *-add command now); then we can figure out which subset of the current CLI can be mapped to it. Paolo
Re: Summary of Re: Making QEMU easier for management tools and applications
On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi wrote: > I'm in favor of simplifying QEMU at the expense of an incompatible CLI > change in QEMU 6.0. If we want to do wholesale incompatible changes to the CLI I think we definitely need some kind of tool where a user can say "here's my old command line, what's the new style equivalent?". Otherwise we're going to have a deluge of user issues where their old working setups broke and QEMU didn't give them any useful hints about why. thanks -- PMM
Re: Summary of Re: Making QEMU easier for management tools and applications
On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster wrote: > = Ways to provide machine-friendly initial configuration = > > Two ways to provide machine-friendly initial configuration on par with > QMP have been proposed: > > 1. Extend QMP > >Machines use the CLI only to configure a QMP socket. The remainder >of the CLI becomes human-only, with much relaxed compatibility rules. > > 2. QAPIfy the CLI > >Provide a machine-friendly CLI based on QAPI and JSON. The current >CLI becomes human-only, with much relaxed compatibility rules. Do we keep the existing CLI around in both cases? I'm concerned that we're still following the HMP/QMP approach, which has left QEMU with the legacy HMP monitor that we still haven't removed. I'm in favor of simplifying QEMU at the expense of an incompatible CLI change in QEMU 6.0. A project like this could prototype incompatible CLI changes in a separate git tree. If it achieves the desired unification (CLI, QMP, configuration file) and simplification (less code, legacy removal) then it can be merged for an upcoming QEMU major release. Stefan
Re: Summary of Re: Making QEMU easier for management tools and applications
Markus Armbruster writes: [...] > = Ways to provide machine-friendly initial configuration = > > Two ways to provide machine-friendly initial configuration on par with > QMP have been proposed: > > 1. Extend QMP > >Machines use the CLI only to configure a QMP socket. The remainder >of the CLI becomes human-only, with much relaxed compatibility rules. > > 2. QAPIfy the CLI > >Provide a machine-friendly CLI based on QAPI and JSON. The current >CLI becomes human-only, with much relaxed compatibility rules. > >Aside: I looked into cleaning up the human-only CLI at the same time, >but the need to maintain compatibility until the transition to the >machine-friendly CLI is complete makes this hard. It needs to be >cleaned up, though. More on that below. Forgot to write down: we're talking not just about qemu-system-FOO, but any executable with a non-trivial command line for use by machines. This includes qemu-img, future qemu-storage-daemon, possibly qemu-nbd and more. Of these, qemu-storage-daemon will have a QMP monitor. The others don't have one so far. [...]