Re: Summary of Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Wed, Feb 12, 2020 at 01:54:42PM +, Stefan Hajnoczi wrote:
> On Mon, Feb 10, 2020 at 05:43:13PM +0100, Markus Armbruster wrote:
> > Stefan Hajnoczi  writes:
> > 
> > > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster  
> > > wrote:
> > >> = Ways to provide machine-friendly initial configuration =
> > >>
> > >> Two ways to provide machine-friendly initial configuration on par with
> > >> QMP have been proposed:
> > >>
> > >> 1. Extend QMP
> > >>
> > >>Machines use the CLI only to configure a QMP socket.  The remainder
> > >>of the CLI becomes human-only, with much relaxed compatibility rules.
> > >>
> > >> 2. QAPIfy the CLI
> > >>
> > >>Provide a machine-friendly CLI based on QAPI and JSON.  The current
> > >>CLI becomes human-only, with much relaxed compatibility rules.
> > >
> > > Do we keep the existing CLI around in both cases?  I'm concerned that
> > > we're still following the HMP/QMP approach, which has left QEMU with
> > > the legacy HMP monitor that we still haven't removed.
> > 
> > The "HMP is legacy" idea is relatively recent.
> > 
> > I think having separate interfaces for humans and machines makes sense,
> > we just need to give both the attention and care they need and deserve.
> > 
> > I think a human-friendly monitor is has its use, but it should ideally
> > be done differently than we do HMP now.
> > 
> > Likewise, human-friendly initial configuration should exist, but it
> > should ideally be done differently than we do HMP now.
> > 
> > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > > change in QEMU 6.0.
> > 
> > I'm convinced the current CLI needs cleanup badly, and that means
> > incompatible change.  The question is how and when to change it.
> > 
> > Here's how I'd like us to do it:
> > 
> > 1. Create machine-friendly initial configuration interface separate from
> >the existing CLI
> > 
> >Doesn't mean it cannot be a CLI.
> > 
> > 2. Develop it step by step to feature parity with existing CLI
> > 
> >If we identify misfeatures we don't want anymore, we should
> >immediately deprecate them in the existing CLI instead.
> > 
> > 2. Transition machine users to this new interface
> > 
> > 3. Declare the existing CLI to be like HMP: for humans, may change
> >incompatibly
> > 
> > 4. Clean up existing CLI step by step to wrap around the
> >machine-friendly interface
> > 
> >Whatever we deprecated in step 2 goes to the bit bucket instead.
> > 
> >I'm open to replacing the existing CLI by a separate wrapper process
> >instead.
> > 
> >Capability to translate to the machine-friendly interface is
> >desirable, so human users can easily transition to the
> >machine-friendly interface when they run into a need to automate.
> > 
> > The risk is of course that we fail at step 4 and remain stuck with the
> > CLI mess we've made.
> 
> Yes, QEMU does not have a good track record of successfully converting
> to new APIs and then removing old code.
> 
> My worry is that this effort will result in the addition of even more
> code but we'll still be stuck with the old cruft (both in the user
> visible interface and in the implementation).

This is why I think any new CLI ought to be done in a new binary,
not qemu-system-. I think it is an easier proposition to
sell to people that this is a clean break if we make it a new
binary. The mere fact the binary exists will make people curious
about it. If we add new stuff to existing binaries, it is
essentially invisible unless you look for it.  Separate binaries
would also make life better for documentation IMHO, as we can
clearly distinguish legacy and modern in the docs. Indeed the
new binary doc shoudl be completely separate, so when people
learn about it, they're not distracted by legacy.

This way, even if we don't delete qemu-system- for a long time,
the new binary would not be polluted by the legacy cruft, even if
it still exists in some internal places.

Ideally the goal would be that QemuOpts be entirely missing from
any code linked into the new binary. This will be challenging given
some of the places QemuOpts embeds itself. Perhaps we can split some
of the source files to isolate the QemuOpts usage. The block layer
is the biggest challenge here.

> But we won't get anywhere if we don't try :).  This sounds like a
> significant project and I wonder if others would be willing to help if
> you can break down the tasks for them.



Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Summary of Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

On Mon, Feb 10, 2020 at 05:43:13PM +0100, Markus Armbruster wrote:
> Stefan Hajnoczi  writes:
> 
> > On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster  wrote:
> >> = Ways to provide machine-friendly initial configuration =
> >>
> >> Two ways to provide machine-friendly initial configuration on par with
> >> QMP have been proposed:
> >>
> >> 1. Extend QMP
> >>
> >>Machines use the CLI only to configure a QMP socket.  The remainder
> >>of the CLI becomes human-only, with much relaxed compatibility rules.
> >>
> >> 2. QAPIfy the CLI
> >>
> >>Provide a machine-friendly CLI based on QAPI and JSON.  The current
> >>CLI becomes human-only, with much relaxed compatibility rules.
> >
> > Do we keep the existing CLI around in both cases?  I'm concerned that
> > we're still following the HMP/QMP approach, which has left QEMU with
> > the legacy HMP monitor that we still haven't removed.
> 
> The "HMP is legacy" idea is relatively recent.
> 
> I think having separate interfaces for humans and machines makes sense,
> we just need to give both the attention and care they need and deserve.
> 
> I think a human-friendly monitor is has its use, but it should ideally
> be done differently than we do HMP now.
> 
> Likewise, human-friendly initial configuration should exist, but it
> should ideally be done differently than we do HMP now.
> 
> > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > change in QEMU 6.0.
> 
> I'm convinced the current CLI needs cleanup badly, and that means
> incompatible change.  The question is how and when to change it.
> 
> Here's how I'd like us to do it:
> 
> 1. Create machine-friendly initial configuration interface separate from
>the existing CLI
> 
>Doesn't mean it cannot be a CLI.
> 
> 2. Develop it step by step to feature parity with existing CLI
> 
>If we identify misfeatures we don't want anymore, we should
>immediately deprecate them in the existing CLI instead.
> 
> 2. Transition machine users to this new interface
> 
> 3. Declare the existing CLI to be like HMP: for humans, may change
>incompatibly
> 
> 4. Clean up existing CLI step by step to wrap around the
>machine-friendly interface
> 
>Whatever we deprecated in step 2 goes to the bit bucket instead.
> 
>I'm open to replacing the existing CLI by a separate wrapper process
>instead.
> 
>Capability to translate to the machine-friendly interface is
>desirable, so human users can easily transition to the
>machine-friendly interface when they run into a need to automate.
> 
> The risk is of course that we fail at step 4 and remain stuck with the
> CLI mess we've made.

Yes, QEMU does not have a good track record of successfully converting
to new APIs and then removing old code.

My worry is that this effort will result in the addition of even more
code but we'll still be stuck with the old cruft (both in the user
visible interface and in the implementation).

But we won't get anywhere if we don't try :).  This sounds like a
significant project and I wonder if others would be willing to help if
you can break down the tasks for them.

Stefan


signature.asc
Description: PGP signature

Re: Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Stefan Hajnoczi  writes:

> On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster  wrote:
>> = Ways to provide machine-friendly initial configuration =
>>
>> Two ways to provide machine-friendly initial configuration on par with
>> QMP have been proposed:
>>
>> 1. Extend QMP
>>
>>Machines use the CLI only to configure a QMP socket.  The remainder
>>of the CLI becomes human-only, with much relaxed compatibility rules.
>>
>> 2. QAPIfy the CLI
>>
>>Provide a machine-friendly CLI based on QAPI and JSON.  The current
>>CLI becomes human-only, with much relaxed compatibility rules.
>
> Do we keep the existing CLI around in both cases?  I'm concerned that
> we're still following the HMP/QMP approach, which has left QEMU with
> the legacy HMP monitor that we still haven't removed.

The "HMP is legacy" idea is relatively recent.

I think having separate interfaces for humans and machines makes sense,
we just need to give both the attention and care they need and deserve.

I think a human-friendly monitor is has its use, but it should ideally
be done differently than we do HMP now.

Likewise, human-friendly initial configuration should exist, but it
should ideally be done differently than we do HMP now.

> I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> change in QEMU 6.0.

I'm convinced the current CLI needs cleanup badly, and that means
incompatible change.  The question is how and when to change it.

Here's how I'd like us to do it:

1. Create machine-friendly initial configuration interface separate from
   the existing CLI

   Doesn't mean it cannot be a CLI.

2. Develop it step by step to feature parity with existing CLI

   If we identify misfeatures we don't want anymore, we should
   immediately deprecate them in the existing CLI instead.

2. Transition machine users to this new interface

3. Declare the existing CLI to be like HMP: for humans, may change
   incompatibly

4. Clean up existing CLI step by step to wrap around the
   machine-friendly interface

   Whatever we deprecated in step 2 goes to the bit bucket instead.

   I'm open to replacing the existing CLI by a separate wrapper process
   instead.

   Capability to translate to the machine-friendly interface is
   desirable, so human users can easily transition to the
   machine-friendly interface when they run into a need to automate.

The risk is of course that we fail at step 4 and remain stuck with the
CLI mess we've made.

> A project like this could prototype incompatible CLI changes in a
> separate git tree.  If it achieves the desired unification (CLI, QMP,
> configuration file) and simplification (less code, legacy removal)
> then it can be merged for an upcoming QEMU major release.

That's effectively a really long-lived feature branch.  Painful.  If it
it what it takes, we do it.  I hope the process I just sketched permits
more incremental development.

We can explore conflict-free in qemu-storage-daemon.

Re: Summary of Re: Making QEMU easier for management tools and applications

[Peter Maydell]

On Mon, 10 Feb 2020 at 11:08, Daniel P. Berrangé  wrote:
>
> On Mon, Feb 10, 2020 at 11:01:48AM +, Peter Maydell wrote:
> > On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi  wrote:
> > > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > > change in QEMU 6.0.
> >
> > If we want to do wholesale incompatible changes to the CLI
> > I think we definitely need some kind of tool where a user
> > can say "here's my old command line, what's the new style
> > equivalent?". Otherwise we're going to have a deluge of
> > user issues where their old working setups broke and
> > QEMU didn't give them any useful hints about why.
>
> There is a risk that if we promise to have a fully automated conversion
> that it will be alot of work, and could force us to introduce hacks into
> the new impl just to satisfy conversion.  IMHO we shouldn't be afraid of
> declaring that some parts of the old syntax can NOT be directly transformed
> into new syntax, simply for the sake of making a new impl more practical
> to move forward with.

Agreed, but we should at least be able to handle the easy
stuff and say "this is the general kind of new option syntax
and set of options you want" for most of the rest.

> An alternative approach to mitigate the disruption is to *not* make any
> incompatible changes to qemu-system-. Instead introduce new binaries
> with the new syntax and any other architectural  changes we wish to make.
> The old binaries can be deprecated but remain around for an extended
> period of time, to give people and apps time to migrate. We can provide
> rough guidance and perhaps partially automated conversion to help people
> move, but not aim for a 100% automated conversion.

I think our history of failing to actually complete transitions
would predict that we'd end up with both the old and the new
binaries essentially forever.

thanks
-- PMM

Re: Summary of Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Mon, Feb 10, 2020 at 11:01:48AM +, Peter Maydell wrote:
> On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi  wrote:
> > I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> > change in QEMU 6.0.
> 
> If we want to do wholesale incompatible changes to the CLI
> I think we definitely need some kind of tool where a user
> can say "here's my old command line, what's the new style
> equivalent?". Otherwise we're going to have a deluge of
> user issues where their old working setups broke and
> QEMU didn't give them any useful hints about why.

There is a risk that if we promise to have a fully automated conversion
that it will be alot of work, and could force us to introduce hacks into
the new impl just to satisfy conversion.  IMHO we shouldn't be afraid of
declaring that some parts of the old syntax can NOT be directly transformed
into new syntax, simply for the sake of making a new impl more practical
to move forward with.

An alternative approach to mitigate the disruption is to *not* make any
incompatible changes to qemu-system-. Instead introduce new binaries
with the new syntax and any other architectural  changes we wish to make.
The old binaries can be deprecated but remain around for an extended
period of time, to give people and apps time to migrate. We can provide
rough guidance and perhaps partially automated conversion to help people
move, but not aim for a 100% automated conversion. 

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Summary of Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 10/02/20 11:56, Stefan Hajnoczi wrote:
> On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster  wrote:
>> = Ways to provide machine-friendly initial configuration =
>>
>> Two ways to provide machine-friendly initial configuration on par with
>> QMP have been proposed:
>>
>> 1. Extend QMP
>>
>>Machines use the CLI only to configure a QMP socket.  The remainder
>>of the CLI becomes human-only, with much relaxed compatibility rules.
>>
>> 2. QAPIfy the CLI
>>
>>Provide a machine-friendly CLI based on QAPI and JSON.  The current
>>CLI becomes human-only, with much relaxed compatibility rules.
> 
> Do we keep the existing CLI around in both cases?  I'm concerned that
> we're still following the HMP/QMP approach, which has left QEMU with
> the legacy HMP monitor that we still haven't removed.
> 
> I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> change in QEMU 6.0.
> 
> A project like this could prototype incompatible CLI changes in a
> separate git tree.  If it achieves the desired unification (CLI, QMP,
> configuration file) and simplification (less code, legacy removal)
> then it can be merged for an upcoming QEMU major release.

I think Daniel had a good point in suggesting a (possibly) throwaway
fork for either (1) or (2).  Let's see what kind of change is needed to
do 100% QMP-based configuration of guests (or at least to QMP-ify
configuration of devices and backends---things that can already have an
*-add command now); then we can figure out which subset of the current
CLI can be mapped to it.

Paolo

Re: Summary of Re: Making QEMU easier for management tools and applications

[Peter Maydell]

On Mon, 10 Feb 2020 at 10:57, Stefan Hajnoczi  wrote:
> I'm in favor of simplifying QEMU at the expense of an incompatible CLI
> change in QEMU 6.0.

If we want to do wholesale incompatible changes to the CLI
I think we definitely need some kind of tool where a user
can say "here's my old command line, what's the new style
equivalent?". Otherwise we're going to have a deluge of
user issues where their old working setups broke and
QEMU didn't give them any useful hints about why.

thanks
-- PMM

Re: Summary of Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

On Tue, Feb 4, 2020 at 3:54 PM Markus Armbruster  wrote:
> = Ways to provide machine-friendly initial configuration =
>
> Two ways to provide machine-friendly initial configuration on par with
> QMP have been proposed:
>
> 1. Extend QMP
>
>Machines use the CLI only to configure a QMP socket.  The remainder
>of the CLI becomes human-only, with much relaxed compatibility rules.
>
> 2. QAPIfy the CLI
>
>Provide a machine-friendly CLI based on QAPI and JSON.  The current
>CLI becomes human-only, with much relaxed compatibility rules.

Do we keep the existing CLI around in both cases?  I'm concerned that
we're still following the HMP/QMP approach, which has left QEMU with
the legacy HMP monitor that we still haven't removed.

I'm in favor of simplifying QEMU at the expense of an incompatible CLI
change in QEMU 6.0.

A project like this could prototype incompatible CLI changes in a
separate git tree.  If it achieves the desired unification (CLI, QMP,
configuration file) and simplification (less code, legacy removal)
then it can be merged for an upcoming QEMU major release.

Stefan

Re: qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications)

[Dr. David Alan Gilbert]

* John Snow (js...@redhat.com) wrote:
> I'm forking the subject as I believe Markus wanted to focus on the
> machine interface aspect.
> 
> I feel that a new human interface is *related* to that goal: the
> splitting of, and commitment to, separate human and machine interfaces
> powered by a single root schema.
> 
> I am a big believer in making QEMU usable directly to human users as I
> feel the pipeline of "tinker to deployment" is important for a
> successful project, for many reasons:
> 
> - QEMU should be easy to pick up and learn.
> 
> - Supporting QEMU's use directly as an "end-user" program increases
> proficiency in the user population at large, which (can) lead to better
> answers and engagement on e.g. Reddit, StackOverflow, IRC
> 
> - Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
> kubevirt) should be a smooth and gradual process as additional
> complexity is desired.
> 
> - Focusing on QEMU's usability allows our project to be consumed easier
> by new cloud-focused projects. If they are already familiar with (and
> happy with) our project, it is more likely to be used instead of seeking
> out alternatives. This is about reducing friction.
> 
> So, for those reasons ... even though I feel that a machine-focused API
> is our #1 priority as it caters to our existing users, we should also
> focus on what it will take to grow mindshare for QEMU's value in the
> ecosystem.
> 
> Slick interfaces and documentation go a long, long way to doing that.
> 
> So: I feel that any new machine-only paradigm or overhaul needs to be
> accompanied with some new sugar to help the medicine go down, so-to-speak.
> 
> On 2/5/20 8:09 AM, Kevin Wolf wrote:
> > Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
>  The other part that it needs to solve is how to be available by default
>  without specifying anything on the command line. Basically, if I press
>  Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
>  implemented internally or by an external Python process, I don't mind.
> >>>
> >>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> >>> use HMP on stdin).
> >>
> >> I don't think it would be that hard, actually.
> >>
> >> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> >> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> >> to do is launch the Python child process, pass it a pair of pipes for
> >> communication and forward everything between the pipes and the chardev.
> >>
> >> (That's the theory anyway.)
> > 
> > If someone is interested, I did a quick proof-of-concept hack:
> > 
> > https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> > 
> > It doesn't clean up anything properly (including the qmp-shell processes
> > it starts), but it spawns a usable qmp-shell on a user-specified
> > character device. stdio seems to work, though without readline
> > functionality (I suppose I still have line-buffering somewhere), vc
> > doesn't really work at all yet.
> > 
> > Try it out like this:
> > 
> > $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
> > monitor_qmp_event: 1
> > Welcome to the QMP low-level shell!
> > Connected to QEMU 4.2.50
> > 
> > (QEMU) query-version
> > {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": 
> > "v4.2.0-1188-gd95a3885a9"}}
> > (QEMU) quit
> > 
> > (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> > refactorings in the storage daemon branch, so why not try both at once?)
> > 
> > Polishing this to make it mergable would still require substantial work,
> > so at the moment I'm not planning to do this. But if someone wants to
> > pick it up, feel free (just let us know).
> > 
> > Hm, in fact... A qmp-shell GSoC project?
> > 
> 
> That would be great. I worry that we should have a clear vision for the
> syntax before we give this project to an intern, though. With a clear
> vision and an outline for deliverables, it's an incredibly appropriate
> project.
> 
> Some things I think we want to define before we start:
> 
> 1. What are we trying to achieve with a standalone shell?
> 2. What syntax should it use?
> 
> I think those are the hardest parts.
> 
> 
> Below, some musings:
> 
> - An integrated QMP shell would be a great usability boost to users of
> bare QEMU.
> 
> - It is undesirable in general to support two interfaces. Feature
> disparity is a problem, as is needing to document and test two separate
> interfaces. The quality disparity between the two is also an issue.
> 
> - Offering HMP via the GTK interface but not QMP is a discoverability
> problem. Unfamiliar users might assume that HMP is our flagship
> interface. It is not.
> 
> - We are unlikely to re-expand HMP to cover everything QMP does; writing
> a QMP shell that makes QMP easy to interface with is a better solution
> for removing redundancy and complexity.
> 
> - I suspect that the target 

qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications)

[John Snow]

I'm forking the subject as I believe Markus wanted to focus on the
machine interface aspect.

I feel that a new human interface is *related* to that goal: the
splitting of, and commitment to, separate human and machine interfaces
powered by a single root schema.

I am a big believer in making QEMU usable directly to human users as I
feel the pipeline of "tinker to deployment" is important for a
successful project, for many reasons:

- QEMU should be easy to pick up and learn.

- Supporting QEMU's use directly as an "end-user" program increases
proficiency in the user population at large, which (can) lead to better
answers and engagement on e.g. Reddit, StackOverflow, IRC

- Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
kubevirt) should be a smooth and gradual process as additional
complexity is desired.

- Focusing on QEMU's usability allows our project to be consumed easier
by new cloud-focused projects. If they are already familiar with (and
happy with) our project, it is more likely to be used instead of seeking
out alternatives. This is about reducing friction.

So, for those reasons ... even though I feel that a machine-focused API
is our #1 priority as it caters to our existing users, we should also
focus on what it will take to grow mindshare for QEMU's value in the
ecosystem.

Slick interfaces and documentation go a long, long way to doing that.

So: I feel that any new machine-only paradigm or overhaul needs to be
accompanied with some new sugar to help the medicine go down, so-to-speak.

On 2/5/20 8:09 AM, Kevin Wolf wrote:
> Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
 The other part that it needs to solve is how to be available by default
 without specifying anything on the command line. Basically, if I press
 Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
 implemented internally or by an external Python process, I don't mind.
>>>
>>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
>>> use HMP on stdin).
>>
>> I don't think it would be that hard, actually.
>>
>> If you have a -qmp-shell option that takes a chardev and defaults to vc,
>> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
>> to do is launch the Python child process, pass it a pair of pipes for
>> communication and forward everything between the pipes and the chardev.
>>
>> (That's the theory anyway.)
> 
> If someone is interested, I did a quick proof-of-concept hack:
> 
> https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> 
> It doesn't clean up anything properly (including the qmp-shell processes
> it starts), but it spawns a usable qmp-shell on a user-specified
> character device. stdio seems to work, though without readline
> functionality (I suppose I still have line-buffering somewhere), vc
> doesn't really work at all yet.
> 
> Try it out like this:
> 
> $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
> monitor_qmp_event: 1
> Welcome to the QMP low-level shell!
> Connected to QEMU 4.2.50
> 
> (QEMU) query-version
> {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": 
> "v4.2.0-1188-gd95a3885a9"}}
> (QEMU) quit
> 
> (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> refactorings in the storage daemon branch, so why not try both at once?)
> 
> Polishing this to make it mergable would still require substantial work,
> so at the moment I'm not planning to do this. But if someone wants to
> pick it up, feel free (just let us know).
> 
> Hm, in fact... A qmp-shell GSoC project?
> 

That would be great. I worry that we should have a clear vision for the
syntax before we give this project to an intern, though. With a clear
vision and an outline for deliverables, it's an incredibly appropriate
project.

Some things I think we want to define before we start:

1. What are we trying to achieve with a standalone shell?
2. What syntax should it use?

I think those are the hardest parts.


Below, some musings:

- An integrated QMP shell would be a great usability boost to users of
bare QEMU.

- It is undesirable in general to support two interfaces. Feature
disparity is a problem, as is needing to document and test two separate
interfaces. The quality disparity between the two is also an issue.

- Offering HMP via the GTK interface but not QMP is a discoverability
problem. Unfamiliar users might assume that HMP is our flagship
interface. It is not.

- We are unlikely to re-expand HMP to cover everything QMP does; writing
a QMP shell that makes QMP easy to interface with is a better solution
for removing redundancy and complexity.

- I suspect that the target audience for users of naked QEMU are:
  - QEMU developers
  - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
researching, testing, and debugging integration.
  - Devops professionals testing, implementing and debugging
configuration & infrastructure
  - 

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
> > > The other part that it needs to solve is how to be available by default
> > > without specifying anything on the command line. Basically, if I press
> > > Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > > implemented internally or by an external Python process, I don't mind.
> > 
> > That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> > use HMP on stdin).
> 
> I don't think it would be that hard, actually.
> 
> If you have a -qmp-shell option that takes a chardev and defaults to vc,
> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
> to do is launch the Python child process, pass it a pair of pipes for
> communication and forward everything between the pipes and the chardev.
> 
> (That's the theory anyway.)

If someone is interested, I did a quick proof-of-concept hack:

https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell

It doesn't clean up anything properly (including the qmp-shell processes
it starts), but it spawns a usable qmp-shell on a user-specified
character device. stdio seems to work, though without readline
functionality (I suppose I still have line-buffering somewhere), vc
doesn't really work at all yet.

Try it out like this:

$ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
monitor_qmp_event: 1
Welcome to the QMP low-level shell!
Connected to QEMU 4.2.50

(QEMU) query-version
{"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": 
"v4.2.0-1188-gd95a3885a9"}}
(QEMU) quit

(Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
refactorings in the storage daemon branch, so why not try both at once?)

Polishing this to make it mergable would still require substantial work,
so at the moment I'm not planning to do this. But if someone wants to
pick it up, feel free (just let us know).

Hm, in fact... A qmp-shell GSoC project?

Kevin

Re: Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Markus Armbruster  writes:

[...]
> = Ways to provide machine-friendly initial configuration =
>
> Two ways to provide machine-friendly initial configuration on par with
> QMP have been proposed:
>
> 1. Extend QMP
>
>Machines use the CLI only to configure a QMP socket.  The remainder
>of the CLI becomes human-only, with much relaxed compatibility rules.
>
> 2. QAPIfy the CLI
>
>Provide a machine-friendly CLI based on QAPI and JSON.  The current
>CLI becomes human-only, with much relaxed compatibility rules.
>
>Aside: I looked into cleaning up the human-only CLI at the same time,
>but the need to maintain compatibility until the transition to the
>machine-friendly CLI is complete makes this hard.  It needs to be
>cleaned up, though.  More on that below.

Forgot to write down: we're talking not just about qemu-system-FOO, but
any executable with a non-trivial command line for use by machines.
This includes qemu-img, future qemu-storage-daemon, possibly qemu-nbd
and more.  Of these, qemu-storage-daemon will have a QMP monitor.  The
others don't have one so far.

[...]

Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

This thread has accumulated more than a hundred messages over seven
weeks.  We need a summary.

The conversation has gone in several directions.  In this message, I'll
cover just the one I consider most important: machine-friendly initial
configuration.  I'll do the rest later, if I have stamina left.

= QMP is fine for machines, CLI is not =

We have two major external host interfaces: the CLI and QMP.  Compare:

QMP
* Purpose: control at run-time
* Commands with argument and return value
* Events with argument value
* Simple type system
* Based on QAPI
* Regular, well-defined syntax: QAPI schema on top of JSON
* Documentation is required; actual documentation is of mixed quality
* Introspectable, except for a few mostly QOM-related holes where we
  bypass QAPI
* Machine-friendly, complemented by human-only HMP

CLI
* Purpose: initial configuration
* Options with argument
* Based on a crazy zoo of QemuOpts (with and without dotted keys), QAPI
  (only a few recent options), ad hoc parsers
* Option argument syntax is mostly (variations of) key=value,...
* QemuOpts' "type system" is "list of key-value pairs, where value can
  be string, bool, (unsigned) integer, or size", optionally restricted
  to known keys with known value types.
* QAPI type system is the same as in QMP
* Introspection is completely inadequate: misses options, incorrect
  option names, misses option arguments partly or completely
* Configuration files with INI-like syntax, completely inadequate: can't
  do most options
* Not machine-friendly
* Maintaining it is a pain, evolving it is worse

The inadequacy of the CLI has become a serious issue.

= Ways to provide machine-friendly initial configuration =

Two ways to provide machine-friendly initial configuration on par with
QMP have been proposed:

1. Extend QMP

   Machines use the CLI only to configure a QMP socket.  The remainder
   of the CLI becomes human-only, with much relaxed compatibility rules.

2. QAPIfy the CLI

   Provide a machine-friendly CLI based on QAPI and JSON.  The current
   CLI becomes human-only, with much relaxed compatibility rules.

   Aside: I looked into cleaning up the human-only CLI at the same time,
   but the need to maintain compatibility until the transition to the
   machine-friendly CLI is complete makes this hard.  It needs to be
   cleaned up, though.  More on that below.

To extend QMP, we wrap QMP commands around the internal initial
configuration interfaces.  These QMP commands take arguments, but don't
return anything.  Many of them will only make sense during initial
configuration.  We'll want to express that in the schema, and enforce it
in the QMP core.  Others will behave differently, e.g. cold plug during
initial configuration, hot plug once the guest runs.

Configuration files are just QMP commands from the initial configuration
subset read from a file instead of a chardev.  JSON is a poor choice for
configuration files, and QMP's verbosity makes it poorer.  We'll want
more suitable concrete syntax for configuration files.

To improve the CLI, we wrap QAPI-based CLI options around the internal
initial configuration interfaces.  We'll want some infrastructure to
generate CLI option boilerplate, just like we generate QMP command
boilerplate.

Configuration files are just CLI options read from a file instead of
argv[].  Again, we'll want a more suitable concrete syntax there.

With an improved CLI, I'd expect machines to use configuration files so
they don't have to mess with shell quoting.  With extended QMP, they'd
perhaps rather reuse their existing QMP code to send the configuration
down a socket.  Less efficient, but I doubt it'll matter.

In both cases, the internal configuration interfaces need to be
converted from QemuOpts to QAPI types.

The one clear advantage of extending QMP is the ability to mix query
commands with initial configuration.

This is also a clear disadvantage: we need to make it work.  Existing
QMP commands may rely on initial configuration to be complete.  We may
have to allow only QMP commands we carefully checked.

How important is the advantage?

= Cleaning up the human-only CLI =

In both cases, the existing CLI remains in a human-only role.  That's a
truckload of messy code to maintain.  Not good.  Can we replace it by
thin wrappers around the machine-friendly alternative, ideally
incrementally?  Can we generate a useful part of these wrappers?

If we QAPIfy the CLI, the QAPI generator already knows the CLI, and only
has to be taught the general human-friendly key=value,... syntax, plus a
way to specify exceptions.

If we extend QMP, we'll additionally need some of the CLI QAPIfication
infrastructure for this.  Hmm.

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Andrea Bolognani  writes:

> On Fri, 2020-01-31 at 07:50 +0100, Markus Armbruster wrote:
>> Kevin Wolf  writes:
>> > Much of this threads plays with the though that maybe we don't need any
>> > compatibility and make the radical conclusion that we don't need any
>> > human-friendly interface at all. Keeping full compatibility is the other
>> > extreme.
>> > 
>> > There might be some middle ground where we break compatibility where the
>> > old way can't easily be maintained with the new infrastructure, but
>> > don't give up on the idea of being used by humans.
>> 
>> I'm not sure the connection between maintaining compatibility and
>> supporting human use is as strong as you seem to imply.
>> 
>> As far as I can tell, the "maybe we don't need any compatibility"
>> discussion is about the CLI.  I'd rephrase it as "maybe we need a
>> machine-friendly CLI on par with QMP more than we need compatibility to
>> the current CLI".
>> 
>> "We don't need any human-friendly interface at all" comes in not because
>> machine-friendly necessarily precludes human-friendly, but only if we're
>> unwilling (unable?) to do the extra work for it.
>> 
>> Compare the monitor:
>> 
>> * QMP is primarily for machines.  We promise stability: no incompatible
>>   changes without clear communicaton of intent and a grace period.  We
>>   provide machine clients tools to deal with the interface evolution,
>>   e.g. query-qmp-schema.
>> 
>> * HMP is exclusively for humans.  It may change at any time.
>> 
>> For the CLI, we don't have such a separation, and our offerings for
>> dealing with interface evolution are wholly inadequate.  We *need* to do
>> better for machines.
>> 
>> Now, the monitor also informs us about the cost of providing a
>> completely separate interface for humans.
>> 
>> Elsewhere in this thread, we discussed layering (a replacement for) HMP
>> on top of QMP cleanly, possibly in a separate process, possibly written
>> in a high-level language like Python.
>> 
>> HMP predates QMP.  We reworked it so the HMP commands are implemented on
>> top of the QMP commands, or at least on top of common helpers.  But this
>> is not quite the same as layering HMP on top of QMP.
>> 
>> If we decide to radically break the CLI so we can start over, we get to
>> decide whether and how to do a human-friendly CLI, in particular how it
>> relates to the machine-friendly CLI.
>
> Does a machine-friendly CLI need to exist at all? Once you decide
> that throwing away the current one is acceptable, you might as well
> reduce the maintainance burden by requiring that software only
> communicates with QEMU via QMP.

We need to provide a machine-friendly interface for initial
configuration.

To provide it, we can extend machine-friendly QMP, or make CLI
machine-friendly like QMP.

In both cases, we need to QAPIfy initial configuration.  I believe
that's going to be a big chunk of the work.

To extend QMP, we wrap the QAPIfied initial configuration in QMP
commands.  Many of them will only make sense during initial
configuration.  We'll want to express that in the schema, and enforce it
in the QMP core.

To improve the CLI, we wrap them in CLI options.  We'll want some
infrastructure to generate the boilerplate, just like we generate QMP
command boilerplate.

With an improved CLI, configuration files are just CLI options read from
a file instead of argv[].  We'll want a more suitable concrete syntax
there, of course.

With extended QMP, configuration files are just QMP commands from the
initial configuration subset read from a file instead of a chardev.
Again, we'll likely want more suitable concrete syntax there.

With an improved CLI, I'd expect machines to use configuration files so
they don't have to mess with shell quoting.  With extended QMP, they'd
perhaps rather reuse their existing QMP code to send the configuration
down a socket.  Less efficient, but I doubt it'll matter.

> Does a human-friendly CLI need to be part of QEMU? We have built so
> much amazing infrastructure on top of QEMU, and as of today none of
> that work is benefiting people who run it directly.

I think we do, for developers and for users who don't need the amazing
infrastructure we built on top of QEMU.  When you use QEMU for
virtualization in production, you generally need it, but there are
other, simpler uses, e.g. certain hardware emulation uses.

A human-friendly CLI can be built both as a wrapper around the
machine-friendly CLI and as a wrapper around the initial configuration
subset of QMP.

Once a machine-friendly replacement for the current CLI is in place, we
can relax the current CLI's compatibility requirements.  I figure we
need this to be able to turn it into a wrapper with reasonable effort.
Hopefully, we can generate much of the wrapping boilerplate.

> As a proof of concept, I have spent a couple of hours writing the
> attached shell script, which I hope will illustrate my point.
>
> Usage is extremely simple: just do something like
>
>   $ 

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Paolo Bonzini  writes:

> On 03/02/20 10:54, Markus Armbruster wrote:
>>> Types are documented as above; however types other than link<> and 
>>> child<>, which are QAPI types, can be user-defined types (structs, 
>>> enums) and this is not included in (1).
>> Specifically, three of four kinds of type names are documented:
>> primitive, child, scalar.  The fourth kind is not, and it can be
>> anything.  It need not be a QAPI type name.  In any case, you just have
>> to know what the type name means.
>
> It is not enforced, but it is supposed to be only QAPI type names
> (primitive or not), child or link.

It's not even documented.

Actual names include "struct tm", "guest statistics", and (my
favourite) "struct".

Re: Making QEMU easier for management tools and applications

[Andrea Bolognani]

On Fri, 2020-01-31 at 07:50 +0100, Markus Armbruster wrote:
> Kevin Wolf  writes:
> > Much of this threads plays with the though that maybe we don't need any
> > compatibility and make the radical conclusion that we don't need any
> > human-friendly interface at all. Keeping full compatibility is the other
> > extreme.
> > 
> > There might be some middle ground where we break compatibility where the
> > old way can't easily be maintained with the new infrastructure, but
> > don't give up on the idea of being used by humans.
> 
> I'm not sure the connection between maintaining compatibility and
> supporting human use is as strong as you seem to imply.
> 
> As far as I can tell, the "maybe we don't need any compatibility"
> discussion is about the CLI.  I'd rephrase it as "maybe we need a
> machine-friendly CLI on par with QMP more than we need compatibility to
> the current CLI".
> 
> "We don't need any human-friendly interface at all" comes in not because
> machine-friendly necessarily precludes human-friendly, but only if we're
> unwilling (unable?) to do the extra work for it.
> 
> Compare the monitor:
> 
> * QMP is primarily for machines.  We promise stability: no incompatible
>   changes without clear communicaton of intent and a grace period.  We
>   provide machine clients tools to deal with the interface evolution,
>   e.g. query-qmp-schema.
> 
> * HMP is exclusively for humans.  It may change at any time.
> 
> For the CLI, we don't have such a separation, and our offerings for
> dealing with interface evolution are wholly inadequate.  We *need* to do
> better for machines.
> 
> Now, the monitor also informs us about the cost of providing a
> completely separate interface for humans.
> 
> Elsewhere in this thread, we discussed layering (a replacement for) HMP
> on top of QMP cleanly, possibly in a separate process, possibly written
> in a high-level language like Python.
> 
> HMP predates QMP.  We reworked it so the HMP commands are implemented on
> top of the QMP commands, or at least on top of common helpers.  But this
> is not quite the same as layering HMP on top of QMP.
> 
> If we decide to radically break the CLI so we can start over, we get to
> decide whether and how to do a human-friendly CLI, in particular how it
> relates to the machine-friendly CLI.

Does a machine-friendly CLI need to exist at all? Once you decide
that throwing away the current one is acceptable, you might as well
reduce the maintainance burden by requiring that software only
communicates with QEMU via QMP.

Does a human-friendly CLI need to be part of QEMU? We have built so
much amazing infrastructure on top of QEMU, and as of today none of
that work is benefiting people who run it directly.


As a proof of concept, I have spent a couple of hours writing the
attached shell script, which I hope will illustrate my point.

Usage is extremely simple: just do something like

  $ ./virt-run debian-10-openstack-amd64.qcow2

and after a few seconds the guest display will appear on your screen.

Behind the scenes, it uses a number of existing high-level tools:

  * virt-inspector, to figure out what guest OS is installed in the
image;

  * virt-install, to produce a domain XML tailored to that specific
guest OS and to create the corresponding libvirt domain;

  * virt-viewer, to provide the UI.

All these tools use libvirt under the hood, and additionally
virt-install uses libosinfo to obtain information about the guest
OS, such as whether or not it supports Virtio devices and how much
memory it needs to run smoothly.


The result is that, if you run

  $ qemu-system-x86_64 -hda debian-10-openstack-amd64.qcow2

you will get

  * a single CPU emulated with TCG;

  * 128 MiB of memory;

  * emulated I/O devices;

whereas the script will give you

  * 2 CPUs accelerated with KVM;

  * 1 GiB of memory;

  * Virtio devices for pretty much everything, including a
virtio-rng device that will for example speed up the first boot
significantly if SSH keys need to be (re)created.

Unsurprisingly, performance is different: when QEMU is invoked
directly, the login prompt for this specific image shows up after
~40 seconds, whereas when we use the script it only takes ~13 seconds
to get there. And the command line is just as simple, if not more so!

All of the above was obtained by hastily cobbling together existing
tools with <100 lines of shell scripting. Imagine how much better it
could be if we actually put some serious work in!


With my argument hopefully demonstrated: I think an architecture akin
to the one Dan has outlined earlier[1] would be a great direction to
take. QEMU can continue to focus on its core competency, that is,
virtual hardware, and leave most of the user interaction up to the
software interacting with its JSON-based API.


Obviously QEMU developers, for their own use, could still benefit
from having access to a user interface that doesn't require either
rigging up libvirt support or messing with JSON 

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 03/02/20 10:54, Markus Armbruster wrote:
>> Types are documented as above; however types other than link<> and 
>> child<>, which are QAPI types, can be user-defined types (structs, 
>> enums) and this is not included in (1).
> Specifically, three of four kinds of type names are documented:
> primitive, child, scalar.  The fourth kind is not, and it can be
> anything.  It need not be a QAPI type name.  In any case, you just have
> to know what the type name means.

It is not enforced, but it is supposed to be only QAPI type names
(primitive or not), child or link.

Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Paolo Bonzini  writes:

> On 31/01/20 18:23, Markus Armbruster wrote:
>> Kashyap Chamarthy  writes:
>> docs/devel/qapi-code-gen.txt has the same problem: it's mostly internal
>> stuff, but there's also introspection, which is an external interface.
>
> We should move introspection to docs/interop.  Any takers?
>
>>> Rigorous support for introspection both of runtime objects and type
>>> capabilities
>> 
>> For a value of "rigorous".
>> 
>> Let me propose QAPI's query-qmp-schema as the tin standard[*] of
>> introspection:
>> 
>> * It's documented
>> 
>> * It comes with something that can pass as a type system
>> 
>> * It actually tells you the full truth.
>
> Well, not all statements age equally well.  But compared to netdev_add 
> and device_add, it was still an improvement. :)

I think "rigorous" is to be read as ambition, not description.  Sadly,
QOM stagnated before getting close to realizing its ambitions.

> It's certainly worse than QAPI *now*, but it's not nonexistent:
>
>> * Documentation
>> 
>>   QAPI: docs/devel/qapi-code-gen.txt section "Client JSON Protocol
>>   introspection"
>> 
>>   QOM: Nada
>
> Well, there is qom.json.  So slightly more than nothing, though I have 
> already found an off-by-one error and it could definitely be improved:
>
>  @type: the type of the property.  This will typically come in one of four
> forms:
>
> 1) A primitive type such as 'u8', 'u16', 'bool', 'str', or 'double'.
>These types are mapped to the appropriate JSON type.
>
> 2) A child type in the form 'child' where subtype is a qdev
>device type name.  Child properties create the composition tree.
>
> 3) A link type in the form 'link' where subtype is a qdev
>device type name.  Link properties form the device model graph.

If the off-by-one you found is "four forms" followed by a list of three,
then the error isn't "four", it's the omitted fourth list item.

Anyway, I grant you "nada" was an exaggeration for effect.

Note that the doc comments are reference documentation.  QAPI got that,
too.  Reference documentation is useful, but no replacement for an
explainer like qapi-code-gen.txt.

>> * Type system
>> 
>>   QAPI: A few built-in types specified in the documentation, type
>>   constructors for complex types.
>> 
>>   QOM: Types are strings, and you just need to know what they mean.
>>   Some string patterns are special: link, child, STR[INT], and
>>   you just need to know what that means, too.
>
> str[int] is not a type as far as I understood it, it's a property name.  

You're right.

> Types are documented as above; however types other than link<> and 
> child<>, which are QAPI types, can be user-defined types (structs, 
> enums) and this is not included in (1).

Specifically, three of four kinds of type names are documented:
primitive, child, scalar.  The fourth kind is not, and it can be
anything.  It need not be a QAPI type name.  In any case, you just have
to know what the type name means.

In contrast, QAPI introspection defines all types in terms of type
constructors and primitive types.  It completely hides user-defined QAPI
type names, so they don't become ABI.

Do you think we could somehow appropriate QAPI's type system for QOM
introspection?

>> * Full truth
>> 
>>   QAPI: If you can access it at the interface, you can also see it in
>>   introspection.
>> 
>>   QOM: Type introspection can show you only the properties of a freshly
>>   created object.  Properties that get created only later are invisible.
>>   Properties that depend on global state are unreliable.  Object
>>   introspection is reliable, but only for that object in its current
>>   state.
>
> Right, that's array properties and at least theoretically child 
> properties (I don't know if there are examples).
>
> Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 31/01/20 18:23, Markus Armbruster wrote:
> Kashyap Chamarthy  writes:
> docs/devel/qapi-code-gen.txt has the same problem: it's mostly internal
> stuff, but there's also introspection, which is an external interface.

We should move introspection to docs/interop.  Any takers?

>> Rigorous support for introspection both of runtime objects and type
>> capabilities
> 
> For a value of "rigorous".
> 
> Let me propose QAPI's query-qmp-schema as the tin standard[*] of
> introspection:
> 
> * It's documented
> 
> * It comes with something that can pass as a type system
> 
> * It actually tells you the full truth.

Well, not all statements age equally well.  But compared to netdev_add 
and device_add, it was still an improvement. :)

It's certainly worse than QAPI *now*, but it's not nonexistent:

> * Documentation
> 
>   QAPI: docs/devel/qapi-code-gen.txt section "Client JSON Protocol
>   introspection"
> 
>   QOM: Nada

Well, there is qom.json.  So slightly more than nothing, though I have 
already found an off-by-one error and it could definitely be improved:

 @type: the type of the property.  This will typically come in one of four
forms:

1) A primitive type such as 'u8', 'u16', 'bool', 'str', or 'double'.
   These types are mapped to the appropriate JSON type.

2) A child type in the form 'child' where subtype is a qdev
   device type name.  Child properties create the composition tree.

3) A link type in the form 'link' where subtype is a qdev
   device type name.  Link properties form the device model graph.


> * Type system
> 
>   QAPI: A few built-in types specified in the documentation, type
>   constructors for complex types.
> 
>   QOM: Types are strings, and you just need to know what they mean.
>   Some string patterns are special: link, child, STR[INT], and
>   you just need to know what that means, too.

str[int] is not a type as far as I understood it, it's a property name.  
Types are documented as above; however types other than link<> and 
child<>, which are QAPI types, can be user-defined types (structs, 
enums) and this is not included in (1).

> * Full truth
> 
>   QAPI: If you can access it at the interface, you can also see it in
>   introspection.
> 
>   QOM: Type introspection can show you only the properties of a freshly
>   created object.  Properties that get created only later are invisible.
>   Properties that depend on global state are unreliable.  Object
>   introspection is reliable, but only for that object in its current
>   state.

Right, that's array properties and at least theoretically child 
properties (I don't know if there are examples).

Paolo

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Markus Armbruster  writes:

> Paolo Bonzini  writes:
>
>> Il dom 2 feb 2020, 10:22 Kevin Wolf  ha scritto:
>>
>>> Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
>>> > On 1/28/20 6:54 AM, Kevin Wolf wrote:
>>> >
>>> > > >
>>> > > > The arguments as dotted keys:
>>> > > >
>>> > > >  id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>>> > > >
>>> > > > Observe there's quite some of nesting.  While that's somewhat
>>> cumbersome
>>> > > > in JSON, it's a lot worse with dotted keys, because there nesting
>>> means
>>> > > > repeated key prefixes.  I could give much worse examples, actually.
>>> > >
>>> > > This is true, but even without the repeated keys (e.g. in a syntax that
>>> > > would use brackets), it would still be unnecessarily verbose and
>>> > > probably hard to remember:
>>> > >
>>> > >  id=bar,backend={type=file,data={out=/tmp/bar.log}}
>>>
>>> [...] I actually think that a syntax like this might make sense for
>>> something like qmp-shell. It might even be more convenient on the
>>> command line than dotted keys if you get a lot of repetition (despite
>>> the required quoting), but it's strictly speaking incompatible because
>>> you could use {} in strings today.
>>>
>>
>> If you are willing to feed schema info to the parser, in principle you
>> could keep backwards compatibility. There would be limitations such as
>> putting the discriminator before the fields, so I am not sure it's a good
>> idea.
>
> Problem: the 'any' type, where the schema doesn't provide the necessary
> information.
>
> Problem: 'gen': false, where we pass the arguments raw, ignoring the
> schema.
>
> If we didn't restrict alternate types so severly, it would also be a
> problem.  For instance, with
>
> { 'alternate': 'Alt',
>   'data': { 'one': 'number',
> 'two': 'str' } }
>
> we don't know what to do for value "on" branch to take for value 42.
> Not a problem because we reject this alternate.  See
> tests/qapi-schema/alternate-conflict-*json for more examples.
>
>> Better QOM introspection would be a requirement, too.
>
> I guess this is what you believe is needed to solve these problems.

Here's how we currently solve them.

We have four pipelines from text to QAPI objects:

1. JSON:
JSON   QObject wrapped QAPI
   text --> parser --> QObject --> in plain qobject --> visit --> object
input visitor

2. keyval:
keyval QObject wrapped QAPI
   text --> parser --> QObject --> in keyval qobj.  --> visit --> object
input visitor

3. "string":
text wrapped   QAPI
   text ->in string --> visit --> object
input visitor

4. QemuOpts:
 opts   QemOpts wrappedQAPI 
   text --> parser --> QemuOpts -->   in options--> visit --> object
visitor


In the JSON pipeline, the parser produces an appropriately typed
QObject, and the qobject input visitor checks it against the schema.

Dotted keys syntax doesn't tell us which scalar type to use, so the
parser uses QString for all scalars.  We then have to use a special
keyval qobject input visitor that expects strings instead of
appropriately typed scalars.  If the resulting QAPI object contains
'any' types, then those remain unconverted.  To visit them, you must use
the keyval qobject input visitor, not the plan one.

These pipelines share a tail:

   Input wrapped   QAPI
   in appropriate   --> visit --> object
   input visitor

The (still few) QAPIfied CLI options exploit this: if the option
argument looks like JSON, we feed it to the JSON half of the joint
pipeline, else to the keyval half.  The entire part before "visit" is
qobject_input_visitor_new_str(): it returns the QObject wrapped in the
appropriate visitor.

To go from QObject wrapped in visitor to a QAPI object of type T, we
generally call generated function visit_type_T().

The string pipeline is used chiefly for QOM.  The visit part is
generally manual, in the property set() method.  The string visitor is
seriously limited: it supports only scalars and lists of certain
integers.

The QemuOpts pipeline is used by a few CLI options and HMP commands,
such as -netdev, -numa, -object.  It is also almost as seriously
limited.

The code for "lists of certain integers" will make your eyes bleed.

I'd love to replace the string and the options pipeline by the keyval
pipeline.

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Paolo Bonzini  writes:

> Il dom 2 feb 2020, 10:22 Kevin Wolf  ha scritto:
>
>> Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
>> > On 1/28/20 6:54 AM, Kevin Wolf wrote:
>> >
>> > > >
>> > > > The arguments as dotted keys:
>> > > >
>> > > >  id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>> > > >
>> > > > Observe there's quite some of nesting.  While that's somewhat
>> cumbersome
>> > > > in JSON, it's a lot worse with dotted keys, because there nesting
>> means
>> > > > repeated key prefixes.  I could give much worse examples, actually.
>> > >
>> > > This is true, but even without the repeated keys (e.g. in a syntax that
>> > > would use brackets), it would still be unnecessarily verbose and
>> > > probably hard to remember:
>> > >
>> > >  id=bar,backend={type=file,data={out=/tmp/bar.log}}
>>
>> [...] I actually think that a syntax like this might make sense for
>> something like qmp-shell. It might even be more convenient on the
>> command line than dotted keys if you get a lot of repetition (despite
>> the required quoting), but it's strictly speaking incompatible because
>> you could use {} in strings today.
>>
>
> If you are willing to feed schema info to the parser, in principle you
> could keep backwards compatibility. There would be limitations such as
> putting the discriminator before the fields, so I am not sure it's a good
> idea.

Problem: the 'any' type, where the schema doesn't provide the necessary
information.

Problem: 'gen': false, where we pass the arguments raw, ignoring the
schema.

If we didn't restrict alternate types so severly, it would also be a
problem.  For instance, with

{ 'alternate': 'Alt',
  'data': { 'one': 'number',
'two': 'str' } }

we don't know what to do for value "on" branch to take for value 42.
Not a problem because we reject this alternate.  See
tests/qapi-schema/alternate-conflict-*json for more examples.

> Better QOM introspection would be a requirement, too.

I guess this is what you believe is needed to solve these problems.

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

Il dom 2 feb 2020, 10:22 Kevin Wolf  ha scritto:

> Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
> > On 1/28/20 6:54 AM, Kevin Wolf wrote:
> >
> > > >
> > > > The arguments as dotted keys:
> > > >
> > > >  id=bar,backend.type=file,backend.data.out=/tmp/bar.log
> > > >
> > > > Observe there's quite some of nesting.  While that's somewhat
> cumbersome
> > > > in JSON, it's a lot worse with dotted keys, because there nesting
> means
> > > > repeated key prefixes.  I could give much worse examples, actually.
> > >
> > > This is true, but even without the repeated keys (e.g. in a syntax that
> > > would use brackets), it would still be unnecessarily verbose and
> > > probably hard to remember:
> > >
> > >  id=bar,backend={type=file,data={out=/tmp/bar.log}}
>
> [...] I actually think that a syntax like this might make sense for
> something like qmp-shell. It might even be more convenient on the
> command line than dotted keys if you get a lot of repetition (despite
> the required quoting), but it's strictly speaking incompatible because
> you could use {} in strings today.
>

If you are willing to feed schema info to the parser, in principle you
could keep backwards compatibility. There would be limitations such as
putting the discriminator before the fields, so I am not sure it's a good
idea.

Better QOM introspection would be a requirement, too.

Paolo


> Kevin
>
>

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 31.01.2020 um 13:27 hat Eric Blake geschrieben:
> On 1/28/20 6:54 AM, Kevin Wolf wrote:
> 
> > > 
> > > The arguments as dotted keys:
> > > 
> > >  id=bar,backend.type=file,backend.data.out=/tmp/bar.log
> > > 
> > > Observe there's quite some of nesting.  While that's somewhat cumbersome
> > > in JSON, it's a lot worse with dotted keys, because there nesting means
> > > repeated key prefixes.  I could give much worse examples, actually.
> > 
> > This is true, but even without the repeated keys (e.g. in a syntax that
> > would use brackets), it would still be unnecessarily verbose and
> > probably hard to remember:
> > 
> >  id=bar,backend={type=file,data={out=/tmp/bar.log}}
> 
> With shells like bash, that would need quoting to avoid unintended brace
> expansions.  It is not the end of the world to require shell quoting (and
> passing JSON on the command line definitely needs it), but a syntax that
> avoids shell quoting is marginally easier to type and reason about.

My point was that even with such a simplified syntax (ignoring all the
implication of using it on the command line), the additional nesting
that simple unions give you would still be bad.

That said, I actually think that a syntax like this might make sense for
something like qmp-shell. It might even be more convenient on the
command line than dotted keys if you get a lot of repetition (despite
the required quoting), but it's strictly speaking incompatible because
you could use {} in strings today.

Kevin

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Kashyap Chamarthy  writes:

> On Fri, Jan 31, 2020 at 12:02:05PM +0100, Paolo Bonzini wrote:
>> Il ven 31 gen 2020, 11:36 Peter Maydell  ha
>> scritto:
>
> [...]
>
>> The advantage of putting them in the header is that you have them all in
>> one place (inline functions and structs must be in the header). In practice
>> that balances for me the disadvantage of having some comments far from the
>> code they document, which increases the risk of bitrot especially for
>> comments such as "called with lock X held".
>> 
>> I definitely agree that the overview/introduction/conventions
>> > side of things is where we'd benefit most if somebody wanted
>> > to try to tackle that. We could roll
>> > https://wiki.qemu.org/Documentation/QOMConventions
>> > into that if we had a better place to put that info.
>> >
>> 
>> I am travelling this weekend so I might try to do some kind of thread
>> summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
>> conversion and patch submission. ;-)
>
> Thanks!  Happy to be the 'scribe' ;-)  I have a skeltal
> qemu-object-model.rst file sitting with some initial content based on
> various sources, including one of your presentations[*] from 2014.
> I'll wait for your new Wiki link to incorporate that content.
>
> (Minor aside: I'm not sure if this file should be in docs/interop/ dir,
> which IIRC, is for things that are 'external' interfaces.  And I learn
> that QOM is used both internally in and as an external interface, e.g.
> whenever a device is being created, machine types, CPU config, etc.)

docs/devel/qapi-code-gen.txt has the same problem: it's mostly internal
stuff, but there's also introspection, which is an external interface.

>
> - - -
>
> I've re-skimmed your scarily-titled "QOM exegesis and apocalypse" 2014
> KVM Forum talk slides[*], where the "Why QOM?" slide says:
>
> All device creation, device configuration, backend creation and
> backed configuration done through a single interface
> 
> Rigorous support for introspection both of runtime objects and type
> capabilities

For a value of "rigorous".

Let me propose QAPI's query-qmp-schema as the tin standard[*] of
introspection:

* It's documented

* It comes with something that can pass as a type system

* It actually tells you the full truth.

Now compare to QOM:

* Documentation

  QAPI: docs/devel/qapi-code-gen.txt section "Client JSON Protocol
  introspection"

  QOM: Nada

* Type system

  QAPI: A few built-in types specified in the documentation, type
  constructors for complex types.

  QOM: Types are strings, and you just need to know what they mean.
  Some string patterns are special: link, child, STR[INT], and
  you just need to know what that means, too.

* Full truth

  QAPI: If you can access it at the interface, you can also see it in
  introspection.

  QOM: Type introspection can show you only the properties of a freshly
  created object.  Properties that get created only later are invisible.
  Properties that depend on global state are unreliable.  Object
  introspection is reliable, but only for that object in its current
  state.

> Me wonders how much of the above "Why" still holds true today.  Although
> further slides give more clues on what worked and what didn't.
>
> I'll wait for fresher details from your upcoming Wiki :-)
>
> [*] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf

[*] Gold was too expensive, but we had some tin on hand, so...

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Paolo Bonzini  writes:

> Il ven 31 gen 2020, 11:36 Peter Maydell  ha
> scritto:
>
>> On Fri, 31 Jan 2020 at 06:11, Markus Armbruster  wrote:
>> > Beware, personal opinion.
>> >
>> > When you put documentation next to the code it documents (which you
>> > absolutely should, because it's your only realistic chance to keep the
>> > two in sync), then extracting API comments is useful, because it
>> > collects them in one place.
>> >
>> > It's of next to no use to me when the comments are all in the same place
>> > already, namely the header.
>>
>
> The advantage of putting them in the header is that you have them all in
> one place (inline functions and structs must be in the header). In practice
> that balances for me the disadvantage of having some comments far from the
> code they document, which increases the risk of bitrot especially for
> comments such as "called with lock X held".

With suitable doc generation from source, we can have them next to the
code *and* all in one place, namely the generated interface docs.

>> I definitely agree that the overview/introduction/conventions
>> side of things is where we'd benefit most if somebody wanted
>> to try to tackle that. We could roll
>> https://wiki.qemu.org/Documentation/QOMConventions
>> into that if we had a better place to put that info.
>>
>
> I am travelling this weekend so I might try to do some kind of thread
> summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
> conversion and patch submission. ;-)

That would be awesome!

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 31/01/20 16:37, Christophe de Dinechin wrote:
> 
> 
>> On 31 Jan 2020, at 08:46, Paolo Bonzini  wrote:
>>
>> On 31/01/20 07:11, Markus Armbruster wrote:
>>> May I present you Armbru's Comment Trust Levels:
>>>
>>> ACTL2: The comment may be overly terse or incomplete, but the
>>> probability for it to be outright wrong is low.
>>>
>>> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
>>> code.
>>>
>>> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
>>> Signifying nothing.
>>>
>>> Most comments in decently maintained code are at ACTL1.
>>>
>>> Around the time initial QOM development solidified, object.h's comments
>>> were ACTL2.  The neglect that is now clearly visible there makes me
>>> downgrade to ACTL1.
>>>
>>> Paolo will have a more informed and possibly different opinion.
>>
>> I think around initial development it was ACTL3, now it's around 1.8.
> 
> Interestingly, the initial doc suggested ACTL to be an whole value between
> 0 and 2. Now it’s fractional and value 3 has a meaning you can guess
> to be “above 2”…

I won't say if this was accidental or not. :
Paolo

> How fast this happened tells you everything you need to know about
> documentation/specifications and actual implementations :-)

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Christophe de Dinechin]

> On 31 Jan 2020, at 08:46, Paolo Bonzini  wrote:
> 
> On 31/01/20 07:11, Markus Armbruster wrote:
>> May I present you Armbru's Comment Trust Levels:
>> 
>> ACTL2: The comment may be overly terse or incomplete, but the
>> probability for it to be outright wrong is low.
>> 
>> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
>> code.
>> 
>> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
>> Signifying nothing.
>> 
>> Most comments in decently maintained code are at ACTL1.
>> 
>> Around the time initial QOM development solidified, object.h's comments
>> were ACTL2.  The neglect that is now clearly visible there makes me
>> downgrade to ACTL1.
>> 
>> Paolo will have a more informed and possibly different opinion.
> 
> I think around initial development it was ACTL3, now it's around 1.8.

Interestingly, the initial doc suggested ACTL to be an whole value between
0 and 2. Now it’s fractional and value 3 has a meaning you can guess
to be “above 2”…

How fast this happened tells you everything you need to know about
documentation/specifications and actual implementations :-)

> 
> Paolo
> 

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Kashyap Chamarthy]

On Fri, Jan 31, 2020 at 12:02:05PM +0100, Paolo Bonzini wrote:
> Il ven 31 gen 2020, 11:36 Peter Maydell  ha
> scritto:

[...]

> The advantage of putting them in the header is that you have them all in
> one place (inline functions and structs must be in the header). In practice
> that balances for me the disadvantage of having some comments far from the
> code they document, which increases the risk of bitrot especially for
> comments such as "called with lock X held".
> 
> I definitely agree that the overview/introduction/conventions
> > side of things is where we'd benefit most if somebody wanted
> > to try to tackle that. We could roll
> > https://wiki.qemu.org/Documentation/QOMConventions
> > into that if we had a better place to put that info.
> >
> 
> I am travelling this weekend so I might try to do some kind of thread
> summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
> conversion and patch submission. ;-)

Thanks!  Happy to be the 'scribe' ;-)  I have a skeltal
qemu-object-model.rst file sitting with some initial content based on
various sources, including one of your presentations[*] from 2014.
I'll wait for your new Wiki link to incorporate that content.

(Minor aside: I'm not sure if this file should be in docs/interop/ dir,
which IIRC, is for things that are 'external' interfaces.  And I learn
that QOM is used both internally in and as an external interface, e.g.
whenever a device is being created, machine types, CPU config, etc.)

- - -

I've re-skimmed your scarily-titled "QOM exegesis and apocalypse" 2014
KVM Forum talk slides[*], where the "Why QOM?" slide says:

All device creation, device configuration, backend creation and
backed configuration done through a single interface

Rigorous support for introspection both of runtime objects and type
capabilities

Me wonders how much of the above "Why" still holds true today.  Although
further slides give more clues on what worked and what didn't.

I'll wait for fresher details from your upcoming Wiki :-)

[*] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf

-- 
/kashyap

Re: Making QEMU easier for management tools and applications

[Eric Blake]

On 1/28/20 6:54 AM, Kevin Wolf wrote:



The arguments as dotted keys:

 id=bar,backend.type=file,backend.data.out=/tmp/bar.log

Observe there's quite some of nesting.  While that's somewhat cumbersome
in JSON, it's a lot worse with dotted keys, because there nesting means
repeated key prefixes.  I could give much worse examples, actually.


This is true, but even without the repeated keys (e.g. in a syntax that
would use brackets), it would still be unnecessarily verbose and
probably hard to remember:

 id=bar,backend={type=file,data={out=/tmp/bar.log}}


With shells like bash, that would need quoting to avoid unintended brace 
expansions.  It is not the end of the world to require shell quoting 
(and passing JSON on the command line definitely needs it), but a syntax 
that avoids shell quoting is marginally easier to type and reason about.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Making QEMU easier for management tools and applications

[Eric Blake]

On 1/28/20 9:36 AM, Markus Armbruster wrote:

Cc: Eric for netdev_add QAPIfication.




These are the known "cheats" in QMP.  There's also netdev_add, but Eric
has patches to QAPIfy it properly.  Eric, I hope you can dust them off.


Yes, I'll move that up on my todo list.

When I originally did the work, we hesitated because we did not have a 
deprecation policy, and the QAPIfied version did not accept both string 
and int the way the old code did. I don't know if we want to do a hard 
break or start a deprecation clock now, but that can be decided as part 
of dusting off the series.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

Il ven 31 gen 2020, 11:36 Peter Maydell  ha
scritto:

> On Fri, 31 Jan 2020 at 06:11, Markus Armbruster  wrote:
> > Beware, personal opinion.
> >
> > When you put documentation next to the code it documents (which you
> > absolutely should, because it's your only realistic chance to keep the
> > two in sync), then extracting API comments is useful, because it
> > collects them in one place.
> >
> > It's of next to no use to me when the comments are all in the same place
> > already, namely the header.
>

The advantage of putting them in the header is that you have them all in
one place (inline functions and structs must be in the header). In practice
that balances for me the disadvantage of having some comments far from the
code they document, which increases the risk of bitrot especially for
comments such as "called with lock X held".

I definitely agree that the overview/introduction/conventions
> side of things is where we'd benefit most if somebody wanted
> to try to tackle that. We could roll
> https://wiki.qemu.org/Documentation/QOMConventions
> into that if we had a better place to put that info.
>

I am travelling this weekend so I might try to do some kind of thread
summary and brain dump in the wiki. I'll leave to Kashyap to do the rST
conversion and patch submission. ;-)

Paolo


> thanks
> -- PMM
>
>

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Peter Maydell]

On Fri, 31 Jan 2020 at 06:11, Markus Armbruster  wrote:
> Beware, personal opinion.
>
> When you put documentation next to the code it documents (which you
> absolutely should, because it's your only realistic chance to keep the
> two in sync), then extracting API comments is useful, because it
> collects them in one place.
>
> It's of next to no use to me when the comments are all in the same place
> already, namely the header.

To throw in a personal opinion on the other side, API comments
should be in the header, not the .c file, because they're
your external interface and as an external consumer of that
interface I shouldn't have to go digging around in your
implementation source file to find the documentation.

Since Paolo put in the effort to upstream the kerneldoc
Sphinx plugin, it's now relatively simple to pull in
the doc comments into a rST file, so you might as well I
guess, though I agree that the cumulative benefit over
just reading the .h file is not enormous.

I definitely agree that the overview/introduction/conventions
side of things is where we'd benefit most if somebody wanted
to try to tackle that. We could roll
https://wiki.qemu.org/Documentation/QOMConventions
into that if we had a better place to put that info.

thanks
-- PMM

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Kashyap Chamarthy]

On Fri, Jan 31, 2020 at 07:11:15AM +0100, Markus Armbruster wrote:
> Kashyap Chamarthy  writes:

[...]

> > What can be done to improve QOM documentation (or lack thereof)?
> 
> Are you trying to push us from idle grousing to actually improve things?
> No fair!

I first wrote it as a semi-complaint, and then I caught myself: "you're
not helping"; so I rephrased it to be more constructive. :-)

> > From a couple of hurried `grep` queries in the QEMU tree, there seems to
> > be no explicit qom.rst|txt, or qemu-object-model.txt|rst or some such.
> > (I hope I haven't missed any other files.)
> 
> As far as I know, all we have is the lovingly[*] written comments in
> include/qom/object.h.  Sadly, we've let them rot in places.  In
> particular, many newer functions are undocumented.
> 
> This is *reference* documentation.  What we lack (sorely!) is an
> overview / friendly introduction, and a design document with rationale.
> Reconstructing rationale now would involve guesswork.

Me nods; that (guesswork in retroactive rationale construction) makes
matters slightly more difficult.

[...]

> > Opening qom/object.h[2], indeed there is copious amounts of docs,
> > expressed as commented-out text.  Two questions:
> >
> > - How much of this is still accurate?  (Sorry, if that's a loaded
> >   question.)
> 
> May I present you Armbru's Comment Trust Levels:
>
> ACTL2: The comment may be overly terse or incomplete, but the
> probability for it to be outright wrong is low.
> 
> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
> code.
> 
> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
> Signifying nothing.
> 
> Most comments in decently maintained code are at ACTL1.

Noted.  (And thanks for the useful reference scale :-))

> Around the time initial QOM development solidified, object.h's comments
> were ACTL2.  The neglect that is now clearly visible there makes me
> downgrade to ACTL1.
> 
> Paolo will have a more informed and possibly different opinion.
> 
> > - If at least 60% is still accurate, is it valuable to extract and
> >   publish it as rendered rST, as part of the on-going QEMU Docs
> >   improvement?
> 
> Beware, personal opinion.
> 
> When you put documentation next to the code it documents (which you
> absolutely should, because it's your only realistic chance to keep the
> two in sync), then extracting API comments is useful, because it
> collects them in one place.
> 
> It's of next to no use to me when the comments are all in the same place
> already, namely the header.

Yeah, reasonable point.

> > (b) The other clue is also from the same post, where Eduardo provides
> > pointers to past KVM Forum presentations by MarkusA, PaoloB,
> > AndreasF on QOM, Qdev et al.
> >
> > Is it worth slapping all these references (with a clear intro and
> > outro) into a qom.rst file in QEMU tree, even if only for
> > reference/context?  Or are these references, in-tree docs in
> > object.h out-of-date beyond repair?  
> 
> Uff.
> 
> My qdev talks predate the rebase onto QOM.  They may well confuse /
> mislead as much as inform now.

Good to know.  (We don't want to add additional sources of confusion.)

> > If it is useful, I'm happy to get the initial doc going, secure in the
> > knowledge that more clueful people than me will chip in during the
> > review :-)
> 
> Ha, nerd sniping!

:-)

Thanks for the response; it was useful.

[...]

-- 
/kashyap

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Paolo Bonzini  writes:

> On 31/01/20 07:50, Markus Armbruster wrote:
 Consider chardev-add.  Example:

 {"execute": "chardev-add",
  "arguments": {"id": "bar",
"backend": {"type": "file",
"data": {"out": "/tmp/bar.log"

 The arguments as dotted keys:

 id=bar,backend.type=file,backend.data.out=/tmp/bar.log

 Observe there's quite some of nesting.  While that's somewhat cumbersome
 in JSON, it's a lot worse with dotted keys, because there nesting means
 repeated key prefixes.  I could give much worse examples, actually.
>>> This is true, but even without the repeated keys (e.g. in a syntax that
>>> would use brackets), it would still be unnecessarily verbose and
>>> probably hard to remember:
>>>
>>> id=bar,backend={type=file,data={out=/tmp/bar.log}}
>> No argument.  It's unnecessarily verbose in JSON, too.
>> 
>
> I think we should be able to switch chardevs to -object/object_add these
> days.  Not right now, but it may be possible.

Intriguing idea.  Would avoid the ugliness of chardev-add-2.

>Introducing a warning
> when chardev and object ids conflict would be a start.

Yes.

Perhaps even any kind instead of just chardev and object IDs.

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 31/01/20 07:50, Markus Armbruster wrote:
>>> Consider chardev-add.  Example:
>>>
>>> {"execute": "chardev-add",
>>>  "arguments": {"id": "bar",
>>>"backend": {"type": "file",
>>>"data": {"out": "/tmp/bar.log"
>>>
>>> The arguments as dotted keys:
>>>
>>> id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>>>
>>> Observe there's quite some of nesting.  While that's somewhat cumbersome
>>> in JSON, it's a lot worse with dotted keys, because there nesting means
>>> repeated key prefixes.  I could give much worse examples, actually.
>> This is true, but even without the repeated keys (e.g. in a syntax that
>> would use brackets), it would still be unnecessarily verbose and
>> probably hard to remember:
>>
>> id=bar,backend={type=file,data={out=/tmp/bar.log}}
> No argument.  It's unnecessarily verbose in JSON, too.
> 

I think we should be able to switch chardevs to -object/object_add these
days.  Not right now, but it may be possible.  Introducing a warning
when chardev and object ids conflict would be a start.

Paolo

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Paolo Bonzini]

On 31/01/20 07:11, Markus Armbruster wrote:
> May I present you Armbru's Comment Trust Levels:
> 
> ACTL2: The comment may be overly terse or incomplete, but the
> probability for it to be outright wrong is low.
> 
> ACTL1: Treat as helpful guidance (with gratitude), but trust only the
> code.
> 
> ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
> Signifying nothing.
> 
> Most comments in decently maintained code are at ACTL1.
> 
> Around the time initial QOM development solidified, object.h's comments
> were ACTL2.  The neglect that is now clearly visible there makes me
> downgrade to ACTL1.
> 
> Paolo will have a more informed and possibly different opinion.

I think around initial development it was ACTL3, now it's around 1.8.

Paolo

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf  writes:

> Am 28.01.2020 um 13:36 hat Markus Armbruster geschrieben:
>> Kevin Wolf  writes:
>> 
>> > Am 27.01.2020 um 21:11 hat John Snow geschrieben:
>> [...]
>> >> (The argument here is: It's a little harder and a little longer to type,
>> >> but the benefits from the schema organization may improve productivity
>> >> of using QEMU directly instead of harming it.)
>> >
>> > I think this is a false dichotomy.
>> >
>> > You can have everything defined by the schema and properly documented
>> > and still have a non-JSON command line. Translating the QAPI schema to
>> > a command line option is a solved problem, this is exactly how
>> > -blockdev works.
>> >
>> > The unsolved part is how to compatibly convert the existing options. If
>> > you're willing to sacrifice compatibility, great. Then we can just
>> > define stuff in the QAPI schema and still keep a command line syntax
>> > that is usable for humans. The code for mapping a QAPI type to the
>> > argument of an option is basically already there.
>> 
>> Correct.
>> 
>> Solving that problem took time, but that's sunk cost now.
>> 
>> > The only question is "is compatibility important"? If the answer is no,
>> > then we'll be there in no time.
>> 
>> I doubt we'll be there in no time, but certainly much sooner than if we
>> have to grapple with compatibility to a byzantine CLI nobody truly
>> understands.
>> 
>> There's one known issue caused by having "a non-JSON command line"
>> (actually: dotted keys as sugar for JSON): pressure to reduce nesting.
>> 
>> Consider chardev-add.  Example:
>> 
>> {"execute": "chardev-add",
>>  "arguments": {"id": "bar",
>>"backend": {"type": "file",
>>"data": {"out": "/tmp/bar.log"
>> 
>> The arguments as dotted keys:
>> 
>> id=bar,backend.type=file,backend.data.out=/tmp/bar.log
>> 
>> Observe there's quite some of nesting.  While that's somewhat cumbersome
>> in JSON, it's a lot worse with dotted keys, because there nesting means
>> repeated key prefixes.  I could give much worse examples, actually.
>
> This is true, but even without the repeated keys (e.g. in a syntax that
> would use brackets), it would still be unnecessarily verbose and
> probably hard to remember:
>
> id=bar,backend={type=file,data={out=/tmp/bar.log}}

No argument.  It's unnecessarily verbose in JSON, too.

>> We'd rather have something like
>> 
>> id=bar,type=file,out=/tmp/bar.log
>> 
>> Back to JSON:
>> 
>> "arguments": {"id": "bar", "type": "file", "out": "/tmp/bar.log"}
>> 
>> QAPI can do this, but it uses feature that predate chardev-add.
>> 
>> We don't want to duplicate the chardev-add schema in modern, flattened
>> form for the CLI.
>> 
>> So the compatibility problem actually shifts to QMP: can we evolve the
>> existing QMP command compatibly at a reasonable cost in design, coding
>> and complexity to support flat arguments?
>
> Well, first of all: Do we need compatibility? If we don't, then we can
> just make the change.

The trouble with flattening this one is QMP, where we promise stability.

> Much of this threads plays with the though that maybe we don't need any
> compatibility and make the radical conclusion that we don't need any
> human-friendly interface at all. Keeping full compatibility is the other
> extreme.
>
> There might be some middle ground where we break compatibility where the
> old way can't easily be maintained with the new infrastructure, but
> don't give up on the idea of being used by humans.

I'm not sure the connection between maintaining compatibility and
supporting human use is as strong as you seem to imply.

As far as I can tell, the "maybe we don't need any compatibility"
discussion is about the CLI.  I'd rephrase it as "maybe we need a
machine-friendly CLI on par with QMP more than we need compatibility to
the current CLI".

"We don't need any human-friendly interface at all" comes in not because
machine-friendly necessarily precludes human-friendly, but only if we're
unwilling (unable?) to do the extra work for it.

Compare the monitor:

* QMP is primarily for machines.  We promise stability: no incompatible
  changes without clear communicaton of intent and a grace period.  We
  provide machine clients tools to deal with the interface evolution,
  e.g. query-qmp-schema.

* HMP is exclusively for humans.  It may change at any time.

For the CLI, we don't have such a separation, and our offerings for
dealing with interface evolution are wholly inadequate.  We *need* to do
better for machines.

Now, the monitor also informs us about the cost of providing a
completely separate interface for humans.

Elsewhere in this thread, we discussed layering (a replacement for) HMP
on top of QMP cleanly, possibly in a separate process, possibly written
in a high-level language like Python.

HMP predates QMP.  We reworked it so the HMP commands are implemented on
top of the QMP commands, or at least on top of common helpers.  But 

Re: Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Markus Armbruster]

Kashyap Chamarthy  writes:

> On Wed, Jan 15, 2020 at 03:02:48PM +0100, Markus Armbruster wrote:
>> Daniel P. Berrangé  writes:
>
> [Changed the subject-line to indicate deviation from the original
> topic.]
>
> [...]
>
>> > Libvirt is of course happy to switch to something else instead of
>> > qom-set for these features if QEMU wants to provide a safer
>> > alternative.
>> 
>> Noted.
>> 
>> libvirt's use of qom-set is okay.  What's not okay is the near-complete
>> lack of QOM documentation, its poor QMP interface, in part due to
>> non-integration with QAPI, and last but not least the lack of QOM
>> leadership leaving it adrift.
>
> What can be done to improve QOM documentation (or lack thereof)?

Are you trying to push us from idle grousing to actually improve things?
No fair!

> From a couple of hurried `grep` queries in the QEMU tree, there seems to
> be no explicit qom.rst|txt, or qemu-object-model.txt|rst or some such.
> (I hope I haven't missed any other files.)

As far as I know, all we have is the lovingly[*] written comments in
include/qom/object.h.  Sadly, we've let them rot in places.  In
particular, many newer functions are undocumented.

This is *reference* documentation.  What we lack (sorely!) is an
overview / friendly introduction, and a design document with rationale.
Reconstructing rationale now would involve guesswork.

> Let's dig further.  Ah, I come across this helpful 2016 blog post[1]
> ("An incomplete list of QEMU APIs") by Eduardo from my bookmarks.  Here
> I get some clues:
>
> (a) In the section titled "QOM", Eduardo writes:
>
> "QOM is short for QEMU Object Model and was introduced in 2011.
> It is heavily documented on its header file
> [include/qom/object.h]" 
>
> Opening qom/object.h[2], indeed there is copious amounts of docs,
> expressed as commented-out text.  Two questions:
>
> - How much of this is still accurate?  (Sorry, if that's a loaded
>   question.)

May I present you Armbru's Comment Trust Levels:

ACTL2: The comment may be overly terse or incomplete, but the
probability for it to be outright wrong is low.

ACTL1: Treat as helpful guidance (with gratitude), but trust only the
code.

ACTL0: It is a tale Told by an idiot[**], full of sound and fury,
Signifying nothing.

Most comments in decently maintained code are at ACTL1.

Around the time initial QOM development solidified, object.h's comments
were ACTL2.  The neglect that is now clearly visible there makes me
downgrade to ACTL1.

Paolo will have a more informed and possibly different opinion.

> - If at least 60% is still accurate, is it valuable to extract and
>   publish it as rendered rST, as part of the on-going QEMU Docs
>   improvement?

Beware, personal opinion.

When you put documentation next to the code it documents (which you
absolutely should, because it's your only realistic chance to keep the
two in sync), then extracting API comments is useful, because it
collects them in one place.

It's of next to no use to me when the comments are all in the same place
already, namely the header.

> (b) The other clue is also from the same post, where Eduardo provides
> pointers to past KVM Forum presentations by MarkusA, PaoloB,
> AndreasF on QOM, Qdev et al.
>
> Is it worth slapping all these references (with a clear intro and
> outro) into a qom.rst file in QEMU tree, even if only for
> reference/context?  Or are these references, in-tree docs in
> object.h out-of-date beyond repair?  

Uff.

My qdev talks predate the rebase onto QOM.  They may well confuse /
mislead as much as inform now.

> If it is useful, I'm happy to get the initial doc going, secure in the
> knowledge that more clueful people than me will chip in during the
> review :-)

Ha, nerd sniping!

> [1] https://habkost.net/posts/2016/11/incomplete-list-of-qemu-apis.html
> [2] https://git.qemu.org/?p=qemu.git;a=blob;f=include/qom/object.h
> [3] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf


[*] Absolutely no irony intended.  Honest, officer!

[**] Don't take it personally, we're all part-time idiots.  Besides,
this is literature.

Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications]

[Kashyap Chamarthy]

On Wed, Jan 15, 2020 at 03:02:48PM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé  writes:

[Changed the subject-line to indicate deviation from the original
topic.]

[...]

> > Libvirt is of course happy to switch to something else instead of
> > qom-set for these features if QEMU wants to provide a safer
> > alternative.
> 
> Noted.
> 
> libvirt's use of qom-set is okay.  What's not okay is the near-complete
> lack of QOM documentation, its poor QMP interface, in part due to
> non-integration with QAPI, and last but not least the lack of QOM
> leadership leaving it adrift.

What can be done to improve QOM documentation (or lack thereof)?

>From a couple of hurried `grep` queries in the QEMU tree, there seems to
be no explicit qom.rst|txt, or qemu-object-model.txt|rst or some such.
(I hope I haven't missed any other files.)

Let's dig further.  Ah, I come across this helpful 2016 blog post[1]
("An incomplete list of QEMU APIs") by Eduardo from my bookmarks.  Here
I get some clues:

(a) In the section titled "QOM", Eduardo writes:

"QOM is short for QEMU Object Model and was introduced in 2011.
It is heavily documented on its header file
[include/qom/object.h]" 

Opening qom/object.h[2], indeed there is copious amounts of docs,
expressed as commented-out text.  Two questions:

- How much of this is still accurate?  (Sorry, if that's a loaded
  question.)

- If at least 60% is still accurate, is it valuable to extract and
  publish it as rendered rST, as part of the on-going QEMU Docs
  improvement?

(b) The other clue is also from the same post, where Eduardo provides
pointers to past KVM Forum presentations by MarkusA, PaoloB,
AndreasF on QOM, Qdev et al.

Is it worth slapping all these references (with a clear intro and
outro) into a qom.rst file in QEMU tree, even if only for
reference/context?  Or are these references, in-tree docs in
object.h out-of-date beyond repair?  

If it is useful, I'm happy to get the initial doc going, secure in the
knowledge that more clueful people than me will chip in during the
review :-)

[1] https://habkost.net/posts/2016/11/incomplete-list-of-qemu-apis.html
[2] https://git.qemu.org/?p=qemu.git;a=blob;f=include/qom/object.h
[3] http://www.linux-kvm.org/images/9/90/Kvmforum14-qom.pdf

-- 
/kashyap

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/28/20 5:16 AM, Markus Armbruster wrote:
> I'm deeply skeptical of any solution that starts with inventing a
> new language.  Start with examining existing ones instead.

I promise that we are in violent agreement.

Later in the mail:

"Any prior art, other projects, and reading anyone can recommend would
be nice."

--js

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Cc: Eric for netdev_add QAPIfication.

Kevin Wolf  writes:

> Am 28.01.2020 um 11:16 hat Daniel P. Berrangé geschrieben:
>> On Mon, Jan 27, 2020 at 11:38:49PM +0100, Paolo Bonzini wrote:
>> > Il lun 27 gen 2020, 21:11 John Snow  ha scritto:
>> > 
>> > >
>> > > > ./qemu-core <> > > {
>> > > "machine": "Q35",
>> > > "memory": "2GiB",
>> > > "accel": "kvm"
>> > > }
>> > > EOF
>> > >
>> > 
>> > And now you have to keep all the syntactic sugar that is in vl.c. I don't
>> > think a redesign of -readconfig should accept anything less verbose than
>> > 
>> > - machine:
>> > type: q35
>> > ram:
>> >type: memory-backend-hostmem
>> >size: 2G
>> > - accel:
>> >   - type: kvm
>> > 
>> > And this is not even taking into account disks.
>> > 
>> > I like the idea of refactoring all the syntactic sugar into a pre-pass on
>> > command line options. This is not an entirely new idea, see
>> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.
>> > 
>> > I am afraid that this thread is derailing a bit, with lots of pipe dreams
>> > and no actionable items. How do we get it back in track?
>> 
>> To me the one thing that is clear. No matter what approach we want to
>> take to QEMU configuration/interaction/CLI/etc, one critical bit of
>> work is a pre-requisite...
>> 
>> ...we must finish[1] the QAPI modelling of QEMU's features in some
>> short, finite timeframe. We can't let it drag on for another 5 years
>> or more. I'd say we need a timeframe that is 2 years max, preferrably
>> 1 year.
>> 
>> I don't think we can achieve this by leaving the task up to to the
>> QAPI maintainers alone. It is unreasonable to put such a burden to
>> on a small number of people to both implement & review it all. We
>> need to consider it a project wide priority item so that we can get
>> broader involvement across all maintainers, in closing the gaps.

Thank you, Daniel.  More on it below.

>> I'm not sure if we have any clear list of where our known gaps exist ?
>
> I don't know about a full list, but I've been discussing command line
> QAPIfication a bit with Markus recently because we had the idea of using
> qemu-storage-daemon as a guinea pig for it.

I still like that idea.  We can explore a 100% QAPIfied CLI there with
minimal disruption elsewhere, and without CLI compatibility concerns.
Constraints due to the shared QAPI schema remain, unless we freely
duplicate stuff, which would probably be a bad idea.

> The big one seems to be QOM (and qdev). object-add and device-add are
> both not defined in terms of QAPI. One of them uses an "any" type (which
> results in QObjects with arbitrary content being passed), the other one
> "gen": false (which avoids generating anything from the schema).

These are the known "cheats" in QMP.  There's also netdev_add, but Eric
has patches to QAPIfy it properly.  Eric, I hope you can dust them off.

For CLI, we have numerous options to QAPIfy.  Some of them are trivial.
Others involve QAPIfying substantial code behind them: I don't want a
QAPIfied option to immediately stuff everything into QemuOpts for
outmoded internal interfaces.  Such shortcuts is what got us into the
netdev_add mess.

QAPIfying internal interfaces is one of the areas where we QAPI guys
will need help.

> I know that some more options exist that have unusual syntax and are
> hard to convert to QAPI while maintaining command line compatibility.

Weird semantics can also get in the way.  For instance, when we replaced
-drive with -blockdev, we used the opportunity to ditch block backend
auto-deletion.

> Maybe that should be solved by just designing new options and
> deprecating the old ones.

Re: Making QEMU easier for management tools and applications

[Gerd Hoffmann]

> > Consider chardev-add.  Example:
> > 
> > {"execute": "chardev-add",
> >  "arguments": {"id": "bar",
> >"backend": {"type": "file",
> >"data": {"out": "/tmp/bar.log"

> Well, first of all: Do we need compatibility?

Well, libvirt uses that, so yes.  Or at least support both old and new
(chardev-add-v2?) in parallel for while to give libvirt time to switch
over.

cheers,
  Gerd

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 28.01.2020 um 13:36 hat Markus Armbruster geschrieben:
> Kevin Wolf  writes:
> 
> > Am 27.01.2020 um 21:11 hat John Snow geschrieben:
> [...]
> >> (The argument here is: It's a little harder and a little longer to type,
> >> but the benefits from the schema organization may improve productivity
> >> of using QEMU directly instead of harming it.)
> >
> > I think this is a false dichotomy.
> >
> > You can have everything defined by the schema and properly documented
> > and still have a non-JSON command line. Translating the QAPI schema to
> > a command line option is a solved problem, this is exactly how
> > -blockdev works.
> >
> > The unsolved part is how to compatibly convert the existing options. If
> > you're willing to sacrifice compatibility, great. Then we can just
> > define stuff in the QAPI schema and still keep a command line syntax
> > that is usable for humans. The code for mapping a QAPI type to the
> > argument of an option is basically already there.
> 
> Correct.
> 
> Solving that problem took time, but that's sunk cost now.
> 
> > The only question is "is compatibility important"? If the answer is no,
> > then we'll be there in no time.
> 
> I doubt we'll be there in no time, but certainly much sooner than if we
> have to grapple with compatibility to a byzantine CLI nobody truly
> understands.
> 
> There's one known issue caused by having "a non-JSON command line"
> (actually: dotted keys as sugar for JSON): pressure to reduce nesting.
> 
> Consider chardev-add.  Example:
> 
> {"execute": "chardev-add",
>  "arguments": {"id": "bar",
>"backend": {"type": "file",
>"data": {"out": "/tmp/bar.log"
> 
> The arguments as dotted keys:
> 
> id=bar,backend.type=file,backend.data.out=/tmp/bar.log
> 
> Observe there's quite some of nesting.  While that's somewhat cumbersome
> in JSON, it's a lot worse with dotted keys, because there nesting means
> repeated key prefixes.  I could give much worse examples, actually.

This is true, but even without the repeated keys (e.g. in a syntax that
would use brackets), it would still be unnecessarily verbose and
probably hard to remember:

id=bar,backend={type=file,data={out=/tmp/bar.log}}

> We'd rather have something like
> 
> id=bar,type=file,out=/tmp/bar.log
> 
> Back to JSON:
> 
> "arguments": {"id": "bar", "type": "file", "out": "/tmp/bar.log"}
> 
> QAPI can do this, but it uses feature that predate chardev-add.
> 
> We don't want to duplicate the chardev-add schema in modern, flattened
> form for the CLI.
> 
> So the compatibility problem actually shifts to QMP: can we evolve the
> existing QMP command compatibly at a reasonable cost in design, coding
> and complexity to support flat arguments?

Well, first of all: Do we need compatibility? If we don't, then we can
just make the change.

Much of this threads plays with the though that maybe we don't need any
compatibility and make the radical conclusion that we don't need any
human-friendly interface at all. Keeping full compatibility is the other
extreme.

There might be some middle ground where we break compatibility where the
old way can't easily be maintained with the new infrastructure, but
don't give up on the idea of being used by humans.

Kevin

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf  writes:

> Am 27.01.2020 um 21:11 hat John Snow geschrieben:
[...]
>> (The argument here is: It's a little harder and a little longer to type,
>> but the benefits from the schema organization may improve productivity
>> of using QEMU directly instead of harming it.)
>
> I think this is a false dichotomy.
>
> You can have everything defined by the schema and properly documented
> and still have a non-JSON command line. Translating the QAPI schema to
> a command line option is a solved problem, this is exactly how
> -blockdev works.
>
> The unsolved part is how to compatibly convert the existing options. If
> you're willing to sacrifice compatibility, great. Then we can just
> define stuff in the QAPI schema and still keep a command line syntax
> that is usable for humans. The code for mapping a QAPI type to the
> argument of an option is basically already there.

Correct.

Solving that problem took time, but that's sunk cost now.

> The only question is "is compatibility important"? If the answer is no,
> then we'll be there in no time.

I doubt we'll be there in no time, but certainly much sooner than if we
have to grapple with compatibility to a byzantine CLI nobody truly
understands.

There's one known issue caused by having "a non-JSON command line"
(actually: dotted keys as sugar for JSON): pressure to reduce nesting.

Consider chardev-add.  Example:

{"execute": "chardev-add",
 "arguments": {"id": "bar",
   "backend": {"type": "file",
   "data": {"out": "/tmp/bar.log"

The arguments as dotted keys:

id=bar,backend.type=file,backend.data.out=/tmp/bar.log

Observe there's quite some of nesting.  While that's somewhat cumbersome
in JSON, it's a lot worse with dotted keys, because there nesting means
repeated key prefixes.  I could give much worse examples, actually.

We'd rather have something like

id=bar,type=file,out=/tmp/bar.log

Back to JSON:

"arguments": {"id": "bar", "type": "file", "out": "/tmp/bar.log"}

QAPI can do this, but it uses feature that predate chardev-add.

We don't want to duplicate the chardev-add schema in modern, flattened
form for the CLI.

So the compatibility problem actually shifts to QMP: can we evolve the
existing QMP command compatibly at a reasonable cost in design, coding
and complexity to support flat arguments?

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 27.01.2020 um 21:29 hat Dr. David Alan Gilbert geschrieben:
> * Kevin Wolf (kw...@redhat.com) wrote:
> > A command history already exists for qmp-shell. It's better than bash
> > because it doesn't mix QMP history with whatever else I do on my
> > computer.
> > 
> > Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> > implemented, it could be a lot more useful than bash completion because
> > it could offer key completion based on the QMP schema.
> > 
> > This is in fact a big part of the problem that qmp-shell really needs to
> > solve before it can replace HMP: How to make writing commands at least
> > almost as simple as with HMP. If I can just press tab a few times to
> > cycle through the available options for the command, that would already
> > be a massive improvement over writing JSON manually (which you would
> > still have to do with your text-file based approach, without any
> > QMP-specific support).
> 
> Doing all that in a python process (i.e. an actual python shell with a
> bunch of qemu commands added) seems easyish.

It does. Polishing everything for the perfect experience could be a bit
more work, but just querying the schema and providing some
autocompletion from it sounds easy enough.

> > The other part that it needs to solve is how to be available by default
> > without specifying anything on the command line. Basically, if I press
> > Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> > implemented internally or by an external Python process, I don't mind.
> 
> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
> use HMP on stdin).

I don't think it would be that hard, actually.

If you have a -qmp-shell option that takes a chardev and defaults to vc,
you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
to do is launch the Python child process, pass it a pair of pipes for
communication and forward everything between the pipes and the chardev.

(That's the theory anyway.)

Kevin

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 28.01.2020 um 11:16 hat Daniel P. Berrangé geschrieben:
> On Mon, Jan 27, 2020 at 11:38:49PM +0100, Paolo Bonzini wrote:
> > Il lun 27 gen 2020, 21:11 John Snow  ha scritto:
> > 
> > >
> > > > ./qemu-core < > > {
> > > "machine": "Q35",
> > > "memory": "2GiB",
> > > "accel": "kvm"
> > > }
> > > EOF
> > >
> > 
> > And now you have to keep all the syntactic sugar that is in vl.c. I don't
> > think a redesign of -readconfig should accept anything less verbose than
> > 
> > - machine:
> > type: q35
> > ram:
> >type: memory-backend-hostmem
> >size: 2G
> > - accel:
> >   - type: kvm
> > 
> > And this is not even taking into account disks.
> > 
> > I like the idea of refactoring all the syntactic sugar into a pre-pass on
> > command line options. This is not an entirely new idea, see
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.
> > 
> > I am afraid that this thread is derailing a bit, with lots of pipe dreams
> > and no actionable items. How do we get it back in track?
> 
> To me the one thing that is clear. No matter what approach we want to
> take to QEMU configuration/interaction/CLI/etc, one critical bit of
> work is a pre-requisite...
> 
> ...we must finish[1] the QAPI modelling of QEMU's features in some
> short, finite timeframe. We can't let it drag on for another 5 years
> or more. I'd say we need a timeframe that is 2 years max, preferrably
> 1 year.
> 
> I don't think we can achieve this by leaving the task up to to the
> QAPI maintainers alone. It is unreasonable to put such a burden to
> on a small number of people to both implement & review it all. We
> need to consider it a project wide priority item so that we can get
> broader involvement across all maintainers, in closing the gaps.
> 
> I'm not sure if we have any clear list of where our known gaps exist ?

I don't know about a full list, but I've been discussing command line
QAPIfication a bit with Markus recently because we had the idea of using
qemu-storage-daemon as a guinea pig for it.

The big one seems to be QOM (and qdev). object-add and device-add are
both not defined in terms of QAPI. One of them uses an "any" type (which
results in QObjects with arbitrary content being passed), the other one
"gen": false (which avoids generating anything from the schema).

I know that some more options exist that have unusual syntax and are
hard to convert to QAPI while maintaining command line compatibility.
Maybe that should be solved by just designing new options and
deprecating the old ones.

Kevin

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 27.01.2020 um 21:11 hat John Snow geschrieben:
> On 1/27/20 6:56 AM, Kevin Wolf wrote:
> > Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
> >> Kevin Wolf  writes:
> >>> basically just a pretty-printed version of it with the consequence that
> >>> it needs to be stored in an external file and there is no reasonable way
> >>> to keep it in my shell history.
> >>
> >> There is a reasonable way to keep it in my file system, though.  I find
> >> that decidedly superior.
> > 
> > That depends a lot on your use case.
> > 
> > If you have a long-lived production VM that you always run with the same
> > configuration, then yes, having a config file for it in the file system
> > is what you probably want. Currently, for this case, people directly
> > using QEMU tend to write a script that contains the command line. I
> > think I do have such scripts somewhere, but their number is very small.
> > 
> > My common case is short-lived VMs with configurations that change very
> > often between QEMU invocations. Here the command line is decidedly
> > superior.
> > 
> > Requiring me to create a file in the file system each time and to
> > remember deleting it after I'm done feels about as convenient as a *nix
> > shell that doesn't accept parameters for commands on the command line,
> > but instead requires you to write a one-off script first and then run
> > that.
> > 
> > Kevin
> > 
> 
> > ./qemu-core < {
> "machine": "Q35",
> "memory": "2GiB",
> "accel": "kvm"
> }
> EOF

I'm not sure why everybody is trying to come up with crude workarounds
when all I said is that keeping things in a file is not always
"decidedly superior".

Of course, I would have points to make about that workaround (like that
I often want to use stdin for an interactive monitor, or that I don't
see how this is superior to --machine Q35 --memory 2GiB --accel kvm),
but that's all beside the point.

> No file required, cooperates with readline, avoids crunchy,
> hard-to-maintain CLI syntax. Directly and easily translatable to a
> stored-file configuration. All configuration and documentation is
> centralized via QAPI.
> 
> A little worse to type manually, yes. Maybe not bad /enough/ for me to
> want to rescue the CLI which prevents full QAPI-fication and a working
> configuration file.
> 
> Arguably, a well documented configuration schema will be much easier to
> browse, discover, and use than a labyrinthine CLI with many stub
> definitions whose options are not exposed in the documentation.
> 
> (The argument here is: It's a little harder and a little longer to type,
> but the benefits from the schema organization may improve productivity
> of using QEMU directly instead of harming it.)

I think this is a false dichotomy.

You can have everything defined by the schema and properly documented
and still have a non-JSON command line. Translating the QAPI schema to
a command line option is a solved problem, this is exactly how
-blockdev works.

The unsolved part is how to compatibly convert the existing options. If
you're willing to sacrifice compatibility, great. Then we can just
define stuff in the QAPI schema and still keep a command line syntax
that is usable for humans. The code for mapping a QAPI type to the
argument of an option is basically already there.

The only question is "is compatibility important"? If the answer is no,
then we'll be there in no time.

Kevin

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Mon, Jan 27, 2020 at 11:38:49PM +0100, Paolo Bonzini wrote:
> Il lun 27 gen 2020, 21:11 John Snow  ha scritto:
> 
> >
> > > ./qemu-core < > {
> > "machine": "Q35",
> > "memory": "2GiB",
> > "accel": "kvm"
> > }
> > EOF
> >
> 
> And now you have to keep all the syntactic sugar that is in vl.c. I don't
> think a redesign of -readconfig should accept anything less verbose than
> 
> - machine:
> type: q35
> ram:
>type: memory-backend-hostmem
>size: 2G
> - accel:
>   - type: kvm
> 
> And this is not even taking into account disks.
> 
> I like the idea of refactoring all the syntactic sugar into a pre-pass on
> command line options. This is not an entirely new idea, see
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.
> 
> I am afraid that this thread is derailing a bit, with lots of pipe dreams
> and no actionable items. How do we get it back in track?

To me the one thing that is clear. No matter what approach we want to
take to QEMU configuration/interaction/CLI/etc, one critical bit of
work is a pre-requisite...

...we must finish[1] the QAPI modelling of QEMU's features in some
short, finite timeframe. We can't let it drag on for another 5 years
or more. I'd say we need a timeframe that is 2 years max, preferrably
1 year.

I don't think we can achieve this by leaving the task up to to the
QAPI maintainers alone. It is unreasonable to put such a burden to
on a small number of people to both implement & review it all. We
need to consider it a project wide priority item so that we can get
broader involvement across all maintainers, in closing the gaps.

I'm not sure if we have any clear list of where our known gaps exist ?

If not a good first action point is do make such a list, so we know
what we're trying to attack.

Regards,
Daniel

[1] For a definition of "finish" that may or may not be equal to
100% of all possible features. It might be valid to declare
we only care about "finishing" QAPI for features used by
libvirt, since those represent the "common case" usage.
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow  writes:

> On 1/27/20 9:35 AM, Kevin Wolf wrote:
>> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
>>> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
 Well, sure. The context of this email was qmp-shell though, which is
 meant to help facilitate the entry of JSON commands so that you *can*
 indeed just forego the CLI/HMP entirely.

 If you are of the opinion that every user of QEMU should be copy/pasting
 JSON straight into a socket and we should delete qmp-shell, that's
 certainly a fine opinion.
>>>
>>> I think part of the pain of qmp-shell comes from the very fact that
>>> it is trying to be an interactive shell. This points people towards
>>> interactively typing in the commands, which is horrific when you get
>>> anywhere near the JSON, or even dot-notation traditional commands.
>>>
>>> If it was just a qmp-client that was single shot, we'd encourage
>>> people to create the JSON in a sensible way - vim/emacs/whatever.
>> 
>> I don't see how this is sensible. QMP commands are something that I
>> reuse even less than VM configurations, so creating a one-off file for
>> each would take me a lot more time and I would still have to type the
>> same JSON code that I have to type with -qmp stdio.
>> 
>> The reason it is and should be an interactive shell is that I'm
>> interacting with it. Switching back and forth between a text editor and
>> a shell to actually send the command to QEMU would make things only even
>> more cumbersome than they already are.
>> 
>>> Bash/dash/zsh/$whatever is their interactive shell, with massively
>>> more features than qmp-shell. You have command history, autocomplete,
>>> conditional and looping constructs, and everything a normal shell
>>> offers.
>> 
>> If I wanted to program a QMP client, I would use Python. For me,
>> conditionals and loops are completely out of scope for a QMP shell. I
>> just want an easy way to tell QEMU to do something specific.
>> 
>> A command history already exists for qmp-shell. It's better than bash
>> because it doesn't mix QMP history with whatever else I do on my
>> computer.
>> 
>> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
>> implemented, it could be a lot more useful than bash completion because
>> it could offer key completion based on the QMP schema.
>> 
>
> It does have tab completion for command names, but it does not know
> about or remember argument fields. It does not have autocomplete or
> typing hints like FiSH or bash ^r.
>
> I would like to change this, actually, by making the docstrings in QAPI
> schema a first class citizen of the spec and allowing them to be
> introspectable via the socket directly.
>
> (I.e., you can get the list of arguments and the docstrings that
> accompany them over the wire so you can display it in the client.)

You need doc strings for help, but not for completion.

query-qmp-schema's reply is already rather big: 130KiB.

The size of docs in an JSON encoding useful to qmp-shell we can only
estimate.  Plain text docs/interop/qemu-qmp-ref.txt is 505KiB.

I guess we'd make the doc strings optional.

> Problem I'm having with qmp-shell is, like Kevin says below ...
>
>> This is in fact a big part of the problem that qmp-shell really needs to
>> solve before it can replace HMP: How to make writing commands at least
>> almost as simple as with HMP. If I can just press tab a few times to
>> cycle through the available options for the command, that would already
>> be a massive improvement over writing JSON manually (which you would
>> still have to do with your text-file based approach, without any
>> QMP-specific support).
>> 
>
> ... I can't figure out how to make writing commands simple.
>
> When you have a "simple" command, the abstraction works OK; you can type
> key=val pairs and go about your way.

qmp-shell tries to improve on the JSON syntax by inventing its own,
but...

> As soon as you have anything nested, the gossamer-thin illusion is
> destroyed.

... it does a halfhearted job.

I'm deeply skeptical of any solution that starts with inventing a
new language.  Start with examining existing ones instead.

>I investigated making this a little easier by adding a parser
> that could read directly from stdin and would allow multi-line JSON
> inputs as arguments.
>
> (Like the python shell does it, for example: When you have a dictionary
> opening brace, it lets you continue to the next line.)
>
> I was a little disheartened that most JSON parsers in python expect to
> consume buffered objects and generally consume the entire buffer -- it
> didn't seem to play nice with the idea of wanting to parse from STDIN
> directly.
>
>
> So:
>
> - I think qmp-shell is worth having, especially if polished
> (autocomplete, docstrings, argument hints, etc).
>
> - Kevin mentioned getting this into the GTK shell. I think that would be
> great, as a step to help phase out HMP.
>
> - I think getting rid of HMP is good 

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/27/20 5:38 PM, Paolo Bonzini wrote:
> 
> 
> Il lun 27 gen 2020, 21:11 John Snow  > ha scritto:
> 
> 
> > ./qemu-core < {
>     "machine": "Q35",
>     "memory": "2GiB",
>     "accel": "kvm"
> }
> EOF
> 
> 
> And now you have to keep all the syntactic sugar that is in vl.c. I

Didn't mean to suggest literal values, I just grabbed some from the top
of my head. Let's pretend I didn't use sugared ones. This is a
distraction from the point.

The point is: you can specify JSON on the command line, as stdin to QEMU
as a here document. No config file needed.

> don't think a redesign of -readconfig should accept anything less
> verbose than
> 
> - machine:
>     type: q35
>     ram:
>        type: memory-backend-hostmem
>        size: 2G
> - accel:
>   - type: kvm
> 
> And this is not even taking into account disks.
> 

Though, point taken, even minimal configurations might get too cumbersome.

> I like the idea of refactoring all the syntactic sugar into a pre-pass
> on command line options. This is not an entirely new idea,
> see https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.

This could be very good, as long as the transformation was knowable in
some way. (Documented, modeled in the schema somehow, observable
post-transformation, whichever.)

(wow, 2010.)

> 
> I am afraid that this thread is derailing a bit, with lots of pipe
> dreams and no actionable items. How do we get it back in track?
> 

That depends on what your opinion of "on track" is. If there are
specific angles you find interesting, it would be easiest to start a
thread around the framing you find productive.

--js

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

Il lun 27 gen 2020, 21:11 John Snow  ha scritto:

>
> > ./qemu-core < {
> "machine": "Q35",
> "memory": "2GiB",
> "accel": "kvm"
> }
> EOF
>

And now you have to keep all the syntactic sugar that is in vl.c. I don't
think a redesign of -readconfig should accept anything less verbose than

- machine:
type: q35
ram:
   type: memory-backend-hostmem
   size: 2G
- accel:
  - type: kvm

And this is not even taking into account disks.

I like the idea of refactoring all the syntactic sugar into a pre-pass on
command line options. This is not an entirely new idea, see
https://www.mail-archive.com/qemu-devel@nongnu.org/msg35024.html.

I am afraid that this thread is derailing a bit, with lots of pipe dreams
and no actionable items. How do we get it back in track?

Paolo


> No file required, cooperates with readline, avoids crunchy,
> hard-to-maintain CLI syntax. Directly and easily translatable to a
> stored-file configuration. All configuration and documentation is
> centralized via QAPI.
>
> A little worse to type manually, yes. Maybe not bad /enough/ for me to
> want to rescue the CLI which prevents full QAPI-fication and a working
> configuration file.
>
> Arguably, a well documented configuration schema will be much easier to
> browse, discover, and use than a labyrinthine CLI with many stub
> definitions whose options are not exposed in the documentation.
>
> (The argument here is: It's a little harder and a little longer to type,
> but the benefits from the schema organization may improve productivity
> of using QEMU directly instead of harming it.)
>
> --js
>
>

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/27/20 9:35 AM, Kevin Wolf wrote:
> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
>> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>>> Well, sure. The context of this email was qmp-shell though, which is
>>> meant to help facilitate the entry of JSON commands so that you *can*
>>> indeed just forego the CLI/HMP entirely.
>>>
>>> If you are of the opinion that every user of QEMU should be copy/pasting
>>> JSON straight into a socket and we should delete qmp-shell, that's
>>> certainly a fine opinion.
>>
>> I think part of the pain of qmp-shell comes from the very fact that
>> it is trying to be an interactive shell. This points people towards
>> interactively typing in the commands, which is horrific when you get
>> anywhere near the JSON, or even dot-notation traditional commands.
>>
>> If it was just a qmp-client that was single shot, we'd encourage
>> people to create the JSON in a sensible way - vim/emacs/whatever.
> 
> I don't see how this is sensible. QMP commands are something that I
> reuse even less than VM configurations, so creating a one-off file for
> each would take me a lot more time and I would still have to type the
> same JSON code that I have to type with -qmp stdio.
> 
> The reason it is and should be an interactive shell is that I'm
> interacting with it. Switching back and forth between a text editor and
> a shell to actually send the command to QEMU would make things only even
> more cumbersome than they already are.
> 
>> Bash/dash/zsh/$whatever is their interactive shell, with massively
>> more features than qmp-shell. You have command history, autocomplete,
>> conditional and looping constructs, and everything a normal shell
>> offers.
> 
> If I wanted to program a QMP client, I would use Python. For me,
> conditionals and loops are completely out of scope for a QMP shell. I
> just want an easy way to tell QEMU to do something specific.
> 
> A command history already exists for qmp-shell. It's better than bash
> because it doesn't mix QMP history with whatever else I do on my
> computer.
> 
> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> implemented, it could be a lot more useful than bash completion because
> it could offer key completion based on the QMP schema.
> 

It does have tab completion for command names, but it does not know
about or remember argument fields. It does not have autocomplete or
typing hints like FiSH or bash ^r.

I would like to change this, actually, by making the docstrings in QAPI
schema a first class citizen of the spec and allowing them to be
introspectable via the socket directly.

(I.e., you can get the list of arguments and the docstrings that
accompany them over the wire so you can display it in the client.)

Problem I'm having with qmp-shell is, like Kevin says below ...

> This is in fact a big part of the problem that qmp-shell really needs to
> solve before it can replace HMP: How to make writing commands at least
> almost as simple as with HMP. If I can just press tab a few times to
> cycle through the available options for the command, that would already
> be a massive improvement over writing JSON manually (which you would
> still have to do with your text-file based approach, without any
> QMP-specific support).
> 

... I can't figure out how to make writing commands simple.

When you have a "simple" command, the abstraction works OK; you can type
key=val pairs and go about your way.

As soon as you have anything nested, the gossamer-thin illusion is
destroyed. I investigated making this a little easier by adding a parser
that could read directly from stdin and would allow multi-line JSON
inputs as arguments.

(Like the python shell does it, for example: When you have a dictionary
opening brace, it lets you continue to the next line.)

I was a little disheartened that most JSON parsers in python expect to
consume buffered objects and generally consume the entire buffer -- it
didn't seem to play nice with the idea of wanting to parse from STDIN
directly.


So:

- I think qmp-shell is worth having, especially if polished
(autocomplete, docstrings, argument hints, etc).

- Kevin mentioned getting this into the GTK shell. I think that would be
great, as a step to help phase out HMP.

- I think getting rid of HMP is good because I don't care for the idea
of supporting two monitor protocols. One schema, one parser, one truth.

- I do, however, like the idea of having a non-rigorous monitor that
lets us get away with murder when we need to. HMP is useful for
debugging, prototypes and other things where the rigor and permanence of
a QAPI schema feels too burdensome.

- So: maybe a turbocharged qmp-shell can offer some similar kinds of
convenience commands that are build on top of real QMP. Sugar logic and
other fanciful things could be implemented there in qmp-shell as
extensions. You'd get a stock pile of them with your QEMU install that
help you do certain tasks quickly and trivially.

- 

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* Kevin Wolf (kw...@redhat.com) wrote:
> Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
> > On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
> > > Well, sure. The context of this email was qmp-shell though, which is
> > > meant to help facilitate the entry of JSON commands so that you *can*
> > > indeed just forego the CLI/HMP entirely.
> > > 
> > > If you are of the opinion that every user of QEMU should be copy/pasting
> > > JSON straight into a socket and we should delete qmp-shell, that's
> > > certainly a fine opinion.
> > 
> > I think part of the pain of qmp-shell comes from the very fact that
> > it is trying to be an interactive shell. This points people towards
> > interactively typing in the commands, which is horrific when you get
> > anywhere near the JSON, or even dot-notation traditional commands.
> > 
> > If it was just a qmp-client that was single shot, we'd encourage
> > people to create the JSON in a sensible way - vim/emacs/whatever.
> 
> I don't see how this is sensible. QMP commands are something that I
> reuse even less than VM configurations, so creating a one-off file for
> each would take me a lot more time and I would still have to type the
> same JSON code that I have to type with -qmp stdio.
> 
> The reason it is and should be an interactive shell is that I'm
> interacting with it. Switching back and forth between a text editor and
> a shell to actually send the command to QEMU would make things only even
> more cumbersome than they already are.
> 
> > Bash/dash/zsh/$whatever is their interactive shell, with massively
> > more features than qmp-shell. You have command history, autocomplete,
> > conditional and looping constructs, and everything a normal shell
> > offers.
> 
> If I wanted to program a QMP client, I would use Python. For me,
> conditionals and loops are completely out of scope for a QMP shell. I
> just want an easy way to tell QEMU to do something specific.
> 
> A command history already exists for qmp-shell. It's better than bash
> because it doesn't mix QMP history with whatever else I do on my
> computer.
> 
> Autocomplete in qmp-shell doesn't exist, as far as I know, but if
> implemented, it could be a lot more useful than bash completion because
> it could offer key completion based on the QMP schema.
> 
> This is in fact a big part of the problem that qmp-shell really needs to
> solve before it can replace HMP: How to make writing commands at least
> almost as simple as with HMP. If I can just press tab a few times to
> cycle through the available options for the command, that would already
> be a massive improvement over writing JSON manually (which you would
> still have to do with your text-file based approach, without any
> QMP-specific support).

Doing all that in a python process (i.e. an actual python shell with a
bunch of qemu commands added) seems easyish.

> The other part that it needs to solve is how to be available by default
> without specifying anything on the command line. Basically, if I press
> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
> implemented internally or by an external Python process, I don't mind.

That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
use HMP on stdin).

Dave

> > The only strong reason for qmp-shell to be interactive would be if
> > the initial protoocl handshake was too slow. I can't see that being
> > a problem with QMP.
> 
> Speed would be the least of my concerns. This is about manual use, and
> it already takes me a while to type in my commands.
> 
> > Example usage:
> > 
> > 1. Launch the QEMU runtime for the desired target
> > 
> >  $ qemu-runtime-x86_64 myvm.sock
> > 
> > 2. Load the configuration to define the VM
> > 
> >$ cat myvm.yaml
> >commands:
> >  - machine_declare:
> >  name: pc-q35-5.0
> >  ...
> >  - blockdev_add:
> >  ...
> >  - device_add:
> >  ...
> >  - blockdev_add:
> >  ...
> >  - device_add:
> >  ...
> >$ qemu-client myvm.sock myvm.yaml
> > 
> > 
> > 3. Hotplug a disk
> > 
> >$ cat mynewdisk.yaml
> >commands:
> >  - blockdev_add:
> >  ...
> >  - device_add:
> >  ...
> >$ qemu-client myvm.sock mynewdisk.yaml
> > 
> > 
> > 3. Hotunplug a disk
> > 
> >$ cat myolddisk.yaml
> >commands:
> >  - device_del:
> >  ...
> >  - blockdev_del:
> >  ...
> >$ qemu-client myvm.sock myolddisk.yaml
> 
> Just to compare, this is what the human user oriented flow looks like
> today:
> 
> 1. qemu-system-x86_64 -M pc-q35-5.0 -drive if=virtio,... -cdrom ...
> 
> 2. 
>(qemu) drive_add ...
>
> 
> 3. 
>(qemu) device_del ...
>
> 
> This is what we're competing with, and honestly I don't see how your
> qemu-runtime-*/qemu-client based flow comes even close to it in terms of
> usability.
> 
> QMP, JSON and YAML may be nice machine interfaces, but having nice
> machine interfaces doesn't mean that 

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* Kevin Wolf (kw...@redhat.com) wrote:
> Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
> > Kevin Wolf  writes:
> > 
> > > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> > >> This is finally something I'd consider to be on a par with the
> > >> original QEMU syntax, before we added hierarchical data. You
> > >> have the minimal possible amount of syntax here. No commas,
> > >> no quotes, no curly brackets, etc.
> > >
> > > This seems to have the same problems as the QEMU command line (how do
> > > you distinguish strings from ints, from bools, from null?).
> > 
> > True: YAML provides only string scalars.
> > 
> > TOML provides strings, integers, floats, booleans, and several flavors
> > of time.  It lacks null.
> > 
> > > It's
> > > basically just a pretty-printed version of it with the consequence that
> > > it needs to be stored in an external file and there is no reasonable way
> > > to keep it in my shell history.
> > 
> > There is a reasonable way to keep it in my file system, though.  I find
> > that decidedly superior.
> 
> That depends a lot on your use case.
> 
> If you have a long-lived production VM that you always run with the same
> configuration, then yes, having a config file for it in the file system
> is what you probably want. Currently, for this case, people directly
> using QEMU tend to write a script that contains the command line. I
> think I do have such scripts somewhere, but their number is very small.
> 
> My common case is short-lived VMs with configurations that change very
> often between QEMU invocations. Here the command line is decidedly
> superior.

I can imagine you could do something similar to gdb's --eval-command
option which lets you pass a command to be executed at startup; gdb's
syntax is a bit painful but it feels like that could be fixed, e.g. I
have:

gdb --eval-command='set pagination off' --eval-command='set confirm no' 
--eval-command='handle SIGPIPE print pas
s nostop' --eval-command='handle SIGBUS print pass nostop' --eval-command=r 
--eval-command='thread apply all bt full'
 --eval-command='info proc mappings' --eval-command='info threads' 
--eval-command=q --args $QEMUTODEBUG "$@"

in a script.

Dave

> Requiring me to create a file in the file system each time and to
> remember deleting it after I'm done feels about as convenient as a *nix
> shell that doesn't accept parameters for commands on the command line,
> but instead requires you to write a one-off script first and then run
> that.
> 
> Kevin
> 
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/27/20 6:56 AM, Kevin Wolf wrote:
> Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
>> Kevin Wolf  writes:
>>
>>> Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
 This is finally something I'd consider to be on a par with the
 original QEMU syntax, before we added hierarchical data. You
 have the minimal possible amount of syntax here. No commas,
 no quotes, no curly brackets, etc.
>>>
>>> This seems to have the same problems as the QEMU command line (how do
>>> you distinguish strings from ints, from bools, from null?).
>>
>> True: YAML provides only string scalars.
>>
>> TOML provides strings, integers, floats, booleans, and several flavors
>> of time.  It lacks null.
>>
>>> It's
>>> basically just a pretty-printed version of it with the consequence that
>>> it needs to be stored in an external file and there is no reasonable way
>>> to keep it in my shell history.
>>
>> There is a reasonable way to keep it in my file system, though.  I find
>> that decidedly superior.
> 
> That depends a lot on your use case.
> 
> If you have a long-lived production VM that you always run with the same
> configuration, then yes, having a config file for it in the file system
> is what you probably want. Currently, for this case, people directly
> using QEMU tend to write a script that contains the command line. I
> think I do have such scripts somewhere, but their number is very small.
> 
> My common case is short-lived VMs with configurations that change very
> often between QEMU invocations. Here the command line is decidedly
> superior.
> 
> Requiring me to create a file in the file system each time and to
> remember deleting it after I'm done feels about as convenient as a *nix
> shell that doesn't accept parameters for commands on the command line,
> but instead requires you to write a one-off script first and then run
> that.
> 
> Kevin
> 

> ./qemu-core <

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/25/20 6:41 AM, Paolo Bonzini wrote:
> On 20/01/20 10:55, Stefan Hajnoczi wrote:
>>>
>>> [1] https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html
>> John and I discussed async events in the past.  qmp-shell currently uses
>> the input() built-in function.  If we modify it with a
>> select(2)/poll(2)-style function that monitors both stdin and the QMP
>> socket then it could print QMP events as soon as they are received.
> 
> I think it should be rewritten using async/await.  A simple example:
> 
> import asyncio
> import sys
> from concurrent.futures import ThreadPoolExecutor
> 
> async def ainput(prompt: str = ""):
> with ThreadPoolExecutor(1, "ainput") as executor:
> return (await asyncio.get_event_loop().run_in_executor(
> executor, sys.stdin.readline
> )).rstrip()
> 
> async def numbers():
> i = 1
> while True:
> print(i)
> i = i + 1
> await asyncio.sleep(1)
> 
> async def main():
> name = await ainput("What's your name? ")
> print("Hello, {}!".format(name))
> 
> asyncio.get_event_loop().create_task(numbers())
> asyncio.get_event_loop().run_until_complete(main())
> 
> This would be a great Summer of Code project.  Even an autocompletion
> interface using readline should be possible.
> 
> Paolo
> 

I wrote an async version at one point; I had problems integrating
asyncio with readline functionality.

--js

Re: Making QEMU easier for management tools and applications

[Gerd Hoffmann]

 Hi,

> > We already have qapi schema for -blockdev + -display + -audiodev +
> > -chardev, creating a configuration struct which supports these
> > shouldn't be that much effort.  Then move over QemuOpts one by one.
> 
> chardev-add uses a QAPI type that has too deep nesting to be user
> friendly.

Yea, right, it is relatively old and doesn't use stuff like base types
which didn't exist back then ...

> > Once config file support is complete enough we should be able to offload
> > backward compatibility command line parsing to some script which
> > transforms the cli into a config file.  Which is probably better than
> > trying to add -writeconfig to qemu as we don't have to do the string
> > processing in C then.
> 
> If we get a launcher script anyway, I would argue that the system
> emulator binary written in C should only support JSON (like it already
> does) and the script can easily translate from YAML to JSON.

That makes sense indeed.

cheers,
  Gerd

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 24.01.2020 um 10:50 hat Daniel P. Berrangé geschrieben:
> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
> > Well, sure. The context of this email was qmp-shell though, which is
> > meant to help facilitate the entry of JSON commands so that you *can*
> > indeed just forego the CLI/HMP entirely.
> > 
> > If you are of the opinion that every user of QEMU should be copy/pasting
> > JSON straight into a socket and we should delete qmp-shell, that's
> > certainly a fine opinion.
> 
> I think part of the pain of qmp-shell comes from the very fact that
> it is trying to be an interactive shell. This points people towards
> interactively typing in the commands, which is horrific when you get
> anywhere near the JSON, or even dot-notation traditional commands.
> 
> If it was just a qmp-client that was single shot, we'd encourage
> people to create the JSON in a sensible way - vim/emacs/whatever.

I don't see how this is sensible. QMP commands are something that I
reuse even less than VM configurations, so creating a one-off file for
each would take me a lot more time and I would still have to type the
same JSON code that I have to type with -qmp stdio.

The reason it is and should be an interactive shell is that I'm
interacting with it. Switching back and forth between a text editor and
a shell to actually send the command to QEMU would make things only even
more cumbersome than they already are.

> Bash/dash/zsh/$whatever is their interactive shell, with massively
> more features than qmp-shell. You have command history, autocomplete,
> conditional and looping constructs, and everything a normal shell
> offers.

If I wanted to program a QMP client, I would use Python. For me,
conditionals and loops are completely out of scope for a QMP shell. I
just want an easy way to tell QEMU to do something specific.

A command history already exists for qmp-shell. It's better than bash
because it doesn't mix QMP history with whatever else I do on my
computer.

Autocomplete in qmp-shell doesn't exist, as far as I know, but if
implemented, it could be a lot more useful than bash completion because
it could offer key completion based on the QMP schema.

This is in fact a big part of the problem that qmp-shell really needs to
solve before it can replace HMP: How to make writing commands at least
almost as simple as with HMP. If I can just press tab a few times to
cycle through the available options for the command, that would already
be a massive improvement over writing JSON manually (which you would
still have to do with your text-file based approach, without any
QMP-specific support).

The other part that it needs to solve is how to be available by default
without specifying anything on the command line. Basically, if I press
Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
implemented internally or by an external Python process, I don't mind.

> The only strong reason for qmp-shell to be interactive would be if
> the initial protoocl handshake was too slow. I can't see that being
> a problem with QMP.

Speed would be the least of my concerns. This is about manual use, and
it already takes me a while to type in my commands.

> Example usage:
> 
> 1. Launch the QEMU runtime for the desired target
> 
>  $ qemu-runtime-x86_64 myvm.sock
> 
> 2. Load the configuration to define the VM
> 
>$ cat myvm.yaml
>commands:
>  - machine_declare:
>  name: pc-q35-5.0
>...
>  - blockdev_add:
>  ...
>  - device_add:
>  ...
>  - blockdev_add:
>  ...
>  - device_add:
>  ...
>$ qemu-client myvm.sock myvm.yaml
> 
> 
> 3. Hotplug a disk
> 
>$ cat mynewdisk.yaml
>commands:
>  - blockdev_add:
>  ...
>  - device_add:
>  ...
>$ qemu-client myvm.sock mynewdisk.yaml
> 
> 
> 3. Hotunplug a disk
> 
>$ cat myolddisk.yaml
>commands:
>  - device_del:
>  ...
>  - blockdev_del:
>  ...
>$ qemu-client myvm.sock myolddisk.yaml

Just to compare, this is what the human user oriented flow looks like
today:

1. qemu-system-x86_64 -M pc-q35-5.0 -drive if=virtio,... -cdrom ...

2. 
   (qemu) drive_add ...
   

3. 
   (qemu) device_del ...
   

This is what we're competing with, and honestly I don't see how your
qemu-runtime-*/qemu-client based flow comes even close to it in terms of
usability.

QMP, JSON and YAML may be nice machine interfaces, but having nice
machine interfaces doesn't mean that you shouldn't also have something
that is suitable for humans. qmp-shell is trying to be that, and while
it leaves much to be desired in its current state, replacing it with
even more machine-friendly stuff that is cumbersome for humans isn't the
right answer.

Kevin

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Daniel P. Berrangé  writes:

> On Sat, Jan 25, 2020 at 11:18:41AM +0100, Markus Armbruster wrote:
>> Kevin Wolf  writes:
>> 
>> > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
>> >> So if we allowed YAML instead of JSON, now we get...
>> >> 
>> >>   $ cat qemu.yaml
[...]
>> >> This is finally something I'd consider to be on a par with the
>> >> original QEMU syntax, before we added hierarchical data. You
>> >> have the minimal possible amount of syntax here. No commas,
>> >> no quotes, no curly brackets, etc.
>> >
>> > This seems to have the same problems as the QEMU command line (how do
>> > you distinguish strings from ints, from bools, from null?).
>> 
>> True: YAML provides only string scalars.
>
> Actually, YAML provides  strings, integers, booleans & nulls, the same
> level of typing at JSON, but with added benefit of supporting comments
> officially:
>
>   https://rollout.io/blog/yaml-tutorial-everything-you-need-get-started/

You're right.  I read section 2.3 Scalars of the YAML spec[*], and
neglected to study sections 2.4. Tags and 10.3. Core Schema.


[*] https://yaml.org/spec/1.2/spec.html

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 27.01.2020 um 09:35 hat Gerd Hoffmann geschrieben:
>   Hi,
> 
> > We build a "configuration" struct in QAPI, and extend from there.
> > 
> > (2) We offer "--config myconfig.yaml" as an option for specifying
> > options.
> 
> Yes.
> 
> > This precludes the use of *any* traditional command line flags.
> 
> Hmm.  Given that the transition effort will probably take a while
> I think it makes sense to allow mixing config file and cmd line
> switches, if it is only for testing the config file parsing and
> processing.  Maybe have a (temporary) -x-config for that?
> 
> We already have qapi schema for -blockdev + -display + -audiodev +
> -chardev, creating a configuration struct which supports these
> shouldn't be that much effort.  Then move over QemuOpts one by one.

chardev-add uses a QAPI type that has too deep nesting to be user
friendly. Markus and I are trying to use mostly QAPIfied command line
options for qemu-storage-daemon, and --chardev is one that we agree must
be simplified to be bearable.

> The most tricky part here probably is -device support.

Possibly, though -object might not be much easier.

> > - We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
> > Supporting two deserialization layers could lead to strange
> > discrepancies between the two formats at runtime. So, admittedly, using
> > JSON would be *even easier*, as we could re-use the same parsers already
> > battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.
> 
> Well, we can support multiple formats, much like openshift accepts both
> json and yaml.  json is pretty much there already, so we could start
> with that and add yaml later on.  Possibly even the ini-style syntax
> accepted by -readconfig today, although I'm not sure there is much
> benefit in that.
> 
> > - We will break compatibility with our existing CLI. People will not be
> > happy about this, especially, perhaps, embedded board and TCG developers
> > who use fairly minimal command lines regularly.
> 
> Once config file support is complete enough we should be able to offload
> backward compatibility command line parsing to some script which
> transforms the cli into a config file.  Which is probably better than
> trying to add -writeconfig to qemu as we don't have to do the string
> processing in C then.

If we get a launcher script anyway, I would argue that the system
emulator binary written in C should only support JSON (like it already
does) and the script can easily translate from YAML to JSON.

Kevin

Re: Making QEMU easier for management tools and applications

[Peter Maydell]

On Mon, 27 Jan 2020 at 11:56, Kevin Wolf  wrote:
> If you have a long-lived production VM that you always run with the same
> configuration, then yes, having a config file for it in the file system
> is what you probably want. Currently, for this case, people directly
> using QEMU tend to write a script that contains the command line. I
> think I do have such scripts somewhere, but their number is very small.

I have some similar scripts, which I use for launching one-off
"run and then kill soon" VMs, mostly as test setups. The
advantage of a script is that you get an actual programming
language and can do things like "substitute in the name of
the directory the script lives in" when setting up parameters
that are filenames, or easily support "this is a default, and
you can override it with an environment variable". So I'd
still need to have a script even if the script changed from
"run QEMU with these command line options" to "create a temp
file with this config and run QEMU on it".

thanks
-- PMM

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 25.01.2020 um 11:18 hat Markus Armbruster geschrieben:
> Kevin Wolf  writes:
> 
> > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> >> This is finally something I'd consider to be on a par with the
> >> original QEMU syntax, before we added hierarchical data. You
> >> have the minimal possible amount of syntax here. No commas,
> >> no quotes, no curly brackets, etc.
> >
> > This seems to have the same problems as the QEMU command line (how do
> > you distinguish strings from ints, from bools, from null?).
> 
> True: YAML provides only string scalars.
> 
> TOML provides strings, integers, floats, booleans, and several flavors
> of time.  It lacks null.
> 
> > It's
> > basically just a pretty-printed version of it with the consequence that
> > it needs to be stored in an external file and there is no reasonable way
> > to keep it in my shell history.
> 
> There is a reasonable way to keep it in my file system, though.  I find
> that decidedly superior.

That depends a lot on your use case.

If you have a long-lived production VM that you always run with the same
configuration, then yes, having a config file for it in the file system
is what you probably want. Currently, for this case, people directly
using QEMU tend to write a script that contains the command line. I
think I do have such scripts somewhere, but their number is very small.

My common case is short-lived VMs with configurations that change very
often between QEMU invocations. Here the command line is decidedly
superior.

Requiring me to create a file in the file system each time and to
remember deleting it after I'm done feels about as convenient as a *nix
shell that doesn't accept parameters for commands on the command line,
but instead requires you to write a one-off script first and then run
that.

Kevin

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Sat, Jan 25, 2020 at 11:18:41AM +0100, Markus Armbruster wrote:
> Kevin Wolf  writes:
> 
> > Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> >> So if we allowed YAML instead of JSON, now we get...
> >> 
> >>   $ cat qemu.yaml
> >>   ---
> >>   arguments:
> >>   - arg: object
> >> data:
> >>   type: secret
> >>   id: libvirt-5-storage-secret0
> >>   data: 
> >> 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
> >>   keyid: masterKey0
> >>   iv: AAECAwQFBgcICQoLDA0ODw==
> >>   format: base64
> >>   - arg: object
> >> data:
> >>   type: secret
> >>   id: libvirt-5-format-luks-secret0
> >>   data: 
> >> 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
> >>   keyid: masterKey0
> >>   iv: AAECAwQFBgcICQoLDA0ODw==
> >>   format: base64
> >>   - arg: blockdev
> >> data:
> >>   node-name: libvirt-5-format
> >>   read-only: false
> >>   driver: qcow2
> >>   encrypt:
> >> format: luks
> >> key-secret: libvirt-5-format-luks-secret0
> >>   file:
> >> driver: iscsi
> >> portal: example.org:6000
> >> target: iqn.1992-01.com.example:storage
> >> lun: 1
> >> transport: tcp
> >> user: myname
> >> password-secret: libvirt-5-storage-secret0
> >> node-name: libvirt-5-storage
> >> auto-read-only: true
> >> discard: unmap
> >>   $ qemu-system-x86_64 -f qemu.yaml
> >> 
> >> This is finally something I'd consider to be on a par with the
> >> original QEMU syntax, before we added hierarchical data. You
> >> have the minimal possible amount of syntax here. No commas,
> >> no quotes, no curly brackets, etc.
> >
> > This seems to have the same problems as the QEMU command line (how do
> > you distinguish strings from ints, from bools, from null?).
> 
> True: YAML provides only string scalars.

Actually, YAML provides  strings, integers, booleans & nulls, the same
level of typing at JSON, but with added benefit of supporting comments
officially:

  https://rollout.io/blog/yaml-tutorial-everything-you-need-get-started/

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Sat, Jan 25, 2020 at 12:52:30PM +0100, Paolo Bonzini wrote:
> On 24/01/20 10:50, Daniel P. Berrangé wrote:
> >   * qemu-launcher-$TARGET
> > 
> > A binary that is able to launch qemu-runtime-$TARGET
> > with jailers active.
> > 
> > This has no command line arguments except for a pair
> > of UNIX socket paths. One is a QMP server, the other
> > is the path for the QMP of qemu-runtime-$TARGET.
> > 
> > Commands it processes will be in automatically proxied
> > through to the qemu-runtime-$TARGET QMP, with appropriate
> > jailer updates being done in between.
> > 
> 
> What would be the advantage of this over the Libvirt embedded driver?
> Especially if you include in the picture something like libvirt-go-xml
> (or libvirt-GObject, does it still exist?) that hides the XML from the
> code that uses it.

Yes, libvirt-gobject still exists & is used by GNOME Boxes.

> The main complication in the launcher is hotplug, which means that a
> simple "do a couple bind mounts, unshare, drop privileges and forget
> about it" approach doesn't work.  Proxying QMP commands doesn't seem
> that easy, and I don't see much code being shared between the launcher
> and QEMU; if the existing QEMU code is not suitable for Libvirt, it
> wouldn't be suitable for a qemu.git launcher either.

Mostly this suggestion is a response to discussions elsewhere in this
thread expressing a desire for a way to do QEMU jailing natively. In
this proposal I'm describing an approach that might satisfy that desire,
while ignoring whatever exists in the current ecosystem including
libvirt.

So yes, there is clearly overlap here with libvirt, and this tool is
not an easy thing to build to a level which is feature comparable
with what libvirt offers. For this launcher to be credible I think
it would have to be something libvirt were capable of using itself
and that's a challenge. So quite possibly the libvirt embedded driver
is indeed a better bet.  


> Also, as you mentioned earlier, QEMU wants to keep its vocabulary
> lower-level, and therefore the launcher's vocabulary would end up
> diverging from QEMU.  Some example:
> 
> - QEMU wants a qemu-pr-helper socket path, the launcher would take care
> of launching qemu-pr-helper itself
> 
> - QEMU wants the complete configuration on the migration destination,
> the launcher might take care of sending it from the source?
> 
> At this point, you get something that looks very much like Libvirt and,
> especially if you include live migration, it has to take into account
> the same compatibility considerations as Libvirt.

Yes indeed.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Mon, Jan 27, 2020 at 10:06:45AM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé  writes:
> >   * qemu-system-$TARGET
> >
> > The current binaries that exist today.
> >
> > qemu-system-$TARGET should not be part of our formal
> > stability promise. We won't gratuitously / knowingly
> > break without good reason, but we will accept that
> > breakage can happen. Stability is only offered by
> > the qemu-{runtime,launcher}-$TARGET.
> >
> > Several choices for their future in long term:
> >
> >   - Leave them as-is and basically ignore them
> > whereever practical going forward, so we
> > minimally worry about backcompat breakage
> >
> >   - Plan to re-write them so that they are simply
> > a shim the forks+execs qemu-runtime-$TARGET
> > and does syntax translation from CLI/HMP/QMP.
> >
> >   - Deprecate them with a view to deletion entirely
> > in $NNN years. For some large-ish value of NNN,
> > given how well known they are
> 
> How do the other complex executables like qemu-img, qemu-nbd fit into
> this picture?
> 
> Do they become redundant somehow for non-human users?
> 
> If not, will they get a QMP-only sibling, like qemu-system-$TARGET gets
> qemu-runtime-$TARGET?

I've not really thought about the other tools when writing this.

In libvirt we do see much of the same pain points in dealing with
these tools, as we do with QEMU emulator targets though. eg need
to probe features, the complexity of specifying block devices.
So there's clearly some conceptual overlap here to consider.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Paolo Bonzini  writes:

> Il mer 15 gen 2020, 10:21 Markus Armbruster  ha scritto:
>
>> > We don’t want the QAPI to let arbitrary fields of a QOM object
>> > be modified, do we?
>>
>> We already do: QMP command qom-set.  If it breaks your guest, you get to
>> keep the pieces.
>>
>
> That's not true. We chose not to make that a "recommended" interface, and
> instead we add new commands. However that's mostly to avoid tying our hands
> and not making too much of QOM part of the API. But I would be very
> surprised if a guest could be broken with qom-set.
>
> This was definitely not the case when QOM was introduced in a half-baked
> state, but let's not indulge in self-flagellation more than it's actually
> necessary.

Nobody thinks of qom-set when adding a QOM property.

qom-set runs a setter function that can do anything.  Is that safe at
any point of time?  Depends.

Fortunately, most setters merely write to some object instance variable.

Often, the object's code reads the instance variable only in realize(),
and never writes to it.  qom-set is safe then, but not terribly useful.

More interesting are the ones that are read and written throughout the
object's life cycle.  Is qom-set safe at any point of time?  Depends.

Your belief qom-set can't break a guest in practice is reassuring.

However, the point I was trying to make with "you get to keep the
pieces" is precisely that we're "not making too much of QOM part of the
API".

An additional point I'd like to repeat now: which parts of QOM are is
pretty much anybody's guess.

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Daniel P. Berrangé  writes:

> On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
>> 
>> 
>> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
[...]
>> > I guess my point is that with a scrap & startover view point, we
>> > should arguably completely ignore the design question of how to
>> > flatten JSON for humans/command line, as it is the wrong problem.
>> > Instead focus on the problem of making use of JSON the best way
>> > to deal with QEMU both functionally and practically, for humans
>> > and machines alike.

Note: I understand "JSON" to stand for "JSON and possibly another
concrete syntax better suited for humans".  Your examples below show
YAML in that role.

[...]
> Here's one conceptual vision of how a better QEMU might look:
>
>   * qemu-runtime-$TARGET
>
> A binary that contains the implementation for the machine
> emulator for $TARGET.
>
> This has no command line arguments except for a UNIX
> socket path which is a QMP server
>
>
>   * qemu-launcher-$TARGET
>
> A binary that is able to launch qemu-runtime-$TARGET
> with jailers active.
>
> This has no command line arguments except for a pair
> of UNIX socket paths. One is a QMP server, the other
> is the path for the QMP of qemu-runtime-$TARGET.
>
> Commands it processes will be in automatically proxied
> through to the qemu-runtime-$TARGET QMP, with appropriate
> jailer updates being done in between.

See Paolo's reply.

>   * qemu-client
>
> A binary that speaks QMP, connects, runs a single command,
> disconnects.

The single command is specified how?

Ah, "Example usage" below shows it's taken from a file argument.

> It is used to talk to either qemu-runtime-$TARGET or
> qemu-launcher-$TARGET, depending on whether the mgmt app
> or user wants to be making use of the jailer facilities
> or not.  

Effectively ditches the CLI in favor of QMP.  One major external host
interface instead of two.

The connection to "making use of JSON the best way to deal with QEMU
both functionally and practically, for humans and machines alike" is
"make it possible to use just QMP for *everything*".  Correct?

We've toyed with the "Just QMP" idea almost since QMP exists.  We never
went beyond "wouldn't it be nice if".

The main chunk of work is providing the CLI functionality we need in
QMP.  Requires QAPIfication of the parts that aren't QAPIfied, yet.

Since parts of CLI overlap with QMP, providing CLI functionality need
not require new QMP commands.  For instance, CLI does cold plug, QMP
does hot plug.  The existing QMP interface could do both: hot while the
machine runs, else cold.

Some functionality makes sense only during initial startup.  Implied in
CLI.  In QMP, it has to be tied to the run state.  No big deal.

"Just QMP" is of course not the only way to do "JSON first".  Another
one is bringing JSON to the CLI.  I explored that: CLI QAPIfication.

Requires the same QAPIfication as "Just QMP" does.

Where "Just QMP" is a hard compatibility break by design, "JSON in CLI"
gives us a choice: we can choose to break compatibility, e.g. by
creating the exact same set of new executables you propose.  Or we can
try to evolve the existing CLI compatibly.  The latter is a tar pit,
because the existing CLI is.  But it's a tar pit *we* get to choose.  We
can't blame "JSON in CLI" for it, only ourselves.

Even though I've worked on "JSON in CLI", I'm not overly attached to it.
If we decide "Just QMP" is the better path forward, I'll happily go
along.  I believe the bulk of the work will be the same: QAPIfying
stuff.

>   * qemu-system-$TARGET
>
> The current binaries that exist today.
>
> qemu-system-$TARGET should not be part of our formal
> stability promise. We won't gratuitously / knowingly
> break without good reason, but we will accept that
> breakage can happen. Stability is only offered by
> the qemu-{runtime,launcher}-$TARGET.
>
> Several choices for their future in long term:
>
>   - Leave them as-is and basically ignore them
> whereever practical going forward, so we
>   minimally worry about backcompat breakage
>
>   - Plan to re-write them so that they are simply
> a shim the forks+execs qemu-runtime-$TARGET
>   and does syntax translation from CLI/HMP/QMP.
>
>   - Deprecate them with a view to deletion entirely
> in $NNN years. For some large-ish value of NNN,
>   given how well known they are

How do the other complex executables like qemu-img, qemu-nbd fit into
this picture?

Do they become redundant somehow for non-human users?

If not, will they get a QMP-only sibling, like qemu-system-$TARGET gets
qemu-runtime-$TARGET?

> Example usage:
>
> 1. Launch the QEMU runtime for the desired target
>
>  $ qemu-runtime-x86_64 myvm.sock
>
> 2. Load the configuration to define the VM
>
>$ cat myvm.yaml
>commands:
>  - machine_declare:
>  name: pc-q35-5.0
>...
>  - 

Re: Making QEMU easier for management tools and applications

[Gerd Hoffmann]

  Hi,

> We build a "configuration" struct in QAPI, and extend from there.
> 
> (2) We offer "--config myconfig.yaml" as an option for specifying
> options.

Yes.

> This precludes the use of *any* traditional command line flags.

Hmm.  Given that the transition effort will probably take a while
I think it makes sense to allow mixing config file and cmd line
switches, if it is only for testing the config file parsing and
processing.  Maybe have a (temporary) -x-config for that?

We already have qapi schema for -blockdev + -display + -audiodev +
-chardev, creating a configuration struct which supports these
shouldn't be that much effort.  Then move over QemuOpts one by one.

The most tricky part here probably is -device support.

> (4) Just scrap the existing config file system entirely. It was never
> finished and does not help solve the existing problem that we do not
> have a schema for our configuration syntax.

Yep.  Zero reason to keep that once everything transitioned to qapi.

> - We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
> Supporting two deserialization layers could lead to strange
> discrepancies between the two formats at runtime. So, admittedly, using
> JSON would be *even easier*, as we could re-use the same parsers already
> battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.

Well, we can support multiple formats, much like openshift accepts both
json and yaml.  json is pretty much there already, so we could start
with that and add yaml later on.  Possibly even the ini-style syntax
accepted by -readconfig today, although I'm not sure there is much
benefit in that.

> - We will break compatibility with our existing CLI. People will not be
> happy about this, especially, perhaps, embedded board and TCG developers
> who use fairly minimal command lines regularly.

Once config file support is complete enough we should be able to offload
backward compatibility command line parsing to some script which
transforms the cli into a config file.  Which is probably better than
trying to add -writeconfig to qemu as we don't have to do the string
processing in C then.

cheers,
  Gerd

Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin]

> On 23 Jan 2020, at 18:58, John Snow  wrote:
> 
> 
> 
> On 1/23/20 2:19 AM, Markus Armbruster wrote:
>> John Snow  writes:
>> 
>>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
> * scripts/qmp/qmp-shell
> 
>  Half-hearted attempt at a human-friendly wrapper around the JSON
>  syntax.  I have no use for this myself.
 I use this fairly often as its a useful debugging / experimentation
 / trouble shooting tool. There's similar ish functionality in
 virsh qemu-monitor-command. I think there's scope of a supported
 tool here that can talk to libvirt or a UNIX socket for doing
 QMP commands, with a friendlier syntax & pretty printing. 
 
>>> 
>>> qmp-shell is one of my go-to tools for working through bitmap workflows
>>> where we don't have convenience commands yet, as some of the setups
>>> required for fleecing et al involve quite a number of steps.
>>> 
>>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>>> commands neatly organized in a format where I can visually reduce them
>>> to their components at a glance.
>>> 
>>> (What I mean is: It's hard to remember which QMP commands you've barfed
>>> into a terminal because JSON is hard to read and looks very visually
>>> repetitive.)
>>> 
>>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>>> a new REPL that was json-aware in some manner such that you could write
>>> multi-line commands like this:
>>> 
 example-command arg={
>>>  "hello": "world"
>>> }
>>> 
>>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>>> into Python as-is simply take a file pointer and consume the entirety of
>>> the rest of the stream -- they don't play very nice with incomplete
>>> input or input that may have trailing data, e.g.:
>>> 
 example-command arg={
>>>  "hello": "world"
>>> } arg2={
>>>  "oops!": "more json!"
>>> }
>> 
>> QMP is in the same boat: it needs to process input that isn't
>> necessarily full expressions (JSON-text in the RFC's grammar).
>> 
>> Any conventional parser can be made streaming by turning it into a
>> coroutine.  This is probably the simplest solution for handwritten
>> streaming LL parsers, because it permits recursive descent.  In Python,
>> I'd try a generator.
>> 
>> Our actual solution for QMP predates coroutine support in QEMU, and is
>> rather hamfisted:
>> 
>> * Streaming lexer: it gets fed characters one at a time, and when its
>>  state machine says "token complete", it feeds the token to the
>>  "streamer".
>> 
>> * "Streamer": gets fed tokens one at a time, buffers them up counting
>>  curly and square bracket nesting until the nesting is zero, then
>>  passes the buffered tokens to the parser.
>> 
>> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>>  a full expression.
>> 
>> The best I can say about this is that it works.  The streamer's token
>> buffer eats a lot of memory compared to a real streaming parser, but in
>> practice, it's a drop in the bucket.
>> 
> 
> I looked into this at one point. I forget why I didn't like it. I had
> some notion that I should replace this one too, but forget exactly why.
> Maybe it wasn't that bad, if I've forgotten.
> 
>>> Also, due to the nature of JSON as being a single discrete object and
>>> never a stream of objects, no existing JSON parser really supports the
>>> idea of ever seeing more than one object per buffer.
>> 
>> That plainly sucks.
>> 
>>> ...So I investigated writing a proper grammar for qmp-shell.
>> 
>> Any parser must start with a proper grammar.  If it doesn't, it's a toy,
>> or a highway to madness.
>> 
>>> Unfortunately, this basically means including the JSON grammar as a
>>> subset of the shell grammar and writing your own parser for it entirely.
>> 
>> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
>> paper, so JSON sticks out left and right.
>> 
>> Scrap and start over.
>> 
>>> I looked into using Python's own lexer; but it's designed to lex
>>> *python*, not *json*. I got a prototype lexer working for this purpose
>>> under a grammar that I think reflects JSON, but I got that sinking
>>> feeling that it was all more trouble than it was worth, and scrapped
>>> working on it any further.
>> 
>> Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
>> weird derivative of JSON in 239 SLOC.
>> 
>>> I did not find any other flex/yacc-like tools that seemed properly
>>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>>> writing a new parser.
>> 
>> While I recommend use of tools for parsing non-trivial grammars (you'll
>> screw up, they won't), they're massive overkill for JSON.
>> 
>>> I'd love to offer a nice robust QMP shell that is available for use by
>>> end users, but the syntax of the shell will need some major considerations.
>> 
>> Scrap and start over.
>> 
>> [...]
>> 
> 
> Yes, I agree: Scrap and start over.
> 
> What SHOULD the syntax look like, though? 

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

Il mer 15 gen 2020, 10:21 Markus Armbruster  ha scritto:

> > We don’t want the QAPI to let arbitrary fields of a QOM object
> > be modified, do we?
>
> We already do: QMP command qom-set.  If it breaks your guest, you get to
> keep the pieces.
>

That's not true. We chose not to make that a "recommended" interface, and
instead we add new commands. However that's mostly to avoid tying our hands
and not making too much of QOM part of the API. But I would be very
surprised if a guest could be broken with qom-set.

This was definitely not the case when QOM was introduced in a half-baked
state, but let's not indulge in self-flagellation more than it's actually
necessary.



> > As for the “public” aspects of a QOM object, that is static if it
> > comes from QAPI, so why couldn’t we define it there?
>
> QAPI specifies a certain kind of data types.
>
> QOM "specifies" data types as an imperative program for creating them.
> Maximally flexible, and fundamentally at odds with static analysis.
> I've hated this since day one.
>
> There's no hard reason why QOM could not specify static aspects
> declaratively.  It just doesn't, and changing it now would be a
> monumental task.
>
> The imperative program mostly creates identical data types every time.
> In other words, the data types are static.  Two problems.  One,
> converting such a program to an equivalent declaration takes manual
> labor.  Two, there are exceptions, and identifying them is more manual
> labor.
>
> >>> the same way, QAPI can't say anything about the structure of a QOM
> >>> object, and I think that's a problem.
> >>>
> > - which, as far as I
> > understand, is mainly because QOM properties aren't necessarily
> static,
> > so we can't provide a statically defined interface for them. Probably
> > solvable in QEMU, but not without a major effort.
> 
>  Or maybe extend the language so that it’s internal semantics
>  knows about this aspect of QOM?
> >>>
> >>> My point is that for example you can't generate a C struct (which is
> >>> statically defined) from something that has a dynamic structure. The
> >>> best you could do is fall back to key=value even in the C source, both
> >>> key and value being strings. But then you still have to parse these
> >>> strings manually, so it doesn't actually help much compared to a state
> >>> without C bindings.
> >
> > I really don’t understand that point. To me, all these operations
> > seem relatively simple to generate.
>
> Yes, if the QAPI fairy gives us a declarative specification equivalent
> to the existing imperative one, then replacing the hand-written
> imperative code by code generated from the declarative specification
> would be relatively simple.
>
> > I think that what confuses me is when you write “something that has a
> > dynamic structure”. I understand that as referring to the structure
> > defined in the schema. But the schema itself is static. So you can
> > generate static code from it, and it’s already done today.
>
> QAPI data types are static.
>
> QOM types are not.  They're effectively almost static in practice.
>
> > Another hypothesis on my side is that we don’t want, ever, to
> > have the API provide for example the option to create its own
> > arbitrary QOM classes, or to tag arbitrary properties to an object,
> > where by “arbitrary” I mean not explicitly mentioned somewhere in
> > something like the schema.
> >
> > So I suspect you are talking about something else.
>
> Kevin's talking about the imperative code creating different QOM
> properties depending on how and in what context it is run.
>
> QOM is perfectly capable of supporting a QMP command to add arbitrary
> QOM properties to an object at run time, or even add arbitrary QOM
> types, but as you wrote we don't want that.
>
> >> QOM and QAPI sabotage each other.  Ironic, considering they were dreamed
> >> up by the same guy :)
> >>
> >> QAPI is compile-time static by design.
> >>
> >> QOM is run-time dynamic by design.  Some support for static definitions
> >> has been grafted on since.
> >>
> >> We use QAPI type system escapes for QOM.  Defeats QAPI introspection and
> >> doc generation.  We provide separate QOM introspection instead, which is
> >> clumsier and less expressive.  QOM documentation doesn't exist.
> >
> > But back to the original discussion about management tools,
> > do we let upper layers tag their own arbitrary stuff in QOM objects?
>
> We don't want management applications to add QOM properties for their
> own purposes.
>
> >>> Maybe what could be done is covering at least the static properties and
> >>> then having key=value for the dynamic part (which should be the
> >>> exception).
> >>
> >> To make this worthwhile, we'd have to replace dynamic QOM properties by
> >> static ones when possible.  Monumental task.
> >
> > I’m sure you are right, but it’s hard for me to evaluate, given how
> > many ways there are to access an object. Naively, grepping for
> > set_prop and for 

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 24/12/19 14:41, Daniel P. Berrangé wrote:
> I'm wondering if we need to think about a temporary *throwaway* fork of
> QEMU. I wasn't involved in the work, but as an outside observer I think
> it is interesting to see how the NEMU project fork ultimately led to QEMU
> moving forward with a new KConfig approach, and the integration of microvm.
> They had the freedom to develop these new approaches with zero regard
> to back compat or broader QEMU needs that might otherwise hold back dev.

Just a quick note: none of these were developed first in NEMU.  KConfig
was completed *for* NEMU (to make their lives easier) but the code
already existed, and microvm is inspired by Firecracker.  NEMU's main
merit in my opinion was to make QEMU's perception issues clearer, and to
cause more thought on those issues and how to approach it.

That said, I agree that a throwaway experiment on command line and
configuration is worthwhile.  The main thing to think about is to make
convenience options syntactic sugar for something else.  For example, a
couple ideas mentioned in the thread or elsewhere in hallway discussions:

- Make default devices expressible as QemuOpts, so that -writeconfig
able to write out the default devices as well and -readconfig would
imply -nodefaults.

- Reduce the scope of -drive by dropping if=none in favor of -blockdev.
 Leave -drive as a quick way of configuring frontend and backend.

- And somewhat unrelated to configuration files: generate bindings for
QMP clients from the schema.

In the meanwhile, we should deprecate -readconfig/-writeconfig.  Can't
go wrong with that.

Paolo

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 24/01/20 10:50, Daniel P. Berrangé wrote:
>   * qemu-launcher-$TARGET
> 
> A binary that is able to launch qemu-runtime-$TARGET
> with jailers active.
> 
> This has no command line arguments except for a pair
> of UNIX socket paths. One is a QMP server, the other
> is the path for the QMP of qemu-runtime-$TARGET.
> 
> Commands it processes will be in automatically proxied
> through to the qemu-runtime-$TARGET QMP, with appropriate
> jailer updates being done in between.
> 

What would be the advantage of this over the Libvirt embedded driver?
Especially if you include in the picture something like libvirt-go-xml
(or libvirt-GObject, does it still exist?) that hides the XML from the
code that uses it.

The main complication in the launcher is hotplug, which means that a
simple "do a couple bind mounts, unshare, drop privileges and forget
about it" approach doesn't work.  Proxying QMP commands doesn't seem
that easy, and I don't see much code being shared between the launcher
and QEMU; if the existing QEMU code is not suitable for Libvirt, it
wouldn't be suitable for a qemu.git launcher either.

Also, as you mentioned earlier, QEMU wants to keep its vocabulary
lower-level, and therefore the launcher's vocabulary would end up
diverging from QEMU.  Some example:

- QEMU wants a qemu-pr-helper socket path, the launcher would take care
of launching qemu-pr-helper itself

- QEMU wants the complete configuration on the migration destination,
the launcher might take care of sending it from the source?

At this point, you get something that looks very much like Libvirt and,
especially if you include live migration, it has to take into account
the same compatibility considerations as Libvirt.

Thanks,

Paolo

Re: Making QEMU easier for management tools and applications

[Paolo Bonzini]

On 20/01/20 10:55, Stefan Hajnoczi wrote:
>>
>> [1] https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html
> John and I discussed async events in the past.  qmp-shell currently uses
> the input() built-in function.  If we modify it with a
> select(2)/poll(2)-style function that monitors both stdin and the QMP
> socket then it could print QMP events as soon as they are received.

I think it should be rewritten using async/await.  A simple example:

import asyncio
import sys
from concurrent.futures import ThreadPoolExecutor

async def ainput(prompt: str = ""):
with ThreadPoolExecutor(1, "ainput") as executor:
return (await asyncio.get_event_loop().run_in_executor(
executor, sys.stdin.readline
)).rstrip()

async def numbers():
i = 1
while True:
print(i)
i = i + 1
await asyncio.sleep(1)

async def main():
name = await ainput("What's your name? ")
print("Hello, {}!".format(name))

asyncio.get_event_loop().create_task(numbers())
asyncio.get_event_loop().run_until_complete(main())

This would be a great Summer of Code project.  Even an autocompletion
interface using readline should be possible.

Paolo

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf  writes:

> Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
>> On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
>> > John Snow  writes:
>> > 
>> > > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
>> > >> So when configuring objects you'll always provide a JSON/YAML doc.
>> > >> They've got some clever stuff for updating objects where you can
>> > >> provide a JSON patch for only the bits which need changing.
>> > >> 
>> > >> When querying/listing objects by default it displays only a small
>> > >> subset of their config information in a human friendly-ish format.
>> > >> If you want to see everything then you ask for it in JSON/YAML
>> > >> format. There's also an expression language that lets you extract
>> > >> particular pieces of information based on requested properties,
>> > >> and you can filter the list of objects based on attributes and so
>> > >> on.
>> > >> 
>> > >> I think it is fair to say the structure of kubernetes object config
>> > >> is on a par with hierarchical complexity of QEMU. The lack of a simple
>> > >> human targetted data input format does not appear to have negatively
>> > >> impacted the adoption of Kubernetes. It is worth questioning why this
>> > >> is the case, while we feel the human CLI syntax for QEMU is so
>> > >> critically important to QEMU's future ?
>> > 
>> > I consider human CLI syntax for QEMU a mostly solved *design* problem:
>> > dotted keys.  It's an unsolved *implementation* problem: the CLI is a
>> > tangled mess of almost two decades' worth of ideas, and only (some of)
>> > the latest strands actually use dotted keys infrastructure.  The
>> > proposed solution is CLI QAPIfication.  Gives us configuration file(s)
>> > and introspection.
>> > 
>> > Dotted keys are merely yet another concrete syntax.  They're designed to
>> > satisfy the CLI requirements we have, which include a measure of
>> > compatibility to what's in the tangled mess.  They're reasonably usable
>> > for simple stuff, but complex stuff can be too verbose to be readable.
>> > They can't express all of the abstract syntax.  Tolerable, since they
>> > provide an escape to JSON.  I recommend programs use the JSON escape
>> > always.  Awkward for humans due to shell quoting.
>> 
>> I agree that the dotted key syntax is our chosen / solved design
>> for expressing JSON on the CLI. I would also say that, in retrospect,
>> this was a incorrect design decision that is one of the key things
>> responsible for QEMU having a bad reputation for complexity.
>
> I doubt this. Whenever I get a bug report with a command line created by
> libvirt, the command line is huge, but basically nothing in it uses
> dotted syntax. Yes, you may have cache.direct=on in it somewhere, but
> that's not actual nesting.
>
> The problem is the amount of options that is specified by management
> tools, and then humans are looking at it and feel it's way too complex.
>
> Command lines written by human users are usually much simpler because
> they just use QEMU's defaults instead of explicitly specifying
> everything.

Yes, machine-generated configuration is more verbose than what humans
produce.  Machines like it explicit.  It's simpler for them than relying
on defaults.  Bonus: immunity to changing defaults.

But no, verbosity is not the core problem, it merely aggravates the core
problem.  Complex configuration is much harder to read in a CLI syntax
than in a half-decent config file.  Yes, it's prohibitively harder just
for lengthy configurations.  That doesn't make it not harder for short
configurations.

>> We should simply never have tried to invent a way to map the full
>> hiearchy of JSON onto the CLI as the result will always be unpleasant.

It's what I had to do to secure a beach head for QAPI on the command
line coast.

>> The dotted notation is the most verbose way to do this type of
>> configuration, because of the string repetition it requires for
>> nested structures.
>
> True. I would have liked a different syntax that used some kind of
> brackets (at least optionally), but Markus didn't like adding another
> character that must be escaped.

Design thread:

Subject: Non-flat command line option argument syntax
Date: Thu, 02 Feb 2017 20:42:33 +0100
Message-ID: <87bmukmlau@dusky.pond.sub.org>
https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg00555.html

A few quick quotes:

* On JSON vs. dotted keys in CLI: "both variants are basically
  illegible.  This is simply something that belongs into a config file
  rather than the command line.  In a config file, JSON would be a
  better choice."

* On dotted keys vs. structured values: "dotted keys are weird and ugly,
  but at least they don't add to the quoting mess.  Structured values
  look better, except when they do add to the quoting mess.  I'm having
  a hard time deciding which one I like less :)"

* Final verdict: "the whole non-flat command line argument design has
  been an exercise in 

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 2:59 AM, Markus Armbruster wrote:
> Digression: JSON is a poor choice for configuration files.  YAML is a
> complex and confusing beast (it's spec is printed 116 pages, and
> yaml-0.2.2/src is ~7kSLOC).  XML is XML, 'nuff said.  TOML is much
> simpler than either of the two.
JSON has a built-in Python parser. YAML has a widely availably PyYAML
parser module. TOML is not yet in widespread use, but does offer a
python parser.

No XML parser I have ever used appears to be feature-complete to the XML
spec. Most modern toolings support JSON and/or YAML.

I personally like the idea of YAML as it was intended for human-written
configuration files without brackets and other such visual confetti. It
even allows for comments as a first-class citizen of the specification.
This would be useful to documentation efforts.

Hearsay: most of the existing high-level users of the virt stack are in
Go or Python. Python users are quite comfortable with whitespace-markup
formats. Go developers are likely working in the cloud space, which
overwhelmingly uses YAML.


Suggestion:

(1) We use YAML and use Python's built-in YAML parser to build a machine
specification format defined by QAPI.

We build a "configuration" struct in QAPI, and extend from there.

(2) We offer "--config myconfig.yaml" as an option for specifying
options. This precludes the use of *any* traditional command line flags.
Security wonkery permitting, we may also allow QEMU to take a YAML file
on STDIN at boot, if this is easier for cloud management programs to script.

(3) We offer "--writeconfig out.yaml" to convert to the new format... if
possible. It very well may NOT be, given the nature of our wild west
parsing system. I consider this an optional olive branch, not a necessity.

(4) Just scrap the existing config file system entirely. It was never
finished and does not help solve the existing problem that we do not
have a schema for our configuration syntax.

(5) Deprecate the traditional CLI configuration syntax. Remove it only
on a major version transition. Advertise this as our "2.0" break. Use
this opportunity to do heavier refactoring of our worst design
transgressions as desired. Make no effort to add additional features to
the CLI if it doesn't come for free.

(Bonus: If we work with libvirt to adapt to the new format, libvirt
users will enjoy a seamless transition. This is an insanely good
marketing point for libvirt's usefulness, allowing QEMU to adapt as
necessary to remove baggage with minimal interruption to end-users.)

(Downside: Any libvirt XML passing CLI flags directly will lose that
ability. They will need to specify YAML object overrides that plug into
specific parts of the hierarchy instead.)

(Upside to the downside: These overrides are often not supported by
downstream vendors anyway.)


Benefits:

- The entirety of the ugly parsing code in vl.c and strewn about
submodules goes away. Many of the .hx files go away.

- QemuOpts goes away. Everything is handled at the QAPI level via YAML
and QMP parsers. Input errors are handled at the schema level.

- Extending QMP and the CLI occurs with one schema definition change.

- We don't have to write a new python parser; The complexity of YAML
itself poses no particular concern here.

- YAML is in wide use. Support in emacs/vim is widespread. Third party
tools for manipulating YAML documents are widely available and in-use
due to usage in the cloud infrastructure world.

- Once deserialized, JSON and YAML configuration statements are
identical in-memory in Python. Many of the same tools and infrastructure
can be recycled.

- PyYaml uses libYAML, a C library. That same library can be used to
build the same structures that we already use for QMP.

- YAML examples can be trivially generated from the QAPI spec.

- YAML allows for in-line comments, which would be a tremendous feature
for writing documentation and annotating machine configuration files.


Downsides:

- We WOULD need a new YAML parsing layer in QEMU, generated by QAPI.
Supporting two deserialization layers could lead to strange
discrepancies between the two formats at runtime. So, admittedly, using
JSON would be *even easier*, as we could re-use the same parsers already
battle-tested in QEMU. We could get *MAXIMUM* code re-use this way.

- YAML does not allow duplicate keys. On its face, this is not an issue,
we can't support parallel universe configurations anyway. If it's
important to allow "overrides" of base configurations, this might intuit
the need for cascading YAML files. (--config base.yaml --config
overlay.yaml)

- We will break compatibility with our existing CLI. People will not be
happy about this, especially, perhaps, embedded board and TCG developers
who use fairly minimal command lines regularly.

- A lot of iotests and python scripts will need serious love to bring
forward into the new world.



I guess at the end of writing this out, I wind up making a slightly
better case for JSON in terms of 

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 1:52 PM, Dr. David Alan Gilbert wrote:
> * John Snow (js...@redhat.com) wrote:
>>
>>
>> On 1/24/20 1:30 PM, Dr. David Alan Gilbert wrote:
>>> However, I caution that too many people think that QMP has all the type
>>> of diagnostic stuff people want - it's just way too hard and beuracratic
>>> to add a simple diagnostic command to QMP, when you just want to add
>>> something to print some diagnostics out for your corner of qemu.
>>
>> I'm not sure I follow, do you mean bureaucratic in contrast to... HMP?
> 
> I mean, someone who adds a QMP command is expected to be a lot more
> careful about the long term stability and proper structures etc etc.
> Where for HMP, if someone wants to add a command that's useful to the
> diagnostics of their part of QEMU and doesn't break anything else, then
> fine.
> 
> Dave

It sounds like we want a formalized notion of "x-debug-" for QMP
commands where we use feature flags to explicitly mark a command
interface as a debugging interface.

(We use such a command for bitmap iotests, to compute checksums of
bitmaps to ensure they migrate correctly. We make no guarantee that this
is stable or useful.)

I'm not sure if it's wise or not, but we could disable these commands
entirely unless you configure with --enable-qmp-debugging.

Might ease that pain of migrating everything onto QMP.

--js

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* John Snow (js...@redhat.com) wrote:
> 
> 
> On 1/24/20 1:30 PM, Dr. David Alan Gilbert wrote:
> > However, I caution that too many people think that QMP has all the type
> > of diagnostic stuff people want - it's just way too hard and beuracratic
> > to add a simple diagnostic command to QMP, when you just want to add
> > something to print some diagnostics out for your corner of qemu.
> 
> I'm not sure I follow, do you mean bureaucratic in contrast to... HMP?

I mean, someone who adds a QMP command is expected to be a lot more
careful about the long term stability and proper structures etc etc.
Where for HMP, if someone wants to add a command that's useful to the
diagnostics of their part of QEMU and doesn't break anything else, then
fine.

Dave

> --js
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 1:30 PM, Dr. David Alan Gilbert wrote:
> However, I caution that too many people think that QMP has all the type
> of diagnostic stuff people want - it's just way too hard and beuracratic
> to add a simple diagnostic command to QMP, when you just want to add
> something to print some diagnostics out for your corner of qemu.

I'm not sure I follow, do you mean bureaucratic in contrast to... HMP?

--js

Re: Making QEMU easier for management tools and applications

[Dr. David Alan Gilbert]

* John Snow (js...@redhat.com) wrote:
> 
> 
> On 1/24/20 9:38 AM, Kevin Wolf wrote:
> > Hi! :-)
> > 
> > Well, not fighting for it to the death and I'm certainly not married to
> > its exact syntax, but I use HMP all the time because it's convenient to
> > use manually and QMP isn't.
> > 
> > If you want to remove HMP, get us a decent QMP shell first. And ideally
> > a way how to use it without manually configuring a QMP socket and
> > starting up and connecting the shell to it with ten kilometers of
> > command line options on both sides first. HMP is available by default,
> > and so should its replacement be.
> 
> This is partly why I am asking about what a "qmp-shell" should look like
> to be usable, so we can sunset HMP once and for all.
> 
> One idea for a qmp-shell is to implement some of the convenience syntax
> in qmp-shell directly to give us some of the same shorthands, and that
> logic lives in the shell now.

I wonder about generating a set of python bindings for the qmp commands,
and generating wrappers for those, and having the qmp-shell be a python
interpreter with all that loaded.
That way it would be very easy to add extra sugar.

However, I caution that too many people think that QMP has all the type
of diagnostic stuff people want - it's just way too hard and beuracratic
to add a simple diagnostic command to QMP, when you just want to add
something to print some diagnostics out for your corner of qemu.

Dave

> --js
> 
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/24/20 9:38 AM, Kevin Wolf wrote:
> Hi! :-)
> 
> Well, not fighting for it to the death and I'm certainly not married to
> its exact syntax, but I use HMP all the time because it's convenient to
> use manually and QMP isn't.
> 
> If you want to remove HMP, get us a decent QMP shell first. And ideally
> a way how to use it without manually configuring a QMP socket and
> starting up and connecting the shell to it with ten kilometers of
> command line options on both sides first. HMP is available by default,
> and so should its replacement be.

This is partly why I am asking about what a "qmp-shell" should look like
to be usable, so we can sunset HMP once and for all.

One idea for a qmp-shell is to implement some of the convenience syntax
in qmp-shell directly to give us some of the same shorthands, and that
logic lives in the shell now.

--js

Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
> On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
> > John Snow  writes:
> > 
> > > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> > >> So when configuring objects you'll always provide a JSON/YAML doc.
> > >> They've got some clever stuff for updating objects where you can
> > >> provide a JSON patch for only the bits which need changing.
> > >> 
> > >> When querying/listing objects by default it displays only a small
> > >> subset of their config information in a human friendly-ish format.
> > >> If you want to see everything then you ask for it in JSON/YAML
> > >> format. There's also an expression language that lets you extract
> > >> particular pieces of information based on requested properties,
> > >> and you can filter the list of objects based on attributes and so
> > >> on.
> > >> 
> > >> I think it is fair to say the structure of kubernetes object config
> > >> is on a par with hierarchical complexity of QEMU. The lack of a simple
> > >> human targetted data input format does not appear to have negatively
> > >> impacted the adoption of Kubernetes. It is worth questioning why this
> > >> is the case, while we feel the human CLI syntax for QEMU is so
> > >> critically important to QEMU's future ?
> > 
> > I consider human CLI syntax for QEMU a mostly solved *design* problem:
> > dotted keys.  It's an unsolved *implementation* problem: the CLI is a
> > tangled mess of almost two decades' worth of ideas, and only (some of)
> > the latest strands actually use dotted keys infrastructure.  The
> > proposed solution is CLI QAPIfication.  Gives us configuration file(s)
> > and introspection.
> > 
> > Dotted keys are merely yet another concrete syntax.  They're designed to
> > satisfy the CLI requirements we have, which include a measure of
> > compatibility to what's in the tangled mess.  They're reasonably usable
> > for simple stuff, but complex stuff can be too verbose to be readable.
> > They can't express all of the abstract syntax.  Tolerable, since they
> > provide an escape to JSON.  I recommend programs use the JSON escape
> > always.  Awkward for humans due to shell quoting.
> 
> I agree that the dotted key syntax is our chosen / solved design
> for expressing JSON on the CLI. I would also say that, in retrospect,
> this was a incorrect design decision that is one of the key things
> responsible for QEMU having a bad reputation for complexity.

I doubt this. Whenever I get a bug report with a command line created by
libvirt, the command line is huge, but basically nothing in it uses
dotted syntax. Yes, you may have cache.direct=on in it somewhere, but
that's not actual nesting.

The problem is the amount of options that is specified by management
tools, and then humans are looking at it and feel it's way too complex.

Command lines written by human users are usually much simpler because
they just use QEMU's defaults instead of explicitly specifying
everything.

> We should simply never have tried to invent a way to map the full
> hiearchy of JSON onto the CLI as the result will always be unpleasant.
> The dotted notation is the most verbose way to do this type of
> configuration, because of the string repetition it requires for
> nested structures.

True. I would have liked a different syntax that used some kind of
brackets (at least optionally), but Markus didn't like adding another
character that must be escaped.

I think if we want, we can still evolve our human syntax to be more user
friendly. My impression was that we don't want to.

> Lets consider how libvirt uses blockdev for a LUKS volume stored
> in iSCSI
> 
>   $ qemu-system-x86_64 \
>   -object secret,id=libvirt-5-storage-secret0,\
> data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
> keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -object secret,id=libvirt-5-format-luks-secret0,\
> data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
> keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -blockdev '{"driver":"iscsi","portal":"example.org:6000",\
> "target":"iqn.1992-01.com.example:storage","lun":1,"transport":"tcp",\
> "user":"myname","password-secret":"libvirt-5-storage-secret0",\
> "node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' 
> \
>   -blockdev 
> '{"node-name":"libvirt-5-format","read-only":false,"driver":"qcow2",\
> "encrypt":{"format":"luks","key-secret":"libvirt-5-format-luks-secret0"},\
> "file":"libvirt-5-storage"}' \
> 
> We all know JSON is horrible on the CLI, no surprise. So
> 
> Lets use human "friendly" dotted syntax instead:
> 
>   $ qemu-system-x86_64 \
>   -object secret,id=libvirt-5-storage-secret0,\
> data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
> keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>   -object secret,id=libvirt-5-format-luks-secret0,\
> 

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
> John Snow  writes:
> 
> > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> >> So when configuring objects you'll always provide a JSON/YAML doc.
> >> They've got some clever stuff for updating objects where you can
> >> provide a JSON patch for only the bits which need changing.
> >> 
> >> When querying/listing objects by default it displays only a small
> >> subset of their config information in a human friendly-ish format.
> >> If you want to see everything then you ask for it in JSON/YAML
> >> format. There's also an expression language that lets you extract
> >> particular pieces of information based on requested properties,
> >> and you can filter the list of objects based on attributes and so
> >> on.
> >> 
> >> I think it is fair to say the structure of kubernetes object config
> >> is on a par with hierarchical complexity of QEMU. The lack of a simple
> >> human targetted data input format does not appear to have negatively
> >> impacted the adoption of Kubernetes. It is worth questioning why this
> >> is the case, while we feel the human CLI syntax for QEMU is so
> >> critically important to QEMU's future ?
> 
> I consider human CLI syntax for QEMU a mostly solved *design* problem:
> dotted keys.  It's an unsolved *implementation* problem: the CLI is a
> tangled mess of almost two decades' worth of ideas, and only (some of)
> the latest strands actually use dotted keys infrastructure.  The
> proposed solution is CLI QAPIfication.  Gives us configuration file(s)
> and introspection.
> 
> Dotted keys are merely yet another concrete syntax.  They're designed to
> satisfy the CLI requirements we have, which include a measure of
> compatibility to what's in the tangled mess.  They're reasonably usable
> for simple stuff, but complex stuff can be too verbose to be readable.
> They can't express all of the abstract syntax.  Tolerable, since they
> provide an escape to JSON.  I recommend programs use the JSON escape
> always.  Awkward for humans due to shell quoting.

I agree that the dotted key syntax is our chosen / solved design
for expressing JSON on the CLI. I would also say that, in retrospect,
this was a incorrect design decision that is one of the key things
responsible for QEMU having a bad reputation for complexity.

We should simply never have tried to invent a way to map the full
hiearchy of JSON onto the CLI as the result will always be unpleasant.
The dotted notation is the most verbose way to do this type of
configuration, because of the string repetition it requires for
nested structures.

Lets consider how libvirt uses blockdev for a LUKS volume stored
in iSCSI

  $ qemu-system-x86_64 \
  -object secret,id=libvirt-5-storage-secret0,\
data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -object secret,id=libvirt-5-format-luks-secret0,\
data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -blockdev '{"driver":"iscsi","portal":"example.org:6000",\
"target":"iqn.1992-01.com.example:storage","lun":1,"transport":"tcp",\
"user":"myname","password-secret":"libvirt-5-storage-secret0",\
"node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \
  -blockdev 
'{"node-name":"libvirt-5-format","read-only":false,"driver":"qcow2",\
"encrypt":{"format":"luks","key-secret":"libvirt-5-format-luks-secret0"},\
"file":"libvirt-5-storage"}' \

We all know JSON is horrible on the CLI, no surprise. So

Lets use human "friendly" dotted syntax instead:

  $ qemu-system-x86_64 \
  -object secret,id=libvirt-5-storage-secret0,\
data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -object secret,id=libvirt-5-format-luks-secret0,\
data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
  -blockdev driver=qcow2,node-name=libvirt-5-format,read-only=false,\
encrypt.format=luks,encrypt.key-secret=libvirt-5-format-luks-secret0,\
file.driver=iscsi,file.portal=example.org:6000,\
file.target=iqn.1992-01.com.example:storage,file.lun=1,file.transport=tcp,\
file.user=myname,file.password-secret=libvirt-6-storage-secret0,\
file.node-name=libvirt-5-storage,file.auto-read-only=true,file.dicard=unmap

I don't think that's much of an improvement, aside from not having
to worry about matching "}".

If we move to JSON in a config file

  $ cat qemu.json
  {
"arguments": [
  {
"arg": "object",
"data": {
  "type": "secret",
  "id":"libvirt-5-storage-secret0",
  "data": 
"9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
  "keyid": "masterKey0",
  "iv": "AAECAwQFBgcICQoLDA0ODw==",
 

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Thu, Jan 23, 2020 at 04:07:09PM -0500, John Snow wrote:
> 
> 
> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> > IOW, the difficulty with configuring QEMU via JSON is not the fault
> > of JSON itself, it is the lack of knowledge amongst users and docs,
> > compounded by our never ending "improvements" to the human syntax.
> > There are other factors too, such as our only partial coverage of
> > config using JSON - some is only possible via the CLI still.
> > 
> 
> I'm fine with getting rid of HMP entirely, I think. It's a weird
> interface with bizarre behavior that's hard to support.
> 
> There's a few commands in there we just don't support at all, but maybe
> it's time to start deprecating one-by-one any of the individual commands
> that are better served by QMP these days, to send the message that HMP's
> days are numbered.
> 
> Bye-bye!
> 
> As for the CLI, well, that's part of the discussion at hand...
> 
> > 
> > I guess my point is that with a scrap & startover view point, we
> > should arguably completely ignore the design question of how to
> > flatten JSON for humans/command line, as it is the wrong problem.
> > Instead focus on the problem of making use of JSON the best way
> > to deal with QEMU both functionally and practically, for humans
> > and machines alike.
> > 
> 
> Well, sure. The context of this email was qmp-shell though, which is
> meant to help facilitate the entry of JSON commands so that you *can*
> indeed just forego the CLI/HMP entirely.
> 
> If you are of the opinion that every user of QEMU should be copy/pasting
> JSON straight into a socket and we should delete qmp-shell, that's
> certainly a fine opinion.

I think part of the pain of qmp-shell comes from the very fact that
it is trying to be an interactive shell. This points people towards
interactively typing in the commands, which is horrific when you get
anywhere near the JSON, or even dot-notation traditional commands.

If it was just a qmp-client that was single shot, we'd encourage
people to create the JSON in a sensible way - vim/emacs/whatever.

Bash/dash/zsh/$whatever is their interactive shell, with massively
more features than qmp-shell. You have command history, autocomplete,
conditional and looping constructs, and everything a normal shell
offers.

The only strong reason for qmp-shell to be interactive would be if
the initial protoocl handshake was too slow. I can't see that being
a problem with QMP. 

> I'm coming from the side that I love qmp-shell; I find it useful, but it
> has some syntax problems. How do I solve them? Is there a way to solve
> them? QAPI is here to stay, and QAPI involves hierarchical data. That
> data is usually best represented by something like json or yaml, but
> those are hard to type in a shell.
> 
> What do we do about that?

Here's one conceptual vision of how a better QEMU might look:

  * qemu-runtime-$TARGET

A binary that contains the implementation for the machine
emulator for $TARGET.

This has no command line arguments except for a UNIX
socket path which is a QMP server


  * qemu-launcher-$TARGET

A binary that is able to launch qemu-runtime-$TARGET
with jailers active.

This has no command line arguments except for a pair
of UNIX socket paths. One is a QMP server, the other
is the path for the QMP of qemu-runtime-$TARGET.

Commands it processes will be in automatically proxied
through to the qemu-runtime-$TARGET QMP, with appropriate
jailer updates being done in between.


  * qemu-client

A binary that speaks QMP, connects, runs a single command,
disconnects.

It is used to talk to either qemu-runtime-$TARGET or
qemu-launcher-$TARGET, depending on whether the mgmt app
or user wants to be making use of the jailer facilities
or not.  


  * qemu-system-$TARGET

The current binaries that exist today.

qemu-system-$TARGET should not be part of our formal
stability promise. We won't gratuitously / knowingly
break without good reason, but we will accept that
breakage can happen. Stability is only offered by
the qemu-{runtime,launcher}-$TARGET.

Several choices for their future in long term:

  - Leave them as-is and basically ignore them
whereever practical going forward, so we
minimally worry about backcompat breakage

  - Plan to re-write them so that they are simply
a shim the forks+execs qemu-runtime-$TARGET
and does syntax translation from CLI/HMP/QMP.

  - Deprecate them with a view to deletion entirely
in $NNN years. For some large-ish value of NNN,
given how well known they are


Example usage:

1. Launch the QEMU runtime for the desired target

 $ qemu-runtime-x86_64 myvm.sock

2. Load the configuration to define the VM

   $ cat myvm.yaml
   commands:
 - machine_declare:
 name: pc-q35-5.0
 ...
 - blockdev_add:
 ...
 - device_add:
 ...
 - blockdev_add:
   

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow  writes:

> On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
>> On Thu, Jan 23, 2020 at 12:58:45PM -0500, John Snow wrote:
>>> Yes, I agree: Scrap and start over.
>>>
>>> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
>>> is that it offers a convenient way to enter the top-level keys of the
>>> arguments dict. This works absolutely fine right up until you need to
>>> start providing nested definitions.
>>>
>>> For the nesting, we say: "Go ahead and use JSON, but you have to take
>>> all the spaces out."
>>>
>>> This... works, charitably, but is hardly what I would call usable.
>>>
>>> For the CLI, we offer a dot syntax notation that resembles nothing in
>>> particular. It often seems the case that it isn't expressive enough to
>>> map losslessly to JSON. I suspect it doesn't handle siblings very well.
>>>
>>> A proper HMP-esque TUI would likely have need of coming up with its own
>>> pet syntax for commands that avoid complicated nested JSON definitions,
>>> but for effort:value ratio, having a QMP shorthand shell that works
>>> arbitrarily with any command might be a better win.
>>>
>>> Do we still have a general-case problem of how to represent QAPI
>>> structures in plaintext? Will this need to be solved for the CLI, too?
>> 
>> I don't know if you've ever looked at how Kubernetes/OpenShift exposes
>> its functionality on the command line ? I think it is interesting to

Yes, superficially.

>> note that they largely don't try to solve this problem of flattening
>> JSON for humans on the CLI using their client.

The fact that their users are happy with this proves it reasonable.

>> Everything in their world is an object described in JSON/YAML, and
>> there are a small set of generic commands that can act on any type
>> of object. These commands primarily input and output JSON or YAML
>> documents directly. As a user you can pick either format since it
>> can do a lossless conversion in both directions server side.

Like Kubernetes/OpenShift, our configuration / control language has an
abstract syntax that permits use of JSON/YAML as concrete syntax.  They
support both, we support just JSON.  We could support YAML, too.

Digression: JSON is a poor choice for configuration files.  YAML is a
complex and confusing beast (it's spec is printed 116 pages, and
yaml-0.2.2/src is ~7kSLOC).  XML is XML, 'nuff said.  TOML is much
simpler than either of the two.

>> So when configuring objects you'll always provide a JSON/YAML doc.
>> They've got some clever stuff for updating objects where you can
>> provide a JSON patch for only the bits which need changing.
>> 
>> When querying/listing objects by default it displays only a small
>> subset of their config information in a human friendly-ish format.
>> If you want to see everything then you ask for it in JSON/YAML
>> format. There's also an expression language that lets you extract
>> particular pieces of information based on requested properties,
>> and you can filter the list of objects based on attributes and so
>> on.
>> 
>> I think it is fair to say the structure of kubernetes object config
>> is on a par with hierarchical complexity of QEMU. The lack of a simple
>> human targetted data input format does not appear to have negatively
>> impacted the adoption of Kubernetes. It is worth questioning why this
>> is the case, while we feel the human CLI syntax for QEMU is so
>> critically important to QEMU's future ?

I consider human CLI syntax for QEMU a mostly solved *design* problem:
dotted keys.  It's an unsolved *implementation* problem: the CLI is a
tangled mess of almost two decades' worth of ideas, and only (some of)
the latest strands actually use dotted keys infrastructure.  The
proposed solution is CLI QAPIfication.  Gives us configuration file(s)
and introspection.

Dotted keys are merely yet another concrete syntax.  They're designed to
satisfy the CLI requirements we have, which include a measure of
compatibility to what's in the tangled mess.  They're reasonably usable
for simple stuff, but complex stuff can be too verbose to be readable.
They can't express all of the abstract syntax.  Tolerable, since they
provide an escape to JSON.  I recommend programs use the JSON escape
always.  Awkward for humans due to shell quoting.

> Well, if the "human CLI syntax" is "Feed it YAML documents", that's
> perfectly peachy for me, too! We need a good, consistent interface.
> Exactly what that interface is isn't really a blocking concern.

Right.

> Configuring a VM is a complicated process and has a lot of moving
> widgets. Feeding it a YAML file is a reasonable thought.

We've grown used to configuring QEMU with gargantuan command lines.
Déformation profesionelle.

> Having JSON and requiring people to type bastardized and differing
> versions of it in 8 places _is_ a concern. We can't document reasonably
> all of the different flavors and why they differ from one place to the
> next.

Yes.

>   We can unify it. If 

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow  writes:

> On 1/23/20 2:19 AM, Markus Armbruster wrote:
[...]
>> Any conventional parser can be made streaming by turning it into a
>> coroutine.  This is probably the simplest solution for handwritten
>> streaming LL parsers, because it permits recursive descent.  In Python,
>> I'd try a generator.
>> 
>> Our actual solution for QMP predates coroutine support in QEMU, and is
>> rather hamfisted:
>> 
>> * Streaming lexer: it gets fed characters one at a time, and when its
>>   state machine says "token complete", it feeds the token to the
>>   "streamer".
>> 
>> * "Streamer": gets fed tokens one at a time, buffers them up counting
>>   curly and square bracket nesting until the nesting is zero, then
>>   passes the buffered tokens to the parser.
>> 
>> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>>   a full expression.
>> 
>> The best I can say about this is that it works.  The streamer's token
>> buffer eats a lot of memory compared to a real streaming parser, but in
>> practice, it's a drop in the bucket.
>> 
>
> I looked into this at one point. I forget why I didn't like it. I had

Because it's confusing?  Took me a while to get used to it.

> some notion that I should replace this one too, but forget exactly why.
> Maybe it wasn't that bad, if I've forgotten.

Bigger fish to fry.

[...]

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
> On Thu, Jan 23, 2020 at 12:58:45PM -0500, John Snow wrote:
>> Yes, I agree: Scrap and start over.
>>
>> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
>> is that it offers a convenient way to enter the top-level keys of the
>> arguments dict. This works absolutely fine right up until you need to
>> start providing nested definitions.
>>
>> For the nesting, we say: "Go ahead and use JSON, but you have to take
>> all the spaces out."
>>
>> This... works, charitably, but is hardly what I would call usable.
>>
>> For the CLI, we offer a dot syntax notation that resembles nothing in
>> particular. It often seems the case that it isn't expressive enough to
>> map losslessly to JSON. I suspect it doesn't handle siblings very well.
>>
>> A proper HMP-esque TUI would likely have need of coming up with its own
>> pet syntax for commands that avoid complicated nested JSON definitions,
>> but for effort:value ratio, having a QMP shorthand shell that works
>> arbitrarily with any command might be a better win.
>>
>> Do we still have a general-case problem of how to represent QAPI
>> structures in plaintext? Will this need to be solved for the CLI, too?
> 
> I don't know if you've ever looked at how Kubernetes/OpenShift exposes
> its functionality on the command line ? I think it is interesting to
> note that they largely don't try to solve this problem of flattening
> JSON for humans on the CLI using their client.
> 
> Everything in their world is an object described in JSON/YAML, and
> there are a small set of generic commands that can act on any type
> of object. These commands primarily input and output JSON or YAML
> documents directly. As a user you can pick either format since it
> can do a lossless conversion in both directions server side.
> 
> So when configuring objects you'll always provide a JSON/YAML doc.
> They've got some clever stuff for updating objects where you can
> provide a JSON patch for only the bits which need changing.
> 
> When querying/listing objects by default it displays only a small
> subset of their config information in a human friendly-ish format.
> If you want to see everything then you ask for it in JSON/YAML
> format. There's also an expression language that lets you extract
> particular pieces of information based on requested properties,
> and you can filter the list of objects based on attributes and so
> on.
> 
> I think it is fair to say the structure of kubernetes object config
> is on a par with hierarchical complexity of QEMU. The lack of a simple
> human targetted data input format does not appear to have negatively
> impacted the adoption of Kubernetes. It is worth questioning why this
> is the case, while we feel the human CLI syntax for QEMU is so
> critically important to QEMU's future ?
> 

Well, if the "human CLI syntax" is "Feed it YAML documents", that's
perfectly peachy for me, too! We need a good, consistent interface.
Exactly what that interface is isn't really a blocking concern.

Configuring a VM is a complicated process and has a lot of moving
widgets. Feeding it a YAML file is a reasonable thought.

Having JSON and requiring people to type bastardized and differing
versions of it in 8 places _is_ a concern. We can't document reasonably
all of the different flavors and why they differ from one place to the
next. We can unify it. If unifying it means using
JSON/YAML/TOML/MAML[1]* everywhere and abandoning a CLI altogether,
that's just as well.

> 
> 
> Part of it is that the machine oriented data input format via QMP
> suffers from the fact that it came second in QEMU after HMP. As a
> result, 90% of the documentation that illustrates QEMU will use the
> human CLI syntax, varying vintages of that. Since all the docs are
> focused on the HMP/CLI syntax, whenever there's a new feature we
> feel pressured to expose it & document it in the human syntax too.
> 

Decent observation; but there's often no reasonable way to NOT use the
CLI, so I think it's not unreasonable that we try to expose features via
the CLI still.

However, the docs being badly out of date are a problem. We actively
lead people towards harmful / difficult to support paradigms.

> All this results in a situation where JSON is functionally the best
> way to configure QEMU, but practically the worse, since very few
> people understand how to actually use it. This is a vicious circle
> holding back QMP/JSON and making the human syntax an ever greater
> burden for users & maintainers
> 

I do basically agree.

> 
> IOW, the difficulty with configuring QEMU via JSON is not the fault
> of JSON itself, it is the lack of knowledge amongst users and docs,
> compounded by our never ending "improvements" to the human syntax.
> There are other factors too, such as our only partial coverage of
> config using JSON - some is only possible via the CLI still.
> 

I'm fine with getting rid of HMP entirely, I think. It's a weird
interface 

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Thu, Jan 23, 2020 at 12:58:45PM -0500, John Snow wrote:
> Yes, I agree: Scrap and start over.
> 
> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
> is that it offers a convenient way to enter the top-level keys of the
> arguments dict. This works absolutely fine right up until you need to
> start providing nested definitions.
> 
> For the nesting, we say: "Go ahead and use JSON, but you have to take
> all the spaces out."
> 
> This... works, charitably, but is hardly what I would call usable.
> 
> For the CLI, we offer a dot syntax notation that resembles nothing in
> particular. It often seems the case that it isn't expressive enough to
> map losslessly to JSON. I suspect it doesn't handle siblings very well.
> 
> A proper HMP-esque TUI would likely have need of coming up with its own
> pet syntax for commands that avoid complicated nested JSON definitions,
> but for effort:value ratio, having a QMP shorthand shell that works
> arbitrarily with any command might be a better win.
> 
> Do we still have a general-case problem of how to represent QAPI
> structures in plaintext? Will this need to be solved for the CLI, too?

I don't know if you've ever looked at how Kubernetes/OpenShift exposes
its functionality on the command line ? I think it is interesting to
note that they largely don't try to solve this problem of flattening
JSON for humans on the CLI using their client.

Everything in their world is an object described in JSON/YAML, and
there are a small set of generic commands that can act on any type
of object. These commands primarily input and output JSON or YAML
documents directly. As a user you can pick either format since it
can do a lossless conversion in both directions server side.

So when configuring objects you'll always provide a JSON/YAML doc.
They've got some clever stuff for updating objects where you can
provide a JSON patch for only the bits which need changing.

When querying/listing objects by default it displays only a small
subset of their config information in a human friendly-ish format.
If you want to see everything then you ask for it in JSON/YAML
format. There's also an expression language that lets you extract
particular pieces of information based on requested properties,
and you can filter the list of objects based on attributes and so
on.

I think it is fair to say the structure of kubernetes object config
is on a par with hierarchical complexity of QEMU. The lack of a simple
human targetted data input format does not appear to have negatively
impacted the adoption of Kubernetes. It is worth questioning why this
is the case, while we feel the human CLI syntax for QEMU is so
critically important to QEMU's future ?



Part of it is that the machine oriented data input format via QMP
suffers from the fact that it came second in QEMU after HMP. As a
result, 90% of the documentation that illustrates QEMU will use the
human CLI syntax, varying vintages of that. Since all the docs are
focused on the HMP/CLI syntax, whenever there's a new feature we
feel pressured to expose it & document it in the human syntax too.

All this results in a situation where JSON is functionally the best
way to configure QEMU, but practically the worse, since very few
people understand how to actually use it. This is a vicious circle
holding back QMP/JSON and making the human syntax an ever greater
burden for users & maintainers


IOW, the difficulty with configuring QEMU via JSON is not the fault
of JSON itself, it is the lack of knowledge amongst users and docs,
compounded by our never ending "improvements" to the human syntax.
There are other factors too, such as our only partial coverage of
config using JSON - some is only possible via the CLI still.


I guess my point is that with a scrap & startover view point, we
should arguably completely ignore the design question of how to
flatten JSON for humans/command line, as it is the wrong problem.
Instead focus on the problem of making use of JSON the best way
to deal with QEMU both functionally and practically, for humans
and machines alike.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 2:19 AM, Markus Armbruster wrote:
> John Snow  writes:
> 
>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
 * scripts/qmp/qmp-shell

   Half-hearted attempt at a human-friendly wrapper around the JSON
   syntax.  I have no use for this myself.
>>> I use this fairly often as its a useful debugging / experimentation
>>> / trouble shooting tool. There's similar ish functionality in
>>> virsh qemu-monitor-command. I think there's scope of a supported
>>> tool here that can talk to libvirt or a UNIX socket for doing
>>> QMP commands, with a friendlier syntax & pretty printing. 
>>>
>>
>> qmp-shell is one of my go-to tools for working through bitmap workflows
>> where we don't have convenience commands yet, as some of the setups
>> required for fleecing et al involve quite a number of steps.
>>
>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>> commands neatly organized in a format where I can visually reduce them
>> to their components at a glance.
>>
>> (What I mean is: It's hard to remember which QMP commands you've barfed
>> into a terminal because JSON is hard to read and looks very visually
>> repetitive.)
>>
>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>> a new REPL that was json-aware in some manner such that you could write
>> multi-line commands like this:
>>
>>> example-command arg={
>>   "hello": "world"
>> }
>>
>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>> into Python as-is simply take a file pointer and consume the entirety of
>> the rest of the stream -- they don't play very nice with incomplete
>> input or input that may have trailing data, e.g.:
>>
>>> example-command arg={
>>   "hello": "world"
>> } arg2={
>>   "oops!": "more json!"
>> }
> 
> QMP is in the same boat: it needs to process input that isn't
> necessarily full expressions (JSON-text in the RFC's grammar).
> 
> Any conventional parser can be made streaming by turning it into a
> coroutine.  This is probably the simplest solution for handwritten
> streaming LL parsers, because it permits recursive descent.  In Python,
> I'd try a generator.
> 
> Our actual solution for QMP predates coroutine support in QEMU, and is
> rather hamfisted:
> 
> * Streaming lexer: it gets fed characters one at a time, and when its
>   state machine says "token complete", it feeds the token to the
>   "streamer".
> 
> * "Streamer": gets fed tokens one at a time, buffers them up counting
>   curly and square bracket nesting until the nesting is zero, then
>   passes the buffered tokens to the parser.
> 
> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>   a full expression.
> 
> The best I can say about this is that it works.  The streamer's token
> buffer eats a lot of memory compared to a real streaming parser, but in
> practice, it's a drop in the bucket.
> 

I looked into this at one point. I forget why I didn't like it. I had
some notion that I should replace this one too, but forget exactly why.
Maybe it wasn't that bad, if I've forgotten.

>> Also, due to the nature of JSON as being a single discrete object and
>> never a stream of objects, no existing JSON parser really supports the
>> idea of ever seeing more than one object per buffer.
> 
> That plainly sucks.
> 
>> ...So I investigated writing a proper grammar for qmp-shell.
> 
> Any parser must start with a proper grammar.  If it doesn't, it's a toy,
> or a highway to madness.
> 
>> Unfortunately, this basically means including the JSON grammar as a
>> subset of the shell grammar and writing your own parser for it entirely.
> 
> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
> paper, so JSON sticks out left and right.
> 
> Scrap and start over.
> 
>> I looked into using Python's own lexer; but it's designed to lex
>> *python*, not *json*. I got a prototype lexer working for this purpose
>> under a grammar that I think reflects JSON, but I got that sinking
>> feeling that it was all more trouble than it was worth, and scrapped
>> working on it any further.
> 
> Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
> weird derivative of JSON in 239 SLOC.
> 
>> I did not find any other flex/yacc-like tools that seemed properly
>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>> writing a new parser.
> 
> While I recommend use of tools for parsing non-trivial grammars (you'll
> screw up, they won't), they're massive overkill for JSON.
> 
>> I'd love to offer a nice robust QMP shell that is available for use by
>> end users, but the syntax of the shell will need some major considerations.
> 
> Scrap and start over.
> 
> [...]
> 

Yes, I agree: Scrap and start over.

What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
is that it offers a convenient way to enter the top-level keys of the
arguments dict. This works absolutely fine right up until you need to
start providing 

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Thu, Jan 23, 2020 at 01:13:33PM -0500, John Snow wrote:
> 
> 
> On 1/23/20 5:27 AM, Daniel P. Berrangé wrote:
> > On Wed, Jan 22, 2020 at 05:42:10PM -0500, John Snow wrote:
> >>
> >>
> >> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
> >>> Based on experiance in libvirt, this is an even larger job than (4),
> >>> as the feature set here is huge.  Much of it directly ties into the
> >>> config problem, as to deal with SELinux / namespace setup the code
> >>> needs to understand what resources to provide access to. This
> >>> requires a way to express 100% coverage of all QEMU configuration
> >>> in use & analyse it to determine what resources it implies. So this
> >>> ties strongly into QAPI-ification completion.
> >>
> >> Is it totally bonkers to suggest that QEMU provide a method of digesting
> >> a given configuration and returning a configuration object that a
> >> standalone jailer can use?
> >>
> >> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
> >> manager cooperate to produce the jailing configuration, and the jailer
> >> does what we ask it to.
> > 
> > It isn't clear what you mean by "QEMU" here. If this QEMU, the system
> > emulator process, then this is the untrustworthy part of the stack,
> > so the jailer must not use any data that QEMU is providing. In fact
> > during startup the jailer does its work before QEMU even exists.
> > 
> 
> I worried about this. Hence the "Nuts?" ask. It sounds like the ultimate
> problem is nobody can know -- except QEMU -- what permissions are truly
> needed for a given configuration. Even if we had an immaculate API, how
> would anyone except QEMU developers know?
> 
> Trial and error, perhaps, on behalf of the jailer developers. Trial and
> error is not the greatest feature of a security mechanism. Clearly, a
> lot of effort has been spent to get libvirt's implementation correct,
> but Stefan raises the idea that other projects have need of
> understanding how to map QEMU configurations to appropriate jails.
> 
> Worse, it could still change on a whim. We (QEMU developers) probably
> are not used to thinking of permitted syscall lists as ABI that we
> strive to maintain. It can change.
> 
> How do we make this easier in a way that doesn't trust QEMU? I feel like
> QEMU needs to provide *some* kind of information that can be used to
> build better jailing configurations...

I think the key thing to understand is that once you involve a "jailer"
then "QEMU" is no longer a single thing we can refer to. It is immediately
split into two pieces, "the launcher" and "the runtime", with a jailer
sitting in between the pair.

The configuration information needed by the launcher is a superset
of the information needed by the runtime. The launcher is trustworthy
and the runtime is untrustworthy.

In the current world qemu-system-* is the runtime and libvirt is
the launcher, and they each have their own completely different
vocabulary for configuration.

Essentially what you're talking about is a way to bring the concept
of the launcher under the responsibility of the QEMU project, and
have much closer alignment between the configuration for the launcher
and the runtime. One would still need to be the superset of the
other, but they could be 90% common. 

Having "the launcher" be part of the QEMU project deliverables is
a completely reasonable concept. Libvirt only took on that role
itself because QEMU has never provided any solution for this
problem itself.  I don't think this is a technically difficult
task. The issues are simply around the practicality of the size/
scope of the work required in order to achieve deliver it for the
diverse range of QEMU functionality & use cases.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 5:27 AM, Daniel P. Berrangé wrote:
> On Wed, Jan 22, 2020 at 05:42:10PM -0500, John Snow wrote:
>>
>>
>> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
>>> Based on experiance in libvirt, this is an even larger job than (4),
>>> as the feature set here is huge.  Much of it directly ties into the
>>> config problem, as to deal with SELinux / namespace setup the code
>>> needs to understand what resources to provide access to. This
>>> requires a way to express 100% coverage of all QEMU configuration
>>> in use & analyse it to determine what resources it implies. So this
>>> ties strongly into QAPI-ification completion.
>>
>> Is it totally bonkers to suggest that QEMU provide a method of digesting
>> a given configuration and returning a configuration object that a
>> standalone jailer can use?
>>
>> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
>> manager cooperate to produce the jailing configuration, and the jailer
>> does what we ask it to.
> 
> It isn't clear what you mean by "QEMU" here. If this QEMU, the system
> emulator process, then this is the untrustworthy part of the stack,
> so the jailer must not use any data that QEMU is providing. In fact
> during startup the jailer does its work before QEMU even exists.
> 

I worried about this. Hence the "Nuts?" ask. It sounds like the ultimate
problem is nobody can know -- except QEMU -- what permissions are truly
needed for a given configuration. Even if we had an immaculate API, how
would anyone except QEMU developers know?

Trial and error, perhaps, on behalf of the jailer developers. Trial and
error is not the greatest feature of a security mechanism. Clearly, a
lot of effort has been spent to get libvirt's implementation correct,
but Stefan raises the idea that other projects have need of
understanding how to map QEMU configurations to appropriate jails.

Worse, it could still change on a whim. We (QEMU developers) probably
are not used to thinking of permitted syscall lists as ABI that we
strive to maintain. It can change.

How do we make this easier in a way that doesn't trust QEMU? I feel like
QEMU needs to provide *some* kind of information that can be used to
build better jailing configurations...

> There are aspects to the confinement that use / rely on knowledge that
> QEMU doesn't normally have, or are expressed in a different way that
> which QEMU uses, or needs to take a different imlpementation approach to
> that which QEMU normally has.
> 
> For networking, for example, from QEMU's config POV, there's just a
> TAP file descriptor. There are then a huge number of ways in which
> that TAP FD has been connected to the network in the host that are
> invisible to QEMU. Plain bridge, openvswitch bridge, macvtap device
> all with varying configs. Knowledge of this is relevant to the manager
> process and the jailer but irrelevant to QEMU.
> 
> When configuring disks we have technical issues. For example we need
> to identify the full backing chain and grant the appropriate permissions
> on this. Even if there was a libqemublock.so, libvirt would not use this
> because the QEMU storage code design is not reliable & minimal enough.
> For example to just query the backing file, QEMU opens the qcow2 and
> parses all the data about it, building up L1/L2 tables, and other
> data structures involved. It is trivial to create qcow2 files which
> result in both memory and CPU denial of service merely from opening
> the file.  Libvirt's approach to this is minimalist just having a
> data table of offsets to the key fields in each file format. So we
> can extract the backing file & its format without reading anything
> else from the disk.
> 
> When configuring chardevs there is a choice of how to do it - we
> could just pass the UNIX socket path in, or we could create the
> UNIX socket ourselves & pass in the pre-opened FD. Both are equally
> functional from QEMU's POV and the end user's POV, but passing a
> pre-opened FD is more convenient for libvirt's needs as it allowed
> for race-free startups sychronization between libvirt & QEMU, or
> rather QMP.  The different options here though, have different
> needs on the jailer, because extra steps are needed when passing
> pre-opened FD to get the SELinux labelling right. QEMU doesn't
> know which approach the mgmt app will want to take, so we can't
> ask QEMU how the jailer should be configured - the mgmt app needs
> to make that decision.
> 
> Essentially we have 2 configuration formats - the high level one
> that the mgmt app layer uses & the low level one that QEMU uses.
> The component in the stack which maps between the two config
> formats, is that one that has the knowledge to configure the
> jailer. This isn't QEMU. It is whatever is immediately above QEMU,
> currently libvirt, but something conceptually equivalent to the
> role libvirt's QEMU driver impl fills.
> 
> Regards,
> Daniel
> 

-- 
—js

Re: Making QEMU easier for management tools and applications

[Daniel P . Berrangé]

On Wed, Jan 22, 2020 at 05:42:10PM -0500, John Snow wrote:
> 
> 
> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
> > Based on experiance in libvirt, this is an even larger job than (4),
> > as the feature set here is huge.  Much of it directly ties into the
> > config problem, as to deal with SELinux / namespace setup the code
> > needs to understand what resources to provide access to. This
> > requires a way to express 100% coverage of all QEMU configuration
> > in use & analyse it to determine what resources it implies. So this
> > ties strongly into QAPI-ification completion.
> 
> Is it totally bonkers to suggest that QEMU provide a method of digesting
> a given configuration and returning a configuration object that a
> standalone jailer can use?
> 
> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
> manager cooperate to produce the jailing configuration, and the jailer
> does what we ask it to.

It isn't clear what you mean by "QEMU" here. If this QEMU, the system
emulator process, then this is the untrustworthy part of the stack,
so the jailer must not use any data that QEMU is providing. In fact
during startup the jailer does its work before QEMU even exists.

There are aspects to the confinement that use / rely on knowledge that
QEMU doesn't normally have, or are expressed in a different way that
which QEMU uses, or needs to take a different imlpementation approach to
that which QEMU normally has.

For networking, for example, from QEMU's config POV, there's just a
TAP file descriptor. There are then a huge number of ways in which
that TAP FD has been connected to the network in the host that are
invisible to QEMU. Plain bridge, openvswitch bridge, macvtap device
all with varying configs. Knowledge of this is relevant to the manager
process and the jailer but irrelevant to QEMU.

When configuring disks we have technical issues. For example we need
to identify the full backing chain and grant the appropriate permissions
on this. Even if there was a libqemublock.so, libvirt would not use this
because the QEMU storage code design is not reliable & minimal enough.
For example to just query the backing file, QEMU opens the qcow2 and
parses all the data about it, building up L1/L2 tables, and other
data structures involved. It is trivial to create qcow2 files which
result in both memory and CPU denial of service merely from opening
the file.  Libvirt's approach to this is minimalist just having a
data table of offsets to the key fields in each file format. So we
can extract the backing file & its format without reading anything
else from the disk.

When configuring chardevs there is a choice of how to do it - we
could just pass the UNIX socket path in, or we could create the
UNIX socket ourselves & pass in the pre-opened FD. Both are equally
functional from QEMU's POV and the end user's POV, but passing a
pre-opened FD is more convenient for libvirt's needs as it allowed
for race-free startups sychronization between libvirt & QEMU, or
rather QMP.  The different options here though, have different
needs on the jailer, because extra steps are needed when passing
pre-opened FD to get the SELinux labelling right. QEMU doesn't
know which approach the mgmt app will want to take, so we can't
ask QEMU how the jailer should be configured - the mgmt app needs
to make that decision.

Essentially we have 2 configuration formats - the high level one
that the mgmt app layer uses & the low level one that QEMU uses.
The component in the stack which maps between the two config
formats, is that one that has the knowledge to configure the
jailer. This isn't QEMU. It is whatever is immediately above QEMU,
currently libvirt, but something conceptually equivalent to the
role libvirt's QEMU driver impl fills.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow  writes:

> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>> * scripts/qmp/qmp-shell
>>>
>>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>>   syntax.  I have no use for this myself.
>> I use this fairly often as its a useful debugging / experimentation
>> / trouble shooting tool. There's similar ish functionality in
>> virsh qemu-monitor-command. I think there's scope of a supported
>> tool here that can talk to libvirt or a UNIX socket for doing
>> QMP commands, with a friendlier syntax & pretty printing. 
>> 
>
> qmp-shell is one of my go-to tools for working through bitmap workflows
> where we don't have convenience commands yet, as some of the setups
> required for fleecing et al involve quite a number of steps.
>
> I can copy-paste raw JSON into a socket, but personally I like seeing my
> commands neatly organized in a format where I can visually reduce them
> to their components at a glance.
>
> (What I mean is: It's hard to remember which QMP commands you've barfed
> into a terminal because JSON is hard to read and looks very visually
> repetitive.)
>
> I tried to rewrite qmp-shell late last year, actually. I wanted to write
> a new REPL that was json-aware in some manner such that you could write
> multi-line commands like this:
>
>> example-command arg={
>   "hello": "world"
> }
>
> This requires, sadly, a streamable JSON parser. Most JSON parsers built
> into Python as-is simply take a file pointer and consume the entirety of
> the rest of the stream -- they don't play very nice with incomplete
> input or input that may have trailing data, e.g.:
>
>> example-command arg={
>   "hello": "world"
> } arg2={
>   "oops!": "more json!"
> }

QMP is in the same boat: it needs to process input that isn't
necessarily full expressions (JSON-text in the RFC's grammar).

Any conventional parser can be made streaming by turning it into a
coroutine.  This is probably the simplest solution for handwritten
streaming LL parsers, because it permits recursive descent.  In Python,
I'd try a generator.

Our actual solution for QMP predates coroutine support in QEMU, and is
rather hamfisted:

* Streaming lexer: it gets fed characters one at a time, and when its
  state machine says "token complete", it feeds the token to the
  "streamer".

* "Streamer": gets fed tokens one at a time, buffers them up counting
  curly and square bracket nesting until the nesting is zero, then
  passes the buffered tokens to the parser.

* Non-streaming parser: it gets fed a sequence of tokens that constitute
  a full expression.

The best I can say about this is that it works.  The streamer's token
buffer eats a lot of memory compared to a real streaming parser, but in
practice, it's a drop in the bucket.

> Also, due to the nature of JSON as being a single discrete object and
> never a stream of objects, no existing JSON parser really supports the
> idea of ever seeing more than one object per buffer.

That plainly sucks.

> ...So I investigated writing a proper grammar for qmp-shell.

Any parser must start with a proper grammar.  If it doesn't, it's a toy,
or a highway to madness.

> Unfortunately, this basically means including the JSON grammar as a
> subset of the shell grammar and writing your own parser for it entirely.

Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
paper, so JSON sticks out left and right.

Scrap and start over.

> I looked into using Python's own lexer; but it's designed to lex
> *python*, not *json*. I got a prototype lexer working for this purpose
> under a grammar that I think reflects JSON, but I got that sinking
> feeling that it was all more trouble than it was worth, and scrapped
> working on it any further.

Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
weird derivative of JSON in 239 SLOC.

> I did not find any other flex/yacc-like tools that seemed properly
> idiomatic or otherwise heavily specialized. I gave up on the idea of
> writing a new parser.

While I recommend use of tools for parsing non-trivial grammars (you'll
screw up, they won't), they're massive overkill for JSON.

> I'd love to offer a nice robust QMP shell that is available for use by
> end users, but the syntax of the shell will need some major considerations.

Scrap and start over.

[...]

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow  writes:

> On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
>> Based on experiance in libvirt, this is an even larger job than (4),
>> as the feature set here is huge.  Much of it directly ties into the
>> config problem, as to deal with SELinux / namespace setup the code
>> needs to understand what resources to provide access to. This
>> requires a way to express 100% coverage of all QEMU configuration
>> in use & analyse it to determine what resources it implies. So this
>> ties strongly into QAPI-ification completion.
>
> Is it totally bonkers to suggest that QEMU provide a method of digesting
> a given configuration and returning a configuration object that a
> standalone jailer can use?
>
> So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
> manager cooperate to produce the jailing configuration, and the jailer
> does what we ask it to.
>
> Nuts?

With the nuts-o-meter calibrated for QEMU CLI: nope, this hardly moves
the needle.

Re: Making QEMU easier for management tools and applications

[John Snow]

On 12/24/19 8:00 AM, Daniel P. Berrangé wrote:
> Based on experiance in libvirt, this is an even larger job than (4),
> as the feature set here is huge.  Much of it directly ties into the
> config problem, as to deal with SELinux / namespace setup the code
> needs to understand what resources to provide access to. This
> requires a way to express 100% coverage of all QEMU configuration
> in use & analyse it to determine what resources it implies. So this
> ties strongly into QAPI-ification completion.

Is it totally bonkers to suggest that QEMU provide a method of digesting
a given configuration and returning a configuration object that a
standalone jailer can use?

So we have a QEMU manager, the generic jailer, and QEMU. QEMU and the
manager cooperate to produce the jailing configuration, and the jailer
does what we ask it to.

Nuts?

--js

Re: Making QEMU easier for management tools and applications

[John Snow]

On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>> * scripts/qmp/qmp-shell
>>
>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>   syntax.  I have no use for this myself.
> I use this fairly often as its a useful debugging / experimentation
> / trouble shooting tool. There's similar ish functionality in
> virsh qemu-monitor-command. I think there's scope of a supported
> tool here that can talk to libvirt or a UNIX socket for doing
> QMP commands, with a friendlier syntax & pretty printing. 
> 

qmp-shell is one of my go-to tools for working through bitmap workflows
where we don't have convenience commands yet, as some of the setups
required for fleecing et al involve quite a number of steps.

I can copy-paste raw JSON into a socket, but personally I like seeing my
commands neatly organized in a format where I can visually reduce them
to their components at a glance.

(What I mean is: It's hard to remember which QMP commands you've barfed
into a terminal because JSON is hard to read and looks very visually
repetitive.)

I tried to rewrite qmp-shell late last year, actually. I wanted to write
a new REPL that was json-aware in some manner such that you could write
multi-line commands like this:

> example-command arg={
  "hello": "world"
}

This requires, sadly, a streamable JSON parser. Most JSON parsers built
into Python as-is simply take a file pointer and consume the entirety of
the rest of the stream -- they don't play very nice with incomplete
input or input that may have trailing data, e.g.:

> example-command arg={
  "hello": "world"
} arg2={
  "oops!": "more json!"
}

Also, due to the nature of JSON as being a single discrete object and
never a stream of objects, no existing JSON parser really supports the
idea of ever seeing more than one object per buffer.

...So I investigated writing a proper grammar for qmp-shell.
Unfortunately, this basically means including the JSON grammar as a
subset of the shell grammar and writing your own parser for it entirely.

I looked into using Python's own lexer; but it's designed to lex
*python*, not *json*. I got a prototype lexer working for this purpose
under a grammar that I think reflects JSON, but I got that sinking
feeling that it was all more trouble than it was worth, and scrapped
working on it any further.

I did not find any other flex/yacc-like tools that seemed properly
idiomatic or otherwise heavily specialized. I gave up on the idea of
writing a new parser.

I'd love to offer a nice robust QMP shell that is available for use by
end users, but the syntax of the shell will need some major considerations.


(Coffee break)


The other thing I wanted to do was offer a TUI-esque backlog of
commands; something reminiscent of mutt or TUI mail clients. Arrow keys
could be used to browse the list of previous TX/RX objects,
spacebar/enter to view them in detail; '?' to pull up the relevant QAPI
documentation in a side-pane. That sort of thing.

I thought it would be nice to offer an "IRC-esque" view of the QMP
protocol where new events stream in from the bottom, and new commands
can be typed into the buffer bar. (I found no convincing way to combine
e.g. readline with a TUI interface. It seemed like you get one or the
other.)

Allowing qmp-shell to show events received asynchronously would also be
a big leap up from the current shell.

It's a lot of fantasy stuff I wasn't *quite* able to tie together.

--js

Re: Making QEMU easier for management tools and applications

[Marc-André Lureau]

Hi

On Tue, Jan 21, 2020 at 3:32 PM Stefan Hajnoczi  wrote:
>
> On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> > Stefan Hajnoczi  writes:
> >
> > > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> > >> Christophe de Dinechin  writes:
> > >> >> On 15 Jan 2020, at 10:20, Markus Armbruster  wrote:
> > >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> > >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> > >>   documented (in qemu-options.hx), their use with qom-set is not.
> > >
> > > I'm happy to use a different interface.
> > >
> > > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> > > be a step backwards.
> >
> > No argument.
> >
> > > Maybe the QAPI code generator could map something like this:
> > >
> > >   { 'command': 'iothread-set-poll-params',
> > > 'data': {
> > > 'id': 'str',
> > > '*max-ns': 'uint64',
> > > '*grow': 'uint64',
> > > '*shrink': 'uint64'
> > > },
> > > 'map-to-qom-set': 'IOThread'
> > >   }
> > >
> > > And turn it into QOM accessors on the IOThread object.
> >
> > I think a generic "set this configuration to that value" command is just
> > fine.  qom-set fails on several counts, though:
> >
> > * Tolerable: qom-set is not actually generic, it applies only to QOM.
> >
> > * qom-set lets you set tons of stuff that is not meant to be changed at
> >   run time.  If it breaks your guest, you get to keep the pieces.
> >
> > * There is virtually no documentation on what can be set to what values,
> >   and their semantics.
> >
> > In its current state, QOM is a user interface superfund site.
>
> Thoughts about a solution:
>
> Static QOM properties should be declared via QAPI instead of
> imperatively via QOM APIs.  That way they are introspectable and type
> information is present in the schema.
>
> The QAPI code generator could emit a function that is callable from
> .class_init().  This eliminates the need to manually call
> object_class_property_add().

I have this in mind too. First step is probably to move as much as
possible as class properties. Please review
https://patchew.org/QEMU/20200110153039.1379601-1-marcandre.lur...@redhat.com/,
I have more of this kind of qdev/qom cleanups pending.



-- 
Marc-André Lureau

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

On Tue, Jan 21, 2020 at 06:42:47AM +0100, Markus Armbruster wrote:
> Stefan Hajnoczi  writes:
> 
> > On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> >> Christophe de Dinechin  writes:
> >> >> On 15 Jan 2020, at 10:20, Markus Armbruster  wrote:
> >> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
> >>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
> >>   documented (in qemu-options.hx), their use with qom-set is not.
> >
> > I'm happy to use a different interface.
> >
> > Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> > be a step backwards.
> 
> No argument.
> 
> > Maybe the QAPI code generator could map something like this:
> >
> >   { 'command': 'iothread-set-poll-params',
> > 'data': {
> > 'id': 'str',
> > '*max-ns': 'uint64',
> > '*grow': 'uint64',
> > '*shrink': 'uint64'
> > },
> > 'map-to-qom-set': 'IOThread'
> >   }
> >
> > And turn it into QOM accessors on the IOThread object.
> 
> I think a generic "set this configuration to that value" command is just
> fine.  qom-set fails on several counts, though:
> 
> * Tolerable: qom-set is not actually generic, it applies only to QOM.
> 
> * qom-set lets you set tons of stuff that is not meant to be changed at
>   run time.  If it breaks your guest, you get to keep the pieces.
> 
> * There is virtually no documentation on what can be set to what values,
>   and their semantics.
> 
> In its current state, QOM is a user interface superfund site.

Thoughts about a solution:

Static QOM properties should be declared via QAPI instead of
imperatively via QOM APIs.  That way they are introspectable and type
information is present in the schema.

The QAPI code generator could emit a function that is callable from
.class_init().  This eliminates the need to manually call
object_class_property_add().

Stefan


signature.asc
Description: PGP signature

Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Stefan Hajnoczi  writes:

> On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
>> Christophe de Dinechin  writes:
>> >> On 15 Jan 2020, at 10:20, Markus Armbruster  wrote:
>> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>>   documented (in qemu-options.hx), their use with qom-set is not.
>
> I'm happy to use a different interface.
>
> Writing a boilerplate "iothread-set-poll-params" QMP command in C would
> be a step backwards.

No argument.

> Maybe the QAPI code generator could map something like this:
>
>   { 'command': 'iothread-set-poll-params',
> 'data': {
> 'id': 'str',
>   '*max-ns': 'uint64',
>   '*grow': 'uint64',
>   '*shrink': 'uint64'
> },
> 'map-to-qom-set': 'IOThread'
>   }
>
> And turn it into QOM accessors on the IOThread object.

I think a generic "set this configuration to that value" command is just
fine.  qom-set fails on several counts, though:

* Tolerable: qom-set is not actually generic, it applies only to QOM.

* qom-set lets you set tons of stuff that is not meant to be changed at
  run time.  If it breaks your guest, you get to keep the pieces.

* There is virtually no documentation on what can be set to what values,
  and their semantics.

In its current state, QOM is a user interface superfund site.

Re: Making QEMU easier for management tools and applications

[Kashyap Chamarthy]

On Mon, Jan 20, 2020 at 09:55:54AM +, Stefan Hajnoczi wrote:
> On Thu, Jan 16, 2020 at 12:03:14PM +0100, Kashyap Chamarthy wrote:

[...]

> > I'm reasonably happy with it (particularly the persistent history
> > captured in ~/.qmp-shell_history), and it has some "known issues" that
> > can trip up a new user.  The one that immediately jumps to mind:
> > asynchronous events won't be printed without a prompt from the user --
> > e.g. after a `blockdev-commit`, you won't see BLOCK_JOB_{READY,
> > COMPLETED} events printed unless you manually hit enter from the
> > 'qmp-shell'.

[...]

> John and I discussed async events in the past.  qmp-shell currently uses
> the input() built-in function.  If we modify it with a
> select(2)/poll(2)-style function that monitors both stdin and the QMP
> socket then it could print QMP events as soon as they are received.
> 
> There might be a nicer way of doing it, but pseudo-code for the idea is:
> 
>   def input_with_events(prompt):
>   while True:
>   print(prompt, end='', flush=True)
>   readable_files = select([sys.stdin, qmp_socket])
>   if qmp_socket in readable_files:
>   print_qmp_events()
> 
>   # stdin is ready, read a line
>   return input()

Thanks for the suggestion.  The Python 'select' module[1] indeed seems
to provide access to select() and poll() Linux system calls.

[1] https://docs.python.org/3/library/select.html

-- 
/kashyap

Re: Making QEMU easier for management tools and applications

[Stefan Hajnoczi]

On Wed, Jan 15, 2020 at 01:15:17PM +0100, Markus Armbruster wrote:
> Christophe de Dinechin  writes:
> >> On 15 Jan 2020, at 10:20, Markus Armbruster  wrote:
> * qemuMonitorJSONSetIOThread() uses it to control iothread's properties
>   poll-max-ns, poll-grow, poll-shrink.  Their use with -object is
>   documented (in qemu-options.hx), their use with qom-set is not.

I'm happy to use a different interface.

Writing a boilerplate "iothread-set-poll-params" QMP command in C would
be a step backwards.

Maybe the QAPI code generator could map something like this:

  { 'command': 'iothread-set-poll-params',
'data': {
'id': 'str',
'*max-ns': 'uint64',
'*grow': 'uint64',
'*shrink': 'uint64'
},
'map-to-qom-set': 'IOThread'
  }

And turn it into QOM accessors on the IOThread object.

Stefan


signature.asc
Description: PGP signature