Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Kevin Wolf]

Am 06.06.2018 um 16:55 hat Michael S. Tsirkin geschrieben:
> On Wed, Jun 06, 2018 at 10:42:33AM -0300, Eduardo Habkost wrote:
> > > If we want a grand vision where a single file stores the whole VM, why
> > > not invest the work and make it right from the start?
> > 
> > We don't want a grand vision where a single file stores the whole
> > VM.  This is exactly what I would like to avoid, by not inventing
> > a whole different appliance file format.
> 
> Besides, trying to get a grand vision from the start is a sure
> way to never have the design leave the drawing board.
> 
> What we are asking for at this point is a way to stick a named blob in
> an image that people can use with qemu without jumping through hoops.
> 
> It seems like a generic enough addition that it seems highly likely
> to be useful down the road and harmless enough that maintaining
> it won't become a burden.
> 
> Can we agree on that as a first step, so we can build that foundation
> and move on to actually building ways to use it?

As you don't seem to believe Max, here's my opinion: No.

I'm okay with adding some well-specified information to qcow2 (I'm
thinking of a JSON document that is validated against a schema) where
the meaning is clear from the qcow2 spec, for all users.

Allowing undefined blobs would add data to qcow2 images whose meaning is
only understood by whatever highlevel tool that wrote it, and thereby
fragment the one qcow2 format that we have today into many subformats
whose specs are scattered all over the net (if they even exist). This is
not where I want to go.

So before I'll even think of adding some extra information, we're back
to what I already said two weeks ago: Show me what you want to store
there. If we go with my proposal to use JSON, show me the JSON schema
(with doc comments, obviously) and we'll talk.

Kevin

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Richard W.M. Jones]

On Wed, Jun 06, 2018 at 01:32:47PM +0200, Max Reitz wrote:
> ext2?

I wrote an nbdkit plugin for ext2/ext3/ext4 last week.

  https://github.com/libguestfs/nbdkit/tree/master/plugins/ext2

It uses libext2fs from e2fsprogs and I think there are some lessons
for anyone who wants to use ext2 to store disk images.

(1) You cannot have more than one host process accessing a single
filesystem image, even read-only.  This is because opening an ext2+
filesystem even read-only causes writes, replaying the journal (for
ext3+) or writing to the superblock.

I'm sure there are some common use-cases such as overlays sharing a
common backing store which are excluded by this restriction.

(2) Within a single process you cannot have more than one libext2fs
handle open on the filesystem image.  This could make qemu block
drivers a bit awkward (although not impossible) because if two
instances of an ext2 qemu block driver both opened different disks in
the same filesystem they'd need to share a handle.

(3) You can resize files in the filesystem, although because we're
waiting for the NBD resize extension to be finalized my plugin does
not do that.

(4) Trim/discard appears to be possible (and it should be possible to
punch holes in the filesystem image) but I couldn't actually
understand how to make it work.  Also fast zeroing.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Mon, 11 Jun 2018 05:06:53 +0300
"Michael S. Tsirkin"  wrote:

> On Sat, Jun 09, 2018 at 11:34:03PM +0200, Max Reitz wrote:

> > Dave was saying that the worst thing about the whole q35 thing is
> > that users download an image and have no idea why it isn't
> > working.  Figuring that out may take a long time, because nothing
> > is even throwing an error message.
> > 
> > If we had a new format, users couldn't even run it in qemu, so they
> > would quickly figure out that in order to run this VM, they need to
> > update their stack.  
> 
> Since then users of old software can't use your
> image at all, then most people simply will not create
> the new fangled image format.

They will have to because without the new fangles the images just don't
work.

> 
> > If we just add this information to qcow2, those users with outdated
> > qemu versions would again have to figure out why the image isn't
> > working.  
> 
> By that metric compatibility is never worth it unless you have
> ability to add new functionality retroactively to existing
> software.

Compatibility is worth it when you add a new extension that is useful
but not critical. When the image does not work without interpreting the
extended metadata, the failure is hard to diagnose, and old versions of
qemu just ignore the extended metadata the extension failed the primary
purpose: making images work out of the box.

Thanks

Michal

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Sat, Jun 09, 2018 at 11:34:03PM +0200, Max Reitz wrote:
> qemu would be very easy to use if it didn't offer any configuration
> options.  The problem is that is offers a huge load of configuration
> options and it is not reasonable to expect every user to know all of them.

Right but once one user id find a way to make a specific guest work in a
specific VM, it should not be as hard as it is to replicate the success.

...

> > This just means we should make it flexible enough to possibly
> > support more uses. It does not mean we need to make it
> > read mail on day 1.
> 
> So you are saying that we may end up with multiple parties storing
> (meta-)data independently in the qcow2 file?

Absolutely.

> before there are concrete design documents, and on why it doesn't matter
> that you focus on qemu whereas others focus on the management layer.

Right. That's what I'm saying.

> Though it would mean opaque storage, as I've said, and that storing data
> in the qcow2 file still probably does not make sense for some of the
> possible use cases.  For instance, you propose storing data for qemu
> proper without any management layer, but this poses the question again
> of who should interpret that data.  (A management layer may just query
> the image before launching qemu and then set the appropriate options,
> but it gets difficult for the block layer to open the qcow2 image and
> change the machine type when qemu is already running.  Though maybe you
> could at least error out when incompatible options have been used to
> launch qemu.  Hm.  Another question was who'd be supposed to store the
> data.)
> 
> [...]
> 
> >> And if you make the format decidedly qcow2-independent, the whole
> >> "putting it into qcow2 is the simplest implementation" argument becomes
> >> rather weak.
> > 
> > I don't see why. Yes I think it's a separate format that we should just
> > allow storing in qcow2 for usability.
> 
> It becomes weak because storing it in qcow2 would no longer be the
> simplest implementation if you'd need to be able to read it from a file
> outside of a qcow2 image anyway.
> 
> It still may be the easiest use case for users.
> 
> [...]
> 
> >> But all of that writing once again comes down to this: You are talking
> >> about qemu.  Dave is talking about something higher in the management
> >> layer.  Those are different things, and as I said, we first need to find
> >> common ground there.
> > 
> > The common ground is that both me and Dave find it useful to store meta-data
> > in the disk image.
> 
> Though it seems to me that you have very different ideas on how to store
> it.  As far as I have understood, Dave just wants to store a bit of data
> that might even go into the image header, whereas you'd prefer a
> full-blown infrastructure for binary storage and large objects, because
> someone might want that at some point.
> 
> That isn't to say I personally prefer either, it just means that those
> are different and deciding on one naturally changes what to do.
> 
> There may be even other approaches, I don't know.

I thought I heard Dave utter "key-value store" at some point,
which likely precludes "go directly into the header".


> And my idea was that we should evaluate the different use cases for
> storing arbitrary metadata in a qcow2 file, and then we'd see whether it
> does or doesn't make sense to do so, for each case.

As block guys maybe you could ask more specific questions then.
E.g. "would 1/2K of data be sufficient for these purposes"?
That's a more valid point than a generic "tells us what it's
for" question.

> I think making qemu store something in the qcow2 file doesn't bring too
> much because first, qemu couldn't really interpret that information by
> itself (it could at best detect conflicting configuration, though in my
> head checking qemu configuration in the qcow2 driver screams complexity;
> and storing that information automatically would be problematic); and
> secondly, I don't think that we can solve the complexity that is modern
> qemu configuration by just putting it into a qcow2 file.  That will only
> work as long as the user doesn't want to change anything.  There is
> probably much more to say, but all of that would deserve its own thread.

But I do think we can solve only doing it once and not per user
of an image.

> Regarding designing an appliance format, there has already been some
> discussion, so I won't say anything about that now, except that it too
> should go into its own thread so that people are aware and don't think
> it's just a qcow2 issue.
> 
> >> This is exactly why I said "where to store it heavily depends on what we
> >> want to store and how we want to use it."  As long as we don't know
> >> that, all of us are using strawman arguments where some other party
> >> suddenly chimes in and says "no, no, no, this is not what I'm talking
> >> about".  Yes, maybe you aren't, but someone else is.
> >>
> >> [...]
> > 
> > Looks like discussion has run its 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-07 23:43, Michael S. Tsirkin wrote:
> On Wed, Jun 06, 2018 at 07:06:27PM +0200, Max Reitz wrote:

[...]

>> Er, yeah, OK.  But it was my understanding that we decided that we have
>> a management layer on top of qemu to make things simple.
> 
> Who's we?

Everyone I'm usually talking to when it comes to adding features to the
block layer, be it on the mailing list or at KVM Forum.

Also my own judgment, when I look at how extensively the block layer
uses QMP, which by definition is not meant to be used by humans.

So this is strongly biased by my immediate environment.

>   I don't think the QEMU community completely gave up on people
> using QEMU directly. It will need to be much more user-friendly than it
> is right now.

You are the very first person I hear this from.  Don't take this as
disagreement, it's just that most of the energy recently seem to go into
making qemu interoperate more easily with management layers than with
end users.  Those goals are not necessarily mutually exclusive, but one
thing this changes did was to make "modern" qemu configuration rather
verbose, which is not what I'd call user-friendly.

>   But it's possible. Fabrice built an emulator in
> javascript, you go to a URL bam it runs a VM.

So?

You can still give qemu an image directly and bam it runs a VM.  It's
just that you get an i440 with an IDE HDD, probably like the JS qemu.

The whole point of this thread was that you might want to run q35, and I
don't know, maybe virtio-blk/scsi.  So exactly what you don't get from
the JS VM.

qemu would be very easy to use if it didn't offer any configuration
options.  The problem is that is offers a huge load of configuration
options and it is not reasonable to expect every user to know all of them.

One way to solve this is to add a management layer which knows all of
the options and may choose good defaults based on knowledge that the
qemu process itself doesn't have (i.e. "put implementation in qemu, and
choose the policy in the management layer").  This is at least what the
block layer usually chooses to do.

>> Also, this is once more a case of first deciding what we want at all.
> 
> Who's we here again?

The participants of this discussion.  Maybe I should have said "you"
because it doesn't necessarily concern myself.

OTOH, we don't have an appliance solution so far (as far as I'm aware),
so I suppose it concerns all of the qemu community, and probably
everyone working somewhere up any qemu management stack as well.

>  Different people want different things. Enough
> people seem to want to store tagged data with a disk image that it might
> be worth someone's while to try to add that capability for starters to
> qemu-img.

I disagree, and I will explain why below. [1]

>> Dave wants configuration options for the upper management layer which
>> are completely opaque to qemu.  That has nothing to do whatsoever with
>> the usability of qemu itself.
> 
> That's why I keep saying, let's start with implementing a mechanism,
> worry about policy later if at all.

[1] I wholeheartedly disagree.

Say you have this mechanism now.  Whatever mechanism it is, because
there isn't even any consensus on that (you'd like key-value binary
object storage, Dave just wants a simple strong with some key-value
format that hasn't been defined in detail, although he has given some
proposals).

Then you need to implement something on top of it, store some values,
interpret them somehow.  And this is exactly what I am asking right now.
 What do you want to store?  Where do you want to interpret it?

I do not understand why you think it's harder to decide that now than
after you have extended qcow2.

I absolutely do think designing the qcow2 extension at least becomes
simpler if you do that design now.


There is a case to be made that one shouldn't worry too much about the
future.  Sometimes you just gotta start somewhere and see where it
leads, maybe the initial design was crap, too bad, then you need to
start over.  But at least you got something done.

But we have talked for half a week now.  In my very personal opinion we
definitely haven't reached the point yet where we just need to start
with something.  Appliances would be a big thing, no need to rush it.

[...]

> I think what we are seeing here is many people jumping on the
> bandwagon and finding more and more uses for ability to store
> meta-data in the qcow2 file.

I don't see that here, but it may very well be true that other people
may find it useful even for other purposes than appliances.

> This just means we should make it flexible enough to possibly
> support more uses. It does not mean we need to make it
> read mail on day 1.

So you are saying that we may end up with multiple parties storing
(meta-)data independently in the qcow2 file?

That would be an argument on why we'd want opaque metadata storage
before there are concrete design documents, and on why it doesn't matter

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Daniel P. Berrangé (berra...@redhat.com) wrote:
> On Fri, Jun 08, 2018 at 09:21:30AM +0100, Dr. David Alan Gilbert wrote:
> > * Laszlo Ersek (ler...@redhat.com) wrote:
> > > On 06/07/18 12:54, Andrea Bolognani wrote:
> > > > On Thu, 2018-06-07 at 11:36 +0100, Daniel P. Berrangé wrote:
> > > >> On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> > > >>> Another problem which Laszlo mentioned is the varstore isn't portable
> > > >>> between UEFI implementations, or if the UEFI is compiled with
> > > >>> different options.  You can even imagine shipping multiple
> > > >>> varstores(!) which argues for a tar-like format.
> > > >>
> > > >> Could we perhaps imagine shipping the actual UEFI bios, rather
> > > >> than only the varstore.  The bios blob runs in guest context,
> > > >> so there shouldn't be able security concerns from hosting
> > > >> vendors with running user provided bios. Mostly its a matter
> > > >> of confidence that the interface between bios & qemu is stable
> > > >> which feels easier than assuming varstore vs different bios is
> > > >> portable.
> > > > 
> > > > That sounds sensible, and further reinforces the idea that we
> > > > need way more than a single string baked into the qcow2 file.
> > > > 
> > > 
> > > Sorry for arriving late (thanks Rich for the Fwd).
> > > 
> > > The contents of the non-volatile UEFI variables should be considered
> > > part of (permanent) guest state, such as disk contents. Therefore I'd
> > > argue for bundling the varstore file with the disk image(s).
> > > 
> > > In turn, the best way to ensure comaptibility between varstore and
> > > firmware binary is to just bundle the firmware binary as well. It's
> > > generally not large (x86) or if it is, it compresses extremely well
> > > (aarch64). For extra politeness, image providers can bundle a text file
> > > with their firmware build options (like a kernel config), possibly even
> > > a JSON document conforming to the new firmware schema (qemu commit
> > > 3a0adfc9bfcf), but that's not a hard requirement I guess.
> > > 
> > > If such a VM is to be migrated between hosts, I'd expect the host admin
> > > to take care of installing the fw binary on all eligible hosts.
> > 
> > There's no way they can do that if they're just importing VMs from
> > templates that include the image; who is going to keep track of which
> > BIOSs are needed where?
> 
> It isn't that unusual a requirement. When Openstack deploys a VM, it
> has the user provided image as a base file, and then creates  qcow2
> overlay.  If the VM is cold migrated (ie not running) to another
> host, OpenStack has to make sure the same base file gets copied across
> to the new host so that the overlay still works. Copying the BIOS file
> and vars state across at the same time is no more difficult than what
> its already doing.

I'm kind of OK with management layers doing it; but Laszlo was
suggesting it was an admins problem;  if we can make it something
manageable by higher levels that's OK.
(Although I'm still concerned that making images with a UEFI image in
that's portable is still not going to work).

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Fri, Jun 08, 2018 at 09:21:30AM +0100, Dr. David Alan Gilbert wrote:
> * Laszlo Ersek (ler...@redhat.com) wrote:
> > On 06/07/18 12:54, Andrea Bolognani wrote:
> > > On Thu, 2018-06-07 at 11:36 +0100, Daniel P. Berrangé wrote:
> > >> On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> > >>> Another problem which Laszlo mentioned is the varstore isn't portable
> > >>> between UEFI implementations, or if the UEFI is compiled with
> > >>> different options.  You can even imagine shipping multiple
> > >>> varstores(!) which argues for a tar-like format.
> > >>
> > >> Could we perhaps imagine shipping the actual UEFI bios, rather
> > >> than only the varstore.  The bios blob runs in guest context,
> > >> so there shouldn't be able security concerns from hosting
> > >> vendors with running user provided bios. Mostly its a matter
> > >> of confidence that the interface between bios & qemu is stable
> > >> which feels easier than assuming varstore vs different bios is
> > >> portable.
> > > 
> > > That sounds sensible, and further reinforces the idea that we
> > > need way more than a single string baked into the qcow2 file.
> > > 
> > 
> > Sorry for arriving late (thanks Rich for the Fwd).
> > 
> > The contents of the non-volatile UEFI variables should be considered
> > part of (permanent) guest state, such as disk contents. Therefore I'd
> > argue for bundling the varstore file with the disk image(s).
> > 
> > In turn, the best way to ensure comaptibility between varstore and
> > firmware binary is to just bundle the firmware binary as well. It's
> > generally not large (x86) or if it is, it compresses extremely well
> > (aarch64). For extra politeness, image providers can bundle a text file
> > with their firmware build options (like a kernel config), possibly even
> > a JSON document conforming to the new firmware schema (qemu commit
> > 3a0adfc9bfcf), but that's not a hard requirement I guess.
> > 
> > If such a VM is to be migrated between hosts, I'd expect the host admin
> > to take care of installing the fw binary on all eligible hosts.
> 
> There's no way they can do that if they're just importing VMs from
> templates that include the image; who is going to keep track of which
> BIOSs are needed where?

It isn't that unusual a requirement. When Openstack deploys a VM, it
has the user provided image as a base file, and then creates  qcow2
overlay.  If the VM is cold migrated (ie not running) to another
host, OpenStack has to make sure the same base file gets copied across
to the new host so that the overlay still works. Copying the BIOS file
and vars state across at the same time is no more difficult than what
its already doing.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Laszlo Ersek (ler...@redhat.com) wrote:
> On 06/07/18 12:54, Andrea Bolognani wrote:
> > On Thu, 2018-06-07 at 11:36 +0100, Daniel P. Berrangé wrote:
> >> On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> >>> Another problem which Laszlo mentioned is the varstore isn't portable
> >>> between UEFI implementations, or if the UEFI is compiled with
> >>> different options.  You can even imagine shipping multiple
> >>> varstores(!) which argues for a tar-like format.
> >>
> >> Could we perhaps imagine shipping the actual UEFI bios, rather
> >> than only the varstore.  The bios blob runs in guest context,
> >> so there shouldn't be able security concerns from hosting
> >> vendors with running user provided bios. Mostly its a matter
> >> of confidence that the interface between bios & qemu is stable
> >> which feels easier than assuming varstore vs different bios is
> >> portable.
> > 
> > That sounds sensible, and further reinforces the idea that we
> > need way more than a single string baked into the qcow2 file.
> > 
> 
> Sorry for arriving late (thanks Rich for the Fwd).
> 
> The contents of the non-volatile UEFI variables should be considered
> part of (permanent) guest state, such as disk contents. Therefore I'd
> argue for bundling the varstore file with the disk image(s).
> 
> In turn, the best way to ensure comaptibility between varstore and
> firmware binary is to just bundle the firmware binary as well. It's
> generally not large (x86) or if it is, it compresses extremely well
> (aarch64). For extra politeness, image providers can bundle a text file
> with their firmware build options (like a kernel config), possibly even
> a JSON document conforming to the new firmware schema (qemu commit
> 3a0adfc9bfcf), but that's not a hard requirement I guess.
> 
> If such a VM is to be migrated between hosts, I'd expect the host admin
> to take care of installing the fw binary on all eligible hosts.

There's no way they can do that if they're just importing VMs from
templates that include the image; who is going to keep track of which
BIOSs are needed where?

Dave

> Regarding compat between QEMU and firmware binary, I see three cases:
> 
> (1) Static requirements presented by the firmware for the QEMU
> configuration. (Such as -D SMM_REQUIRE.) With the domain configuration
> captured one way or another alongside the disk image anyway, this should
> not be a problem.
> 
> (2) New firmware launched on old QEMU. The firmware generally detects or
> negotiates features with QEMU, so this should be safe.
> 
> (Discounting firmware regressions, of course -- for example, search
>  for
> the string "I messed up".)
> 
> (3) Old firmware launched on new QEMU. This scenario has given us a lot
> more grief than (2), but I think for the appliance distribution use
> case, it can be folded into case (1) above -- specify the machine type
> too in the domain config, and that should be compatible with the old
> firmware.
> 
> (The handling of (3) is not uniform between upstream QEMU and various
> downstreams. For example, consider
> . This was a latent bug in
> OVMF that got exposed by a new QEMU (due to a valid QEMU change), even
> when using old machine types. The upstream solution was to fix edk2 and
> stick with QEMU as-was (although the agreement around that hadn't been
> universal). Conversely, one downstream solution was to restrict the
> otherwise valid QEMU change to new machine types
> .)
> 
> 
> All in all I agree with Daniel's proposal; it seems to be the most
> robust one.
> 
> And, I too recall that, under AMD SEV, users will be supposed to, or
> allowed to, provide their own firmware binaries.
> 
> Thanks!
> Laszlo
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Wed, Jun 06, 2018 at 07:06:27PM +0200, Max Reitz wrote:
> On 2018-06-06 17:09, Michael S. Tsirkin wrote:
> > On Wed, Jun 06, 2018 at 04:51:39PM +0200, Max Reitz wrote:
> >> On 2018-06-06 16:31, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> > 
> >>>
> >>> 
> >>>
> >>> The problem with having a separate file is that you either have to 
> >>> copy
> >>> it around with the image 
> >>
> >> Which is just an inconvenience.
> >
> > It's more than that;  if it's a separate file then the tools can't
> > rely on users supplying it, and frankly they won't and they'll still
> > just supply an image.
> 
>  At which point you throw an error and tell them to specify the config 
>  file.
> >>>
> >>> No:
> >>>a) At the moment they get away with it for images since they're all
> >>>   'pc' and the management layers do the right thing.
> >>
> >> So so far nobody has complained?  I don't really see the problem then.
> >>
> >> If deploying a disk and using all the defaults works out for users,
> >> great.  If they want more options, apparently they already know they
> >> have to provide some config.
> > 
> > QEMU's usability is terrible. There are tons of tools out there to try
> > to tame it, but of course they lack the knowledge of the VM internals
> > that QEMU has.
> 
> Er, yeah, OK.  But it was my understanding that we decided that we have
> a management layer on top of qemu to make things simple.

Who's we? I don't think the QEMU community completely gave up on people
using QEMU directly. It will need to be much more user-friendly than it
is right now. But it's possible. Fabrice built an emulator in
javascript, you go to a URL bam it runs a VM.

> Also, this is once more a case of first deciding what we want at all.

Who's we here again? Different people want different things. Enough
people seem to want to store tagged data with a disk image that it might
be worth someone's while to try to add that capability for starters to
qemu-img.

> Dave wants configuration options for the upper management layer which
> are completely opaque to qemu.  That has nothing to do whatsoever with
> the usability of qemu itself.

That's why I keep saying, let's start with implementing a mechanism,
worry about policy later if at all.

> >>>b) They'll give the wrong config file - then you'd need to add a flag
> >>>  to detect that - which means you'd need to add something to the
> >>>  qcow to match it to the config; loop back to teh start!
> >>
> >> I'm not sure how seriously I should take this argument.  Do stupid
> >> things, win stupid prizes.
> >>
> >> If that's the issue, add a UUID to qcow2 files and reference it from the
> >> config file.
> >>
> >>> We should make this EASY for users.
> >>
> >> To me, having a simple config file they can edit manually certainly
> >> seems simpler than having to use specific tools to edit it inside of the
> >> qcow2 file.
> > 
> > I think you are one of the happy users familiar with qemu intricacies
> > and/or using a tool on top that does it for you.
> 
> Yeah, virt-manager and sometimes libvirt directly.  Works nicely.  In
> any case, having to manage more than a single file was never one of my
> worries.  In fact, I never had to manage any file because both tools do
> it for me.
> 
> And again, I don't know what the usability of qemu has to do with what
> Dave is proposing.
> 
> [...]

I think what we are seeing here is many people jumping on the
bandwagon and finding more and more uses for ability to store
meta-data in the qcow2 file.

This just means we should make it flexible enough to possibly
support more uses. It does not mean we need to make it
read mail on day 1.

> >> Because I think (maybe I'm wrong, though) where to store it heavily
> >> depends on what we want to store and how we want to use it.
> > 
> > I don't really see why.
> 
> For instance, supporting full-blown appliances would mean supporting
> multiple images.  Maybe in multiple formats.  Maybe the user wants
> runtime performance and is willing to give up a bit of installation time
> for that (e.g. for unpacking an archive).
> 
> In any case, if we want to be able to configure every kind of VM, tying
> everything to qcow2 seems like a bad idea.  First defining a format and
> then deciding on whether it makes sense to be able to put it into qcow2
> for certain subcases seems much more reasonable.
> 
> And if you make the format decidedly qcow2-independent, the whole
> "putting it into qcow2 is the simplest implementation" argument becomes
> rather weak.

I don't see why. Yes I think it's a separate format that we should just
allow storing in qcow2 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Thu, Jun 07, 2018 at 12:54:33PM +0200, Andrea Bolognani wrote:
> On Thu, 2018-06-07 at 11:36 +0100, Daniel P. Berrangé wrote:
> > On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> > > Another problem which Laszlo mentioned is the varstore isn't portable
> > > between UEFI implementations, or if the UEFI is compiled with
> > > different options.  You can even imagine shipping multiple
> > > varstores(!) which argues for a tar-like format.
> > 
> > Could we perhaps imagine shipping the actual UEFI bios, rather
> > than only the varstore.  The bios blob runs in guest context,
> > so there shouldn't be able security concerns from hosting
> > vendors with running user provided bios. Mostly its a matter
> > of confidence that the interface between bios & qemu is stable
> > which feels easier than assuming varstore vs different bios is
> > portable.
> 
> That sounds sensible, and further reinforces the idea that we
> need way more than a single string baked into the qcow2 file.

I don't think anyone said we want a single string.
What was proposed is a set of key value pairs with
values being binary blobs.

> -- 
> Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Thu, Jun 07, 2018 at 11:36:20AM +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> > On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> > > Something that I haven't seen mentioned in the thread - and this
> > > looks like as good a point as any to jump in - is that for q35
> > > guests using EFI as well as aarch64 guests the "one click import"
> > > experience requires not only hints about the machine (and firmware!)
> > > type, but also a copy of the EFI variable store:
> > > 
> > >   $ virt-builder fedora-27 --arch aarch64 --notes
> > >   Fedora® 27 Server (aarch64)
> > > 
> > >   [...]
> > > 
> > >   You will need to use the associated UEFI NVRAM variables file:
> > > http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz
> > 
> > This is true, although only sometimes.  If the bootloader[*] has a
> > working fallback path then usually it is able to boot and reset the
> > UEFI varstore back to the correct values.  We have had bugs before
> > where the fallback path was not working, eg:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1353689 (yours!)
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1558793
> > 
> > Another problem which Laszlo mentioned is the varstore isn't portable
> > between UEFI implementations, or if the UEFI is compiled with
> > different options.  You can even imagine shipping multiple
> > varstores(!) which argues for a tar-like format.
> 
> Could we perhaps imagine shipping the actual UEFI bios, rather
> than only the varstore.

That's pretty unusual, UEFI is designed to abstract away the
hardware. It normally ships with the hardware.

I don't think it's a good idea to stick firmware itself in the image:
updating guest images is already a problem, at least we can easily fix
firmware bugs by dnf update on the host.

> The bios blob runs in guest context,
> so there shouldn't be able security concerns from hosting
> vendors with running user provided bios.

It seems possible that users that do supply their own firmware
will want to save it with the image. I don't think
we should do it for the standard firmware.


> Mostly its a matter
> of confidence that the interface between bios & qemu is stable
> which feels easier than assuming varstore vs different bios is
> portable. IIRC, shipping actual UEFI BIOS is something that was
> desirable for AMD SEV usage.

For SEV storing the un-encrypted binary, having QEMU read it out and write
it into guest memory isn't any better than shipping it with QEMU.

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Gerd Hoffmann]

  Hi,

> > I could be wrong, but I feel like it's significantly less likely
> > that a random QEMU binary won't like a random EFI ROM than it is
> > for a random EFI ROM to not like a random EFI NVRAM.
> 
> True, but it's not that rare to find SeaBIOS+qemu version problems;

Hmm?  Any recent examples?  Since we switched over to have qemu generate
the acpi tables instead of expecting the firmware doing it (qemu 1.5 or
1.6 IIRC) there where no hard lockstep updates.  Only soft dependencies
a'la "if you want use the new qemu feature foo you also need a seabios
supporting the new feature foo".

> so I'll assume the same happens with EFI.

We try to avoid it but sometimes it doesn't work out as we like.

cheers,
  Gerd

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Laszlo Ersek]

On 06/07/18 12:51, Andrea Bolognani wrote:
> On Thu, 2018-06-07 at 11:32 +0100, Richard W.M. Jones wrote:
>> On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
>>> Something that I haven't seen mentioned in the thread - and this
>>> looks like as good a point as any to jump in - is that for q35
>>> guests using EFI as well as aarch64 guests the "one click import"
>>> experience requires not only hints about the machine (and firmware!)
>>> type, but also a copy of the EFI variable store:
>>>
>>>   $ virt-builder fedora-27 --arch aarch64 --notes
>>>   Fedora® 27 Server (aarch64)
>>>
>>>   [...]
>>>
>>>   You will need to use the associated UEFI NVRAM variables file:
>>> http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz
>>
>> This is true, although only sometimes.  If the bootloader[*] has a
>> working fallback path then usually it is able to boot and reset the
>> UEFI varstore back to the correct values.  We have had bugs before
>> where the fallback path was not working, eg:
>>
>>   https://bugzilla.redhat.com/show_bug.cgi?id=1353689 (yours!)
>>   https://bugzilla.redhat.com/show_bug.cgi?id=1558793
>
> [...]
>> [*] I'm not sure exactly which bit of the bootloader does this,
>> whether it's UEFI itself, or the grub-efi in the guest.
>
> IIUC the UEFI spec itself reserves certain file names in the ESP
> for this fallback mechanism; it's then up to the guest operating
> system to actually install something appropriate there.
>
> In Fedora and RHEL, shim is what takes care of it (except when it
> doesn't ;), but in Debian and Ubuntu AFAIK shim is not included
> and the fallback path doesn't work at all, which makes providing
> the NVRAM file a hard requirement to boot such guests.

Quoting the UEFI-2.7 spec:

> 3.4.3 Boot Option Variables Default Boot Behavior
>
> [...] the boot options require a standard default behavior in the
> exceptional case that valid boot options are not present on a
> platform. The default behavior must be invoked any time the BootOrder
> variable does not exist or only points to nonexistent boot options, or
> if no entry in BootOrder can successfully be executed.
>
> If system firmware supports boot option recovery as described in
> Section 3.4, system firmware must include a PlatformRecovery
> variable specifying a short-form File Path Media Device Path (see
> Section 3.1.2) containing the platform default file path for removable
> media (see Table 11). [...]

(Note from Laszlo: think '\EFI\BOOT\BOOTX64.EFI' on the system disk's
EFI System Partition.)

> It is expected that this default boot will load an operating system or
> a maintenance utility.
>
> If this is an operating system setup program it is then responsible
> for setting the requisite environment variables for subsequent boots.
> [...]

More details:
.

Thanks
Laszlo

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Laszlo Ersek]

On 06/07/18 12:54, Andrea Bolognani wrote:
> On Thu, 2018-06-07 at 11:36 +0100, Daniel P. Berrangé wrote:
>> On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
>>> Another problem which Laszlo mentioned is the varstore isn't portable
>>> between UEFI implementations, or if the UEFI is compiled with
>>> different options.  You can even imagine shipping multiple
>>> varstores(!) which argues for a tar-like format.
>>
>> Could we perhaps imagine shipping the actual UEFI bios, rather
>> than only the varstore.  The bios blob runs in guest context,
>> so there shouldn't be able security concerns from hosting
>> vendors with running user provided bios. Mostly its a matter
>> of confidence that the interface between bios & qemu is stable
>> which feels easier than assuming varstore vs different bios is
>> portable.
> 
> That sounds sensible, and further reinforces the idea that we
> need way more than a single string baked into the qcow2 file.
> 

Sorry for arriving late (thanks Rich for the Fwd).

The contents of the non-volatile UEFI variables should be considered
part of (permanent) guest state, such as disk contents. Therefore I'd
argue for bundling the varstore file with the disk image(s).

In turn, the best way to ensure comaptibility between varstore and
firmware binary is to just bundle the firmware binary as well. It's
generally not large (x86) or if it is, it compresses extremely well
(aarch64). For extra politeness, image providers can bundle a text file
with their firmware build options (like a kernel config), possibly even
a JSON document conforming to the new firmware schema (qemu commit
3a0adfc9bfcf), but that's not a hard requirement I guess.

If such a VM is to be migrated between hosts, I'd expect the host admin
to take care of installing the fw binary on all eligible hosts.


Regarding compat between QEMU and firmware binary, I see three cases:

(1) Static requirements presented by the firmware for the QEMU
configuration. (Such as -D SMM_REQUIRE.) With the domain configuration
captured one way or another alongside the disk image anyway, this should
not be a problem.

(2) New firmware launched on old QEMU. The firmware generally detects or
negotiates features with QEMU, so this should be safe.

(Discounting firmware regressions, of course -- for example, search
 for
the string "I messed up".)

(3) Old firmware launched on new QEMU. This scenario has given us a lot
more grief than (2), but I think for the appliance distribution use
case, it can be folded into case (1) above -- specify the machine type
too in the domain config, and that should be compatible with the old
firmware.

(The handling of (3) is not uniform between upstream QEMU and various
downstreams. For example, consider
. This was a latent bug in
OVMF that got exposed by a new QEMU (due to a valid QEMU change), even
when using old machine types. The upstream solution was to fix edk2 and
stick with QEMU as-was (although the agreement around that hadn't been
universal). Conversely, one downstream solution was to restrict the
otherwise valid QEMU change to new machine types
.)


All in all I agree with Daniel's proposal; it seems to be the most
robust one.

And, I too recall that, under AMD SEV, users will be supposed to, or
allowed to, provide their own firmware binaries.

Thanks!
Laszlo

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Kashyap Chamarthy]

On Wed, Jun 06, 2018 at 03:36:53PM -0300, Eduardo Habkost wrote:
> On Wed, Jun 06, 2018 at 08:33:39PM +0200, Michal Suchánek wrote:
> [...]
> > Lastly we are missing a developer of a management layer committed to
> > support such appliances.
> 
> This is important.  Without developers of management tools
> willing to help specify the requirements and implement the
> feature, all the work in the lower layers would be useless.

FWIW, I'm following along from the OpenStack 'Nova' (it allows you to
provision VMs / QEMU instances) point of view.

Here is a bug (filed by Eduardo) that is tracking what needs to fixed in
Nova:

https://bugzilla.redhat.com/show_bug.cgi?id=1581414 -- OpenStack
shouldn't break if the default machine-type in QEMU is "q35"

Refer to comment#6 and comment#11 for some analysis as to where Nova
makes assumptions for machine types.  (There is one instance of it.)

Related: Elsewhere on this KM-long thread, Dan Berrangé and myself have
noted how Nova allows configuring machine types today -- either via
setting a config attribute (in /etc/nova/nova.conf) per Compute node
(where QEMU processes are launched) or via setting a metadata property
per template disk image, which is used to launch Nova instances (VMs).

-- 
/kashyap

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Andrea Bolognani (abolo...@redhat.com) wrote:
> On Thu, 2018-06-07 at 15:45 +0100, Dr. David Alan Gilbert wrote:
> > * Andrea Bolognani (abolo...@redhat.com) wrote:
> > > On Thu, 2018-06-07 at 14:49 +0100, Dr. David Alan Gilbert wrote:
> > > > Including the nvram and efi makes me nervous; but I can see why together
> > > > they might work.  However, there's no guarantee that EFI has been tested
> > > > with the QEMU it's used on and ... that could be trouble.
> > > 
> > > If the QEMU binary doesn't support EFI, then a guest expecting
> > > EFI won't be able to start regardless of where the EFI ROM came
> > > from.
> > 
> > No, I mean if the QEMU doesn't support that *particular* EFI.
> 
> I could be wrong, but I feel like it's significantly less likely
> that a random QEMU binary won't like a random EFI ROM than it is
> for a random EFI ROM to not like a random EFI NVRAM.

True, but it's not that rare to find SeaBIOS+qemu version problems;
so I'll assume the same happens with EFI.

> > > > Also, if we're going to start including the EFI rom then that would have
> > > > to be migrated with the VM so that after a restart on a different host
> > > > it's still using the right ROM that's compatible with it's varfile.
> > > 
> > > That's a problem that needs to be addressed anyway, because even
> > > as it is now you could easily find yourself trying and failing
> > > to migrate a guest between two hosts that have different and
> > > incompatible EFI ROMs installed.
> > 
> > True; although I was working on the basis that vendors who cared about
> > migration compatibility would couple the EFI versions with machine types
> > to ensure that the variable data didn't become incompatible.
> 
> As far as I know, nobody is actually doing this at the moment.

I'm assuming we'll have to.

Dave


> -- 
> Andrea Bolognani / Red Hat / Virtualization
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Andrea Bolognani]

On Thu, 2018-06-07 at 15:45 +0100, Dr. David Alan Gilbert wrote:
> * Andrea Bolognani (abolo...@redhat.com) wrote:
> > On Thu, 2018-06-07 at 14:49 +0100, Dr. David Alan Gilbert wrote:
> > > Including the nvram and efi makes me nervous; but I can see why together
> > > they might work.  However, there's no guarantee that EFI has been tested
> > > with the QEMU it's used on and ... that could be trouble.
> > 
> > If the QEMU binary doesn't support EFI, then a guest expecting
> > EFI won't be able to start regardless of where the EFI ROM came
> > from.
> 
> No, I mean if the QEMU doesn't support that *particular* EFI.

I could be wrong, but I feel like it's significantly less likely
that a random QEMU binary won't like a random EFI ROM than it is
for a random EFI ROM to not like a random EFI NVRAM.

> > > Also, if we're going to start including the EFI rom then that would have
> > > to be migrated with the VM so that after a restart on a different host
> > > it's still using the right ROM that's compatible with it's varfile.
> > 
> > That's a problem that needs to be addressed anyway, because even
> > as it is now you could easily find yourself trying and failing
> > to migrate a guest between two hosts that have different and
> > incompatible EFI ROMs installed.
> 
> True; although I was working on the basis that vendors who cared about
> migration compatibility would couple the EFI versions with machine types
> to ensure that the variable data didn't become incompatible.

As far as I know, nobody is actually doing this at the moment.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Andrea Bolognani (abolo...@redhat.com) wrote:
> On Thu, 2018-06-07 at 14:49 +0100, Dr. David Alan Gilbert wrote:
> > Including the nvram and efi makes me nervous; but I can see why together
> > they might work.  However, there's no guarantee that EFI has been tested
> > with the QEMU it's used on and ... that could be trouble.
> 
> If the QEMU binary doesn't support EFI, then a guest expecting
> EFI won't be able to start regardless of where the EFI ROM came
> from.

No, I mean if the QEMU doesn't support that *particular* EFI.

> > Also, if we're going to start including the EFI rom then that would have
> > to be migrated with the VM so that after a restart on a different host
> > it's still using the right ROM that's compatible with it's varfile.
> 
> That's a problem that needs to be addressed anyway, because even
> as it is now you could easily find yourself trying and failing
> to migrate a guest between two hosts that have different and
> incompatible EFI ROMs installed.

True; although I was working on the basis that vendors who cared about
migration compatibility would couple the EFI versions with machine types
to ensure that the variable data didn't become incompatible.

Dave

> -- 
> Andrea Bolognani / Red Hat / Virtualization
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Andrea Bolognani]

On Thu, 2018-06-07 at 14:49 +0100, Dr. David Alan Gilbert wrote:
> Including the nvram and efi makes me nervous; but I can see why together
> they might work.  However, there's no guarantee that EFI has been tested
> with the QEMU it's used on and ... that could be trouble.

If the QEMU binary doesn't support EFI, then a guest expecting
EFI won't be able to start regardless of where the EFI ROM came
from.

> Also, if we're going to start including the EFI rom then that would have
> to be migrated with the VM so that after a restart on a different host
> it's still using the right ROM that's compatible with it's varfile.

That's a problem that needs to be addressed anyway, because even
as it is now you could easily find yourself trying and failing
to migrate a guest between two hosts that have different and
incompatible EFI ROMs installed.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Daniel P. Berrangé (berra...@redhat.com) wrote:
> On Thu, Jun 07, 2018 at 01:17:24PM +0200, Andrea Bolognani wrote:
> > On Thu, 2018-06-07 at 11:22 +0100, Daniel P. Berrangé wrote:
> > > On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> > > > While hints might be considered a reasonable fit for qcow2, I think
> > > > it's pretty hard to argue for embedding the NVRAM file in there,
> > > > which to me signals quite clearly that an archive containing the
> > > > disk image(s) *and* the configuration hints *and* other ancillary
> > > > files such as the NVRAM is the only way to build a solution that's
> > > > not dead on arrival.
> > > 
> > > On a similar theme, I can imagine users wanting to provide a TPM
> > > data blob too, and for AMD SEV we'd need to be able to provide a
> > > DH key, and session blob too IIUC.
> > 
> > I'm not familiar with the technologies you're talking about, but
> > all that sounds like something very security sensitive and not
> > something eg. the Fedora project would want to bake into their
> > cloud images.
> > 
> > Perhaps we should keep in mind that this kind of archive format
> > lends itself quite naturally to both generic ready-made images and
> > custom, fully configured images: in the former case it would only
> > contain the few things mentione above, while in the latter it might
> > also have security sensitive data that's specific to the deployment
> > it's going to be used against.
> 
> I don't thonk there's any such distinction. A downstream user
> may build generic ready-made images, or fully configured app
> specific images. Both can contain the security sensitive data.

Including the nvram and efi makes me nervous; but I can see why together
they might work.  However, there's no guarantee that EFI has been tested
with the QEMU it's used on and ... that could be trouble.
Also, if we're going to start including the EFI rom then that would have
to be migrated with the VM so that after a restart on a different host
it's still using the right ROM that's compatible with it's varfile.

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Thu, Jun 07, 2018 at 01:17:24PM +0200, Andrea Bolognani wrote:
> On Thu, 2018-06-07 at 11:22 +0100, Daniel P. Berrangé wrote:
> > On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> > > While hints might be considered a reasonable fit for qcow2, I think
> > > it's pretty hard to argue for embedding the NVRAM file in there,
> > > which to me signals quite clearly that an archive containing the
> > > disk image(s) *and* the configuration hints *and* other ancillary
> > > files such as the NVRAM is the only way to build a solution that's
> > > not dead on arrival.
> > 
> > On a similar theme, I can imagine users wanting to provide a TPM
> > data blob too, and for AMD SEV we'd need to be able to provide a
> > DH key, and session blob too IIUC.
> 
> I'm not familiar with the technologies you're talking about, but
> all that sounds like something very security sensitive and not
> something eg. the Fedora project would want to bake into their
> cloud images.
> 
> Perhaps we should keep in mind that this kind of archive format
> lends itself quite naturally to both generic ready-made images and
> custom, fully configured images: in the former case it would only
> contain the few things mentione above, while in the latter it might
> also have security sensitive data that's specific to the deployment
> it's going to be used against.

I don't thonk there's any such distinction. A downstream user
may build generic ready-made images, or fully configured app
specific images. Both can contain the security sensitive data.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Andrea Bolognani]

On Wed, 2018-06-06 at 17:32 +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 06, 2018 at 10:36:20AM -0500, Eric Blake wrote:
> > But for the new config to be useful, you have to modify at least one tool in
> > the path.  At which point, it is just as easy to say: "libvirt is now smart
> > enough to read the config file out of a .qcow2 to know that it should prefer
> > a q35 machine" as it is to say "libvirt is now smart enough to treat a .tar
> > file containing .qcow2 and a config file that states that it should prefer a
> > q35 machine", and either approach requires just a single file for the user
> > to download.
> 
> Just to be clear, libvirt isn't going to do either of those things.
> 
> Whether there is metadata stuffed inside qcow2, or in a metdata file
> inside a tar file, libvirt is not going to look inside either of them.
> The XML is the only place libvirt deals with the hardware config.
> 
> Extracting machine type is always going to be a job for the layer above
> such as OpenStack/OVirt/Virt-manager/etc. They will then decide whether
> or not they want to honour that info, and if so, put it into the XML
> they give to libvirt.
> 
> As mentioned elsewhere, IMHO, it is more friendly to those tools
> to use pre-existing formats, eg TAR and XML/JSON, for which
> their respective programming langauges already have APIs/parsers.

Something that I haven't seen mentioned in the thread - and this
looks like as good a point as any to jump in - is that for q35
guests using EFI as well as aarch64 guests the "one click import"
experience requires not only hints about the machine (and firmware!)
type, but also a copy of the EFI variable store:

  $ virt-builder fedora-27 --arch aarch64 --notes
  Fedora® 27 Server (aarch64)

  [...]

  You will need to use the associated UEFI NVRAM variables file:
http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz

While hints might be considered a reasonable fit for qcow2, I think
it's pretty hard to argue for embedding the NVRAM file in there,
which to me signals quite clearly that an archive containing the
disk image(s) *and* the configuration hints *and* other ancillary
files such as the NVRAM is the only way to build a solution that's
not dead on arrival.

It's pretty easy then to imagine using something like

  $ virt-builder \
fedora-27 \
--arch aarch64 \
--format qva \
--output f27-aarch64.qva

or download the equivalent from some website, followed by

  $ virt-install \
--name f27-aarch64 \
--import \
--input f27-aarch.qva

or the equivalent pointy-clicky import step and having things
Just Work™, provided sufficient hints are included in the archive;
the user, or the management application, would of course be able
to override such hints at import time.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Andrea Bolognani]

On Thu, 2018-06-07 at 11:36 +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> > Another problem which Laszlo mentioned is the varstore isn't portable
> > between UEFI implementations, or if the UEFI is compiled with
> > different options.  You can even imagine shipping multiple
> > varstores(!) which argues for a tar-like format.
> 
> Could we perhaps imagine shipping the actual UEFI bios, rather
> than only the varstore.  The bios blob runs in guest context,
> so there shouldn't be able security concerns from hosting
> vendors with running user provided bios. Mostly its a matter
> of confidence that the interface between bios & qemu is stable
> which feels easier than assuming varstore vs different bios is
> portable.

That sounds sensible, and further reinforces the idea that we
need way more than a single string baked into the qcow2 file.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Andrea Bolognani]

On Thu, 2018-06-07 at 11:22 +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> > While hints might be considered a reasonable fit for qcow2, I think
> > it's pretty hard to argue for embedding the NVRAM file in there,
> > which to me signals quite clearly that an archive containing the
> > disk image(s) *and* the configuration hints *and* other ancillary
> > files such as the NVRAM is the only way to build a solution that's
> > not dead on arrival.
> 
> On a similar theme, I can imagine users wanting to provide a TPM
> data blob too, and for AMD SEV we'd need to be able to provide a
> DH key, and session blob too IIUC.

I'm not familiar with the technologies you're talking about, but
all that sounds like something very security sensitive and not
something eg. the Fedora project would want to bake into their
cloud images.

Perhaps we should keep in mind that this kind of archive format
lends itself quite naturally to both generic ready-made images and
custom, fully configured images: in the former case it would only
contain the few things mentione above, while in the latter it might
also have security sensitive data that's specific to the deployment
it's going to be used against.

For non-vanilla images, it might be interesting to include the
libvirt XML in its entirety, which would make it trivial to keep
around a full-contained copy of a guest that can be imported back
into libvirt with a single click; on the other hand, the management
layer might want to override that, and for generic images we
probably want to avoid the security implications of people
importing potentially untrusted configurations into the system
libvirt instance and stick to just a few hints instead.

So there's at least two partially overlapping use cases right
there. Fun :)

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Thu, Jun 07, 2018 at 11:32:18AM +0100, Richard W.M. Jones wrote:
> On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> > Something that I haven't seen mentioned in the thread - and this
> > looks like as good a point as any to jump in - is that for q35
> > guests using EFI as well as aarch64 guests the "one click import"
> > experience requires not only hints about the machine (and firmware!)
> > type, but also a copy of the EFI variable store:
> > 
> >   $ virt-builder fedora-27 --arch aarch64 --notes
> >   Fedora® 27 Server (aarch64)
> > 
> >   [...]
> > 
> >   You will need to use the associated UEFI NVRAM variables file:
> > http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz
> 
> This is true, although only sometimes.  If the bootloader[*] has a
> working fallback path then usually it is able to boot and reset the
> UEFI varstore back to the correct values.  We have had bugs before
> where the fallback path was not working, eg:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1353689 (yours!)
>   https://bugzilla.redhat.com/show_bug.cgi?id=1558793
> 
> Another problem which Laszlo mentioned is the varstore isn't portable
> between UEFI implementations, or if the UEFI is compiled with
> different options.  You can even imagine shipping multiple
> varstores(!) which argues for a tar-like format.

Could we perhaps imagine shipping the actual UEFI bios, rather
than only the varstore.  The bios blob runs in guest context,
so there shouldn't be able security concerns from hosting
vendors with running user provided bios. Mostly its a matter
of confidence that the interface between bios & qemu is stable
which feels easier than assuming varstore vs different bios is
portable. IIRC, shipping actual UEFI BIOS is something that was
desirable for AMD SEV usage.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Richard W.M. Jones (rjo...@redhat.com) wrote:
> On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> > Something that I haven't seen mentioned in the thread - and this
> > looks like as good a point as any to jump in - is that for q35
> > guests using EFI as well as aarch64 guests the "one click import"
> > experience requires not only hints about the machine (and firmware!)
> > type, but also a copy of the EFI variable store:
> > 
> >   $ virt-builder fedora-27 --arch aarch64 --notes
> >   Fedora® 27 Server (aarch64)
> > 
> >   [...]
> > 
> >   You will need to use the associated UEFI NVRAM variables file:
> > http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz
> 
> This is true, although only sometimes.  If the bootloader[*] has a
> working fallback path then usually it is able to boot and reset the
> UEFI varstore back to the correct values.  We have had bugs before
> where the fallback path was not working, eg:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1353689 (yours!)
>   https://bugzilla.redhat.com/show_bug.cgi?id=1558793
> 
> Another problem which Laszlo mentioned is the varstore isn't portable
> between UEFI implementations, or if the UEFI is compiled with
> different options.  You can even imagine shipping multiple
> varstores(!) which argues for a tar-like format.

Given that level of incompatibility with var stores (which I've seen
myself) I don't see how you can distribute them with images.

Dave

> > While hints might be considered a reasonable fit for qcow2, I think
> > it's pretty hard to argue for embedding the NVRAM file in there,
> > which to me signals quite clearly that an archive containing the
> > disk image(s) *and* the configuration hints *and* other ancillary
> > files such as the NVRAM is the only way to build a solution that's
> > not dead on arrival.
> 
> The tar argument is quite strong.  Just not the wretched OVA/OVF :-)
> 
> > It's pretty easy then to imagine using something like
> > 
> >   $ virt-builder \
> > fedora-27 \
> > --arch aarch64 \
> > --format qva \
> > --output f27-aarch64.qva
> > 
> > or download the equivalent from some website, followed by
> > 
> >   $ virt-install \
> > --name f27-aarch64 \
> > --import \
> > --input f27-aarch.qva
> > 
> > or the equivalent pointy-clicky import step and having things
> > Just Work™, provided sufficient hints are included in the archive;
> > the user, or the management application, would of course be able
> > to override such hints at import time.
> 
> RFEs for virt-builder & virt-install one day :-)
> 
> Rich.
> 
> [*] I'm not sure exactly which bit of the bootloader does this,
> whether it's UEFI itself, or the grub-efi in the guest.
> 
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Richard W.M. Jones]

On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> Something that I haven't seen mentioned in the thread - and this
> looks like as good a point as any to jump in - is that for q35
> guests using EFI as well as aarch64 guests the "one click import"
> experience requires not only hints about the machine (and firmware!)
> type, but also a copy of the EFI variable store:
> 
>   $ virt-builder fedora-27 --arch aarch64 --notes
>   Fedora® 27 Server (aarch64)
> 
>   [...]
> 
>   You will need to use the associated UEFI NVRAM variables file:
> http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz

This is true, although only sometimes.  If the bootloader[*] has a
working fallback path then usually it is able to boot and reset the
UEFI varstore back to the correct values.  We have had bugs before
where the fallback path was not working, eg:

  https://bugzilla.redhat.com/show_bug.cgi?id=1353689 (yours!)
  https://bugzilla.redhat.com/show_bug.cgi?id=1558793

Another problem which Laszlo mentioned is the varstore isn't portable
between UEFI implementations, or if the UEFI is compiled with
different options.  You can even imagine shipping multiple
varstores(!) which argues for a tar-like format.

> While hints might be considered a reasonable fit for qcow2, I think
> it's pretty hard to argue for embedding the NVRAM file in there,
> which to me signals quite clearly that an archive containing the
> disk image(s) *and* the configuration hints *and* other ancillary
> files such as the NVRAM is the only way to build a solution that's
> not dead on arrival.

The tar argument is quite strong.  Just not the wretched OVA/OVF :-)

> It's pretty easy then to imagine using something like
> 
>   $ virt-builder \
> fedora-27 \
> --arch aarch64 \
> --format qva \
> --output f27-aarch64.qva
> 
> or download the equivalent from some website, followed by
> 
>   $ virt-install \
> --name f27-aarch64 \
> --import \
> --input f27-aarch.qva
> 
> or the equivalent pointy-clicky import step and having things
> Just Work™, provided sufficient hints are included in the archive;
> the user, or the management application, would of course be able
> to override such hints at import time.

RFEs for virt-builder & virt-install one day :-)

Rich.

[*] I'm not sure exactly which bit of the bootloader does this,
whether it's UEFI itself, or the grub-efi in the guest.


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Thu, Jun 07, 2018 at 12:02:29PM +0200, Andrea Bolognani wrote:
> On Wed, 2018-06-06 at 17:32 +0100, Daniel P. Berrangé wrote:
> > On Wed, Jun 06, 2018 at 10:36:20AM -0500, Eric Blake wrote:
> > > But for the new config to be useful, you have to modify at least one tool 
> > > in
> > > the path.  At which point, it is just as easy to say: "libvirt is now 
> > > smart
> > > enough to read the config file out of a .qcow2 to know that it should 
> > > prefer
> > > a q35 machine" as it is to say "libvirt is now smart enough to treat a 
> > > .tar
> > > file containing .qcow2 and a config file that states that it should 
> > > prefer a
> > > q35 machine", and either approach requires just a single file for the user
> > > to download.
> > 
> > Just to be clear, libvirt isn't going to do either of those things.
> > 
> > Whether there is metadata stuffed inside qcow2, or in a metdata file
> > inside a tar file, libvirt is not going to look inside either of them.
> > The XML is the only place libvirt deals with the hardware config.
> > 
> > Extracting machine type is always going to be a job for the layer above
> > such as OpenStack/OVirt/Virt-manager/etc. They will then decide whether
> > or not they want to honour that info, and if so, put it into the XML
> > they give to libvirt.
> > 
> > As mentioned elsewhere, IMHO, it is more friendly to those tools
> > to use pre-existing formats, eg TAR and XML/JSON, for which
> > their respective programming langauges already have APIs/parsers.
> 
> Something that I haven't seen mentioned in the thread - and this
> looks like as good a point as any to jump in - is that for q35
> guests using EFI as well as aarch64 guests the "one click import"
> experience requires not only hints about the machine (and firmware!)
> type, but also a copy of the EFI variable store:
> 
>   $ virt-builder fedora-27 --arch aarch64 --notes
>   Fedora® 27 Server (aarch64)
> 
>   [...]
> 
>   You will need to use the associated UEFI NVRAM variables file:
> http://libguestfs.org/download/builder/fedora-27-aarch64-nvram.xz
> 
> While hints might be considered a reasonable fit for qcow2, I think
> it's pretty hard to argue for embedding the NVRAM file in there,
> which to me signals quite clearly that an archive containing the
> disk image(s) *and* the configuration hints *and* other ancillary
> files such as the NVRAM is the only way to build a solution that's
> not dead on arrival.

On a similar theme, I can imagine users wanting to provide a TPM
data blob too, and for AMD SEV we'd need to be able to provide a
DH key, and session blob too IIUC.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Gerd Hoffmann]

  Hi,

> our actual qcow2v3 improvements, so no one ended up switching to that. So,
> to some extent, various high-level consumers still have the notion that
> 'raw' files are better/safer/faster than 'qcow2' files because of an
> anecdote from years ago, even if we have since fixed the speed parity and
> added locking to eliminate careless data loss.

When I use raw images the reasons are different ones.  Most of the time
it is that I want use the image with something which isn't qemu and thus
doesn't understand qcow2.  Booting a image as container for example
(systemd-nspawn -i $image.raw).

cheers,
  Gerd

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/06/2018 09:57 AM, Eric Blake wrote:

On 06/06/2018 09:43 AM, Michael S. Tsirkin wrote:

On Wed, Jun 06, 2018 at 01:02:53PM +0200, Max Reitz wrote:

Yeah, but why make qcow2 that format?  That's what I completely fail to
understand.


Because why not? It's cheap to add it there and is much easier
than teaching people about a new container format.


tar is not a new container format, but it is a new format to various 
toolchains - that said, if we popularize tar as the format for including 
a config file alongside a qcow2 image, it's not that hard to fix the 
stack to start passing that file around as the new preferred file type.


On a completely different front, 'qcow2' as a file format comes with 
some psychological baggage.  If someone was using it 8 years ago, before 
we did coroutine optimizations, it was noticeably slower than raw, and 
relatively easier to get into a corrupted image condition that resulted 
in data loss.  Just one VM lost, and it leaves a sour taste in your 
mouth, where you are unwilling to trust that file format (even though 
the file format was not necessarily the cause of the corruption). 
Marketing-wise, we failed with our improvements ('qcow2v3' is so much 
more of a mouthful than 'qcow3'), and it took years to flip the defaults 
from v2 as the default to v3 as the default (moreso in downstream 
distros than upstream), in part because we couldn't convince people of 
the improvements they would be gaining by moving to v3.  Historically, 
there's also 'qed' which was promised as a way to fix some of the poor 
performance of qcow2, but which ended up not being any better than our 
actual qcow2v3 improvements, so no one ended up switching to that. So, 
to some extent, various high-level consumers still have the notion that 
'raw' files are better/safer/faster than 'qcow2' files because of an 
anecdote from years ago, even if we have since fixed the speed parity 
and added locking to eliminate careless data loss.


If we DO add a new tar-file block driver to qemu, that could serve as a 
marketing opportunity to convince people that the new format has all of 
the features that you can't get from just a raw file, and does not 
suffer from the slowness or data corruption they were worried about in 
qcow2.  Thus, even if our new format is just a thin wrapper around a 
config file plus the existing qcow2v3 we already know and love, the mere 
fact that it is a new format may get people to move away from raw images 
in situations where just the name 'qcow2' is unable to do so, at which 
point we can help them take advantage of the features made possible by 
qcow2.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eduardo Habkost]

On Wed, Jun 06, 2018 at 08:33:39PM +0200, Michal Suchánek wrote:
[...]
> Lastly we are missing a developer of a management layer committed to
> support such appliances.

This is important.  Without developers of management tools
willing to help specify the requirements and implement the
feature, all the work in the lower layers would be useless.

-- 
Eduardo

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 20:02:54 +0200
Max Reitz  wrote:

> On 2018-06-06 17:25, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 16:55:08 +0200
> > Max Reitz  wrote:
> >   
> >> On 2018-06-06 16:41, Dr. David Alan Gilbert wrote:  
> >>> * Max Reitz (mre...@redhat.com) wrote:

> >> Users using a whole VM stack plus management, but then handling two
> >> files instead of one is too much to ask?  
> > 
> > What you don't seem to realize is there are cases when there is an
> > 'administrator' who has set up the VM stack plus management and 'joe
> > user' who wants to run some random VM on that stack.
> > 
> > And if you download an appliance compatible with the stack it should
> > just work. For a long time the 'appliance' for qemu based
> > virtualization was a simple qcow2 file which was sized sufficiently
> > for the VM to run but shrunk for transport. And although it is
> > technically wrong it JustWorked(tm).  
> 
> Hm, yes.  As I replied to Dave, I understand, but I would think this
> then requires a real appliance solution.  I think you do want such a
> solution, but Dave doesn't.

Yes, Dave wants a poor man's half-assed appliance and insists on it not
being an appliance. Duh.

In the pc world to maintain the status quo with minimum changes you
only need to know if the image uses EFI or legacy BIOS and you can
maintain the illusion that the TimeProvenSolution(tm) JustWorks(tm).

Sneaking that single piece of information somewhere seems to be the
goal here.

> 
> My problem is that I cannot accept Dave's arguments on why to include
> this blob in qcow2 if someone else already plans on making that blob
> the basis for qcow2 appliances.
> 
> And I still do not think that qcow2 is the right format for VM
> appliances.  To convince me, we'd first need a consensus on what the
> appliances are for (Michael seems to want them for qemu directly,

Let's put this straight: qemu as is cannot run appliances. It is not
designed for that and it would be a big feature to create enough
management inside qemu (or around qemu but part of qemu distribution) to
change that.

As it stands qemu always takes the configuration from the outside -
either from the user directly or from a separate management layer.

> apparently you want them for something higher up the stack) and thus
> what they are supposed to be capable of exactly.

Using qcow2 would be kind of cool but it has its limitations and
drawbacks as well.

You could use qcow2 as transport format and convert the VM to use
raw disks or whatever if you need the performance. And you could run
the VM directly from qcow2 without additional processing which is its
advantage.

It would fail miserably with tools not aware of the extra metadata,
however.

Lastly we are missing a developer of a management layer committed to
support such appliances.

> 
> Like, one thing that is important to discuss is this (but please not
> in this thread...): If we agree on making an appliance format (qcow2
> or not), is it for running VMs off or do we just want it for VM
> export/import?  The former might mean we need qcow2, because there is
> no good way to offer good performance with multiple disks otherwise
> (but this would constrain us e.g. in the disk image format -- no raw
> images for you, then).  But the latter can work just fine with a
> normal archival format as long as building/decomposing it is possible
> without copying.

Indeed, using an archive should be good enough for the 1-click download
solution. It will take time to extract but it will typically take even
more time to download or publish. So optimizing the format for speed of
export/import might be misplaced.

Thanks

Michal


pgpJ0HJhOa8Vo.pgp
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 18:10, Eduardo Habkost wrote:
> On Wed, Jun 06, 2018 at 04:17:14PM +0200, Max Reitz wrote:
>> On 2018-06-06 15:45, Michal Suchánek wrote:

[...]

>>> I understand that for some use cases simplifying the distribution of
>>> VMs as much as possible is quite important.
>>
>> I don't because still nobody has explained it to me.
>>
>> The only explanation I got so far was "People are lazy and we have
>> defaults for everything, so we don't throw an error if people forget to
>> pass a configuration file."
> 
> People don't pass a configuration file today because there's no
> standard for such a configuration file.  qcow2 is already used
> today as an appliance file format because there's no better
> option.  People download disk images from appliance and OS
> providers, import them into a cloud system, and it works out of
> the box because (luckily) "pc" is enough for most of them.

OK.

> We can specify a true appliance file format, and ask people to
> use it.  But then providers of single-disk appliances and OSes
> will need to publish two appliance images: qcow2 disk image for
> old systems that don't support the new format, and one in the new
> appliance format, for systems that support it.

True.  This is a valid argument for making a compatible change to qcow2.
 But: If these options are actually important, the qcow2 file is not
really compatible either.  You might end up with a blank screen again.

Depending on how we'd design a new format, it might contain a config
file and a qcow2 file which may be easily extractable.  If so, users of
legacy software could easily extract the qcow2 file and might even be
tempted to look at the config file when it doesn't work, and maybe that
could even help them.

Also, I think that it is not too unreasonable to ask providers to
provide two formats.

But that does not make me disregard your argument.  It is still valid,
especially with your convenience note below.

(Providers could choose whether it is best for them to include the
description in the qcow2 file, or whether to offer an archive that
contains e.g. a raw file.)

>> Which to me still just makes it an inconvenience.
> 
> Well, there are small inconveniences and there are big
> inconveniences that together make a system unnecessarily hard to
> use.  I'd say this one falls somewhere in the middle.

Hm, OK.

> [...]
>> I'm noticing a pattern here, and that is that everybody has a different
>> opinion on what we actually want in the end, and it's just by chance
>> that we find ourselves in two camps ("put it in qcow2" vs. "put it
>> somewhere else").
>>
>> Maybe we should first discuss what we actually want before we can
>> discuss where to put it.
> 
> I'm inclined to agree.  Once we figure out a good VM description
> format, we can justify a proposal to allow embedding the VM
> description in qcow2 for convenience.

OK.

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 17:25, Michal Suchánek wrote:
> On Wed, 6 Jun 2018 16:55:08 +0200
> Max Reitz  wrote:
> 
>> On 2018-06-06 16:41, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:  
>>
>> [...]
>>
 So why is it so dangerous to connect a disk you just downloaded to
 e.g. the wrong machine type?  I assumed it just wouldn't work and
 you'd try again, until you realized that maybe you should read the
 download description and do as it says ("download this config
 file, pass it").  
>>>
>>> That's bad!  Stuff should just-work;  
>>
>> That's how it always should be.  Life's tough, though.
>>
>>>  it currently just works,  
>>
>> Due to sheer blind luck, I'd say.
> 
> It's TimeProvenSolution(tm).
> 
>>
>>>   things
>>> should get better and easier for our users.  
>>
>> Users using a whole VM stack plus management, but then handling two
>> files instead of one is too much to ask?
> 
> What you don't seem to realize is there are cases when there is an
> 'administrator' who has set up the VM stack plus management and 'joe
> user' who wants to run some random VM on that stack.
> 
> And if you download an appliance compatible with the stack it should
> just work. For a long time the 'appliance' for qemu based
> virtualization was a simple qcow2 file which was sized sufficiently for
> the VM to run but shrunk for transport. And although it is technically
> wrong it JustWorked(tm).

Hm, yes.  As I replied to Dave, I understand, but I would think this
then requires a real appliance solution.  I think you do want such a
solution, but Dave doesn't.

My problem is that I cannot accept Dave's arguments on why to include
this blob in qcow2 if someone else already plans on making that blob the
basis for qcow2 appliances.

And I still do not think that qcow2 is the right format for VM
appliances.  To convince me, we'd first need a consensus on what the
appliances are for (Michael seems to want them for qemu directly,
apparently you want them for something higher up the stack) and thus
what they are supposed to be capable of exactly.

Like, one thing that is important to discuss is this (but please not in
this thread...): If we agree on making an appliance format (qcow2 or
not), is it for running VMs off or do we just want it for VM
export/import?  The former might mean we need qcow2, because there is no
good way to offer good performance with multiple disks otherwise (but
this would constrain us e.g. in the disk image format -- no raw images
for you, then).  But the latter can work just fine with a normal
archival format as long as building/decomposing it is possible without
copying.

(I would think that you can move blocks from one file to another, so
with proper alignment you should be able to build/decompose an archive
from/into its members without copying.)

>>>  And anyway, not
>>> working for EFI for exmaple can be just a blank screen.  Seriously
>>> - keep it easy for the user!  
>>
>> Thinking this through makes you end up with appliances.
> 
> And those can in general have more than one disk.

Indeed.  And thus knowing what we want is important so we can make a
good decision on where to store it instead of just focusing on where it
would be apparently simplest.

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 17:05, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 16:31, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> 
>>>
>>> 
>>>
>>> The problem with having a separate file is that you either have to copy
>>> it around with the image 
>>
>> Which is just an inconvenience.
>
> It's more than that;  if it's a separate file then the tools can't
> rely on users supplying it, and frankly they won't and they'll still
> just supply an image.

 At which point you throw an error and tell them to specify the config file.
>>>
>>> No:
>>>a) At the moment they get away with it for images since they're all
>>>   'pc' and the management layers do the right thing.
>>
>> So so far nobody has complained?  I don't really see the problem then.
>>
>> If deploying a disk and using all the defaults works out for users,
>> great.  If they want more options, apparently they already know they
>> have to provide some config.
> 
> This problem all came about because of q35.  We can't change defaults to
> use q35 because importing existing images might break so we need to
> start flagging stuff as q35 - all we're trying to do is make stuff no
> more broken than today.

Well, I really don't want to go into this much further.  I understand
that you see it as a bug prevention.  But as I've said, we've always had
this issue, e.g. with minimum RAM required.  It's nothing new.

But the most important thing is that other people seem to really
disagree on your "I don't want it to get an appliance" stance.

>>>b) They'll give the wrong config file - then you'd need to add a flag
>>>  to detect that - which means you'd need to add something to the
>>>  qcow to match it to the config; loop back to teh start!
>>
>> I'm not sure how seriously I should take this argument.  Do stupid
>> things, win stupid prizes.
>>
>> If that's the issue, add a UUID to qcow2 files and reference it from the
>> config file.
> 
> Is a UUID a small string :-)

Well, it certainly isn't unheard of for disk image formats, so I'm much
more inclined to include it.

Though UUIDs tend to be nasty to handle, but as long as they don't
change...  Oh well.

>>> We should make this EASY for users.
>>
>> To me, having a simple config file they can edit manually certainly
>> seems simpler than having to use specific tools to edit it inside of the
>> qcow2 file.
> 
> The users never touch the tools; they click and import the VM image.

Not really, no.  I know these are the end users, but others (the
providers) probably do want to edit the config file.

[...]

>>> Storing a simple bit
>>> of data with the image avoids that.
>>
>> It is not a simple bit of data, as evidenced by the discussion about
>> storing binary blobs and MIME types going on.
>
> All of the things they've suggested can be done inside that one blob;
> even inside the json (or any other structure in that blob).

 Right, from qcow2's perspective it's a blob of data.  But you can put a
 whole filesystem into a blob of data, and I get the impression that this
 is what some are trying to do.

 Once we store larger amounts of binary data in that blob (which is what
 I'm fearing from comments on MIME types and PNG images), people will
 realize that always having to re-store the whole blob if you modify
 something in the middle is inefficient and that it needs to be
 optimized.  I don't think you want to do that, but we haven't
 implemented any of this yet and people are already asking for such
 binary data inside of the blob.

 I suspect it'll only get worse over time.
 I think the most difficult thing about this discussion is that there are
 different targets.

 You just want to store a bit of information.  OK, good, but then I'd say
 we could even just prepend that to the image file in a small header.
>>>
>>>
>>> I think you're over-reading what people are asking for.
>>> I think the PNG suggestion is again the 'label on the front' for a logo.
>>
>> Which is OK if you store like everything, but very much over the top for
>> your suggestion.  Again, different people want different things and I
>> feel like that is the real discussion we should be having right now and
>> not necessarily where to store it.
>>
>> Because I think (maybe I'm wrong, though) where to store it heavily
>> depends on what we want to store and how we want to use it.
>>
>>> I've not seen anything that's not for either:
>>>   a) The user to know what the image is
>>
>> I thought the use case was 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 17:09, Michael S. Tsirkin wrote:
> On Wed, Jun 06, 2018 at 04:51:39PM +0200, Max Reitz wrote:
>> On 2018-06-06 16:31, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> 
>>>
>>> 
>>>
>>> The problem with having a separate file is that you either have to copy
>>> it around with the image 
>>
>> Which is just an inconvenience.
>
> It's more than that;  if it's a separate file then the tools can't
> rely on users supplying it, and frankly they won't and they'll still
> just supply an image.

 At which point you throw an error and tell them to specify the config file.
>>>
>>> No:
>>>a) At the moment they get away with it for images since they're all
>>>   'pc' and the management layers do the right thing.
>>
>> So so far nobody has complained?  I don't really see the problem then.
>>
>> If deploying a disk and using all the defaults works out for users,
>> great.  If they want more options, apparently they already know they
>> have to provide some config.
> 
> QEMU's usability is terrible. There are tons of tools out there to try
> to tame it, but of course they lack the knowledge of the VM internals
> that QEMU has.

Er, yeah, OK.  But it was my understanding that we decided that we have
a management layer on top of qemu to make things simple.

Also, this is once more a case of first deciding what we want at all.
Dave wants configuration options for the upper management layer which
are completely opaque to qemu.  That has nothing to do whatsoever with
the usability of qemu itself.

>>>b) They'll give the wrong config file - then you'd need to add a flag
>>>  to detect that - which means you'd need to add something to the
>>>  qcow to match it to the config; loop back to teh start!
>>
>> I'm not sure how seriously I should take this argument.  Do stupid
>> things, win stupid prizes.
>>
>> If that's the issue, add a UUID to qcow2 files and reference it from the
>> config file.
>>
>>> We should make this EASY for users.
>>
>> To me, having a simple config file they can edit manually certainly
>> seems simpler than having to use specific tools to edit it inside of the
>> qcow2 file.
> 
> I think you are one of the happy users familiar with qemu intricacies
> and/or using a tool on top that does it for you.

Yeah, virt-manager and sometimes libvirt directly.  Works nicely.  In
any case, having to manage more than a single file was never one of my
worries.  In fact, I never had to manage any file because both tools do
it for me.

And again, I don't know what the usability of qemu has to do with what
Dave is proposing.

[...]
>> Because I think (maybe I'm wrong, though) where to store it heavily
>> depends on what we want to store and how we want to use it.
> 
> I don't really see why.

For instance, supporting full-blown appliances would mean supporting
multiple images.  Maybe in multiple formats.  Maybe the user wants
runtime performance and is willing to give up a bit of installation time
for that (e.g. for unpacking an archive).

In any case, if we want to be able to configure every kind of VM, tying
everything to qcow2 seems like a bad idea.  First defining a format and
then deciding on whether it makes sense to be able to put it into qcow2
for certain subcases seems much more reasonable.

And if you make the format decidedly qcow2-independent, the whole
"putting it into qcow2 is the simplest implementation" argument becomes
rather weak.

>>> I've not seen anything that's not for either:
>>>   a) The user to know what the image is
>>
>> I thought the use case was they just downloaded it.
>>
>> Otherwise, they should manage their filenames reasonably, come on.
>> Seriously, adding a cute picture because users are too stupid to manage
>> their VMs is *not* qcow2's problem.
> 
> QEMU is hard to use right and it is QEMU's problem. Users aren't stupid
> but neither do they have the time to learn internals of the tools they
> use.

Technically, it's the users' problem.  It may be qemu's fault, though.

I will not say it is qemu's fault, because I was always told we have a
management layer to make things simple again.  "qemu worries about
execution, management layer worries about policy" is what I was told.

Also, I have no idea what you are talking about.  I gave a very specific
example.  How is adding a picture to a VM disk image going to help
anyone?  If that's the issue people are facing, I would argue they
probably have a multitude of different issues with using qemu, because I
fully agree with you on that point -- using qemu for complex cases is
hard.  Well, no, it's simple, really, but then you probably won't get
the best out of it.  (As can be 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/06/2018 11:11 AM, Michal Suchánek wrote:


The reason why storing the config inside the qcow2 file is you have one
self-contained file that can be updated by qemu itself.

So you take an image file, point your management console to it, boot
it, change something on it, shut it down, and publish the image as the
new revision of the VM.

With the tar file the management console needs to chew the tar file,
save a copy of the config file and the disk image somewhere, and when
you update the image you have to re-export the tar file.


That's not necessarily true.  We're arguing that as long as the qcow2 
image is last in the tar file, it should still be relatively easy to 
write a qemu block driver that manages tar files for in-place qcow2 
editing.  GNU tar can also do --append or --update to modify the config 
file; although we may have to be careful if that starts to make the 
qcow2 image not the last member in the tar file.  On the other hand, if 
updating the config file is a common enough operation, I would not be 
surprised if we end up teaching the qemu tar driver how to resize images 
even when the qcow2 portion of the tar file is not last, or adding 
qemu-img commands that wrap typical tar file manipulations.




This is a lot of lengthy copying. The upside is that the console can
check that the machine you are exporting is not running an you are not
prone to publishing images in inconsistent state.


Read-only qcow2 within a tar does NOT require lengthy copying.  Updating 
qcow2 within tar might, but then again, we are arguing that with a sane 
layout, copying the ENTIRE qcow2 portion is NOT always going to be 
necessary.  For example, modern Linux has things like 
fallocate(FALLOC_FL_INSERT_RANGE) (on file systems that are capable) for 
enlarging the head of a tar file enough to stick in a larger config file 
at the front, with a LOT less data movement effort when compared to 
moving the qcow2 tail of that same tar file.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Daniel P. Berrangé (berra...@redhat.com) wrote:
> On Wed, Jun 06, 2018 at 10:36:20AM -0500, Eric Blake wrote:
> > On 06/06/2018 10:05 AM, Dr. David Alan Gilbert wrote:
> > 
> > > > If that's the issue, add a UUID to qcow2 files and reference it from the
> > > > config file.
> > > 
> > > Is a UUID a small string :-)
> > 
> > Even better, it's something that you could stick directly in the qcow2
> > header (and which therefore cannot grow to a larger size) - it would be a
> > well-constrained scoped addition.  Maybe the analogy to actual hardware
> > would be that the config file is like a sticky note, and a UUID embedded in
> > the qcow2 file would be the disk serial number; if you are paranoid that the
> > sticky note could be too easily pulled off one disk and put on another, then
> > the sticky note can include the serial number.
> > 
> > > 
> > > > > We should make this EASY for users.
> > > > 
> > > > To me, having a simple config file they can edit manually certainly
> > > > seems simpler than having to use specific tools to edit it inside of the
> > > > qcow2 file.
> > > 
> > > The users never touch the tools; they click and import the VM image.
> > 
> > And if we make it easy to import a tar file as the VM image, then that's
> > still the case.
> > 
> > > > Sure that it isn't a soft constraint?  If most tools can stay unchanged
> > > > but some very specific ones have to be changed, that seems reasonable 
> > > > to me.
> > > 
> > > The hard constraint is the normal path stays unchanged; we can change
> > > the tools to make use of the extra data, but not change what's out
> > > there.
> > 
> > But for the new config to be useful, you have to modify at least one tool in
> > the path.  At which point, it is just as easy to say: "libvirt is now smart
> > enough to read the config file out of a .qcow2 to know that it should prefer
> > a q35 machine" as it is to say "libvirt is now smart enough to treat a .tar
> > file containing .qcow2 and a config file that states that it should prefer a
> > q35 machine", and either approach requires just a single file for the user
> > to download.
> 
> Just to be clear, libvirt isn't going to do either of those things.
> 
> Whether there is metadata stuffed inside qcow2, or in a metdata file
> inside a tar file, libvirt is not going to look inside either of them.
> The XML is the only place libvirt deals with the hardware config.
> 
> Extracting machine type is always going to be a job for the layer above
> such as OpenStack/OVirt/Virt-manager/etc. They will then decide whether
> or not they want to honour that info, and if so, put it into the XML
> they give to libvirt.
> 
> As mentioned elsewhere, IMHO, it is more friendly to those tools
> to use pre-existing formats, eg TAR and XML/JSON, for which
> their respective programming langauges already have APIs/parsers.

Libvirt could provide a wrapper around whichever format to extract the
data and provide it to the upper layer.  It could also validate against
it to see if the constraints were met as a service for an upper layer.

Dave

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Wed, Jun 06, 2018 at 10:36:20AM -0500, Eric Blake wrote:
> On 06/06/2018 10:05 AM, Dr. David Alan Gilbert wrote:
> 
> > > If that's the issue, add a UUID to qcow2 files and reference it from the
> > > config file.
> > 
> > Is a UUID a small string :-)
> 
> Even better, it's something that you could stick directly in the qcow2
> header (and which therefore cannot grow to a larger size) - it would be a
> well-constrained scoped addition.  Maybe the analogy to actual hardware
> would be that the config file is like a sticky note, and a UUID embedded in
> the qcow2 file would be the disk serial number; if you are paranoid that the
> sticky note could be too easily pulled off one disk and put on another, then
> the sticky note can include the serial number.
> 
> > 
> > > > We should make this EASY for users.
> > > 
> > > To me, having a simple config file they can edit manually certainly
> > > seems simpler than having to use specific tools to edit it inside of the
> > > qcow2 file.
> > 
> > The users never touch the tools; they click and import the VM image.
> 
> And if we make it easy to import a tar file as the VM image, then that's
> still the case.
> 
> > > Sure that it isn't a soft constraint?  If most tools can stay unchanged
> > > but some very specific ones have to be changed, that seems reasonable to 
> > > me.
> > 
> > The hard constraint is the normal path stays unchanged; we can change
> > the tools to make use of the extra data, but not change what's out
> > there.
> 
> But for the new config to be useful, you have to modify at least one tool in
> the path.  At which point, it is just as easy to say: "libvirt is now smart
> enough to read the config file out of a .qcow2 to know that it should prefer
> a q35 machine" as it is to say "libvirt is now smart enough to treat a .tar
> file containing .qcow2 and a config file that states that it should prefer a
> q35 machine", and either approach requires just a single file for the user
> to download.

Just to be clear, libvirt isn't going to do either of those things.

Whether there is metadata stuffed inside qcow2, or in a metdata file
inside a tar file, libvirt is not going to look inside either of them.
The XML is the only place libvirt deals with the hardware config.

Extracting machine type is always going to be a job for the layer above
such as OpenStack/OVirt/Virt-manager/etc. They will then decide whether
or not they want to honour that info, and if so, put it into the XML
they give to libvirt.

As mentioned elsewhere, IMHO, it is more friendly to those tools
to use pre-existing formats, eg TAR and XML/JSON, for which
their respective programming langauges already have APIs/parsers.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 10:36:20 -0500
Eric Blake  wrote:

> On 06/06/2018 10:05 AM, Dr. David Alan Gilbert wrote:
> 
> >   
> >>> We should make this EASY for users.  
> >>
> >> To me, having a simple config file they can edit manually certainly
> >> seems simpler than having to use specific tools to edit it inside
> >> of the qcow2 file.  
> > 
> > The users never touch the tools; they click and import the VM
> > image.  
> 
> And if we make it easy to import a tar file as the VM image, then
> that's still the case.
> 
> >> Sure that it isn't a soft constraint?  If most tools can stay
> >> unchanged but some very specific ones have to be changed, that
> >> seems reasonable to me.  
> > 
> > The hard constraint is the normal path stays unchanged; we can
> > change the tools to make use of the extra data, but not change
> > what's out there.  
> 
> But for the new config to be useful, you have to modify at least one 
> tool in the path.  At which point, it is just as easy to say:
> "libvirt is now smart enough to read the config file out of a .qcow2
> to know that it should prefer a q35 machine" as it is to say "libvirt
> is now smart enough to treat a .tar file containing .qcow2 and a
> config file that states that it should prefer a q35 machine", and
> either approach requires just a single file for the user to
> download.  Or, if you are worried about what happens in a too-old
> system that doesn't understand the config file, you have either: "the
> tool didn't know that the .qcow2 contained a config snippet, and
> tried to open the qcow2 file with pc even though q35 would have been
> better" or "the tool didn't know that the .tar file contains a config
> snippet and .qcow2 image, and could not run the image".  Either way,
> the image with the new config data doesn't run unless the user
> realizes they need to upgrade their system - but trying (and failing)
> to run is actually less friendly than flat out claiming unrecognized
> format and failing early.  So going with the tar format actually
> encourages users to upgrade, unlike going with enhancing qcow2.
> 

Yes, that's a good argument.

The reason why storing the config inside the qcow2 file is you have one
self-contained file that can be updated by qemu itself.

So you take an image file, point your management console to it, boot
it, change something on it, shut it down, and publish the image as the
new revision of the VM.

With the tar file the management console needs to chew the tar file,
save a copy of the config file and the disk image somewhere, and when
you update the image you have to re-export the tar file.

This is a lot of lengthy copying. The upside is that the console can
check that the machine you are exporting is not running an you are not
prone to publishing images in inconsistent state.

Thanks

Michal

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eduardo Habkost]

On Wed, Jun 06, 2018 at 04:17:14PM +0200, Max Reitz wrote:
> On 2018-06-06 15:45, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 15:14:03 +0200
> > Max Reitz  wrote:
> > 
> >> On 2018-06-06 14:13, Michal Suchánek wrote:
> >>> On Wed, 6 Jun 2018 13:52:35 +0200
> >>> Max Reitz  wrote:
> >>>   
>  On 2018-06-06 13:43, Michal Suchánek wrote:  
> > On Wed, 6 Jun 2018 13:32:47 +0200
> > Max Reitz  wrote:
> > 
> >> On 2018-06-06 13:19, Michal Suchánek wrote:
> >>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>> Max Reitz  wrote:  
> >>
> >> [...]
> >>
>  What I'm trying to get at is that qcow2 was not designed to be
>  a container format for arbitrary files.  If you want to make it
>  such, I'm sure there are existing formats that work
>  better.  
> >>>
> >>> Such as?  
> >>
> >> ext2?
> >
> > So you want an ext2 driver in qemu instead of expanding qcow2 to
> > work not only for a single disk but also for an appliance?
> 
>  Yes, because ext2 was designed to be a proper filesystem.  I'm not
>  an FS designer.  Well, not a good one anyway.  So I don't trust
>  myself on extending qcow2 to be a good FS -- and why would I, when
>  there are already numerous FS around.  
> >>>
> >>> Do you expect that performance of qemu using qcow2 driver over ext2
> >>> driver will be better than using qcow driver directly with some part
> >>> semi-permanently occupied by a configuration blob? My bet is not.  
> >>
> >> If you want to store multiple disk images in a single file?  I would
> >> think so, yes.  With qcow2, I would assume it leads to
> >> fragmentation.  
> > 
> > How is that different from single disk divided into two partitions
> > internally (without any knowledge on the qcow2 level)?
> 
> From how it's going to be fragmented, there is no difference.  If you
> have multiple partitions and write to them concurrently, thus allocating
> new areas, you get bad fragmentation.
> 
> >> I would hope that proper filesystems can mitigate this.
> > 
> > Not really. Not without much complexity and repeated maintenance.
> 
> Yes, a proper filesystem.  Which we'd get for free with multiple files.
> 
> >>> The ext* drivers are designed to work with kernel VM infrastructure
> >>> which must be tuned for different usage scenarios and you would
> >>> have to duplicate that tuning in qemu to get competitive
> >>> performance. Also you get qcow2 and ext2 metadata which must be
> >>> allocated, managed, etc. You get more storage and performance
> >>> overhead for no good reason.  
> >>
> >> Yes, there is a good reason.  You can add arbitrary configuration
> >> options without having to worry about me.
> > 
> > But I will not be able to use the images in qemu so it will be useless.
> 
> Neither can you with the current proposal because that is about adding
> management layer configuration options which are opaque to qemu.
> 
> > Well, there is FUSE and that is certainly blazing fast and ubiquitous,
> > I am sure.
> 
> If you want to use pre-existing drivers, you'd probably use a loop device.
> 
> Otherwise, you'd use the block layer for accessing the disk.
> 
> If you want blazingly fast, you probably won't use qcow2 anyway.  Or,
> funnily enough, you'd want to probably split the qcow2 file into a
> metadata and a data file, so you get even more files.  (But that is a
> proposal for the future.)
> 
> >> Seriously, though, a real FS would allow you to be more expressive and
> >> really do what you want without having to work around the quirks that
> >> adding a not-real-FS in the most simple way possible to qcow2 would
> >> bring with it.
> >>
> >> Because this is part of my fear, that we now add a very simple blob
> >> for just a sprinkle of data.  But over time it gets more and more
> >> complex because we want to store more and more data to make things
> >> ever more convenient[1], we notice that we need more features, the
> >> format gets more complex, and in the end we have an FS that is just
> >> worse than a real FS.
> >>
> >> [1] And note that if I'm convinced to store VM configuration data in
> >> qemu, I will agree that we can store any data in there and it would be
> >> nice if any VM could be provisioned and used that way.
> >>
> >>> On the other hand, qcow is designed for storing VM disk data and
> >>> hopefully was tuned to do that decently over the years. The primary
> >>> use case remains storing VM disk data. Adding a configuration blob
> >>> does not change that.  
> >>
> >> True.  So the argument is that qcow2 may be worse for storing
> >> arbitrary data, but we don't have performance requirements for that;
> >> but we do have performance requirements for disk data and adding
> >> another format below qcow2 will not make it better.
> >>
> >> I do think it is possible to not make things worse with a format under
> >> qcow2, but that may require additional complexity, that you think is
> >> 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/06/2018 10:05 AM, Dr. David Alan Gilbert wrote:


If that's the issue, add a UUID to qcow2 files and reference it from the
config file.


Is a UUID a small string :-)


Even better, it's something that you could stick directly in the qcow2 
header (and which therefore cannot grow to a larger size) - it would be 
a well-constrained scoped addition.  Maybe the analogy to actual 
hardware would be that the config file is like a sticky note, and a UUID 
embedded in the qcow2 file would be the disk serial number; if you are 
paranoid that the sticky note could be too easily pulled off one disk 
and put on another, then the sticky note can include the serial number.





We should make this EASY for users.


To me, having a simple config file they can edit manually certainly
seems simpler than having to use specific tools to edit it inside of the
qcow2 file.


The users never touch the tools; they click and import the VM image.


And if we make it easy to import a tar file as the VM image, then that's 
still the case.



Sure that it isn't a soft constraint?  If most tools can stay unchanged
but some very specific ones have to be changed, that seems reasonable to me.


The hard constraint is the normal path stays unchanged; we can change
the tools to make use of the extra data, but not change what's out
there.


But for the new config to be useful, you have to modify at least one 
tool in the path.  At which point, it is just as easy to say: "libvirt 
is now smart enough to read the config file out of a .qcow2 to know that 
it should prefer a q35 machine" as it is to say "libvirt is now smart 
enough to treat a .tar file containing .qcow2 and a config file that 
states that it should prefer a q35 machine", and either approach 
requires just a single file for the user to download.  Or, if you are 
worried about what happens in a too-old system that doesn't understand 
the config file, you have either: "the tool didn't know that the .qcow2 
contained a config snippet, and tried to open the qcow2 file with pc 
even though q35 would have been better" or "the tool didn't know that 
the .tar file contains a config snippet and .qcow2 image, and could not 
run the image".  Either way, the image with the new config data doesn't 
run unless the user realizes they need to upgrade their system - but 
trying (and failing) to run is actually less friendly than flat out 
claiming unrecognized format and failing early.  So going with the tar 
format actually encourages users to upgrade, unlike going with enhancing 
qcow2.





Because it's got to be EASY for the customer; seriously - stop punishing
the user for not noticing something.
We've got to help the users, if not we get people asking why their VM
system has given them a black screen, or why the image they just
downloaded didn't work - it's basic user friendliness.

It's not obvious why it's failed; if it was as simple as a nice box
popping up telling them they'd booted it wouldn't be too bad; but some
of them will waste 3 hours trying to figure out wth happened.

*seriously* think about our users.


I am - to me, telling a user that "here is your image, it is a new file 
extension, therefore you need a new qemu before you can even try to use 
it - but once you have that, everything just works" is nicer than "here 
is the same extension you've always used, but it might not work for you 
and it might be 3 hours of your time to figure out why it didn't work".




The implementation is trivial is what I meant, just like the
implementation would be rather simple for qcow2 to store a binary blob
and completely ignore it.


But then you'd have people shipping .newformat files as well as qcow2
files and you'd have to persuade people to start doing that, and they'd
ship both or none or


Why would you intentionally ship a .qcow2 that only works on q35 if 
.newformat is already known to be nicer to users?  Just ship .newformat! 
 People shipping just .qcow2 would be limited to those used to 'pc' as 
the default, but even they could be encouraged to tweak their process to 
make it easier for end consumers to take advantage of .newformat.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 16:55:08 +0200
Max Reitz  wrote:

> On 2018-06-06 16:41, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:  
> 
> [...]
> 
> >> So why is it so dangerous to connect a disk you just downloaded to
> >> e.g. the wrong machine type?  I assumed it just wouldn't work and
> >> you'd try again, until you realized that maybe you should read the
> >> download description and do as it says ("download this config
> >> file, pass it").  
> > 
> > That's bad!  Stuff should just-work;  
> 
> That's how it always should be.  Life's tough, though.
> 
> >  it currently just works,  
> 
> Due to sheer blind luck, I'd say.

It's TimeProvenSolution(tm).

> 
> >   things
> > should get better and easier for our users.  
> 
> Users using a whole VM stack plus management, but then handling two
> files instead of one is too much to ask?

What you don't seem to realize is there are cases when there is an
'administrator' who has set up the VM stack plus management and 'joe
user' who wants to run some random VM on that stack.

And if you download an appliance compatible with the stack it should
just work. For a long time the 'appliance' for qemu based
virtualization was a simple qcow2 file which was sized sufficiently for
the VM to run but shrunk for transport. And although it is technically
wrong it JustWorked(tm).

> 
> >  And anyway, not
> > working for EFI for exmaple can be just a blank screen.  Seriously
> > - keep it easy for the user!  
> 
> Thinking this through makes you end up with appliances.

And those can in general have more than one disk.

Thanks

Michal


pgpjgqBLAuwNd.pgp
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Wed, Jun 06, 2018 at 04:51:39PM +0200, Max Reitz wrote:
> On 2018-06-06 16:31, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> >>> 
> > 
> > 
> > 
> > The problem with having a separate file is that you either have to copy
> > it around with the image 
> 
>  Which is just an inconvenience.
> >>>
> >>> It's more than that;  if it's a separate file then the tools can't
> >>> rely on users supplying it, and frankly they won't and they'll still
> >>> just supply an image.
> >>
> >> At which point you throw an error and tell them to specify the config file.
> > 
> > No:
> >a) At the moment they get away with it for images since they're all
> >   'pc' and the management layers do the right thing.
> 
> So so far nobody has complained?  I don't really see the problem then.
> 
> If deploying a disk and using all the defaults works out for users,
> great.  If they want more options, apparently they already know they
> have to provide some config.

QEMU's usability is terrible. There are tons of tools out there to try
to tame it, but of course they lack the knowledge of the VM internals
that QEMU has.


> >b) They'll give the wrong config file - then you'd need to add a flag
> >  to detect that - which means you'd need to add something to the
> >  qcow to match it to the config; loop back to teh start!
> 
> I'm not sure how seriously I should take this argument.  Do stupid
> things, win stupid prizes.
> 
> If that's the issue, add a UUID to qcow2 files and reference it from the
> config file.
> 
> > We should make this EASY for users.
> 
> To me, having a simple config file they can edit manually certainly
> seems simpler than having to use specific tools to edit it inside of the
> qcow2 file.

I think you are one of the happy users familiar with qemu intricacies
and/or using a tool on top that does it for you.

>  I understand it is an inconvenience and it would be nice to change it,
>  but please understand that I do not want qcow2 to become a filesystem
>  just to relieve an inconvenience.
> >>>
> >>> I very much don't want it to be a filesystem; my reason for writing
> >>> down my spec the way I did was to make it clear that the only
> >>> thing I want of qcow2 is a single blob, no more; I don't want naming
> >>> of the blob or anything else.
> >>>
>  (Note: I understand that you may not want qcow2 to become a filesystem,
>  but I do get the impression from others.)
> >>>
> >>> My aim was to specify it to fulfill the requirements that everyone
> >>> else had asked for, but still only having one unmodifiable blob in qcow.
> >>>
> >   or have an archive. If you have an archive
> > you have to have an unpacking step which then copies, potentially a lot
> > of data taking some reasonable amount of time.
> 
>  I'm sure this can be optimized, but yes, I get that.
> 
>  (If you use e.g. tar and store the image data starting on an FS cluster
>  boundary (64 kB should be more than sufficient), I assume there is a way
>  to extract that data into a new file without copying anything.)
> >>>
> >>> But then we have to modify all the current things that know how to
> >>> handle a qcow2.
> >>
> >> Not in this case because it'd still be a flat qcow2 file in a simple tar
> >> archive.
> >>
> >> But you're right if we had a more complex format (like chunks stored in
> >> a tar file).
> > 
> > My only problem with using the tar like that is that all tools
> > everywhere would need to be updated to be able to parse them.
> > (Note if adding a blob to qcow2 like I'm asking for would break existing
> > qcow2 users then I don't want it either).
> 
> OK, so I suppose this goes back to "let's decide what we really want to
> configure first, and how we can limit our scope in an effective and
> long-lasting way".
> 
> > Storing a simple bit
> > of data with the image avoids that.
> 
>  It is not a simple bit of data, as evidenced by the discussion about
>  storing binary blobs and MIME types going on.
> >>>
> >>> All of the things they've suggested can be done inside that one blob;
> >>> even inside the json (or any other structure in that blob).
> >>
> >> Right, from qcow2's perspective it's a blob of data.  But you can put a
> >> whole filesystem into a blob of data, and I get the impression that this
> >> is what some are trying to do.
> >>
> >> Once we store larger amounts of binary data in that blob (which is what
> >> I'm fearing from comments on MIME types and PNG images), people will
> >> realize that always having to re-store the whole blob if you modify

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 16:31, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> >>> 
> > 
> > 
> > 
> > The problem with having a separate file is that you either have to copy
> > it around with the image 
> 
>  Which is just an inconvenience.
> >>>
> >>> It's more than that;  if it's a separate file then the tools can't
> >>> rely on users supplying it, and frankly they won't and they'll still
> >>> just supply an image.
> >>
> >> At which point you throw an error and tell them to specify the config file.
> > 
> > No:
> >a) At the moment they get away with it for images since they're all
> >   'pc' and the management layers do the right thing.
> 
> So so far nobody has complained?  I don't really see the problem then.
> 
> If deploying a disk and using all the defaults works out for users,
> great.  If they want more options, apparently they already know they
> have to provide some config.

This problem all came about because of q35.  We can't change defaults to
use q35 because importing existing images might break so we need to
start flagging stuff as q35 - all we're trying to do is make stuff no
more broken than today.

> >b) They'll give the wrong config file - then you'd need to add a flag
> >  to detect that - which means you'd need to add something to the
> >  qcow to match it to the config; loop back to teh start!
> 
> I'm not sure how seriously I should take this argument.  Do stupid
> things, win stupid prizes.
> 
> If that's the issue, add a UUID to qcow2 files and reference it from the
> config file.

Is a UUID a small string :-)

> > We should make this EASY for users.
> 
> To me, having a simple config file they can edit manually certainly
> seems simpler than having to use specific tools to edit it inside of the
> qcow2 file.

The users never touch the tools; they click and import the VM image.

>  I understand it is an inconvenience and it would be nice to change it,
>  but please understand that I do not want qcow2 to become a filesystem
>  just to relieve an inconvenience.
> >>>
> >>> I very much don't want it to be a filesystem; my reason for writing
> >>> down my spec the way I did was to make it clear that the only
> >>> thing I want of qcow2 is a single blob, no more; I don't want naming
> >>> of the blob or anything else.
> >>>
>  (Note: I understand that you may not want qcow2 to become a filesystem,
>  but I do get the impression from others.)
> >>>
> >>> My aim was to specify it to fulfill the requirements that everyone
> >>> else had asked for, but still only having one unmodifiable blob in qcow.
> >>>
> >   or have an archive. If you have an archive
> > you have to have an unpacking step which then copies, potentially a lot
> > of data taking some reasonable amount of time.
> 
>  I'm sure this can be optimized, but yes, I get that.
> 
>  (If you use e.g. tar and store the image data starting on an FS cluster
>  boundary (64 kB should be more than sufficient), I assume there is a way
>  to extract that data into a new file without copying anything.)
> >>>
> >>> But then we have to modify all the current things that know how to
> >>> handle a qcow2.
> >>
> >> Not in this case because it'd still be a flat qcow2 file in a simple tar
> >> archive.
> >>
> >> But you're right if we had a more complex format (like chunks stored in
> >> a tar file).
> > 
> > My only problem with using the tar like that is that all tools
> > everywhere would need to be updated to be able to parse them.
> > (Note if adding a blob to qcow2 like I'm asking for would break existing
> > qcow2 users then I don't want it either).
> 
> OK, so I suppose this goes back to "let's decide what we really want to
> configure first, and how we can limit our scope in an effective and
> long-lasting way".
> 
> > Storing a simple bit
> > of data with the image avoids that.
> 
>  It is not a simple bit of data, as evidenced by the discussion about
>  storing binary blobs and MIME types going on.
> >>>
> >>> All of the things they've suggested can be done inside that one blob;
> >>> even inside the json (or any other structure in that blob).
> >>
> >> Right, from qcow2's perspective it's a blob of data.  But you can put a
> >> whole filesystem into a blob of data, and I get the impression that this
> >> is what some are trying to do.
> >>
> >> Once we store larger amounts of binary data in that blob (which is what
> >> I'm fearing from comments on MIME types and PNG images), people will
> >> realize that always having to 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:46, Michael S. Tsirkin wrote:
> On Wed, Jun 06, 2018 at 01:44:02PM +0200, Max Reitz wrote:
>> Because it's a hack, right.  Storing binary data in a qcow2 file,
>> completely ignoring it in qemu (and being completely unusable to any
>> potential other users of the qcow2 format[1]) and only interpreting it
>> somewhere up the stack is a hack.
> 
> It's just a first step and it ensures compatibility with old QEMU
> versions. But down the road I think we will start warning
> user if the machine type does not match, and possibly even
> get the type from there if user didn't supply it.

If it's a first step we should have an idea on what the following steps
should be.

I don't want people to convince me that adding a blob to qcow2 is a good
idea because "we are only going to store two or three values!" and then
that blob grows out of control because "well, now we have it anyway".

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:43, Michael S. Tsirkin wrote:
> On Wed, Jun 06, 2018 at 01:02:53PM +0200, Max Reitz wrote:
>> Yeah, but why make qcow2 that format?  That's what I completely fail to
>> understand.
> 
> Because why not? It's cheap to add it there and is much easier
> than teaching people about a new container format.

"new" container format?  qcow2 is not a container format, so that would
be teaching people something new anyway.

> Eric Blake put it very well I think.  There are several things that
> several people would like to see addressed:
> 
> (1) A sensible list of guest visible aspects of the VM
>   preserving which across VM restarts we deem critical enough to support
>   starting guests.
>   At this point this includes at least architecture and machine type.

If you use a whole management layer, that is trivial anyway (and that
seems to be Dave's assumption), because that management layer can store
its configuration somewhere.  Dave's issue was about downloading foreign
images, not about restarts.

> (2) A compact file format for serializing list (1)

OK.

> (3) Ability to store file (2) in a qcow2 image

I see that people are asking this, yes, although I don't quite get the
point.

> You are asking why store (2) in qcow2 image specifically. The answer is
> it's just one place where we can store it. The answer is we don't need
> to involve qemu-block at all for storing it in other places.
> 
> But for many people it will be handy to have it in the same file, and
> qcow2 is popular enough that many people will be well served if it's
> there.

Note that I'm also asking why it needs to be a single file.
Furthermore, I'm asking what the final scope is intended to be.  I don't
believe qcow2 to be the correct format for a whole appliance, but I do
believe that assuming that people want to provide just a qcow2 file
without anything else means already accepting it *is* an appliance.  But
Dave seems to reject that idea.

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/06/2018 09:43 AM, Michael S. Tsirkin wrote:

On Wed, Jun 06, 2018 at 01:02:53PM +0200, Max Reitz wrote:

Yeah, but why make qcow2 that format?  That's what I completely fail to
understand.


Because why not? It's cheap to add it there and is much easier
than teaching people about a new container format.


tar is not a new container format, but it is a new format to various 
toolchains - that said, if we popularize tar as the format for including 
a config file alongside a qcow2 image, it's not that hard to fix the 
stack to start passing that file around as the new preferred file type.




Eric Blake put it very well I think.  There are several things that
several people would like to see addressed:

(1) A sensible list of guest visible aspects of the VM
   preserving which across VM restarts we deem critical enough to support
   starting guests.
   At this point this includes at least architecture and machine type.


This part is true no matter where we store it. So as I see it, the 
question is now whether to store it in qcow2, or whether to store qcow2 
+ config in a tar file.




(2) A compact file format for serializing list (1)


tar files already serve this purpose (even if we don't like the current 
OVA specification for the config side, it at least proves that tar files 
are usable in this manner)




(3) Ability to store file (2) in a qcow2 image


As Dan pointed out, if we can FIRST popularize storing a config file + 
qcow2 in a tar file, THEN we can (for convenience) let qcow2 directly 
store the same config file, if it helps things (although if popularizing 
tar files works, we wouldn't need it in qcow2).


Maybe the step we want to take now is to add a new block driver to qemu 
that supports tar files including resize.  That is, where we currently do:


qcow2 -> posix file

we would instead popularize:

qcow2 -> tar driver -> posix file

where the tar driver takes care of finding the right subset within the 
overall tar file, AND makes it easy to resize (by updating tar metadata 
any time qcow2 wants to resize larger, and assuming qcow2 is the last 
member of the tar archive)


Perhaps we can even automate it so that the tar driver is automatically 
inserted even when not explicitly specified (after all, you can already 
start qemu by giving just the file name holding a qcow2 image, and qemu 
can figure out to use the qcow2 driver) - although we DO have to be 
careful of avoiding CVEs (probing image formats MUST NOT allow a guest 
to convert a raw image into something that qemu would treat as a tar, 
any more than it would misinterpret raw as qcow2)





You are asking why store (2) in qcow2 image specifically. The answer is
it's just one place where we can store it. The answer is we don't need
to involve qemu-block at all for storing it in other places.


Then why start with qcow2? Let's start with tar files, and enhance 
qemu-block to make tar files containing qcow2 easier to use in qemu (and 
NOT with making qcow2 larger just to bypass tar files).




But for many people it will be handy to have it in the same file, and
qcow2 is popular enough that many people will be well served if it's
there.



--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:41, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:

[...]

>> So why is it so dangerous to connect a disk you just downloaded to e.g.
>> the wrong machine type?  I assumed it just wouldn't work and you'd try
>> again, until you realized that maybe you should read the download
>> description and do as it says ("download this config file, pass it").
> 
> That's bad!  Stuff should just-work;

That's how it always should be.  Life's tough, though.

>  it currently just works,

Due to sheer blind luck, I'd say.

>   things
> should get better and easier for our users.

Users using a whole VM stack plus management, but then handling two
files instead of one is too much to ask?

>  And anyway, not working for
> EFI for exmaple can be just a blank screen.  Seriously - keep it easy
> for the user!

Thinking this through makes you end up with appliances.

> And with 'pc' type VMs being all that's around it does just-work.
> 

[...]

>> [1] Yes, I concede that there are probably no other users of qcow2.  But
>> please forgive me for assuming that qcow2 was in a sense designed to be
>> a rather general image format that not only qemu could use.
>
> What makes it QEMU specific?  It's basically just the same key/value
> setup as OVA, except putting them inside the qcow2.

 Well, not necessarily qemu-specific, but
 ${management_software}-specific, which comes down to the same.  Or,
 well, I'd think that, but hold on.

> We could use the same keys/value definitions as OVA in the blob,
> although their definitions aren't very portable either.

 So you're proposing that we only add options that seem portable for any VM?
>>>
>>> Hmm.  We should probably split them, so there should be general options
>>> (e.g. minimum-ram) but also hypervisor specifics
>>> (qemu.machine-class=q35); but that doesn't mean you can't add keys
>>> for multiple hypervisors into the one blob.  I mean
>>> something like:
>>> minimum-ram = 1G
>>> qemu.machine-class = q35
>>> anothervm.machine-class = 
>>
>> Well, and that's my issue.  Once you have application-specific info, you
>> can go wild.  And I would go wild, without a reasonable and strict
>> requirement that the information we want to store has to fulfill.
>>
>> For the record, I would've liked it if you'd said "only portable
>> options".  But I would have replied that I would fear we'd still end up
>> with someone saying "I'd like to store X and Y, let's just put them into
>> the specification, then they are portable [even if only this stack
>> supports them]" and we wouldn't really have won anything.
> 
> I couldn't second guess every other hypervisor on the planet to know
> whether specifying a machine class would work for them.

If they support the config format...

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:55, Michael S. Tsirkin wrote:
> On Wed, Jun 06, 2018 at 10:42:33AM -0300, Eduardo Habkost wrote:
>>> If we want a grand vision where a single file stores the whole VM, why
>>> not invest the work and make it right from the start?
>>
>> We don't want a grand vision where a single file stores the whole
>> VM.  This is exactly what I would like to avoid, by not inventing
>> a whole different appliance file format.
> 
> Besides, trying to get a grand vision from the start is a sure
> way to never have the design leave the drawing board.

Yes, but with our own non-qcow2 format we could easily start with
something simple and start over later.

> What we are asking for at this point is a way to stick a named blob in
> an image that people can use with qemu without jumping through hoops.
> 
> It seems like a generic enough addition that it seems highly likely
> to be useful down the road and harmless enough that maintaining
> it won't become a burden.

Yes.  Its genericity is a big part of what's bothering me.

> Can we agree on that as a first step, so we can build that foundation
> and move on to actually building ways to use it?

No, because I don't want to agree on putting anything inside qcow2
before I know its scope.

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:31, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
>>> 
> 
> 
> 
> The problem with having a separate file is that you either have to copy
> it around with the image 

 Which is just an inconvenience.
>>>
>>> It's more than that;  if it's a separate file then the tools can't
>>> rely on users supplying it, and frankly they won't and they'll still
>>> just supply an image.
>>
>> At which point you throw an error and tell them to specify the config file.
> 
> No:
>a) At the moment they get away with it for images since they're all
>   'pc' and the management layers do the right thing.

So so far nobody has complained?  I don't really see the problem then.

If deploying a disk and using all the defaults works out for users,
great.  If they want more options, apparently they already know they
have to provide some config.

>b) They'll give the wrong config file - then you'd need to add a flag
>  to detect that - which means you'd need to add something to the
>  qcow to match it to the config; loop back to teh start!

I'm not sure how seriously I should take this argument.  Do stupid
things, win stupid prizes.

If that's the issue, add a UUID to qcow2 files and reference it from the
config file.

> We should make this EASY for users.

To me, having a simple config file they can edit manually certainly
seems simpler than having to use specific tools to edit it inside of the
qcow2 file.

 I understand it is an inconvenience and it would be nice to change it,
 but please understand that I do not want qcow2 to become a filesystem
 just to relieve an inconvenience.
>>>
>>> I very much don't want it to be a filesystem; my reason for writing
>>> down my spec the way I did was to make it clear that the only
>>> thing I want of qcow2 is a single blob, no more; I don't want naming
>>> of the blob or anything else.
>>>
 (Note: I understand that you may not want qcow2 to become a filesystem,
 but I do get the impression from others.)
>>>
>>> My aim was to specify it to fulfill the requirements that everyone
>>> else had asked for, but still only having one unmodifiable blob in qcow.
>>>
>   or have an archive. If you have an archive
> you have to have an unpacking step which then copies, potentially a lot
> of data taking some reasonable amount of time.

 I'm sure this can be optimized, but yes, I get that.

 (If you use e.g. tar and store the image data starting on an FS cluster
 boundary (64 kB should be more than sufficient), I assume there is a way
 to extract that data into a new file without copying anything.)
>>>
>>> But then we have to modify all the current things that know how to
>>> handle a qcow2.
>>
>> Not in this case because it'd still be a flat qcow2 file in a simple tar
>> archive.
>>
>> But you're right if we had a more complex format (like chunks stored in
>> a tar file).
> 
> My only problem with using the tar like that is that all tools
> everywhere would need to be updated to be able to parse them.
> (Note if adding a blob to qcow2 like I'm asking for would break existing
> qcow2 users then I don't want it either).

OK, so I suppose this goes back to "let's decide what we really want to
configure first, and how we can limit our scope in an effective and
long-lasting way".

> Storing a simple bit
> of data with the image avoids that.

 It is not a simple bit of data, as evidenced by the discussion about
 storing binary blobs and MIME types going on.
>>>
>>> All of the things they've suggested can be done inside that one blob;
>>> even inside the json (or any other structure in that blob).
>>
>> Right, from qcow2's perspective it's a blob of data.  But you can put a
>> whole filesystem into a blob of data, and I get the impression that this
>> is what some are trying to do.
>>
>> Once we store larger amounts of binary data in that blob (which is what
>> I'm fearing from comments on MIME types and PNG images), people will
>> realize that always having to re-store the whole blob if you modify
>> something in the middle is inefficient and that it needs to be
>> optimized.  I don't think you want to do that, but we haven't
>> implemented any of this yet and people are already asking for such
>> binary data inside of the blob.
>>
>> I suspect it'll only get worse over time.
>> I think the most difficult thing about this discussion is that there are
>> different targets.
>>
>> You just want to store a bit of information.  OK, good, but then I'd say
>> we could even just prepend that to the image file in a small header.
> 
> 
> I 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Wed, Jun 06, 2018 at 01:44:02PM +0200, Max Reitz wrote:
> Because it's a hack, right.  Storing binary data in a qcow2 file,
> completely ignoring it in qemu (and being completely unusable to any
> potential other users of the qcow2 format[1]) and only interpreting it
> somewhere up the stack is a hack.

It's just a first step and it ensures compatibility with old QEMU
versions. But down the road I think we will start warning
user if the machine type does not match, and possibly even
get the type from there if user didn't supply it.

-- 
MST

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Wed, Jun 06, 2018 at 01:02:53PM +0200, Max Reitz wrote:
> Yeah, but why make qcow2 that format?  That's what I completely fail to
> understand.

Because why not? It's cheap to add it there and is much easier
than teaching people about a new container format.

Eric Blake put it very well I think.  There are several things that
several people would like to see addressed:

(1) A sensible list of guest visible aspects of the VM
  preserving which across VM restarts we deem critical enough to support
  starting guests.
  At this point this includes at least architecture and machine type.

(2) A compact file format for serializing list (1)

(3) Ability to store file (2) in a qcow2 image


You are asking why store (2) in qcow2 image specifically. The answer is
it's just one place where we can store it. The answer is we don't need
to involve qemu-block at all for storing it in other places.

But for many people it will be handy to have it in the same file, and
qcow2 is popular enough that many people will be well served if it's
there.

-- 
MT

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 16:02, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 14:16, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 13:19, Michal Suchánek wrote:
> >>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>> Max Reitz  wrote:
> >>>
>  On 2018-06-06 12:32, Michal Suchánek wrote:
> > On Tue, 29 May 2018 12:14:15 +0200
> > Max Reitz  wrote:
> 
>  [...]
> 
> >> Unless I have got something terribly wrong (which is indeed a
> >> possibility!), to me this proposal means basically to turn qcow2
> >> into (1) a VM description format for qemu, and (2) to turn it into
> >> an archive format on the way.  
> >
> > And if you go all the way you can store multiple disks along with
> > the VM definition so you can have the whole appliance in one file.
> > It conveniently solves the problem of synchronizing snapshots across
> > multiple disk images and the question where to store the machine
> > state if you want to suspend it.   
> 
>  Yeah, but why make qcow2 that format?  That's what I completely fail
>  to understand.
> 
>  If you want to have a single VM description file that contains the VM
>  configuration and some qcow2/raw/whatever files along with it for the
>  guest disk data, sure, go ahead.  But why does the format of the 
>  whole
>  thing need to be qcow2?
> >>>
> >>> Because then qemu can access the disk data from the image directly
> >>> without any need for extraction, copying to different file, etc.
> >>
> >> This does not explain why it needs to be qcow2.  There is absolutely no
> >> reason why you couldn't use qcow2 files in-place inside of another 
> >> file.
> >
> > Because then we'd have to change the whole stack to take advantage of
> > that.  Adding a feature into qcow2 means nothing else changes.
> 
>  Because it's a hack, right.  Storing binary data in a qcow2 file,
>  completely ignoring it in qemu (and being completely unusable to any
>  potential other users of the qcow2 format[1]) and only interpreting it
>  somewhere up the stack is a hack.
> >>>
> >>> It's not a hack!
> >>> Seriously it's not.
> >>> There's nothing wrong with it being aimed higher up the stack than qemu,
> >>
> >> Not really, but storing that information in a disk image file is, from
> >> my perspective.  So far, qcow2 was always just for qemu.  (Hmm...  Maybe
> >> backing links weren't, but at least they were intended for qemu 
> >> originally.)
> >>
> >> So this would mix information for different layers inside qcow2 which to
> >> me sounds weird.  Maybe I just have to get used to it.
> > 
> > The important point is it's explicitly for a different layer; we're not
> > mixing it - the guest can never get to this information.
> 
> Neither can it get to bitmaps, but bitmaps are still for qemu.
> 
> >   It also saves
> > the higher level management layers ever having to look at the data the
> > guest can get to, which is a security advantage.
> 
> Er, well, yes, but guessing configuration options from the guest disk
> contents is definitely a bad idea, I agree on that anyway.
> 
> > From my point of view, it really is the sticky label on the disc rather
> > than the contents of it.
> 
> Sure, which is why I wouldn't put it in qcow2.  Content and meta-content
> is what qcow2 currently stores, but not how to use it.
> 
> >>> the problem we started off with was what happens when a user downloads
> >>> a VM image and tries to import it into their VM system;
> >>
> >> Well, the VM system should choke without a config file. O:-)
> >>
> >>> weve already
> >>> got 2+ layers of management stuff in there - I want the information to
> >>> guide those layers, not form a complete set of configuration.
> >>
> >> But I do.
> >>
> >> If we store some information, I don't see why we don't store all of it.
> > 
> > Hmm, now that generally I don't like:
> 
> Me neither.
> 
> >   a) That really would make it hypervisor specific
> 
> Yes.
> 
> >   b) It would be a lot of data
> 
> Yes.
> 
> >   c) Generally, the supplier of an image doesn't know how the end-user
> >  wants it configured - although for some appliances they might.
> 
> Well, yes.  But just storing very basic information limits the use case
> to a very basic case anyway, doesn't it?  So this wouldn't be worse.
> 
> Everything beyond a very basic use case can expect the user to take 30
> seconds to download and pass a config file.
> 
> >   d) Remember the only problem we had when we 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Wed, Jun 06, 2018 at 03:31:35PM +0100, Dr. David Alan Gilbert wrote:
> > Not in this case because it'd still be a flat qcow2 file in a simple tar
> > archive.
> > 
> > But you're right if we had a more complex format (like chunks stored in
> > a tar file).
> 
> My only problem with using the tar like that is that all tools
> everywhere would need to be updated to be able to parse them.

I feel it is the opposite actually. By adding named blobs or custom
strings to qcow2, we've effectively invented a new type of archive
format, except apps cant use the normal unzip/tar tools/apis they
already have. Instead they need to use qemu-img to read,add,remove
blobs from qcow2. It is very compelling to use an existing archive
format like tar/zip because every language has APIs for dealing
with them and apps probably already do this for things like OVA.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:02, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 14:16, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 13:19, Michal Suchánek wrote:
>>> On Wed, 6 Jun 2018 13:02:53 +0200
>>> Max Reitz  wrote:
>>>
 On 2018-06-06 12:32, Michal Suchánek wrote:
> On Tue, 29 May 2018 12:14:15 +0200
> Max Reitz  wrote:

 [...]

>> Unless I have got something terribly wrong (which is indeed a
>> possibility!), to me this proposal means basically to turn qcow2
>> into (1) a VM description format for qemu, and (2) to turn it into
>> an archive format on the way.  
>
> And if you go all the way you can store multiple disks along with
> the VM definition so you can have the whole appliance in one file.
> It conveniently solves the problem of synchronizing snapshots across
> multiple disk images and the question where to store the machine
> state if you want to suspend it.   

 Yeah, but why make qcow2 that format?  That's what I completely fail
 to understand.

 If you want to have a single VM description file that contains the VM
 configuration and some qcow2/raw/whatever files along with it for the
 guest disk data, sure, go ahead.  But why does the format of the whole
 thing need to be qcow2?
>>>
>>> Because then qemu can access the disk data from the image directly
>>> without any need for extraction, copying to different file, etc.
>>
>> This does not explain why it needs to be qcow2.  There is absolutely no
>> reason why you couldn't use qcow2 files in-place inside of another file.
>
> Because then we'd have to change the whole stack to take advantage of
> that.  Adding a feature into qcow2 means nothing else changes.

 Because it's a hack, right.  Storing binary data in a qcow2 file,
 completely ignoring it in qemu (and being completely unusable to any
 potential other users of the qcow2 format[1]) and only interpreting it
 somewhere up the stack is a hack.
>>>
>>> It's not a hack!
>>> Seriously it's not.
>>> There's nothing wrong with it being aimed higher up the stack than qemu,
>>
>> Not really, but storing that information in a disk image file is, from
>> my perspective.  So far, qcow2 was always just for qemu.  (Hmm...  Maybe
>> backing links weren't, but at least they were intended for qemu originally.)
>>
>> So this would mix information for different layers inside qcow2 which to
>> me sounds weird.  Maybe I just have to get used to it.
> 
> The important point is it's explicitly for a different layer; we're not
> mixing it - the guest can never get to this information.

Neither can it get to bitmaps, but bitmaps are still for qemu.

>   It also saves
> the higher level management layers ever having to look at the data the
> guest can get to, which is a security advantage.

Er, well, yes, but guessing configuration options from the guest disk
contents is definitely a bad idea, I agree on that anyway.

> From my point of view, it really is the sticky label on the disc rather
> than the contents of it.

Sure, which is why I wouldn't put it in qcow2.  Content and meta-content
is what qcow2 currently stores, but not how to use it.

>>> the problem we started off with was what happens when a user downloads
>>> a VM image and tries to import it into their VM system;
>>
>> Well, the VM system should choke without a config file. O:-)
>>
>>> weve already
>>> got 2+ layers of management stuff in there - I want the information to
>>> guide those layers, not form a complete set of configuration.
>>
>> But I do.
>>
>> If we store some information, I don't see why we don't store all of it.
> 
> Hmm, now that generally I don't like:

Me neither.

>   a) That really would make it hypervisor specific

Yes.

>   b) It would be a lot of data

Yes.

>   c) Generally, the supplier of an image doesn't know how the end-user
>  wants it configured - although for some appliances they might.

Well, yes.  But just storing very basic information limits the use case
to a very basic case anyway, doesn't it?  So this wouldn't be worse.

Everything beyond a very basic use case can expect the user to take 30
seconds to download and pass a config file.

>   d) Remember the only problem we had when we got here was how to stop
>  the user shooting themselves in the foot by connecting the wrong
>  image to the wrong VM type.

Hm.  How exactly is that shooting yourself in the foot?  Won't it just
not work?

>   So I'm expecting to use this to
>  

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> > 



> >>> The problem with having a separate file is that you either have to copy
> >>> it around with the image 
> >>
> >> Which is just an inconvenience.
> > 
> > It's more than that;  if it's a separate file then the tools can't
> > rely on users supplying it, and frankly they won't and they'll still
> > just supply an image.
> 
> At which point you throw an error and tell them to specify the config file.

No:
   a) At the moment they get away with it for images since they're all
  'pc' and the management layers do the right thing.
   b) They'll give the wrong config file - then you'd need to add a flag
 to detect that - which means you'd need to add something to the
 qcow to match it to the config; loop back to teh start!

We should make this EASY for users.

> >> I understand it is an inconvenience and it would be nice to change it,
> >> but please understand that I do not want qcow2 to become a filesystem
> >> just to relieve an inconvenience.
> > 
> > I very much don't want it to be a filesystem; my reason for writing
> > down my spec the way I did was to make it clear that the only
> > thing I want of qcow2 is a single blob, no more; I don't want naming
> > of the blob or anything else.
> > 
> >> (Note: I understand that you may not want qcow2 to become a filesystem,
> >> but I do get the impression from others.)
> > 
> > My aim was to specify it to fulfill the requirements that everyone
> > else had asked for, but still only having one unmodifiable blob in qcow.
> > 
> >>>   or have an archive. If you have an archive
> >>> you have to have an unpacking step which then copies, potentially a lot
> >>> of data taking some reasonable amount of time.
> >>
> >> I'm sure this can be optimized, but yes, I get that.
> >>
> >> (If you use e.g. tar and store the image data starting on an FS cluster
> >> boundary (64 kB should be more than sufficient), I assume there is a way
> >> to extract that data into a new file without copying anything.)
> > 
> > But then we have to modify all the current things that know how to
> > handle a qcow2.
> 
> Not in this case because it'd still be a flat qcow2 file in a simple tar
> archive.
> 
> But you're right if we had a more complex format (like chunks stored in
> a tar file).

My only problem with using the tar like that is that all tools
everywhere would need to be updated to be able to parse them.
(Note if adding a blob to qcow2 like I'm asking for would break existing
qcow2 users then I don't want it either).

> >>> Storing a simple bit
> >>> of data with the image avoids that.
> >>
> >> It is not a simple bit of data, as evidenced by the discussion about
> >> storing binary blobs and MIME types going on.
> > 
> > All of the things they've suggested can be done inside that one blob;
> > even inside the json (or any other structure in that blob).
> 
> Right, from qcow2's perspective it's a blob of data.  But you can put a
> whole filesystem into a blob of data, and I get the impression that this
> is what some are trying to do.
> 
> Once we store larger amounts of binary data in that blob (which is what
> I'm fearing from comments on MIME types and PNG images), people will
> realize that always having to re-store the whole blob if you modify
> something in the middle is inefficient and that it needs to be
> optimized.  I don't think you want to do that, but we haven't
> implemented any of this yet and people are already asking for such
> binary data inside of the blob.
> 
> I suspect it'll only get worse over time.
> I think the most difficult thing about this discussion is that there are
> different targets.
> 
> You just want to store a bit of information.  OK, good, but then I'd say
> we could even just prepend that to the image file in a small header.


I think you're over-reading what people are asking for.
I think the PNG suggestion is again the 'label on the front' for a logo.
I've not seen anything that's not for either:
  a) The user to know what the image is
  b) The management layer to know what type of VM to create

> (Note that extending that header would not even be too complicated,
> because you can easily move the qcow2 header somewhere else.  Say you
> move it back by one cluster (e.g. 64 kB), then you just put the cluster
> that was there originally to the end of the file, which is pretty much
> trivial.  Then you copy that original data there and overwrite it with
> the image header.  Done.)
> 
> Others want to store more binary data.  Then this may get inefficient
> and insufficient.  But I'd think at this point it gets really
> problematic to put the data into the 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Wed, Jun 06, 2018 at 12:40:35PM +0100, Richard W.M. Jones wrote:
> I started off a long reply here, but I think you're right.  If we
> cannot make people decide on and use a proper disk image + metadata
> container, then it's also unlikely we'll get them to add sensible
> metadata to their qcow2 images either :-(
> 
> Rich.

But we can do it for them: ask how to run image, store it
in the image forever.

-- 
MST

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Wed, Jun 06, 2018 at 11:14:32AM -0300, Eduardo Habkost wrote:
> On Wed, Jun 06, 2018 at 02:50:10PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Jun 06, 2018 at 03:45:10PM +0200, Michal Suchánek wrote:
> > > 
> > > I think that *if* we want an 'appliance' format that stores a whole VM
> > > in a single file to ease VM distribution then the logical place to look
> > > in qemu is qcow. The reason have been explained at length.
> > 
> > I rather disagree. This is a common problem beyond just QEMU and everyone
> > just uses an existing archive format (TAR, ZIP) for bundling together
> > one or more disk images, metdata for config, and whatever other resources
> > are applicable for the vendor.  This works with any disk format (raw,
> > qcow2, vmdk, vpc, etc) so is preferrable to inventing someting that is
> > specific to qcow2 IMHO.
> 
> Now we have N+1 appliance file formats.  :)
> 
> (We like it or not, qcow2 is already used as an appliance format
> for single-disk VMs in practice.)
> 
> But I agree this must not be specific to qcow2.  The same VM
> description format we agree upon should work with other disk
> formats or with multi-disk appliances.
> 
> If we specify a reasonable VM description format for appliances
> and make it work inside (e.g.) tar files, we will still have the
> option of allowing the description be placed inside qcow2 if we
> really want to.  I don't think we need to finish this qcow2
> bikeshedding exercise right now.

Yes, I think that is sensible, as once we actually try it out in real
world cases, we might then find a tar/zip is sufficient after all and
we don't need to do something extra for qcow2. Also means we can do
experiments without committing to a qcow2 format spec change right
away.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 16:14, Eduardo Habkost wrote:
> On Wed, Jun 06, 2018 at 02:50:10PM +0100, Daniel P. Berrangé wrote:
>> On Wed, Jun 06, 2018 at 03:45:10PM +0200, Michal Suchánek wrote:
>>>
>>> I think that *if* we want an 'appliance' format that stores a whole VM
>>> in a single file to ease VM distribution then the logical place to look
>>> in qemu is qcow. The reason have been explained at length.
>>
>> I rather disagree. This is a common problem beyond just QEMU and everyone
>> just uses an existing archive format (TAR, ZIP) for bundling together
>> one or more disk images, metdata for config, and whatever other resources
>> are applicable for the vendor.  This works with any disk format (raw,
>> qcow2, vmdk, vpc, etc) so is preferrable to inventing someting that is
>> specific to qcow2 IMHO.
> 
> Now we have N+1 appliance file formats.  :)
> 
> (We like it or not, qcow2 is already used as an appliance format
> for single-disk VMs in practice.)
> 
> But I agree this must not be specific to qcow2.  The same VM
> description format we agree upon should work with other disk
> formats or with multi-disk appliances.
> 
> If we specify a reasonable VM description format for appliances
> and make it work inside (e.g.) tar files, we will still have the
> option of allowing the description be placed inside qcow2 if we
> really want to.  I don't think we need to finish this qcow2
> bikeshedding exercise right now.

That actually sounds reasonable to me.

I better not think about it for too long so I don't come up with
something I very much dislike about it.

(Well, now I have.  The thing I still dislike is that we haven't talked
about what we actually want in the end.  Do we want just the very basic
configuration options that are portable?  But what really is the use
case for such basic information?  Won't people demand ever more
configuration options, then?  I certainly would.

And having to handle a full-blown config is probably a pain...)

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eduardo Habkost]

On Wed, Jun 06, 2018 at 02:50:10PM +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 06, 2018 at 03:45:10PM +0200, Michal Suchánek wrote:
> > 
> > I think that *if* we want an 'appliance' format that stores a whole VM
> > in a single file to ease VM distribution then the logical place to look
> > in qemu is qcow. The reason have been explained at length.
> 
> I rather disagree. This is a common problem beyond just QEMU and everyone
> just uses an existing archive format (TAR, ZIP) for bundling together
> one or more disk images, metdata for config, and whatever other resources
> are applicable for the vendor.  This works with any disk format (raw,
> qcow2, vmdk, vpc, etc) so is preferrable to inventing someting that is
> specific to qcow2 IMHO.

Now we have N+1 appliance file formats.  :)

(We like it or not, qcow2 is already used as an appliance format
for single-disk VMs in practice.)

But I agree this must not be specific to qcow2.  The same VM
description format we agree upon should work with other disk
formats or with multi-disk appliances.

If we specify a reasonable VM description format for appliances
and make it work inside (e.g.) tar files, we will still have the
option of allowing the description be placed inside qcow2 if we
really want to.  I don't think we need to finish this qcow2
bikeshedding exercise right now.

-- 
Eduardo

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 14:16, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mre...@redhat.com) wrote:
>  On 2018-06-06 13:19, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:02:53 +0200
> > Max Reitz  wrote:
> >
> >> On 2018-06-06 12:32, Michal Suchánek wrote:
> >>> On Tue, 29 May 2018 12:14:15 +0200
> >>> Max Reitz  wrote:
> >>
> >> [...]
> >>
>  Unless I have got something terribly wrong (which is indeed a
>  possibility!), to me this proposal means basically to turn qcow2
>  into (1) a VM description format for qemu, and (2) to turn it into
>  an archive format on the way.  
> >>>
> >>> And if you go all the way you can store multiple disks along with
> >>> the VM definition so you can have the whole appliance in one file.
> >>> It conveniently solves the problem of synchronizing snapshots across
> >>> multiple disk images and the question where to store the machine
> >>> state if you want to suspend it.   
> >>
> >> Yeah, but why make qcow2 that format?  That's what I completely fail
> >> to understand.
> >>
> >> If you want to have a single VM description file that contains the VM
> >> configuration and some qcow2/raw/whatever files along with it for the
> >> guest disk data, sure, go ahead.  But why does the format of the whole
> >> thing need to be qcow2?
> >
> > Because then qemu can access the disk data from the image directly
> > without any need for extraction, copying to different file, etc.
> 
>  This does not explain why it needs to be qcow2.  There is absolutely no
>  reason why you couldn't use qcow2 files in-place inside of another file.
> >>>
> >>> Because then we'd have to change the whole stack to take advantage of
> >>> that.  Adding a feature into qcow2 means nothing else changes.
> >>
> >> Because it's a hack, right.  Storing binary data in a qcow2 file,
> >> completely ignoring it in qemu (and being completely unusable to any
> >> potential other users of the qcow2 format[1]) and only interpreting it
> >> somewhere up the stack is a hack.
> > 
> > It's not a hack!
> > Seriously it's not.
> > There's nothing wrong with it being aimed higher up the stack than qemu,
> 
> Not really, but storing that information in a disk image file is, from
> my perspective.  So far, qcow2 was always just for qemu.  (Hmm...  Maybe
> backing links weren't, but at least they were intended for qemu originally.)
> 
> So this would mix information for different layers inside qcow2 which to
> me sounds weird.  Maybe I just have to get used to it.

The important point is it's explicitly for a different layer; we're not
mixing it - the guest can never get to this information.  It also saves
the higher level management layers ever having to look at the data the
guest can get to, which is a security advantage.

From my point of view, it really is the sticky label on the disc rather
than the contents of it.

> > the problem we started off with was what happens when a user downloads
> > a VM image and tries to import it into their VM system;
> 
> Well, the VM system should choke without a config file. O:-)
> 
> > weve already
> > got 2+ layers of management stuff in there - I want the information to
> > guide those layers, not form a complete set of configuration.
> 
> But I do.
> 
> If we store some information, I don't see why we don't store all of it.

Hmm, now that generally I don't like:
  a) That really would make it hypervisor specific
  b) It would be a lot of data
  c) Generally, the supplier of an image doesn't know how the end-user
 wants it configured - although for some appliances they might.
  d) Remember the only problem we had when we got here was how to stop
 the user shooting themselves in the foot by connecting the wrong
 image to the wrong VM type.  So I'm expecting to use this to
 contain requirements, nothing more.


> >> That is not necessarily a negative point, hacks can work wonderfully
> >> well, and they usually are simple, that is correct.  But the thing is
> >> that I feel like people have grand visions of what to get out of this.
> >> Imagine, a single file that can configure all and any VM!
> >>
> >> But hacks usually only solve a single issue.  Once you try to extend a
> >> hack, it breaks down and becomes insufficient.
> >>
> >> If we want a grand vision where a single file stores the whole VM, why
> >> not invest the work and make it right from the start?
> > 
> > Because we won't get it right; however much we bikeshed about it
> > we'll just end up with a mess.
> 
> Sure, but the same thing applies to putting it into qcow2.  The
> difference is, for something outside of qcow2, throwing it away and
> starting over is simple.
> 
> When 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Daniel P . Berrangé]

On Wed, Jun 06, 2018 at 03:45:10PM +0200, Michal Suchánek wrote:
> 
> I think that *if* we want an 'appliance' format that stores a whole VM
> in a single file to ease VM distribution then the logical place to look
> in qemu is qcow. The reason have been explained at length.

I rather disagree. This is a common problem beyond just QEMU and everyone
just uses an existing archive format (TAR, ZIP) for bundling together
one or more disk images, metdata for config, and whatever other resources
are applicable for the vendor.  This works with any disk format (raw,
qcow2, vmdk, vpc, etc) so is preferrable to inventing someting that is
specific to qcow2 IMHO.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 15:14:03 +0200
Max Reitz  wrote:

> On 2018-06-06 14:13, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:52:35 +0200
> > Max Reitz  wrote:
> >   
> >> On 2018-06-06 13:43, Michal Suchánek wrote:  
> >>> On Wed, 6 Jun 2018 13:32:47 +0200
> >>> Max Reitz  wrote:
> >>> 
>  On 2018-06-06 13:19, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:02:53 +0200
> > Max Reitz  wrote:  
> 
> [...]
> 
> >> What I'm trying to get at is that qcow2 was not designed to be
> >> a container format for arbitrary files.  If you want to make it
> >> such, I'm sure there are existing formats that work
> >> better.  
> >
> > Such as?  
> 
>  ext2?
> >>>
> >>> So you want an ext2 driver in qemu instead of expanding qcow2 to
> >>> work not only for a single disk but also for an appliance?
> >>
> >> Yes, because ext2 was designed to be a proper filesystem.  I'm not
> >> an FS designer.  Well, not a good one anyway.  So I don't trust
> >> myself on extending qcow2 to be a good FS -- and why would I, when
> >> there are already numerous FS around.  
> > 
> > Do you expect that performance of qemu using qcow2 driver over ext2
> > driver will be better than using qcow driver directly with some part
> > semi-permanently occupied by a configuration blob? My bet is not.  
> 
> If you want to store multiple disk images in a single file?  I would
> think so, yes.  With qcow2, I would assume it leads to
> fragmentation.  

How is that different from single disk divided into two partitions
internally (without any knowledge on the qcow2 level)?

> I would hope that proper filesystems can mitigate this.

Not really. Not without much complexity and repeated maintenance.

> > The ext* drivers are designed to work with kernel VM infrastructure
> > which must be tuned for different usage scenarios and you would
> > have to duplicate that tuning in qemu to get competitive
> > performance. Also you get qcow2 and ext2 metadata which must be
> > allocated, managed, etc. You get more storage and performance
> > overhead for no good reason.  
> 
> Yes, there is a good reason.  You can add arbitrary configuration
> options without having to worry about me.

But I will not be able to use the images in qemu so it will be useless.

Well, there is FUSE and that is certainly blazing fast and ubiquitous,
I am sure.

> 
> Seriously, though, a real FS would allow you to be more expressive and
> really do what you want without having to work around the quirks that
> adding a not-real-FS in the most simple way possible to qcow2 would
> bring with it.
> 
> Because this is part of my fear, that we now add a very simple blob
> for just a sprinkle of data.  But over time it gets more and more
> complex because we want to store more and more data to make things
> ever more convenient[1], we notice that we need more features, the
> format gets more complex, and in the end we have an FS that is just
> worse than a real FS.
> 
> [1] And note that if I'm convinced to store VM configuration data in
> qemu, I will agree that we can store any data in there and it would be
> nice if any VM could be provisioned and used that way.
> 
> > On the other hand, qcow is designed for storing VM disk data and
> > hopefully was tuned to do that decently over the years. The primary
> > use case remains storing VM disk data. Adding a configuration blob
> > does not change that.  
> 
> True.  So the argument is that qcow2 may be worse for storing
> arbitrary data, but we don't have performance requirements for that;
> but we do have performance requirements for disk data and adding
> another format below qcow2 will not make it better.
> 
> I do think it is possible to not make things worse with a format under
> qcow2, but that may require additional complexity, that you think is
> pointless.
> 
> I understand that you think that, but I still believe that putting the
> configuration into qcow2 is just the wrong way around and will fall on
> our feet in the long run.

I think that *if* we want an 'appliance' format that stores a whole VM
in a single file to ease VM distribution then the logical place to look
in qemu is qcow. The reason have been explained at length.

I understand that for some use cases simplifying the distribution of
VMs as much as possible is quite important.

> 
>  Unless I have got something terribly wrong (which is indeed a
>  possibility!), to me this proposal means basically to turn
>  qcow2 into (1) a VM description format for qemu, and (2) to
>  turn it into an archive format on the way.
> >>>
> >>> And if you go all the way you can store multiple disks along
> >>> with the VM definition so you can have the whole appliance in
> >>> one file. It conveniently solves the problem of synchronizing
> >>> snapshots across multiple disk images and the question where
> >>> to store the machine state if you want to suspend it.

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eduardo Habkost]

On Wed, Jun 06, 2018 at 01:44:02PM +0200, Max Reitz wrote:
> On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 13:19, Michal Suchánek wrote:
> >>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>> Max Reitz  wrote:
> >>>
>  On 2018-06-06 12:32, Michal Suchánek wrote:
> > On Tue, 29 May 2018 12:14:15 +0200
> > Max Reitz  wrote:
> 
> [...]
> 
> >> Unless I have got something terribly wrong (which is indeed a
> >> possibility!), to me this proposal means basically to turn qcow2
> >> into (1) a VM description format for qemu, and (2) to turn it into
> >> an archive format on the way.  
> >
> > And if you go all the way you can store multiple disks along with
> > the VM definition so you can have the whole appliance in one file.
> > It conveniently solves the problem of synchronizing snapshots across
> > multiple disk images and the question where to store the machine
> > state if you want to suspend it.   
> 
>  Yeah, but why make qcow2 that format?  That's what I completely fail
>  to understand.
> 
>  If you want to have a single VM description file that contains the VM
>  configuration and some qcow2/raw/whatever files along with it for the
>  guest disk data, sure, go ahead.  But why does the format of the whole
>  thing need to be qcow2?
> >>>
> >>> Because then qemu can access the disk data from the image directly
> >>> without any need for extraction, copying to different file, etc.
> >>
> >> This does not explain why it needs to be qcow2.  There is absolutely no
> >> reason why you couldn't use qcow2 files in-place inside of another file.
> > 
> > Because then we'd have to change the whole stack to take advantage of
> > that.  Adding a feature into qcow2 means nothing else changes.
> 
> Because it's a hack, right.  Storing binary data in a qcow2 file,
> completely ignoring it in qemu (and being completely unusable to any
> potential other users of the qcow2 format[1]) and only interpreting it
> somewhere up the stack is a hack.
> 
> That is not necessarily a negative point, hacks can work wonderfully
> well, and they usually are simple, that is correct.  But the thing is
> that I feel like people have grand visions of what to get out of this.
> Imagine, a single file that can configure all and any VM!
> 
> But hacks usually only solve a single issue.  Once you try to extend a
> hack, it breaks down and becomes insufficient.
> 
> If we want a grand vision where a single file stores the whole VM, why
> not invest the work and make it right from the start?

We don't want a grand vision where a single file stores the whole
VM.  This is exactly what I would like to avoid, by not inventing
a whole different appliance file format.

-- 
Eduardo

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eduardo Habkost]

On Wed, Jun 06, 2018 at 10:44:20AM +0100, Dr. David Alan Gilbert wrote:
> * Eduardo Habkost (ehabk...@redhat.com) wrote:
> > On Tue, Jun 05, 2018 at 10:21:59AM +0100, Dr. David Alan Gilbert wrote:
> > > 
> > > 
> > > This seems to have fizzled out because of a lack of a concrete proposal;
> > > so here is one based on a reply to Max's post:
> > > 
> > > * Max Reitz (mre...@redhat.com) wrote:
> > > 
> > > 
> > > 
> > > > The original problem was that you need to supply a machine type to qemu,
> > > > and that multiple common architectures now have multiple machine types
> > > > and not necessarily all work with a single image.  So far so good, but I
> > > > have two issues here already:
> > > > 
> > > > (1) How is qemu supposed to interpret that information?  If it's stored
> > > > in the image file, I don't see a nice way of retrieving it before the
> > > > machine is initialized, at least not with qemu's current architecture.
> > > 
> > > 
> > > 
> > > > (2) Again, I personally just really don't like saving such information
> > > > in a disk image.  One actual argument I can bring up for that distaste
> > > > is this: Suppose, you have multiple images attached to your VM.  Now the
> > > > VM wants to store the machine type.  Where does it go?  Into all of
> > > > them?
> > > 
> > > 
> > > 
> > > > So I think if we decide to store the machine type, that is kind of a
> > > > slippery slope and then there are good arguments for storing even more
> > > > configuration options in the file, too.  But I really, really don't like
> > > > that.
> > > 
> > > 
> > > 
> > > > For another, how do we store the data?  key-value seems wrong if we want
> > > > to store everything.  JSON might be fine.  But eventually we just want
> > > > basically a qemu configuration file in there, I would think (which may
> > > > support JSON at some point?).   So basically we would store the data as
> > > > a binary blob and let the rest of qemu do its thing with it.  But then
> > > > please tell me why I fought so valiantly against storing random bitmaps
> > > > in qcow2 files.  I hate the idea of making qcow2 a random archive
> > > > format.  We have tar for that.
> > > 
> > > 
> > > 
> > > > tl;dr: I really don't get why it's so hard to supply a config file along
> > > > with a qcow2 image.  Is it so hard for people to realize that a VM does
> > > > not only consist of a disk?
> > > 
> > > Yes! Because in many cases that's all it needs, and it's ready to run
> > > with no unpacking.
> > > 
> > > I think we should have:
> > > 
> > > --
> > > Layer 0:
> > >QCOW provides a way to store a single string of arbitrary (but
> > > limited?) length.
> > >QCOW provides a way to replace the string by a new string.
> > >The original or the new string will be stored after that;
> > >never some mix.
> > >Where a file 'b' has a backing file 'a', 'b' inherits the
> > >string from 'a' unless 'b' has it's own string.
> > >Snapshots inherit their string from the main unless they have
> > >their own string.
> > > 
> > > Layer 1:
> > >The string shall always be a JSON 'object'; i.e. of the form
> > > { "something": ... , "more": ... }
> > > 
> > >The key strings shall be non-null and non-empty and shall
> > >be unique.
> > > 
> > 
> > I'd prefer layer 0+1 to:
> > 
> > 1) Allow multiple entries to be stored (implemented by layer 1
> >in this proposal)
> > 2) Identify each entry with a name (implemented by layer 1 in
> >this proposal)
> > 3) Allow arbitrary binary data to be stored on an entry
> >(not possible with the JSON-based proposal, because JSON
> >strings are not blobs, but Unicode strings).
> > 4) Make it easy to replace only one entry while keeping others
> >intact (not the case here, if all entries are stored in the
> >same JSON string)
> > 
> > I think it would be simpler if layer 0 simply provided a list of
> > names/value pairs, where names are ascii strings, and values are
> > binary data[1].  It would make layer 1 unnecessary, and allow (3)
> > and (4) to happen.
> > 
> > [1] In other words, Rich's proposal of "named blobs":
> > https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html
> 
> My reasoning was just one of simplicity; each layer in this is
> almost trivial.
> The downside to my proposal is it's more expensive if you want
> to change a single key; but as yet no one has suggested why we'd
> want to do it frequently enough to worry.
> 
> My suggestion of JSON was just to try and stop the bikeshedding;
> we seem to be putting a lot of effort into inventing a new
> storage definition for something that so far we need to hold
> one rarely changing short string, which even with all the discussion
> has moved upto maybe 3 or 4 rarely changing pieces of data.

I agree with that feeling, and I don't have a very strong
preference.

I would prefer if qcow2 let us store named blobs, because it
gives us 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 14:03, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrangé (berra...@redhat.com) wrote:
>> On Wed, Jun 06, 2018 at 12:42:28PM +0100, Richard W.M. Jones wrote:
>>> On Wed, Jun 06, 2018 at 12:14:07PM +0100, Dr. David Alan Gilbert wrote:
 The problem with having a separate file is that you either have to copy
 it around with the image or have an archive.  If you have an archive
 you have to have an unpacking step which then copies, potentially a lot
 of data taking some reasonable amount of time.  Storing a simple bit
 of data with the image avoids that.
>>>
>>> This isn't really true.  For OVA (ie. tar) we don't unpack them.
>>> Adding file.offset and file.size in qemu's raw driver was crucial to
>>> that optimization.
>>
>> Though that assumes you're only using the qcow2 file in read-only mode.
>> As soon as you need write access you need to unpack from the OVA so that
>> the qcow2 file can grow its length when new sectors are allocated.
> 
> And the person creating the OVA has to do that taring rather than just
> take the qcow2 they've just used in the VM.

Note that this again can be done efficiently.  You just overwrite the
beginning of the qcow2 file and move the overwritten clusters somewhere
else.

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 14:13, Michal Suchánek wrote:
> On Wed, 6 Jun 2018 13:52:35 +0200
> Max Reitz  wrote:
> 
>> On 2018-06-06 13:43, Michal Suchánek wrote:
>>> On Wed, 6 Jun 2018 13:32:47 +0200
>>> Max Reitz  wrote:
>>>   
 On 2018-06-06 13:19, Michal Suchánek wrote:  
> On Wed, 6 Jun 2018 13:02:53 +0200
> Max Reitz  wrote:

[...]

>> What I'm trying to get at is that qcow2 was not designed to be a
>> container format for arbitrary files.  If you want to make it
>> such, I'm sure there are existing formats that work better.
>
> Such as?

 ext2?  
>>>
>>> So you want an ext2 driver in qemu instead of expanding qcow2 to
>>> work not only for a single disk but also for an appliance?  
>>
>> Yes, because ext2 was designed to be a proper filesystem.  I'm not an
>> FS designer.  Well, not a good one anyway.  So I don't trust myself on
>> extending qcow2 to be a good FS -- and why would I, when there are
>> already numerous FS around.
> 
> Do you expect that performance of qemu using qcow2 driver over ext2
> driver will be better than using qcow driver directly with some part
> semi-permanently occupied by a configuration blob? My bet is not.

If you want to store multiple disk images in a single file?  I would
think so, yes.  With qcow2, I would assume it leads to fragmentation.  I
would hope that proper filesystems can mitigate this.
> The ext* drivers are designed to work with kernel VM infrastructure
> which must be tuned for different usage scenarios and you would have to
> duplicate that tuning in qemu to get competitive performance. Also you
> get qcow2 and ext2 metadata which must be allocated, managed, etc. You
> get more storage and performance overhead for no good reason.

Yes, there is a good reason.  You can add arbitrary configuration
options without having to worry about me.

Seriously, though, a real FS would allow you to be more expressive and
really do what you want without having to work around the quirks that
adding a not-real-FS in the most simple way possible to qcow2 would
bring with it.

Because this is part of my fear, that we now add a very simple blob for
just a sprinkle of data.  But over time it gets more and more complex
because we want to store more and more data to make things ever more
convenient[1], we notice that we need more features, the format gets
more complex, and in the end we have an FS that is just worse than a
real FS.

[1] And note that if I'm convinced to store VM configuration data in
qemu, I will agree that we can store any data in there and it would be
nice if any VM could be provisioned and used that way.

> On the other hand, qcow is designed for storing VM disk data and
> hopefully was tuned to do that decently over the years. The primary use
> case remains storing VM disk data. Adding a configuration blob does not
> change that.

True.  So the argument is that qcow2 may be worse for storing arbitrary
data, but we don't have performance requirements for that; but we do
have performance requirements for disk data and adding another format
below qcow2 will not make it better.

I do think it is possible to not make things worse with a format under
qcow2, but that may require additional complexity, that you think is
pointless.

I understand that you think that, but I still believe that putting the
configuration into qcow2 is just the wrong way around and will fall on
our feet in the long run.

 Unless I have got something terribly wrong (which is indeed a
 possibility!), to me this proposal means basically to turn
 qcow2 into (1) a VM description format for qemu, and (2) to
 turn it into an archive format on the way.  
>>>
>>> And if you go all the way you can store multiple disks along
>>> with the VM definition so you can have the whole appliance in
>>> one file. It conveniently solves the problem of synchronizing
>>> snapshots across multiple disk images and the question where to
>>> store the machine state if you want to suspend it.   
>>
>> Yeah, but why make qcow2 that format?  That's what I completely
>> fail to understand.
>>
>> If you want to have a single VM description file that contains
>> the VM configuration and some qcow2/raw/whatever files along
>> with it for the guest disk data, sure, go ahead.  But why does
>> the format of the whole thing need to be qcow2?
>
> Because then qemu can access the disk data from the image directly
> without any need for extraction, copying to different file,
> etc.

 This does not explain why it needs to be qcow2.  There is
 absolutely no reason why you couldn't use qcow2 files in-place
 inside of another file.  
>>>
>>> qemu cannot read the disk data from the file in-place.  
>>
>> Hu?  Why not?
> 
> Well, it can possibly read the image if it happens to be continuous. It
> will not be able to update it without a fs driver, however.

Yes, but 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 14:00, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
>>> * Max Reitz (mre...@redhat.com) wrote:
 On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> 
>
> This seems to have fizzled out because of a lack of a concrete proposal;
> so here is one based on a reply to Max's post:
>
> * Max Reitz (mre...@redhat.com) wrote:
>
> 
>
>> The original problem was that you need to supply a machine type to qemu,
>> and that multiple common architectures now have multiple machine types
>> and not necessarily all work with a single image.  So far so good, but I
>> have two issues here already:
>>
>> (1) How is qemu supposed to interpret that information?  If it's stored
>> in the image file, I don't see a nice way of retrieving it before the
>> machine is initialized, at least not with qemu's current architecture.
>
> 
>
>> (2) Again, I personally just really don't like saving such information
>> in a disk image.  One actual argument I can bring up for that distaste
>> is this: Suppose, you have multiple images attached to your VM.  Now the
>> VM wants to store the machine type.  Where does it go?  Into all of
>> them?
>
> 
>
>> So I think if we decide to store the machine type, that is kind of a
>> slippery slope and then there are good arguments for storing even more
>> configuration options in the file, too.  But I really, really don't like
>> that.
>
> 
>
>> For another, how do we store the data?  key-value seems wrong if we want
>> to store everything.  JSON might be fine.  But eventually we just want
>> basically a qemu configuration file in there, I would think (which may
>> support JSON at some point?).   So basically we would store the data as
>> a binary blob and let the rest of qemu do its thing with it.  But then
>> please tell me why I fought so valiantly against storing random bitmaps
>> in qcow2 files.  I hate the idea of making qcow2 a random archive
>> format.  We have tar for that.
>
> 
>
>> tl;dr: I really don't get why it's so hard to supply a config file along
>> with a qcow2 image.  Is it so hard for people to realize that a VM does
>> not only consist of a disk?
>
> Yes! Because in many cases that's all it needs, and it's ready to run
> with no unpacking.

 It clearly is not, or we would not have this discussion.

 The disk image is only enough if you want the default values for all of
 qemu's configuration options, because today (and if I were to decide, in
 the future, too) disk images do not configure the VM (well, they
 configure the guest, but not the VM itself).
>>>
>>> The problem with having a separate file is that you either have to copy
>>> it around with the image 
>>
>> Which is just an inconvenience.
> 
> It's more than that;  if it's a separate file then the tools can't
> rely on users supplying it, and frankly they won't and they'll still
> just supply an image.

At which point you throw an error and tell them to specify the config file.

>> I understand it is an inconvenience and it would be nice to change it,
>> but please understand that I do not want qcow2 to become a filesystem
>> just to relieve an inconvenience.
> 
> I very much don't want it to be a filesystem; my reason for writing
> down my spec the way I did was to make it clear that the only
> thing I want of qcow2 is a single blob, no more; I don't want naming
> of the blob or anything else.
> 
>> (Note: I understand that you may not want qcow2 to become a filesystem,
>> but I do get the impression from others.)
> 
> My aim was to specify it to fulfill the requirements that everyone
> else had asked for, but still only having one unmodifiable blob in qcow.
> 
>>>   or have an archive. If you have an archive
>>> you have to have an unpacking step which then copies, potentially a lot
>>> of data taking some reasonable amount of time.
>>
>> I'm sure this can be optimized, but yes, I get that.
>>
>> (If you use e.g. tar and store the image data starting on an FS cluster
>> boundary (64 kB should be more than sufficient), I assume there is a way
>> to extract that data into a new file without copying anything.)
> 
> But then we have to modify all the current things that know how to
> handle a qcow2.

Not in this case because it'd still be a flat qcow2 file in a simple tar
archive.

But you're right if we had a more complex format (like chunks stored in
a tar file).

>>> Storing a simple bit
>>> of data with the image avoids that.
>>
>> It is not a simple bit of data, as evidenced by the discussion about
>> storing binary blobs and MIME types going on.
> 
> All of the things they've suggested can be done inside that one blob;
> 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-06 13:19, Michal Suchánek wrote:
> >>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>> Max Reitz  wrote:
> >>>
>  On 2018-06-06 12:32, Michal Suchánek wrote:
> > On Tue, 29 May 2018 12:14:15 +0200
> > Max Reitz  wrote:
> 
> [...]
> 
> >> Unless I have got something terribly wrong (which is indeed a
> >> possibility!), to me this proposal means basically to turn qcow2
> >> into (1) a VM description format for qemu, and (2) to turn it into
> >> an archive format on the way.  
> >
> > And if you go all the way you can store multiple disks along with
> > the VM definition so you can have the whole appliance in one file.
> > It conveniently solves the problem of synchronizing snapshots across
> > multiple disk images and the question where to store the machine
> > state if you want to suspend it.   
> 
>  Yeah, but why make qcow2 that format?  That's what I completely fail
>  to understand.
> 
>  If you want to have a single VM description file that contains the VM
>  configuration and some qcow2/raw/whatever files along with it for the
>  guest disk data, sure, go ahead.  But why does the format of the whole
>  thing need to be qcow2?
> >>>
> >>> Because then qemu can access the disk data from the image directly
> >>> without any need for extraction, copying to different file, etc.
> >>
> >> This does not explain why it needs to be qcow2.  There is absolutely no
> >> reason why you couldn't use qcow2 files in-place inside of another file.
> > 
> > Because then we'd have to change the whole stack to take advantage of
> > that.  Adding a feature into qcow2 means nothing else changes.
> 
> Because it's a hack, right.  Storing binary data in a qcow2 file,
> completely ignoring it in qemu (and being completely unusable to any
> potential other users of the qcow2 format[1]) and only interpreting it
> somewhere up the stack is a hack.

It's not a hack!
Seriously it's not.
There's nothing wrong with it being aimed higher up the stack than qemu,
the problem we started off with was what happens when a user downloads
a VM image and tries to import it into their VM system; weve already
got 2+ layers of management stuff in there - I want the information to
guide those layers, not form a complete set of configuration.

> That is not necessarily a negative point, hacks can work wonderfully
> well, and they usually are simple, that is correct.  But the thing is
> that I feel like people have grand visions of what to get out of this.
> Imagine, a single file that can configure all and any VM!
> 
> But hacks usually only solve a single issue.  Once you try to extend a
> hack, it breaks down and becomes insufficient.
> 
> If we want a grand vision where a single file stores the whole VM, why
> not invest the work and make it right from the start?

Because we won't get it right; however much we bikeshed about it
we'll just end up with a mess.   The right thing is to put in something
to hold configuration and then review the items of configuration we
add properly as we define them.

> Max
> 
> [1] Yes, I concede that there are probably no other users of qcow2.  But
> please forgive me for assuming that qcow2 was in a sense designed to be
> a rather general image format that not only qemu could use.

What makes it QEMU specific?  It's basically just the same key/value
setup as OVA, except putting them inside the qcow2.
We could use the same keys/value definitions as OVA in the blob,
although their definitions aren't very portable either.

Dave



--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 13:52:35 +0200
Max Reitz  wrote:

> On 2018-06-06 13:43, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:32:47 +0200
> > Max Reitz  wrote:
> >   
> >> On 2018-06-06 13:19, Michal Suchánek wrote:  
> >>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>> Max Reitz  wrote:
> >>> 
>  On 2018-06-06 12:32, Michal Suchánek wrote:
> > On Tue, 29 May 2018 12:14:15 +0200
> > Max Reitz  wrote:  
> 
> [...]
> 
> >> I hate the idea of making qcow2 a random archive format.  
> >
> > What's wrong with that?  
> 
>  The fact that qcow2 isn't.
> 
>  From my perspective it would increase the format's complexity to
>  a point where you could just create a new format altogether.
>  Well, actually, all you do is design a filesystem (or reuse an
>  existing one).
> 
> >> We have tar for that.  
> >
> > It does not support expanding the stored files.  
> 
>  Nor does qcow2, because it does not support storing files at
>  all.
> >>>
> >>> AFAICT from the previous discussion it already does allow storing
> >>> multiple data streams that can be changed independently so it
> >>> basically is an archive format or filesystem except the streams
> >>> are not named nor easily accessible separately outside of
> >>> qemu.
> >>
> >> I don't quite understand what you are referring to.  We have
> >> snapshots, we have bitmaps, yes, but all of that are related
> >> directly to the stored guest disk data.
> >>
> >> The only thing we currently have in qcow2 that is opaque is the VM
> >> state that can be stored in snapshots (and don't hold me
> >> responsible for that).  
> > 
> > But it has to be related to the stored disk data only because of
> > your dislike for stuff not related to the disk data. Not for a
> > technical reason. The format can sustain storing unrelated data.  
> 
> Yes.  Technically, you can even store data without qcow2 knowing,
> except that qemu-img check -r leaks will delete it, but yeah.
> 
>  Secondly, that completely depends on how you use it.  You can
>  freely expand the last file in the archive, for instance.  Also
>  I've seen people store files in chunks so they can indeed resize
>  it.
> 
>  (I'm wondering if we could write a block driver that could
>  provide such a chunk allocation transparently to qcow2...  Note
>  that a qcow2 file does not need to be continuous, so you could
>  in theory indeed store the qcow2 file and its data in completely
>  separate places in a tar file.)
> >>>
> >>> Which basically invents another new filesystem on top of tar for
> >>> no good reason. Especially when we have already support for
> >>> storage format that is capable enough.
> >>
> >> No different from inventing a filesystem on top of qcow2.
> >>
> >> I don't think qcow2 is any more capable than tar.  
> > 
> > It can natively resize the objects it stores.  
> 
> It does not store arbitrary objects.  It stores a guest disk (and
> snapshots), and bitmaps.  These are stored in directory structures so
> new chunks can be dynamically added.  Note that both use slightly
> different structures.
> 
> Yes, it is technically possible to use a similar structure to store
> arbitrary objects.  But currently it does not have that capability.
> 
> >tar cannot do that
> > so you will have to store random nonsense in the tar archive that
> > only makes sense for your fs/tar driver.  
> 
> Right.  But what's the difference to storing random nonsense in a
> qcow2 file that not even makes sense to qemu?
> 
>  What I'm trying to get at is that qcow2 was not designed to be a
>  container format for arbitrary files.  If you want to make it
>  such, I'm sure there are existing formats that work better.
> >>>
> >>> Such as?
> >>
> >> ext2?  
> > 
> > So you want an ext2 driver in qemu instead of expanding qcow2 to
> > work not only for a single disk but also for an appliance?  
> 
> Yes, because ext2 was designed to be a proper filesystem.  I'm not an
> FS designer.  Well, not a good one anyway.  So I don't trust myself on
> extending qcow2 to be a good FS -- and why would I, when there are
> already numerous FS around.

Do you expect that performance of qemu using qcow2 driver over ext2
driver will be better than using qcow driver directly with some part
semi-permanently occupied by a configuration blob? My bet is not.

The ext* drivers are designed to work with kernel VM infrastructure
which must be tuned for different usage scenarios and you would have to
duplicate that tuning in qemu to get competitive performance. Also you
get qcow2 and ext2 metadata which must be allocated, managed, etc. You
get more storage and performance overhead for no good reason.

On the other hand, qcow is designed for storing VM disk data and
hopefully was tuned to do that decently over the years. The primary use
case remains 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Daniel P. Berrangé (berra...@redhat.com) wrote:
> On Wed, Jun 06, 2018 at 12:42:28PM +0100, Richard W.M. Jones wrote:
> > On Wed, Jun 06, 2018 at 12:14:07PM +0100, Dr. David Alan Gilbert wrote:
> > > The problem with having a separate file is that you either have to copy
> > > it around with the image or have an archive.  If you have an archive
> > > you have to have an unpacking step which then copies, potentially a lot
> > > of data taking some reasonable amount of time.  Storing a simple bit
> > > of data with the image avoids that.
> > 
> > This isn't really true.  For OVA (ie. tar) we don't unpack them.
> > Adding file.offset and file.size in qemu's raw driver was crucial to
> > that optimization.
> 
> Though that assumes you're only using the qcow2 file in read-only mode.
> As soon as you need write access you need to unpack from the OVA so that
> the qcow2 file can grow its length when new sectors are allocated.

And the person creating the OVA has to do that taring rather than just
take the qcow2 they've just used in the VM.

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> >>> 
> >>>
> >>> This seems to have fizzled out because of a lack of a concrete proposal;
> >>> so here is one based on a reply to Max's post:
> >>>
> >>> * Max Reitz (mre...@redhat.com) wrote:
> >>>
> >>> 
> >>>
>  The original problem was that you need to supply a machine type to qemu,
>  and that multiple common architectures now have multiple machine types
>  and not necessarily all work with a single image.  So far so good, but I
>  have two issues here already:
> 
>  (1) How is qemu supposed to interpret that information?  If it's stored
>  in the image file, I don't see a nice way of retrieving it before the
>  machine is initialized, at least not with qemu's current architecture.
> >>>
> >>> 
> >>>
>  (2) Again, I personally just really don't like saving such information
>  in a disk image.  One actual argument I can bring up for that distaste
>  is this: Suppose, you have multiple images attached to your VM.  Now the
>  VM wants to store the machine type.  Where does it go?  Into all of
>  them?
> >>>
> >>> 
> >>>
>  So I think if we decide to store the machine type, that is kind of a
>  slippery slope and then there are good arguments for storing even more
>  configuration options in the file, too.  But I really, really don't like
>  that.
> >>>
> >>> 
> >>>
>  For another, how do we store the data?  key-value seems wrong if we want
>  to store everything.  JSON might be fine.  But eventually we just want
>  basically a qemu configuration file in there, I would think (which may
>  support JSON at some point?).   So basically we would store the data as
>  a binary blob and let the rest of qemu do its thing with it.  But then
>  please tell me why I fought so valiantly against storing random bitmaps
>  in qcow2 files.  I hate the idea of making qcow2 a random archive
>  format.  We have tar for that.
> >>>
> >>> 
> >>>
>  tl;dr: I really don't get why it's so hard to supply a config file along
>  with a qcow2 image.  Is it so hard for people to realize that a VM does
>  not only consist of a disk?
> >>>
> >>> Yes! Because in many cases that's all it needs, and it's ready to run
> >>> with no unpacking.
> >>
> >> It clearly is not, or we would not have this discussion.
> >>
> >> The disk image is only enough if you want the default values for all of
> >> qemu's configuration options, because today (and if I were to decide, in
> >> the future, too) disk images do not configure the VM (well, they
> >> configure the guest, but not the VM itself).
> > 
> > The problem with having a separate file is that you either have to copy
> > it around with the image 
> 
> Which is just an inconvenience.

It's more than that;  if it's a separate file then the tools can't
rely on users supplying it, and frankly they won't and they'll still
just supply an image.

> I understand it is an inconvenience and it would be nice to change it,
> but please understand that I do not want qcow2 to become a filesystem
> just to relieve an inconvenience.

I very much don't want it to be a filesystem; my reason for writing
down my spec the way I did was to make it clear that the only
thing I want of qcow2 is a single blob, no more; I don't want naming
of the blob or anything else.

> (Note: I understand that you may not want qcow2 to become a filesystem,
> but I do get the impression from others.)

My aim was to specify it to fulfill the requirements that everyone
else had asked for, but still only having one unmodifiable blob in qcow.

> >   or have an archive. If you have an archive
> > you have to have an unpacking step which then copies, potentially a lot
> > of data taking some reasonable amount of time.
> 
> I'm sure this can be optimized, but yes, I get that.
> 
> (If you use e.g. tar and store the image data starting on an FS cluster
> boundary (64 kB should be more than sufficient), I assume there is a way
> to extract that data into a new file without copying anything.)

But then we have to modify all the current things that know how to
handle a qcow2.

> > Storing a simple bit
> > of data with the image avoids that.
> 
> It is not a simple bit of data, as evidenced by the discussion about
> storing binary blobs and MIME types going on.

All of the things they've suggested can be done inside that one blob;
even inside the json (or any other structure in that blob).

> >>> I think we should have:
> >>>
> >>> --
> >>> Layer 0:
> >>>QCOW provides a way to store a single string of arbitrary (but
> >>> limited?) length.
> >>>QCOW provides a way to replace the string by a new string.
> >>>  

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 13:48, Daniel P. Berrangé wrote:
> On Wed, Jun 06, 2018 at 12:42:28PM +0100, Richard W.M. Jones wrote:
>> On Wed, Jun 06, 2018 at 12:14:07PM +0100, Dr. David Alan Gilbert wrote:
>>> The problem with having a separate file is that you either have to copy
>>> it around with the image or have an archive.  If you have an archive
>>> you have to have an unpacking step which then copies, potentially a lot
>>> of data taking some reasonable amount of time.  Storing a simple bit
>>> of data with the image avoids that.
>>
>> This isn't really true.  For OVA (ie. tar) we don't unpack them.
>> Adding file.offset and file.size in qemu's raw driver was crucial to
>> that optimization.
> 
> Though that assumes you're only using the qcow2 file in read-only mode.
> As soon as you need write access you need to unpack from the OVA so that
> the qcow2 file can grow its length when new sectors are allocated.

Except if the qcow2 file is at the end of the archive.  Then all you
need to do is adjust the length field of the tar file header.

Max



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 13:43, Michal Suchánek wrote:
> On Wed, 6 Jun 2018 13:32:47 +0200
> Max Reitz  wrote:
> 
>> On 2018-06-06 13:19, Michal Suchánek wrote:
>>> On Wed, 6 Jun 2018 13:02:53 +0200
>>> Max Reitz  wrote:
>>>   
 On 2018-06-06 12:32, Michal Suchánek wrote:  
> On Tue, 29 May 2018 12:14:15 +0200
> Max Reitz  wrote:

[...]

>> I hate the idea of making qcow2 a random archive format.
>
> What's wrong with that?

 The fact that qcow2 isn't.

 From my perspective it would increase the format's complexity to a
 point where you could just create a new format altogether.  Well,
 actually, all you do is design a filesystem (or reuse an existing
 one).
  
>> We have tar for that.
>
> It does not support expanding the stored files.

 Nor does qcow2, because it does not support storing files at all.  
>>>
>>> AFAICT from the previous discussion it already does allow storing
>>> multiple data streams that can be changed independently so it
>>> basically is an archive format or filesystem except the streams are
>>> not named nor easily accessible separately outside of qemu.  
>>
>> I don't quite understand what you are referring to.  We have
>> snapshots, we have bitmaps, yes, but all of that are related directly
>> to the stored guest disk data.
>>
>> The only thing we currently have in qcow2 that is opaque is the VM
>> state that can be stored in snapshots (and don't hold me responsible
>> for that).
> 
> But it has to be related to the stored disk data only because of your
> dislike for stuff not related to the disk data. Not for a technical
> reason. The format can sustain storing unrelated data.

Yes.  Technically, you can even store data without qcow2 knowing, except
that qemu-img check -r leaks will delete it, but yeah.

 Secondly, that completely depends on how you use it.  You can
 freely expand the last file in the archive, for instance.  Also
 I've seen people store files in chunks so they can indeed resize
 it.

 (I'm wondering if we could write a block driver that could provide
 such a chunk allocation transparently to qcow2...  Note that a
 qcow2 file does not need to be continuous, so you could in theory
 indeed store the qcow2 file and its data in completely separate
 places in a tar file.)  
>>>
>>> Which basically invents another new filesystem on top of tar for no
>>> good reason. Especially when we have already support for storage
>>> format that is capable enough.  
>>
>> No different from inventing a filesystem on top of qcow2.
>>
>> I don't think qcow2 is any more capable than tar.
> 
> It can natively resize the objects it stores.

It does not store arbitrary objects.  It stores a guest disk (and
snapshots), and bitmaps.  These are stored in directory structures so
new chunks can be dynamically added.  Note that both use slightly
different structures.

Yes, it is technically possible to use a similar structure to store
arbitrary objects.  But currently it does not have that capability.

>tar cannot do that so you
> will have to store random nonsense in the tar archive that only makes
> sense for your fs/tar driver.

Right.  But what's the difference to storing random nonsense in a qcow2
file that not even makes sense to qemu?

 What I'm trying to get at is that qcow2 was not designed to be a
 container format for arbitrary files.  If you want to make it such,
 I'm sure there are existing formats that work better.  
>>>
>>> Such as?  
>>
>> ext2?
> 
> So you want an ext2 driver in qemu instead of expanding qcow2 to work
> not only for a single disk but also for an appliance?

Yes, because ext2 was designed to be a proper filesystem.  I'm not an FS
designer.  Well, not a good one anyway.  So I don't trust myself on
extending qcow2 to be a good FS -- and why would I, when there are
already numerous FS around.

>> It seems to me that you want to make qcow2 a filesystem.  Sure, the FS
>> we'd end up with would probably be simpler than ext2, but I assume
>> thanks to feature creep we'd eventually end up with a qcow2 format
>> that is a worse FS than real FS (especially performance-wise), but
>> that is similarly complex.
> 
> I do not see how the complexity increases drastically by assigning
> user-visible names to some of the data stored in the image. I am not
> familiar with the internals, though.

As I said, qcow2 does not store objects currently.  It stores specific
data structures that differ.  There is no common "object structure".

>> Unless I have got something terribly wrong (which is indeed a
>> possibility!), to me this proposal means basically to turn qcow2
>> into (1) a VM description format for qemu, and (2) to turn it
>> into an archive format on the way.
>
> And if you go all the way you can store multiple disks along with
> the VM definition so you can have 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 6 Jun 2018 13:32:47 +0200
Max Reitz  wrote:

> On 2018-06-06 13:19, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:02:53 +0200
> > Max Reitz  wrote:
> >   
> >> On 2018-06-06 12:32, Michal Suchánek wrote:  
> >>> On Tue, 29 May 2018 12:14:15 +0200
> >>> Max Reitz  wrote:
> >>> 
>  On 2018-05-29 08:44, Kevin Wolf wrote:
> > Am 28.05.2018 um 23:25 hat Richard W.M. Jones geschrieben:  
> >> On Mon, May 28, 2018 at 10:20:54PM +0100, Richard W.M. Jones
> >> wrote:  
> >>> On Mon, May 28, 2018 at 08:38:33PM +0200, Kevin Wolf
> >>> wrote:  
>  Just accessing the image file within a tar archive is
>  possible and we could write a block driver for that (I
>  actually think we should do this), but it restricts you
>  because certain operations like resizing aren't really
>  possible in tar. Unfortunately, resizing is a really common
>  operation for non-raw image formats.  
> >>>
> >>> We do this already in virt-v2v (using file.offset and
> >>> file.size parameters in the raw driver).
> >>>
> >>> For virt-v2v we only need to read the source so resizing isn't
> >>> an issue.  For most of the cases we're talking about the
> >>> downloaded image would also be a template / base image, so I
> >>> suppose only reading would be required too.
> >>>
> >>> I also wrote an nbdkit tar file driver (supports writes, but
> >>> not resizing).
> >>> https://manpages.debian.org/testing/nbdkit-plugin-perl/nbdkit-tar-plugin.1.en.html
> >>>   
> >>
> >> I should add the other thorny issue with OVA files is that the
> >> metadata contains a checksum (SHA1 or SHA256) of the disk
> >> images. If you modify the disk images in-place in the tar file
> >> then you need to recalculate those.  
> >
> > All of this means that OVA isn't really well suited to be used
> > as a native format for VM configuration + images. It's just for
> > sharing read-only images that are converted into another native
> > format before they are used.
> >
> > Which is probably fair for the use case it was made for, but
> > means that we need something else to solve our problem.  
> 
>  Maybe we should first narrow down our problem.  Maybe you have
>  done that already, but I'm quite in the dark still.
> 
>  The original problem was that you need to supply a machine type
>  to qemu, and that multiple common architectures now have multiple
>  machine types and not necessarily all work with a single image.
>  So far so good, but I have two issues here already:
> 
>  (1) How is qemu supposed to interpret that information?  If it's
>  stored in the image file, I don't see a nice way of retrieving it
>  before the machine is initialized, at least not with qemu's
>  current architecture. Once we support configuring qemu solely
>  through QMP, sure, you can do a blockdev-add and then build the
>  machine accordingly.  But that is not here today, and I'm not
>  sure this is a good idea either, because that would mean
>  automagic defaults for the machine-building QMP commands derived
>  from the blockdev-add earlier, which should get a plain "No".
>  Also, having to use QMP to build your machine wouldn't make
>  anything easier; at least not easier than just supplying a
>  configuration file along with the image.
> 
>  (Building the magic into -blockdev might be less horrible, but
>  such magic (adding block devices influences machine defaults) to
>  me still doesn't seem worth not having to supply a config file
>  along with the disk image.)
> 
>  (2) Again, I personally just really don't like saving such
>  information in a disk image.  One actual argument I can bring up
>  for that distaste is this: Suppose, you have multiple images
>  attached to your VM.  Now the VM wants to store the machine type.
>  Where does it go?  Into all of them?  But some of those images
>  may only contain data and might be intended to be shared between
>  multiple VMs.  So those shouldn't receive the mark.  Only disks
>  with binaries should receive them. But what if those binaries are
>  just cross-compiled binaries for some other VM?  Oh no, so not
>  even binaries are a sure indicator...  So I have no idea where
>  the information is supposed to be stored.  In any case, "the
>  first image" just gets an outright "no" from me, and "all
>  images" gets an "I don't think this is a good idea".
> 
>  Loading is fun, too.  OK, so you attach multiple disk images to a
>  VM. Oops, they have varying machine type information...  Now
>  what?  Use the information from the first one?  Definitely no.
>  Just ignore all of the information in such a case and have the
>  user supply the machine type again?  

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Richard W.M. Jones]

On Wed, Jun 06, 2018 at 12:14:07PM +0100, Dr. David Alan Gilbert wrote:
> The problem with having a separate file is that you either have to copy
> it around with the image or have an archive.  If you have an archive
> you have to have an unpacking step which then copies, potentially a lot
> of data taking some reasonable amount of time.  Storing a simple bit
> of data with the image avoids that.

This isn't really true.  For OVA (ie. tar) we don't unpack them.
Adding file.offset and file.size in qemu's raw driver was crucial to
that optimization.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Richard W.M. Jones]

On Wed, Jun 06, 2018 at 01:02:53PM +0200, Max Reitz wrote:
> (I'm wondering if we could write a block driver that could provide such
> a chunk allocation transparently to qcow2...  Note that a qcow2 file
> does not need to be continuous, so you could in theory indeed store the
> qcow2 file and its data in completely separate places in a tar file.)

nbdkit-split-plugin
(https://github.com/libguestfs/nbdkit/tree/master/plugins/split).  It
currently doesn't support resizing although it could do fairly easily.

> Yeah, but why make qcow2 that format?  That's what I completely fail to
> understand.

I started off a long reply here, but I think you're right.  If we
cannot make people decide on and use a proper disk image + metadata
container, then it's also unlikely we'll get them to add sensible
metadata to their qcow2 images either :-(

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 13:19, Michal Suchánek wrote:
> > On Wed, 6 Jun 2018 13:02:53 +0200
> > Max Reitz  wrote:
> > 
> >> On 2018-06-06 12:32, Michal Suchánek wrote:
> >>> On Tue, 29 May 2018 12:14:15 +0200
> >>> Max Reitz  wrote:
> >>>   
>  On 2018-05-29 08:44, Kevin Wolf wrote:  
> > Am 28.05.2018 um 23:25 hat Richard W.M. Jones geschrieben:
> >> On Mon, May 28, 2018 at 10:20:54PM +0100, Richard W.M. Jones
> >> wrote:
> >>> On Mon, May 28, 2018 at 08:38:33PM +0200, Kevin Wolf wrote:
>  Just accessing the image file within a tar archive is possible
>  and we could write a block driver for that (I actually think we
>  should do this), but it restricts you because certain
>  operations like resizing aren't really possible in tar.
>  Unfortunately, resizing is a really common operation for
>  non-raw image formats.
> >>>
> >>> We do this already in virt-v2v (using file.offset and file.size
> >>> parameters in the raw driver).
> >>>
> >>> For virt-v2v we only need to read the source so resizing isn't
> >>> an issue.  For most of the cases we're talking about the
> >>> downloaded image would also be a template / base image, so I
> >>> suppose only reading would be required too.
> >>>
> >>> I also wrote an nbdkit tar file driver (supports writes, but not
> >>> resizing).
> >>> https://manpages.debian.org/testing/nbdkit-plugin-perl/nbdkit-tar-plugin.1.en.html
> >>> 
> >>
> >> I should add the other thorny issue with OVA files is that the
> >> metadata contains a checksum (SHA1 or SHA256) of the disk images.
> >> If you modify the disk images in-place in the tar file then you
> >> need to recalculate those.
> >
> > All of this means that OVA isn't really well suited to be used as
> > a native format for VM configuration + images. It's just for
> > sharing read-only images that are converted into another native
> > format before they are used.
> >
> > Which is probably fair for the use case it was made for, but means
> > that we need something else to solve our problem.
> 
>  Maybe we should first narrow down our problem.  Maybe you have done
>  that already, but I'm quite in the dark still.
> 
>  The original problem was that you need to supply a machine type to
>  qemu, and that multiple common architectures now have multiple
>  machine types and not necessarily all work with a single image.  So
>  far so good, but I have two issues here already:
> 
>  (1) How is qemu supposed to interpret that information?  If it's
>  stored in the image file, I don't see a nice way of retrieving it
>  before the machine is initialized, at least not with qemu's current
>  architecture. Once we support configuring qemu solely through QMP,
>  sure, you can do a blockdev-add and then build the machine
>  accordingly.  But that is not here today, and I'm not sure this is
>  a good idea either, because that would mean automagic defaults for
>  the machine-building QMP commands derived from the blockdev-add
>  earlier, which should get a plain "No". Also, having to use QMP to
>  build your machine wouldn't make anything easier; at least not
>  easier than just supplying a configuration file along with the
>  image.
> 
>  (Building the magic into -blockdev might be less horrible, but such
>  magic (adding block devices influences machine defaults) to me
>  still doesn't seem worth not having to supply a config file along
>  with the disk image.)
> 
>  (2) Again, I personally just really don't like saving such
>  information in a disk image.  One actual argument I can bring up
>  for that distaste is this: Suppose, you have multiple images
>  attached to your VM.  Now the VM wants to store the machine type.
>  Where does it go?  Into all of them?  But some of those images may
>  only contain data and might be intended to be shared between
>  multiple VMs.  So those shouldn't receive the mark.  Only disks
>  with binaries should receive them. But what if those binaries are
>  just cross-compiled binaries for some other VM?  Oh no, so not
>  even binaries are a sure indicator...  So I have no idea where the
>  information is supposed to be stored.  In any case, "the first
>  image" just gets an outright "no" from me, and "all images" gets
>  an "I don't think this is a good idea".
> 
>  Loading is fun, too.  OK, so you attach multiple disk images to a
>  VM. Oops, they have varying machine type information...  Now
>  what?  Use the information from the first one?  Definitely no.
>  Just ignore all of the information in such a case and have the
>  user supply the machine type again?  Possible, but it seems weird
>  to me 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 13:19, Michal Suchánek wrote:
> On Wed, 6 Jun 2018 13:02:53 +0200
> Max Reitz  wrote:
> 
>> On 2018-06-06 12:32, Michal Suchánek wrote:
>>> On Tue, 29 May 2018 12:14:15 +0200
>>> Max Reitz  wrote:
>>>   
 On 2018-05-29 08:44, Kevin Wolf wrote:  
> Am 28.05.2018 um 23:25 hat Richard W.M. Jones geschrieben:
>> On Mon, May 28, 2018 at 10:20:54PM +0100, Richard W.M. Jones
>> wrote:
>>> On Mon, May 28, 2018 at 08:38:33PM +0200, Kevin Wolf wrote:
 Just accessing the image file within a tar archive is possible
 and we could write a block driver for that (I actually think we
 should do this), but it restricts you because certain
 operations like resizing aren't really possible in tar.
 Unfortunately, resizing is a really common operation for
 non-raw image formats.
>>>
>>> We do this already in virt-v2v (using file.offset and file.size
>>> parameters in the raw driver).
>>>
>>> For virt-v2v we only need to read the source so resizing isn't
>>> an issue.  For most of the cases we're talking about the
>>> downloaded image would also be a template / base image, so I
>>> suppose only reading would be required too.
>>>
>>> I also wrote an nbdkit tar file driver (supports writes, but not
>>> resizing).
>>> https://manpages.debian.org/testing/nbdkit-plugin-perl/nbdkit-tar-plugin.1.en.html
>>> 
>>
>> I should add the other thorny issue with OVA files is that the
>> metadata contains a checksum (SHA1 or SHA256) of the disk images.
>> If you modify the disk images in-place in the tar file then you
>> need to recalculate those.
>
> All of this means that OVA isn't really well suited to be used as
> a native format for VM configuration + images. It's just for
> sharing read-only images that are converted into another native
> format before they are used.
>
> Which is probably fair for the use case it was made for, but means
> that we need something else to solve our problem.

 Maybe we should first narrow down our problem.  Maybe you have done
 that already, but I'm quite in the dark still.

 The original problem was that you need to supply a machine type to
 qemu, and that multiple common architectures now have multiple
 machine types and not necessarily all work with a single image.  So
 far so good, but I have two issues here already:

 (1) How is qemu supposed to interpret that information?  If it's
 stored in the image file, I don't see a nice way of retrieving it
 before the machine is initialized, at least not with qemu's current
 architecture. Once we support configuring qemu solely through QMP,
 sure, you can do a blockdev-add and then build the machine
 accordingly.  But that is not here today, and I'm not sure this is
 a good idea either, because that would mean automagic defaults for
 the machine-building QMP commands derived from the blockdev-add
 earlier, which should get a plain "No". Also, having to use QMP to
 build your machine wouldn't make anything easier; at least not
 easier than just supplying a configuration file along with the
 image.

 (Building the magic into -blockdev might be less horrible, but such
 magic (adding block devices influences machine defaults) to me
 still doesn't seem worth not having to supply a config file along
 with the disk image.)

 (2) Again, I personally just really don't like saving such
 information in a disk image.  One actual argument I can bring up
 for that distaste is this: Suppose, you have multiple images
 attached to your VM.  Now the VM wants to store the machine type.
 Where does it go?  Into all of them?  But some of those images may
 only contain data and might be intended to be shared between
 multiple VMs.  So those shouldn't receive the mark.  Only disks
 with binaries should receive them. But what if those binaries are
 just cross-compiled binaries for some other VM?  Oh no, so not
 even binaries are a sure indicator...  So I have no idea where the
 information is supposed to be stored.  In any case, "the first
 image" just gets an outright "no" from me, and "all images" gets
 an "I don't think this is a good idea".

 Loading is fun, too.  OK, so you attach multiple disk images to a
 VM. Oops, they have varying machine type information...  Now
 what?  Use the information from the first one?  Definitely no.
 Just ignore all of the information in such a case and have the
 user supply the machine type again?  Possible, but it seems weird
 to me that qemu would usually guess the machine type, but once you
 attach some random other image to it, it suddenly fails to do
 that.  But maybe it's just me who thinks this is weird.


 OK, so let's go a 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> * Max Reitz (mre...@redhat.com) wrote:
>> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
>>> 
>>>
>>> This seems to have fizzled out because of a lack of a concrete proposal;
>>> so here is one based on a reply to Max's post:
>>>
>>> * Max Reitz (mre...@redhat.com) wrote:
>>>
>>> 
>>>
 The original problem was that you need to supply a machine type to qemu,
 and that multiple common architectures now have multiple machine types
 and not necessarily all work with a single image.  So far so good, but I
 have two issues here already:

 (1) How is qemu supposed to interpret that information?  If it's stored
 in the image file, I don't see a nice way of retrieving it before the
 machine is initialized, at least not with qemu's current architecture.
>>>
>>> 
>>>
 (2) Again, I personally just really don't like saving such information
 in a disk image.  One actual argument I can bring up for that distaste
 is this: Suppose, you have multiple images attached to your VM.  Now the
 VM wants to store the machine type.  Where does it go?  Into all of
 them?
>>>
>>> 
>>>
 So I think if we decide to store the machine type, that is kind of a
 slippery slope and then there are good arguments for storing even more
 configuration options in the file, too.  But I really, really don't like
 that.
>>>
>>> 
>>>
 For another, how do we store the data?  key-value seems wrong if we want
 to store everything.  JSON might be fine.  But eventually we just want
 basically a qemu configuration file in there, I would think (which may
 support JSON at some point?).   So basically we would store the data as
 a binary blob and let the rest of qemu do its thing with it.  But then
 please tell me why I fought so valiantly against storing random bitmaps
 in qcow2 files.  I hate the idea of making qcow2 a random archive
 format.  We have tar for that.
>>>
>>> 
>>>
 tl;dr: I really don't get why it's so hard to supply a config file along
 with a qcow2 image.  Is it so hard for people to realize that a VM does
 not only consist of a disk?
>>>
>>> Yes! Because in many cases that's all it needs, and it's ready to run
>>> with no unpacking.
>>
>> It clearly is not, or we would not have this discussion.
>>
>> The disk image is only enough if you want the default values for all of
>> qemu's configuration options, because today (and if I were to decide, in
>> the future, too) disk images do not configure the VM (well, they
>> configure the guest, but not the VM itself).
> 
> The problem with having a separate file is that you either have to copy
> it around with the image 

Which is just an inconvenience.

I understand it is an inconvenience and it would be nice to change it,
but please understand that I do not want qcow2 to become a filesystem
just to relieve an inconvenience.

(Note: I understand that you may not want qcow2 to become a filesystem,
but I do get the impression from others.)

>   or have an archive. If you have an archive
> you have to have an unpacking step which then copies, potentially a lot
> of data taking some reasonable amount of time.

I'm sure this can be optimized, but yes, I get that.

(If you use e.g. tar and store the image data starting on an FS cluster
boundary (64 kB should be more than sufficient), I assume there is a way
to extract that data into a new file without copying anything.)

> Storing a simple bit
> of data with the image avoids that.

It is not a simple bit of data, as evidenced by the discussion about
storing binary blobs and MIME types going on.

>>> I think we should have:
>>>
>>> --
>>> Layer 0:
>>>QCOW provides a way to store a single string of arbitrary (but
>>> limited?) length.
>>>QCOW provides a way to replace the string by a new string.
>>>The original or the new string will be stored after that;
>>>never some mix.
>>>Where a file 'b' has a backing file 'a', 'b' inherits the
>>>string from 'a' unless 'b' has it's own string.
>>>Snapshots inherit their string from the main unless they have
>>>their own string.
>>>
>>> Layer 1:
>>>The string shall always be a JSON 'object'; i.e. of the form
>>> { "something": ... , "more": ... }
>>>
>>>The key strings shall be non-null and non-empty and shall
>>>be unique.
>>>
>>> Layer 2:
>>>'.'s in the key string shall indicate hierarchy
>>
>> I don't understand why we we'd need dotted syntax when we already have
>> JSON, but that's not my issue.
> 
> I think someone earlier in the thread had asked about how we handled
> hierarchy so I added it.
> 
>>>Key strings shall be listed in qemu's 
>>>   docs/specs/qcow-keys.rst
>>>
>>>   that shall indicate their meaning and the meaning and
>>>   valid 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Peter Krempa]

On Wed, Jun 06, 2018 at 13:02:56 +0200, Max Reitz wrote:
> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:

[...]

> >   (I would suggest in layer2 that the keys are sorted, but
> >   that's a pain to do in some json creators)
> >c) Forcing the registry of keys might avoid silly duplication.
> >   We can but hope.
> >d) I've not said it's a libvirt XML file since that seems
> >   a bit prescriptive.
> > 
> > Some initial suggested keys:
> > 
> >"qemu.machine-types": [ "q35", "i440fx" ]
> >"qemu.min-ram-MB": 1024
> 
> I still don't understand why you'd want to put the configuration into
> qcow2 instead of the other way around.
> 
> Or why you'd want to use a single file at all, because as this whole
> thread shows, a disk image alone is clearly not sufficient to describe a VM.

I concur to many points made here. I think it would be wrong to put
anything besides disk-related data in a disk image.

If we want to have a all-in-one VM image, then it should be something
separate from this.

Some more points against squashing all this irrelevant stuff into qcow2
is that if you use it as a disk image only with any higher level
management you don't care about any stored configuration.

In fact in libvirt we'd need a way to prevent any of the data from the
disk image from  being applied to the configuration as it would create
problems.

Also there is a big difference between storing "suggestions" of
configuration and the actual full configuration itself. The
"suggestions" which storage controller or machine type to use are
sometimes helpful and sometimes not. Users may have their own
preference. And besides this we have the libosinfo project which can
provide suggestions.

I think that if the clear separation between a disk image format and an
all-in-one VM file is not kept it will end up in a big mess.


signature.asc
Description: PGP signature

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> > 
> > 
> > This seems to have fizzled out because of a lack of a concrete proposal;
> > so here is one based on a reply to Max's post:
> > 
> > * Max Reitz (mre...@redhat.com) wrote:
> > 
> > 
> > 
> >> The original problem was that you need to supply a machine type to qemu,
> >> and that multiple common architectures now have multiple machine types
> >> and not necessarily all work with a single image.  So far so good, but I
> >> have two issues here already:
> >>
> >> (1) How is qemu supposed to interpret that information?  If it's stored
> >> in the image file, I don't see a nice way of retrieving it before the
> >> machine is initialized, at least not with qemu's current architecture.
> > 
> > 
> > 
> >> (2) Again, I personally just really don't like saving such information
> >> in a disk image.  One actual argument I can bring up for that distaste
> >> is this: Suppose, you have multiple images attached to your VM.  Now the
> >> VM wants to store the machine type.  Where does it go?  Into all of
> >> them?
> > 
> > 
> > 
> >> So I think if we decide to store the machine type, that is kind of a
> >> slippery slope and then there are good arguments for storing even more
> >> configuration options in the file, too.  But I really, really don't like
> >> that.
> > 
> > 
> > 
> >> For another, how do we store the data?  key-value seems wrong if we want
> >> to store everything.  JSON might be fine.  But eventually we just want
> >> basically a qemu configuration file in there, I would think (which may
> >> support JSON at some point?).   So basically we would store the data as
> >> a binary blob and let the rest of qemu do its thing with it.  But then
> >> please tell me why I fought so valiantly against storing random bitmaps
> >> in qcow2 files.  I hate the idea of making qcow2 a random archive
> >> format.  We have tar for that.
> > 
> > 
> > 
> >> tl;dr: I really don't get why it's so hard to supply a config file along
> >> with a qcow2 image.  Is it so hard for people to realize that a VM does
> >> not only consist of a disk?
> > 
> > Yes! Because in many cases that's all it needs, and it's ready to run
> > with no unpacking.
> 
> It clearly is not, or we would not have this discussion.
> 
> The disk image is only enough if you want the default values for all of
> qemu's configuration options, because today (and if I were to decide, in
> the future, too) disk images do not configure the VM (well, they
> configure the guest, but not the VM itself).

The problem with having a separate file is that you either have to copy
it around with the image or have an archive.  If you have an archive
you have to have an unpacking step which then copies, potentially a lot
of data taking some reasonable amount of time.  Storing a simple bit
of data with the image avoids that.

> > I think we should have:
> > 
> > --
> > Layer 0:
> >QCOW provides a way to store a single string of arbitrary (but
> > limited?) length.
> >QCOW provides a way to replace the string by a new string.
> >The original or the new string will be stored after that;
> >never some mix.
> >Where a file 'b' has a backing file 'a', 'b' inherits the
> >string from 'a' unless 'b' has it's own string.
> >Snapshots inherit their string from the main unless they have
> >their own string.
> > 
> > Layer 1:
> >The string shall always be a JSON 'object'; i.e. of the form
> > { "something": ... , "more": ... }
> > 
> >The key strings shall be non-null and non-empty and shall
> >be unique.
> > 
> > Layer 2:
> >'.'s in the key string shall indicate hierarchy
> 
> I don't understand why we we'd need dotted syntax when we already have
> JSON, but that's not my issue.

I think someone earlier in the thread had asked about how we handled
hierarchy so I added it.

> >Key strings shall be listed in qemu's 
> >   docs/specs/qcow-keys.rst
> > 
> >   that shall indicate their meaning and the meaning and
> >   valid formatting of the value associated with the,
> > 
> >Key strings shall start with either:
> >   qemu.   in which case they must be listed in a file in
> >   the qemu source tree
> > 
> >   a reverse dotted name unique to the submitter, they may
> >   be listed in the same file in the source tree, e.g.
> >   com.redhat.
> 
> So this is just another configuration file format.
> 
> > Layer 3:
> >QEMU shall, for a given qcow2 file be able to dump the
> >key values.
> > 
> > Layer 4:
> >On creating a VM by importing a qcow2, a management layer
> >shall inspect the key/values to influence the configuration
> >of the VM created.   Where it imports multiple qcow2's it
> >shall inspect all the files and flag disagreements.
> > 
> >Management layers shall, on creating a qcow2 shall 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-06 12:32, Michal Suchánek wrote:
> On Tue, 29 May 2018 12:14:15 +0200
> Max Reitz  wrote:
> 
>> On 2018-05-29 08:44, Kevin Wolf wrote:
>>> Am 28.05.2018 um 23:25 hat Richard W.M. Jones geschrieben:  
 On Mon, May 28, 2018 at 10:20:54PM +0100, Richard W.M. Jones
 wrote:  
> On Mon, May 28, 2018 at 08:38:33PM +0200, Kevin Wolf wrote:  
>> Just accessing the image file within a tar archive is possible
>> and we could write a block driver for that (I actually think we
>> should do this), but it restricts you because certain operations
>> like resizing aren't really possible in tar. Unfortunately,
>> resizing is a really common operation for non-raw image
>> formats.  
>
> We do this already in virt-v2v (using file.offset and file.size
> parameters in the raw driver).
>
> For virt-v2v we only need to read the source so resizing isn't an
> issue.  For most of the cases we're talking about the downloaded
> image would also be a template / base image, so I suppose only
> reading would be required too.
>
> I also wrote an nbdkit tar file driver (supports writes, but not
> resizing).
> https://manpages.debian.org/testing/nbdkit-plugin-perl/nbdkit-tar-plugin.1.en.html
>   

 I should add the other thorny issue with OVA files is that the
 metadata contains a checksum (SHA1 or SHA256) of the disk images.
 If you modify the disk images in-place in the tar file then you
 need to recalculate those.  
>>>
>>> All of this means that OVA isn't really well suited to be used as a
>>> native format for VM configuration + images. It's just for sharing
>>> read-only images that are converted into another native format
>>> before they are used.
>>>
>>> Which is probably fair for the use case it was made for, but means
>>> that we need something else to solve our problem.  
>>
>> Maybe we should first narrow down our problem.  Maybe you have done
>> that already, but I'm quite in the dark still.
>>
>> The original problem was that you need to supply a machine type to
>> qemu, and that multiple common architectures now have multiple
>> machine types and not necessarily all work with a single image.  So
>> far so good, but I have two issues here already:
>>
>> (1) How is qemu supposed to interpret that information?  If it's
>> stored in the image file, I don't see a nice way of retrieving it
>> before the machine is initialized, at least not with qemu's current
>> architecture. Once we support configuring qemu solely through QMP,
>> sure, you can do a blockdev-add and then build the machine
>> accordingly.  But that is not here today, and I'm not sure this is a
>> good idea either, because that would mean automagic defaults for the
>> machine-building QMP commands derived from the blockdev-add earlier,
>> which should get a plain "No". Also, having to use QMP to build your
>> machine wouldn't make anything easier; at least not easier than just
>> supplying a configuration file along with the image.
>>
>> (Building the magic into -blockdev might be less horrible, but such
>> magic (adding block devices influences machine defaults) to me still
>> doesn't seem worth not having to supply a config file along with the
>> disk image.)
>>
>> (2) Again, I personally just really don't like saving such information
>> in a disk image.  One actual argument I can bring up for that distaste
>> is this: Suppose, you have multiple images attached to your VM.  Now
>> the VM wants to store the machine type.  Where does it go?  Into all
>> of them?  But some of those images may only contain data and might be
>> intended to be shared between multiple VMs.  So those shouldn't
>> receive the mark.  Only disks with binaries should receive them.
>> But what if those binaries are just cross-compiled binaries for some
>> other VM?  Oh no, so not even binaries are a sure indicator...  So I
>> have no idea where the information is supposed to be stored.  In any
>> case, "the first image" just gets an outright "no" from me, and "all
>> images" gets an "I don't think this is a good idea".
>>
>> Loading is fun, too.  OK, so you attach multiple disk images to a VM.
>> Oops, they have varying machine type information...  Now what?  Use
>> the information from the first one?  Definitely no.  Just ignore all
>> of the information in such a case and have the user supply the
>> machine type again?  Possible, but it seems weird to me that qemu
>> would usually guess the machine type, but once you attach some random
>> other image to it, it suddenly fails to do that.  But maybe it's just
>> me who thinks this is weird.
>>
>>
>> OK, so let's go a step further.  We have stored the machine type
>> information in order to not have to supply a config file with the
>> qcow2 image -- because if we did, it could just contain the machine
>> type and that would be it.
>>
>> So to me it follows naturally that just storing the machine type
>> doesn't make much sense 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Max Reitz]

On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> 
> 
> This seems to have fizzled out because of a lack of a concrete proposal;
> so here is one based on a reply to Max's post:
> 
> * Max Reitz (mre...@redhat.com) wrote:
> 
> 
> 
>> The original problem was that you need to supply a machine type to qemu,
>> and that multiple common architectures now have multiple machine types
>> and not necessarily all work with a single image.  So far so good, but I
>> have two issues here already:
>>
>> (1) How is qemu supposed to interpret that information?  If it's stored
>> in the image file, I don't see a nice way of retrieving it before the
>> machine is initialized, at least not with qemu's current architecture.
> 
> 
> 
>> (2) Again, I personally just really don't like saving such information
>> in a disk image.  One actual argument I can bring up for that distaste
>> is this: Suppose, you have multiple images attached to your VM.  Now the
>> VM wants to store the machine type.  Where does it go?  Into all of
>> them?
> 
> 
> 
>> So I think if we decide to store the machine type, that is kind of a
>> slippery slope and then there are good arguments for storing even more
>> configuration options in the file, too.  But I really, really don't like
>> that.
> 
> 
> 
>> For another, how do we store the data?  key-value seems wrong if we want
>> to store everything.  JSON might be fine.  But eventually we just want
>> basically a qemu configuration file in there, I would think (which may
>> support JSON at some point?).   So basically we would store the data as
>> a binary blob and let the rest of qemu do its thing with it.  But then
>> please tell me why I fought so valiantly against storing random bitmaps
>> in qcow2 files.  I hate the idea of making qcow2 a random archive
>> format.  We have tar for that.
> 
> 
> 
>> tl;dr: I really don't get why it's so hard to supply a config file along
>> with a qcow2 image.  Is it so hard for people to realize that a VM does
>> not only consist of a disk?
> 
> Yes! Because in many cases that's all it needs, and it's ready to run
> with no unpacking.

It clearly is not, or we would not have this discussion.

The disk image is only enough if you want the default values for all of
qemu's configuration options, because today (and if I were to decide, in
the future, too) disk images do not configure the VM (well, they
configure the guest, but not the VM itself).

> I think we should have:
> 
> --
> Layer 0:
>QCOW provides a way to store a single string of arbitrary (but
> limited?) length.
>QCOW provides a way to replace the string by a new string.
>The original or the new string will be stored after that;
>never some mix.
>Where a file 'b' has a backing file 'a', 'b' inherits the
>string from 'a' unless 'b' has it's own string.
>Snapshots inherit their string from the main unless they have
>their own string.
> 
> Layer 1:
>The string shall always be a JSON 'object'; i.e. of the form
> { "something": ... , "more": ... }
> 
>The key strings shall be non-null and non-empty and shall
>be unique.
> 
> Layer 2:
>'.'s in the key string shall indicate hierarchy

I don't understand why we we'd need dotted syntax when we already have
JSON, but that's not my issue.

>Key strings shall be listed in qemu's 
>   docs/specs/qcow-keys.rst
> 
>   that shall indicate their meaning and the meaning and
>   valid formatting of the value associated with the,
> 
>Key strings shall start with either:
>   qemu.   in which case they must be listed in a file in
>   the qemu source tree
> 
>   a reverse dotted name unique to the submitter, they may
>   be listed in the same file in the source tree, e.g.
>   com.redhat.

So this is just another configuration file format.

> Layer 3:
>QEMU shall, for a given qcow2 file be able to dump the
>key values.
> 
> Layer 4:
>On creating a VM by importing a qcow2, a management layer
>shall inspect the key/values to influence the configuration
>of the VM created.   Where it imports multiple qcow2's it
>shall inspect all the files and flag disagreements.
> 
>Management layers shall, on creating a qcow2 shall set the
>keys based on the VM the qcow2 is created for.  If the qcow2
>is created as an additional disk for an exisitng VM it's
>fine to leave the string empty (e.g. for a data disk).

This at least solves the issue of where qemu should store the data (qemu
doesn't care), and how qemu should interpret it (not at all).

But I really, really, really do not like storing arbitrary data in qcow2
files.  I hated it badly enough when qemu knew what to do with it, but I
hate it even more when even qemu has no idea what to do with it.

Having a specification of what everything means in the qemu tree makes
things less unbearable, but not to my liking still.

> 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Tue, 29 May 2018 12:14:15 +0200
Max Reitz  wrote:

> On 2018-05-29 08:44, Kevin Wolf wrote:
> > Am 28.05.2018 um 23:25 hat Richard W.M. Jones geschrieben:  
> >> On Mon, May 28, 2018 at 10:20:54PM +0100, Richard W.M. Jones
> >> wrote:  
> >>> On Mon, May 28, 2018 at 08:38:33PM +0200, Kevin Wolf wrote:  
>  Just accessing the image file within a tar archive is possible
>  and we could write a block driver for that (I actually think we
>  should do this), but it restricts you because certain operations
>  like resizing aren't really possible in tar. Unfortunately,
>  resizing is a really common operation for non-raw image
>  formats.  
> >>>
> >>> We do this already in virt-v2v (using file.offset and file.size
> >>> parameters in the raw driver).
> >>>
> >>> For virt-v2v we only need to read the source so resizing isn't an
> >>> issue.  For most of the cases we're talking about the downloaded
> >>> image would also be a template / base image, so I suppose only
> >>> reading would be required too.
> >>>
> >>> I also wrote an nbdkit tar file driver (supports writes, but not
> >>> resizing).
> >>> https://manpages.debian.org/testing/nbdkit-plugin-perl/nbdkit-tar-plugin.1.en.html
> >>>   
> >>
> >> I should add the other thorny issue with OVA files is that the
> >> metadata contains a checksum (SHA1 or SHA256) of the disk images.
> >> If you modify the disk images in-place in the tar file then you
> >> need to recalculate those.  
> > 
> > All of this means that OVA isn't really well suited to be used as a
> > native format for VM configuration + images. It's just for sharing
> > read-only images that are converted into another native format
> > before they are used.
> > 
> > Which is probably fair for the use case it was made for, but means
> > that we need something else to solve our problem.  
> 
> Maybe we should first narrow down our problem.  Maybe you have done
> that already, but I'm quite in the dark still.
> 
> The original problem was that you need to supply a machine type to
> qemu, and that multiple common architectures now have multiple
> machine types and not necessarily all work with a single image.  So
> far so good, but I have two issues here already:
> 
> (1) How is qemu supposed to interpret that information?  If it's
> stored in the image file, I don't see a nice way of retrieving it
> before the machine is initialized, at least not with qemu's current
> architecture. Once we support configuring qemu solely through QMP,
> sure, you can do a blockdev-add and then build the machine
> accordingly.  But that is not here today, and I'm not sure this is a
> good idea either, because that would mean automagic defaults for the
> machine-building QMP commands derived from the blockdev-add earlier,
> which should get a plain "No". Also, having to use QMP to build your
> machine wouldn't make anything easier; at least not easier than just
> supplying a configuration file along with the image.
> 
> (Building the magic into -blockdev might be less horrible, but such
> magic (adding block devices influences machine defaults) to me still
> doesn't seem worth not having to supply a config file along with the
> disk image.)
> 
> (2) Again, I personally just really don't like saving such information
> in a disk image.  One actual argument I can bring up for that distaste
> is this: Suppose, you have multiple images attached to your VM.  Now
> the VM wants to store the machine type.  Where does it go?  Into all
> of them?  But some of those images may only contain data and might be
> intended to be shared between multiple VMs.  So those shouldn't
> receive the mark.  Only disks with binaries should receive them.
> But what if those binaries are just cross-compiled binaries for some
> other VM?  Oh no, so not even binaries are a sure indicator...  So I
> have no idea where the information is supposed to be stored.  In any
> case, "the first image" just gets an outright "no" from me, and "all
> images" gets an "I don't think this is a good idea".
> 
> Loading is fun, too.  OK, so you attach multiple disk images to a VM.
> Oops, they have varying machine type information...  Now what?  Use
> the information from the first one?  Definitely no.  Just ignore all
> of the information in such a case and have the user supply the
> machine type again?  Possible, but it seems weird to me that qemu
> would usually guess the machine type, but once you attach some random
> other image to it, it suddenly fails to do that.  But maybe it's just
> me who thinks this is weird.
> 
> 
> OK, so let's go a step further.  We have stored the machine type
> information in order to not have to supply a config file with the
> qcow2 image -- because if we did, it could just contain the machine
> type and that would be it.
> 
> So to me it follows naturally that just storing the machine type
> doesn't make much sense if we cannot also store more VM configuration
> in a qcow2 file, because I don't see why you 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Eduardo Habkost (ehabk...@redhat.com) wrote:
> On Tue, Jun 05, 2018 at 10:21:59AM +0100, Dr. David Alan Gilbert wrote:
> > 
> > 
> > This seems to have fizzled out because of a lack of a concrete proposal;
> > so here is one based on a reply to Max's post:
> > 
> > * Max Reitz (mre...@redhat.com) wrote:
> > 
> > 
> > 
> > > The original problem was that you need to supply a machine type to qemu,
> > > and that multiple common architectures now have multiple machine types
> > > and not necessarily all work with a single image.  So far so good, but I
> > > have two issues here already:
> > > 
> > > (1) How is qemu supposed to interpret that information?  If it's stored
> > > in the image file, I don't see a nice way of retrieving it before the
> > > machine is initialized, at least not with qemu's current architecture.
> > 
> > 
> > 
> > > (2) Again, I personally just really don't like saving such information
> > > in a disk image.  One actual argument I can bring up for that distaste
> > > is this: Suppose, you have multiple images attached to your VM.  Now the
> > > VM wants to store the machine type.  Where does it go?  Into all of
> > > them?
> > 
> > 
> > 
> > > So I think if we decide to store the machine type, that is kind of a
> > > slippery slope and then there are good arguments for storing even more
> > > configuration options in the file, too.  But I really, really don't like
> > > that.
> > 
> > 
> > 
> > > For another, how do we store the data?  key-value seems wrong if we want
> > > to store everything.  JSON might be fine.  But eventually we just want
> > > basically a qemu configuration file in there, I would think (which may
> > > support JSON at some point?).   So basically we would store the data as
> > > a binary blob and let the rest of qemu do its thing with it.  But then
> > > please tell me why I fought so valiantly against storing random bitmaps
> > > in qcow2 files.  I hate the idea of making qcow2 a random archive
> > > format.  We have tar for that.
> > 
> > 
> > 
> > > tl;dr: I really don't get why it's so hard to supply a config file along
> > > with a qcow2 image.  Is it so hard for people to realize that a VM does
> > > not only consist of a disk?
> > 
> > Yes! Because in many cases that's all it needs, and it's ready to run
> > with no unpacking.
> > 
> > I think we should have:
> > 
> > --
> > Layer 0:
> >QCOW provides a way to store a single string of arbitrary (but
> > limited?) length.
> >QCOW provides a way to replace the string by a new string.
> >The original or the new string will be stored after that;
> >never some mix.
> >Where a file 'b' has a backing file 'a', 'b' inherits the
> >string from 'a' unless 'b' has it's own string.
> >Snapshots inherit their string from the main unless they have
> >their own string.
> > 
> > Layer 1:
> >The string shall always be a JSON 'object'; i.e. of the form
> > { "something": ... , "more": ... }
> > 
> >The key strings shall be non-null and non-empty and shall
> >be unique.
> > 
> 
> I'd prefer layer 0+1 to:
> 
> 1) Allow multiple entries to be stored (implemented by layer 1
>in this proposal)
> 2) Identify each entry with a name (implemented by layer 1 in
>this proposal)
> 3) Allow arbitrary binary data to be stored on an entry
>(not possible with the JSON-based proposal, because JSON
>strings are not blobs, but Unicode strings).
> 4) Make it easy to replace only one entry while keeping others
>intact (not the case here, if all entries are stored in the
>same JSON string)
> 
> I think it would be simpler if layer 0 simply provided a list of
> names/value pairs, where names are ascii strings, and values are
> binary data[1].  It would make layer 1 unnecessary, and allow (3)
> and (4) to happen.
> 
> [1] In other words, Rich's proposal of "named blobs":
> https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html

My reasoning was just one of simplicity; each layer in this is
almost trivial.
The downside to my proposal is it's more expensive if you want
to change a single key; but as yet no one has suggested why we'd
want to do it frequently enough to worry.

My suggestion of JSON was just to try and stop the bikeshedding;
we seem to be putting a lot of effort into inventing a new
storage definition for something that so far we need to hold
one rarely changing short string, which even with all the discussion
has moved upto maybe 3 or 4 rarely changing pieces of data.

Dave


> 
> 
> > Layer 2:
> >'.'s in the key string shall indicate hierarchy
> >
> >Key strings shall be listed in qemu's 
> >   docs/specs/qcow-keys.rst
> > 
> >   that shall indicate their meaning and the meaning and
> >   valid formatting of the value associated with the,
> > 
> >Key strings shall start with either:
> >   qemu.   in which case they must be listed in a file in
> >   

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Dr. David Alan Gilbert]

* Michael S. Tsirkin (m...@redhat.com) wrote:
> On Tue, Jun 05, 2018 at 03:09:17PM -0500, Eric Blake wrote:
> > On 06/05/2018 02:58 PM, Richard W.M. Jones wrote:
> > > > Binary blobs can always be base64 encoded for representation within
> > > > a valid JSON UTF-8 string (and we already have several QMP
> > > > interfaces that utilize base64 encoding to pass through what is
> > > > otherwise invalid UTF-8).  It does inflate things slightly compared
> > > > to a format that allows a raw length coupled with raw data, but that
> > > > is not necessarily a problem.
> > > 
> > > Of course how we represent them externally and/or while
> > > using QMP / qemu-img to store and retrieve them is up for grabs.
> > > Doesn't JSON allow binary to be encoded?  (Knowing how poorly
> > > done JSON is, I wouldn't be surprised if not)
> > 
> > JSON itself does not have a binary primitive; to pass arbitrary data through
> > JSON you have to first encode that data into something like base64 that can
> > then be represented as a UTF-8 string.  For reference, look at
> > qapi/crypto.json and the definition of QCryptoSecretFormat.
> 
> But there isn't a way to figure out that a string is base64, which
> means each application needs to know whether it's a string or
> a binary.
> 
> How about specifying the encoding in the value?
> 
> string value:
> [A-Za-z][^=\0]=S[^\0]*
> 
> base64 value:
> [A-Za-z][^=\0]=B[A-Za-z0-9+/]*
> 
> or the key:
> S[A-Za-z][^=\0]=[^\0]*
> 
> base64 value:
> B[A-Za-z][^=\0]=[A-Za-z0-9+/]*

Why reinvent the wheel yet again?
The world has enough encodings out there, and we're trying to add
something that's a trivial data store for a frankly trivial use.

While I hate json, it's at least standard.

Dave

> ?
> 
> -- 
> MST
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Gerd Hoffmann]

  Hi,

> Based on this proposal for layer 2, it looks like you expect the
> number of keys used on layer 1 to become large.
> 
> I would prefer a solution that expects a very small set of keys
> for layer 0+1, and point to other specifications of how the blob
> can be interpreted for each key.  This way we can experiment with
> different solutions for layers 2-4, instead of deciding on a
> specific format like JSON.

Well, you never know what people will use this for ...

cheers,
  Gerd

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Gerd Hoffmann]

  Hi,

> By binary I actually meant binary.  The idea is you could
> store things like PNG images in them (for icons).

Guess if we go that route we also want store a mine-type for each entry.

cheers,
  Gerd

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michal Suchánek]

On Wed, 23 May 2018 18:35:31 +0200
Markus Armbruster  wrote:

> Eduardo Habkost  writes:
> 
> > On Wed, May 23, 2018 at 01:19:46PM +0200, Markus Armbruster wrote:  
> >> Eduardo Habkost  writes:
> >>   
> >> > On Mon, May 21, 2018 at 07:44:40PM +0100, Daniel P. Berrangé
> >> > wrote:  
> >> >> On Mon, May 21, 2018 at 03:29:28PM -0300, Eduardo Habkost
> >> >> wrote:  
> >> >> > On Sat, May 19, 2018 at 08:05:06AM +0200, Markus Armbruster
> >> >> > wrote:  
...  
> >> >> > The point here is to allow users to simply copy an existing
> >> >> > disk image, and it will contain enough hints for a cloud
> >> >> > stack to choose reasonable defaults for machine-type and disk
> >> >> > type automatically.  Requiring the user to perform a separate
> >> >> > step to encapsulate the disk image in another file format
> >> >> > defeats the whole purpose of the proposal.  
> >> >> 
> >> >> It doesn't have to mean more work for the user - the application
> >> >> that is used to create the image can do that on their behalf.
> >> >> oVirt for example can import/export OVA files, containing OVF
> >> >> metadata. I could imagine virt-manager, and other tools adding
> >> >> export ability without much trouble if this was deemed a
> >> >> desirable thing. Bundling gives ability to have multiple disk
> >> >> images in one archive, which is something OVF does.  
> >> >
> >> > I have the impression that "the application that is used to
> >> > create the image" is a very large set.  It can be virt-manager,
> >> > virt-install, virt-manager, or even QEMU itself.
> >> >
> >> > Today people can simply create a VM on virt-manager, or run QEMU
> >> > manually, and upload the qcow2 image directly from its original
> >> > location (they don't need to copy/export it).  Don't we want the
> >> > same procedure to keep working instead of requiring users to use
> >> > another tool?  
> >> 
...
> >> With OVF, you solve the problem further up the stack: you do
> >> virtual appliances instead of disk images.
> >>   
> >
> > I guess the main problem is that people are already using disk
> > images as if they were virtual appliances.
> >
> > We can tell people to stop doing that and use OVF, but then we
> > won't make anybody's life any easier: publishers of images might
> > need to generate both qcow2 and OVF images if they want it to
> > work with older hosts; consumers will need to find out if they
> > need qcow2 or OVF.  
> 
> I'm afraid providing for "hints" in QCOW2 could only add problems.  To
> pick the right hints, publishers need to predict how future software
> consuming the image will interpret them.  Consumers may have to
> configure their software to interpret hints in various ways.

That depends on the hint.

If you define that the key libvirt-xml contains the libvirt machine
definition it is up to libvirt to define what exactly the hint contains.

It allows exporting a machine from virt-manager or virsh and importing
into same with exact same properties. Looking at it will probably give
you some idea how to configure the VM in another virtualization
solution.

Similarly OpenStack can define different hint for OpenStack VMs. 

You are free to equip you appliance with multiple hints if you want it
to work it on multiple virtualization solutions. It is much more
efficient than providing multiple appliance images that embed the
same disk image and different metadata. Virtualization solutions are
free to implement conversions to ease importing VMs from different
source, even for something as horrible as OVF.

It is a bit limiting, though.

You cannot make an applicance with multiple disks when the metadata is
embedded in a disk image.

Thanks

Michal

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Tue, Jun 05, 2018 at 03:46:45PM -0500, Eric Blake wrote:
> On 06/05/2018 03:28 PM, Michael S. Tsirkin wrote:
> > On Tue, Jun 05, 2018 at 03:09:17PM -0500, Eric Blake wrote:
> > > On 06/05/2018 02:58 PM, Richard W.M. Jones wrote:
> > > > > Binary blobs can always be base64 encoded for representation within
> > > > > a valid JSON UTF-8 string (and we already have several QMP
> > > > > interfaces that utilize base64 encoding to pass through what is
> > > > > otherwise invalid UTF-8).  It does inflate things slightly compared
> > > > > to a format that allows a raw length coupled with raw data, but that
> > > > > is not necessarily a problem.
> > > > 
> > > > Of course how we represent them externally and/or while
> > > > using QMP / qemu-img to store and retrieve them is up for grabs.
> > > > Doesn't JSON allow binary to be encoded?  (Knowing how poorly
> > > > done JSON is, I wouldn't be surprised if not)
> > > 
> > > JSON itself does not have a binary primitive; to pass arbitrary data 
> > > through
> > > JSON you have to first encode that data into something like base64 that 
> > > can
> > > then be represented as a UTF-8 string.  For reference, look at
> > > qapi/crypto.json and the definition of QCryptoSecretFormat.
> > 
> > But there isn't a way to figure out that a string is base64, which
> > means each application needs to know whether it's a string or
> > a binary.
> 
> Other than if the key name is well-known (and interpretation of the value is
> done according to the key, if it is interpreted at all).
> 
> > 
> > How about specifying the encoding in the value?
> > 
> > string value:
> > [A-Za-z][^=\0]=S[^\0]*
> > 
> > base64 value:
> > [A-Za-z][^=\0]=B[A-Za-z0-9+/]*
> > 
> > or the key:
> > S[A-Za-z][^=\0]=[^\0]*
> > 
> > base64 value:
> > B[A-Za-z][^=\0]=[A-Za-z0-9+/]*
> 
> If we're going to express a tuple of , then let's describe
> it as a 3-tuple, rather than trying to overload that into a 2-tuple
> key=value syntax.

It's a 3 tuple, isn't it?

Foobar=Sabc
1--23

"=" marks end of key but we don't have so many types to need a separator
between type and value.  Would you prefer:

Foobar=S:abc

then, just in case we add a huge number of types down the road?


>  After all, while you may want a high-level layer 1
> representation in JSON or something else easy to hand off to other
> processing, there's no requirement that the layer 0 storage in qcow2 can't
> be a struct in some other layout, even if we have to convert between layouts
> for what is stored in the file vs. what is presented to the user.
>
>
> By that argument, a single JSON object (if we insist on storing a single
> JSON object as layer 0) might be:
> 
> [ { "name": "foo", "type": "raw", "value": "bar" },
>   { "name": "quux", "type": "base64", "value": "aGVsbG8=" } ]

I suspect a lot of what we might want to save would be snippets from
QEMU command line, which are all 0-terminated strings but for which
there is no guarantee they are valid JSON or even UTF8 strings.

We could base64 encode them all but that means some of json advantages
(e.g. readability) are gone, and it will be easy for us to get confused
and forget to encode some string which will lead to security issues.
I'd rather use a format that allows zero terminated strings
as 1st class citizens.


> But figuring out how to add a "Header extension type" to
> docs/interop/qcow2.txt, and whether the key/value payload will usefully fit
> in that header extension (where we are tight on spacing, as ALL extensions
> combined must fit in a single cluster), or whether the header should instead
> have an offset field that points to some other cluster in the qcow2 image,

I'm guessing we need an offset, yes.

> is relatively straightforward, and relatively independent of the even bigger
> design question of whether we want to allow qcow2 as an image format to
> expose an arbitrary data store feature and what types of data should go into
> that store.

I agree it can be decided later but it's not orthogonal in that
it only makes sense to argue what exactly we store there after
we have the capability to store *something* :)

I think we can start small and define the format first, extend qemu-img
- either get/set the file so we can pipe it to some other utility, or
teach it to work with keys/values directly.

Next step will be to start defining contents. We already have two things I
know we want to store: the architecture and the machine type are useful
for bootable images.

Poking at these values from qemu and warning user on mismatch, or even
getting the default machine from there if not specified might
be a step after that.

> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.   +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/05/2018 03:28 PM, Michael S. Tsirkin wrote:

On Tue, Jun 05, 2018 at 03:09:17PM -0500, Eric Blake wrote:

On 06/05/2018 02:58 PM, Richard W.M. Jones wrote:

Binary blobs can always be base64 encoded for representation within
a valid JSON UTF-8 string (and we already have several QMP
interfaces that utilize base64 encoding to pass through what is
otherwise invalid UTF-8).  It does inflate things slightly compared
to a format that allows a raw length coupled with raw data, but that
is not necessarily a problem.


Of course how we represent them externally and/or while
using QMP / qemu-img to store and retrieve them is up for grabs.
Doesn't JSON allow binary to be encoded?  (Knowing how poorly
done JSON is, I wouldn't be surprised if not)


JSON itself does not have a binary primitive; to pass arbitrary data through
JSON you have to first encode that data into something like base64 that can
then be represented as a UTF-8 string.  For reference, look at
qapi/crypto.json and the definition of QCryptoSecretFormat.


But there isn't a way to figure out that a string is base64, which
means each application needs to know whether it's a string or
a binary.


Other than if the key name is well-known (and interpretation of the 
value is done according to the key, if it is interpreted at all).




How about specifying the encoding in the value?

string value:
[A-Za-z][^=\0]=S[^\0]*

base64 value:
[A-Za-z][^=\0]=B[A-Za-z0-9+/]*

or the key:
S[A-Za-z][^=\0]=[^\0]*

base64 value:
B[A-Za-z][^=\0]=[A-Za-z0-9+/]*


If we're going to express a tuple of , then let's 
describe it as a 3-tuple, rather than trying to overload that into a 
2-tuple key=value syntax.  After all, while you may want a high-level 
layer 1 representation in JSON or something else easy to hand off to 
other processing, there's no requirement that the layer 0 storage in 
qcow2 can't be a struct in some other layout, even if we have to convert 
between layouts for what is stored in the file vs. what is presented to 
the user.


By that argument, a single JSON object (if we insist on storing a single 
JSON object as layer 0) might be:


[ { "name": "foo", "type": "raw", "value": "bar" },
  { "name": "quux", "type": "base64", "value": "aGVsbG8=" } ]

But figuring out how to add a "Header extension type" to 
docs/interop/qcow2.txt, and whether the key/value payload will usefully 
fit in that header extension (where we are tight on spacing, as ALL 
extensions combined must fit in a single cluster), or whether the header 
should instead have an offset field that points to some other cluster in 
the qcow2 image, is relatively straightforward, and relatively 
independent of the even bigger design question of whether we want to 
allow qcow2 as an image format to expose an arbitrary data store feature 
and what types of data should go into that store.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Tue, Jun 05, 2018 at 03:09:17PM -0500, Eric Blake wrote:
> On 06/05/2018 02:58 PM, Richard W.M. Jones wrote:
> > > Binary blobs can always be base64 encoded for representation within
> > > a valid JSON UTF-8 string (and we already have several QMP
> > > interfaces that utilize base64 encoding to pass through what is
> > > otherwise invalid UTF-8).  It does inflate things slightly compared
> > > to a format that allows a raw length coupled with raw data, but that
> > > is not necessarily a problem.
> > 
> > Of course how we represent them externally and/or while
> > using QMP / qemu-img to store and retrieve them is up for grabs.
> > Doesn't JSON allow binary to be encoded?  (Knowing how poorly
> > done JSON is, I wouldn't be surprised if not)
> 
> JSON itself does not have a binary primitive; to pass arbitrary data through
> JSON you have to first encode that data into something like base64 that can
> then be represented as a UTF-8 string.  For reference, look at
> qapi/crypto.json and the definition of QCryptoSecretFormat.

But there isn't a way to figure out that a string is base64, which
means each application needs to know whether it's a string or
a binary.

How about specifying the encoding in the value?

string value:
[A-Za-z][^=\0]=S[^\0]*

base64 value:
[A-Za-z][^=\0]=B[A-Za-z0-9+/]*

or the key:
S[A-Za-z][^=\0]=[^\0]*

base64 value:
B[A-Za-z][^=\0]=[A-Za-z0-9+/]*

?

-- 
MST

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/05/2018 02:58 PM, Richard W.M. Jones wrote:

Binary blobs can always be base64 encoded for representation within
a valid JSON UTF-8 string (and we already have several QMP
interfaces that utilize base64 encoding to pass through what is
otherwise invalid UTF-8).  It does inflate things slightly compared
to a format that allows a raw length coupled with raw data, but that
is not necessarily a problem.


Of course how we represent them externally and/or while
using QMP / qemu-img to store and retrieve them is up for grabs.
Doesn't JSON allow binary to be encoded?  (Knowing how poorly
done JSON is, I wouldn't be surprised if not)


JSON itself does not have a binary primitive; to pass arbitrary data 
through JSON you have to first encode that data into something like 
base64 that can then be represented as a UTF-8 string.  For reference, 
look at qapi/crypto.json and the definition of QCryptoSecretFormat.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Tue, Jun 05, 2018 at 02:54:07PM -0500, Eric Blake wrote:
> On 06/05/2018 02:47 PM, Michael S. Tsirkin wrote:
> 
> > > > Layer 1:
> > > > The string shall always be a JSON 'object'; i.e. of the form
> > > >  { "something": ... , "more": ... }
> > > > 
> > > > The key strings shall be non-null and non-empty and shall
> > > > be unique.
> 
> > > 
> > > I think it would be simpler if layer 0 simply provided a list of
> > > names/value pairs, where names are ascii strings, and values are
> > > binary data[1].  It would make layer 1 unnecessary, and allow (3)
> > > and (4) to happen.
> > > 
> > > [1] In other words, Rich's proposal of "named blobs":
> > > https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html
> > 
> > I think simple is beautiful, too. But assuming they
> > really are binary how are blobs encoded?
> > Did binary really mean UTF-8 here?
> 
> Binary blobs can always be base64 encoded for representation within a valid
> JSON UTF-8 string (and we already have several QMP interfaces that utilize
> base64 encoding to pass through what is otherwise invalid UTF-8).  It does
> inflate things slightly compared to a format that allows a raw length
> coupled with raw data, but that is not necessarily a problem.

OK so what's proposed here is something like the following:

[A-Za-z][^=\0]*=[^\0]*

Sound reasonable?

How about a fixed key at the beginning to specify the format?

E.g. 16 first bytes:

qcow2-format=00\0

anything else in first 16 bytes means some other format?

> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.   +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Richard W.M. Jones]

On Tue, Jun 05, 2018 at 02:54:07PM -0500, Eric Blake wrote:
> On 06/05/2018 02:47 PM, Michael S. Tsirkin wrote:
> 
> >>>Layer 1:
> >>>The string shall always be a JSON 'object'; i.e. of the form
> >>> { "something": ... , "more": ... }
> >>>
> >>>The key strings shall be non-null and non-empty and shall
> >>>be unique.
> 
> >>
> >>I think it would be simpler if layer 0 simply provided a list of
> >>names/value pairs, where names are ascii strings, and values are
> >>binary data[1].  It would make layer 1 unnecessary, and allow (3)
> >>and (4) to happen.
> >>
> >>[1] In other words, Rich's proposal of "named blobs":
> >>https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html
> >
> >I think simple is beautiful, too. But assuming they
> >really are binary how are blobs encoded?
> >Did binary really mean UTF-8 here?

By binary I actually meant binary.  The idea is you could
store things like PNG images in them (for icons).

> Binary blobs can always be base64 encoded for representation within
> a valid JSON UTF-8 string (and we already have several QMP
> interfaces that utilize base64 encoding to pass through what is
> otherwise invalid UTF-8).  It does inflate things slightly compared
> to a format that allows a raw length coupled with raw data, but that
> is not necessarily a problem.

Of course how we represent them externally and/or while
using QMP / qemu-img to store and retrieve them is up for grabs.
Doesn't JSON allow binary to be encoded?  (Knowing how poorly
done JSON is, I wouldn't be surprised if not)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eric Blake]

On 06/05/2018 02:47 PM, Michael S. Tsirkin wrote:


Layer 1:
The string shall always be a JSON 'object'; i.e. of the form
 { "something": ... , "more": ... }

The key strings shall be non-null and non-empty and shall
be unique.




I think it would be simpler if layer 0 simply provided a list of
names/value pairs, where names are ascii strings, and values are
binary data[1].  It would make layer 1 unnecessary, and allow (3)
and (4) to happen.

[1] In other words, Rich's proposal of "named blobs":
https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html


I think simple is beautiful, too. But assuming they
really are binary how are blobs encoded?
Did binary really mean UTF-8 here?


Binary blobs can always be base64 encoded for representation within a 
valid JSON UTF-8 string (and we already have several QMP interfaces that 
utilize base64 encoding to pass through what is otherwise invalid 
UTF-8).  It does inflate things slightly compared to a format that 
allows a raw length coupled with raw data, but that is not necessarily a 
problem.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Michael S. Tsirkin]

On Tue, Jun 05, 2018 at 04:03:24PM -0300, Eduardo Habkost wrote:
> On Tue, Jun 05, 2018 at 10:21:59AM +0100, Dr. David Alan Gilbert wrote:
> > 
> > 
> > This seems to have fizzled out because of a lack of a concrete proposal;
> > so here is one based on a reply to Max's post:
> > 
> > * Max Reitz (mre...@redhat.com) wrote:
> > 
> > 
> > 
> > > The original problem was that you need to supply a machine type to qemu,
> > > and that multiple common architectures now have multiple machine types
> > > and not necessarily all work with a single image.  So far so good, but I
> > > have two issues here already:
> > > 
> > > (1) How is qemu supposed to interpret that information?  If it's stored
> > > in the image file, I don't see a nice way of retrieving it before the
> > > machine is initialized, at least not with qemu's current architecture.
> > 
> > 
> > 
> > > (2) Again, I personally just really don't like saving such information
> > > in a disk image.  One actual argument I can bring up for that distaste
> > > is this: Suppose, you have multiple images attached to your VM.  Now the
> > > VM wants to store the machine type.  Where does it go?  Into all of
> > > them?
> > 
> > 
> > 
> > > So I think if we decide to store the machine type, that is kind of a
> > > slippery slope and then there are good arguments for storing even more
> > > configuration options in the file, too.  But I really, really don't like
> > > that.
> > 
> > 
> > 
> > > For another, how do we store the data?  key-value seems wrong if we want
> > > to store everything.  JSON might be fine.  But eventually we just want
> > > basically a qemu configuration file in there, I would think (which may
> > > support JSON at some point?).   So basically we would store the data as
> > > a binary blob and let the rest of qemu do its thing with it.  But then
> > > please tell me why I fought so valiantly against storing random bitmaps
> > > in qcow2 files.  I hate the idea of making qcow2 a random archive
> > > format.  We have tar for that.
> > 
> > 
> > 
> > > tl;dr: I really don't get why it's so hard to supply a config file along
> > > with a qcow2 image.  Is it so hard for people to realize that a VM does
> > > not only consist of a disk?
> > 
> > Yes! Because in many cases that's all it needs, and it's ready to run
> > with no unpacking.
> > 
> > I think we should have:
> > 
> > --
> > Layer 0:
> >QCOW provides a way to store a single string of arbitrary (but
> > limited?) length.
> >QCOW provides a way to replace the string by a new string.
> >The original or the new string will be stored after that;
> >never some mix.
> >Where a file 'b' has a backing file 'a', 'b' inherits the
> >string from 'a' unless 'b' has it's own string.
> >Snapshots inherit their string from the main unless they have
> >their own string.
> > 
> > Layer 1:
> >The string shall always be a JSON 'object'; i.e. of the form
> > { "something": ... , "more": ... }
> > 
> >The key strings shall be non-null and non-empty and shall
> >be unique.
> > 
> 
> I'd prefer layer 0+1 to:
> 
> 1) Allow multiple entries to be stored (implemented by layer 1
>in this proposal)
> 2) Identify each entry with a name (implemented by layer 1 in
>this proposal)
> 3) Allow arbitrary binary data to be stored on an entry
>(not possible with the JSON-based proposal, because JSON
>strings are not blobs, but Unicode strings).
> 4) Make it easy to replace only one entry while keeping others
>intact (not the case here, if all entries are stored in the
>same JSON string)
> 
> I think it would be simpler if layer 0 simply provided a list of
> names/value pairs, where names are ascii strings, and values are
> binary data[1].  It would make layer 1 unnecessary, and allow (3)
> and (4) to happen.
> 
> [1] In other words, Rich's proposal of "named blobs":
> https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html

I think simple is beautiful, too. But assuming they
really are binary how are blobs encoded?
Did binary really mean UTF-8 here?

> 
> > Layer 2:
> >'.'s in the key string shall indicate hierarchy
> >
> >Key strings shall be listed in qemu's 
> >   docs/specs/qcow-keys.rst
> > 
> >   that shall indicate their meaning and the meaning and
> >   valid formatting of the value associated with the,
> > 
> >Key strings shall start with either:
> >   qemu.   in which case they must be listed in a file in
> >   the qemu source tree
> > 
> >   a reverse dotted name unique to the submitter, they may
> >   be listed in the same file in the source tree, e.g.
> >   com.redhat.
> > 
> 
> Based on this proposal for layer 2, it looks like you expect the
> number of keys used on layer 1 to become large.
> 
> I would prefer a solution that expects a very small set of keys
> for layer 0+1, and point to other 

Re: [Qemu-block] [Qemu-devel] storing machine data in qcow images?

[Eduardo Habkost]

On Tue, Jun 05, 2018 at 10:21:59AM +0100, Dr. David Alan Gilbert wrote:
> 
> 
> This seems to have fizzled out because of a lack of a concrete proposal;
> so here is one based on a reply to Max's post:
> 
> * Max Reitz (mre...@redhat.com) wrote:
> 
> 
> 
> > The original problem was that you need to supply a machine type to qemu,
> > and that multiple common architectures now have multiple machine types
> > and not necessarily all work with a single image.  So far so good, but I
> > have two issues here already:
> > 
> > (1) How is qemu supposed to interpret that information?  If it's stored
> > in the image file, I don't see a nice way of retrieving it before the
> > machine is initialized, at least not with qemu's current architecture.
> 
> 
> 
> > (2) Again, I personally just really don't like saving such information
> > in a disk image.  One actual argument I can bring up for that distaste
> > is this: Suppose, you have multiple images attached to your VM.  Now the
> > VM wants to store the machine type.  Where does it go?  Into all of
> > them?
> 
> 
> 
> > So I think if we decide to store the machine type, that is kind of a
> > slippery slope and then there are good arguments for storing even more
> > configuration options in the file, too.  But I really, really don't like
> > that.
> 
> 
> 
> > For another, how do we store the data?  key-value seems wrong if we want
> > to store everything.  JSON might be fine.  But eventually we just want
> > basically a qemu configuration file in there, I would think (which may
> > support JSON at some point?).   So basically we would store the data as
> > a binary blob and let the rest of qemu do its thing with it.  But then
> > please tell me why I fought so valiantly against storing random bitmaps
> > in qcow2 files.  I hate the idea of making qcow2 a random archive
> > format.  We have tar for that.
> 
> 
> 
> > tl;dr: I really don't get why it's so hard to supply a config file along
> > with a qcow2 image.  Is it so hard for people to realize that a VM does
> > not only consist of a disk?
> 
> Yes! Because in many cases that's all it needs, and it's ready to run
> with no unpacking.
> 
> I think we should have:
> 
> --
> Layer 0:
>QCOW provides a way to store a single string of arbitrary (but
> limited?) length.
>QCOW provides a way to replace the string by a new string.
>The original or the new string will be stored after that;
>never some mix.
>Where a file 'b' has a backing file 'a', 'b' inherits the
>string from 'a' unless 'b' has it's own string.
>Snapshots inherit their string from the main unless they have
>their own string.
> 
> Layer 1:
>The string shall always be a JSON 'object'; i.e. of the form
> { "something": ... , "more": ... }
> 
>The key strings shall be non-null and non-empty and shall
>be unique.
> 

I'd prefer layer 0+1 to:

1) Allow multiple entries to be stored (implemented by layer 1
   in this proposal)
2) Identify each entry with a name (implemented by layer 1 in
   this proposal)
3) Allow arbitrary binary data to be stored on an entry
   (not possible with the JSON-based proposal, because JSON
   strings are not blobs, but Unicode strings).
4) Make it easy to replace only one entry while keeping others
   intact (not the case here, if all entries are stored in the
   same JSON string)

I think it would be simpler if layer 0 simply provided a list of
names/value pairs, where names are ascii strings, and values are
binary data[1].  It would make layer 1 unnecessary, and allow (3)
and (4) to happen.

[1] In other words, Rich's proposal of "named blobs":
https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html


> Layer 2:
>'.'s in the key string shall indicate hierarchy
>
>Key strings shall be listed in qemu's 
>   docs/specs/qcow-keys.rst
> 
>   that shall indicate their meaning and the meaning and
>   valid formatting of the value associated with the,
> 
>Key strings shall start with either:
>   qemu.   in which case they must be listed in a file in
>   the qemu source tree
> 
>   a reverse dotted name unique to the submitter, they may
>   be listed in the same file in the source tree, e.g.
>   com.redhat.
> 

Based on this proposal for layer 2, it looks like you expect the
number of keys used on layer 1 to become large.

I would prefer a solution that expects a very small set of keys
for layer 0+1, and point to other specifications of how the blob
can be interpreted for each key.  This way we can experiment with
different solutions for layers 2-4, instead of deciding on a
specific format like JSON.


> Layer 3:
>QEMU shall, for a given qcow2 file be able to dump the
>key values.
> 
> Layer 4:
>On creating a VM by importing a qcow2, a management layer
>shall inspect the key/values to influence the configuration
>of the VM created.