* Max Reitz (mre...@redhat.com) wrote:
> On 2018-06-06 13:14, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mre...@redhat.com) wrote:
> >> On 2018-06-05 11:21, Dr. David Alan Gilbert wrote:
> >>> <reawakening a fizzled out thread>
> >>>
> >>> This seems to have fizzled out because of a lack of a concrete proposal;
> >>> so here is one based on a reply to Max's post:
> >>>
> >>> * Max Reitz (mre...@redhat.com) wrote:
> >>>
> >>> <snip>
> >>>
> >>>> The original problem was that you need to supply a machine type to qemu,
> >>>> and that multiple common architectures now have multiple machine types
> >>>> and not necessarily all work with a single image. So far so good, but I
> >>>> have two issues here already:
> >>>>
> >>>> (1) How is qemu supposed to interpret that information? If it's stored
> >>>> in the image file, I don't see a nice way of retrieving it before the
> >>>> machine is initialized, at least not with qemu's current architecture.
> >>>
> >>> <snip>
> >>>
> >>>> (2) Again, I personally just really don't like saving such information
> >>>> in a disk image. One actual argument I can bring up for that distaste
> >>>> is this: Suppose, you have multiple images attached to your VM. Now the
> >>>> VM wants to store the machine type. Where does it go? Into all of
> >>>> them?
> >>>
> >>> <snip>
> >>>
> >>>> So I think if we decide to store the machine type, that is kind of a
> >>>> slippery slope and then there are good arguments for storing even more
> >>>> configuration options in the file, too. But I really, really don't like
> >>>> that.
> >>>
> >>> <snip>
> >>>
> >>>> For another, how do we store the data? key-value seems wrong if we want
> >>>> to store everything. JSON might be fine. But eventually we just want
> >>>> basically a qemu configuration file in there, I would think (which may
> >>>> support JSON at some point?). So basically we would store the data as
> >>>> a binary blob and let the rest of qemu do its thing with it. But then
> >>>> please tell me why I fought so valiantly against storing random bitmaps
> >>>> in qcow2 files. I hate the idea of making qcow2 a random archive
> >>>> format. We have tar for that.
> >>>
> >>> <snip>
> >>>
> >>>> tl;dr: I really don't get why it's so hard to supply a config file along
> >>>> with a qcow2 image. Is it so hard for people to realize that a VM does
> >>>> not only consist of a disk?
> >>>
> >>> Yes! Because in many cases that's all it needs, and it's ready to run
> >>> with no unpacking.
> >>
> >> It clearly is not, or we would not have this discussion.
> >>
> >> The disk image is only enough if you want the default values for all of
> >> qemu's configuration options, because today (and if I were to decide, in
> >> the future, too) disk images do not configure the VM (well, they
> >> configure the guest, but not the VM itself).
> >
> > The problem with having a separate file is that you either have to copy
> > it around with the image
>
> Which is just an inconvenience.
It's more than that; if it's a separate file then the tools can't
rely on users supplying it, and frankly they won't and they'll still
just supply an image.
> I understand it is an inconvenience and it would be nice to change it,
> but please understand that I do not want qcow2 to become a filesystem
> just to relieve an inconvenience.
I very much don't want it to be a filesystem; my reason for writing
down my spec the way I did was to make it clear that the only
thing I want of qcow2 is a single blob, no more; I don't want naming
of the blob or anything else.
> (Note: I understand that you may not want qcow2 to become a filesystem,
> but I do get the impression from others.)
My aim was to specify it to fulfill the requirements that everyone
else had asked for, but still only having one unmodifiable blob in qcow.
> > or have an archive. If you have an archive
> > you have to have an unpacking step which then copies, potentially a lot
> > of data taking some reasonable amount of time.
>
> I'm sure this can be optimized, but yes, I get that.
>
> (If you use e.g. tar and store the image data starting on an FS cluster
> boundary (64 kB should be more than sufficient), I assume there is a way
> to extract that data into a new file without copying anything.)
But then we have to modify all the current things that know how to
handle a qcow2.
> > Storing a simple bit
> > of data with the image avoids that.
>
> It is not a simple bit of data, as evidenced by the discussion about
> storing binary blobs and MIME types going on.
All of the things they've suggested can be done inside that one blob;
even inside the json (or any other structure in that blob).
> >>> I think we should have:
> >>>
> >>> --------------------------------------------------------------
> >>> Layer 0:
> >>> QCOW provides a way to store a single string of arbitrary (but
> >>> limited?) length.
> >>> QCOW provides a way to replace the string by a new string.
> >>> The original or the new string will be stored after that;
> >>> never some mix.
> >>> Where a file 'b' has a backing file 'a', 'b' inherits the
> >>> string from 'a' unless 'b' has it's own string.
> >>> Snapshots inherit their string from the main unless they have
> >>> their own string.
> >>>
> >>> Layer 1:
> >>> The string shall always be a JSON 'object'; i.e. of the form
> >>> { "something": ... , "more": ... }
> >>>
> >>> The key strings shall be non-null and non-empty and shall
> >>> be unique.
> >>>
> >>> Layer 2:
> >>> '.'s in the key string shall indicate hierarchy
> >>
> >> I don't understand why we we'd need dotted syntax when we already have
> >> JSON, but that's not my issue.
> >
> > I think someone earlier in the thread had asked about how we handled
> > hierarchy so I added it.
> >
> >>> Key strings shall be listed in qemu's
> >>> docs/specs/qcow-keys.rst
> >>>
> >>> that shall indicate their meaning and the meaning and
> >>> valid formatting of the value associated with the,
> >>>
> >>> Key strings shall start with either:
> >>> qemu. in which case they must be listed in a file in
> >>> the qemu source tree
> >>>
> >>> a reverse dotted name unique to the submitter, they may
> >>> be listed in the same file in the source tree, e.g.
> >>> com.redhat.
> >>
> >> So this is just another configuration file format.
> >>
> >>> Layer 3:
> >>> QEMU shall, for a given qcow2 file be able to dump the
> >>> key values.
> >>>
> >>> Layer 4:
> >>> On creating a VM by importing a qcow2, a management layer
> >>> shall inspect the key/values to influence the configuration
> >>> of the VM created. Where it imports multiple qcow2's it
> >>> shall inspect all the files and flag disagreements.
> >>>
> >>> Management layers shall, on creating a qcow2 shall set the
> >>> keys based on the VM the qcow2 is created for. If the qcow2
> >>> is created as an additional disk for an exisitng VM it's
> >>> fine to leave the string empty (e.g. for a data disk).
> >>
> >> This at least solves the issue of where qemu should store the data (qemu
> >> doesn't care), and how qemu should interpret it (not at all).
> >>
> >> But I really, really, really do not like storing arbitrary data in qcow2
> >> files. I hated it badly enough when qemu knew what to do with it, but I
> >> hate it even more when even qemu has no idea what to do with it.
> >>
> >> Having a specification of what everything means in the qemu tree makes
> >> things less unbearable, but not to my liking still.
> >
> > Have you said why you hate it so much?
> > Your hate for it seems to be making a simple solution hard.
>
> Because it's a disk image format. Data therein should be relevant to
> the disk image. I see qcow2 as a representation of data stored on a
> physical storage medium.
What we're missing here is the notes scribbled on the sticky label on
the disc; you rarely need them on a physical drive in a computer,
LUNs on a SAN don't need them that much because they have a full
filesystem and don't move about much. Here we're talking about an image
being downloaded or sent between people.
> Some metadata associated directly with that is fine (such as dirty
> bitmaps, backing chains, things like that). But configuring the whole
> VM seems out of scope to me.
>
> Also, making qcow2 a filesystem is not a simple solution.
>
> ...OK, let me back off here, I may be over-interpreting things and
> throwing opinions of different people into one pot.
>
> Maybe you don't want qcow2 to be a filesystem, and you just want to
> store a single binary blob. Well, OK, that's not that bad. But in any
> case, I wouldn't call it a simple solution anymore.
>
> Yes, storing just the machine type somewhere would be possible with a
> simple solution; but as I said (and the whole thread shows since then),
> this is a slippery slope, and suddenly we arrive at storing arbitrary
> binary data (like images?!) along with MIME types. That will not be
> possible with a simple solution anymore, I don't think.
Right; I was thinking we were too far down that slope to get rid
of all of those requirements, but I was trying to force it back to
being a single blob as far as QCOW2 saw it.
> >>> --------------------------------------------------------------
> >>>
> >>>
> >>> Some reasoning:
> >>> a) I've avoided the problem of when QEMU interprets the value
> >>> by ignoring it and giving it to management layers at the point
> >>> of VM import.
> >>
> >> Yes, but in the process you've made it completely opaque to qemu,
> >> basically, which doesn't really make it better for me. Not that
> >> qemu-specific information in qcow2 files would be what I want, but, well.
> >>
> >> But it does solve technical issues, I concede that.
> >>
> >>> b) I hate JSON, but there again nailing down a fixed format
> >>> seems easiest and it makes the job of QCOW easy - a single
> >>> string.
> >>
> >> Not really. The string can be rather long, so you probably don't want
> >> to store it in the image header, and thus it's just a binary blob from
> >> qcow2's perspective, essentially.
> >
> > Yes, but it's a single blob - I'm not asking for multiple keyed blobs
> > or the ability to update individual blobs; just one blob that I can
> > replace.
>
> OK, you aren't, but others seem to be.
>
> Or, well, you call it a single blob. But actually the current ideas
> seem to be to store a rather large configuration tree with binary data
> in that blob, so to me personally there is absolutely no functional
> difference to just storing a tar file in that blob.
>
> So correct me if I'm wrong, but to me it appears that you effectively
> want to store a filesystem in qcow2.[1] Well, that's better than making
> qcow2 the filesystem, but it still appears just the wrong way around to me.
It's different in the sense that what we end up with is still a qcow2;
anything that just handles qcow2's and can pass them through doesn't
need to do anything different; users don't need to do anything
different. No one has to pack/unpack the file.
> [1] Yes, I know that the guest disk already contains an FS. :-P
>
> >>> (I would suggest in layer2 that the keys are sorted, but
> >>> that's a pain to do in some json creators)
> >>> c) Forcing the registry of keys might avoid silly duplication.
> >>> We can but hope.
> >>> d) I've not said it's a libvirt XML file since that seems
> >>> a bit prescriptive.
> >>>
> >>> Some initial suggested keys:
> >>>
> >>> "qemu.machine-types": [ "q35", "i440fx" ]
> >>> "qemu.min-ram-MB": 1024
> >>
> >> I still don't understand why you'd want to put the configuration into
> >> qcow2 instead of the other way around.
> >>
> >> Or why you'd want to use a single file at all, because as this whole
> >> thread shows, a disk image alone is clearly not sufficient to describe a
> >> VM.
> >>
> >> (Or it may be in simple cases, but then that's because you don't need
> >> any configuration.)
> >
> > Because it avoids the unpacking associated with archives.
>
> I'm not talking about unpacking. I'm talking about a potentially new
> format which allows accessing the qcow2 file in-place. It would
> probably be trivial to write a block driver to allow this.
>
> (And as I wrote in my response to Michal, I suspect that tar could
> actually allow this, even though it would probably not be the ideal format.)
As above, I don't think this is trivial; you have to change all the
layers; lets say it was a tar; you'd have to somehow know that you're
importing one of these special tars, you also have to have a tool to
create them; and you have to worry about whether that alignment
is correct for the storage/memory you're using it with.
Dave
> Max
>
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
