Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Fri, Jun 19, 2020 at 04:40:46PM -0600, Alex Williamson wrote: > On Tue, 9 Jun 2020 20:37:31 -0400 > Yan Zhao wrote: > > > On Fri, Jun 05, 2020 at 03:39:50PM +0100, Dr. David Alan Gilbert wrote: > > > > > > I tried to simplify the problem a bit, but we keep going backwards. > > > > > > If > > > > > > the requirement is that potentially any source device can migrate > > > > > > to any > > > > > > target device and we cannot provide any means other than writing an > > > > > > opaque source string into a version attribute on the target and > > > > > > evaluating the result to determine compatibility, then we're > > > > > > requiring > > > > > > userspace to do an exhaustive search to find a potential match. > > > > > > That > > > > > > sucks. > > > > > > > hi Alex and Dave, > > do you think it's good for us to put aside physical devices and mdev > > aggregation > > for the moment, and use Alex's original idea that > > > > + Userspace should regard two mdev devices compatible when ALL of below > > + conditions are met: > > + (0) The mdev devices are of the same type > > + (1) success when reading from migration_version attribute of one mdev > > device. > > + (2) success when writing migration_version string of one mdev device to > > + migration_version attribute of the other mdev device. > > I think Pandora's box is already opened, if we can't articulate how > this solution would evolve to support features that we know are coming, > why should we proceed with this approach? We've already seen interest > in breaking rule (0) in this thread, so we can't focus the solution on > mdev devices. > > Maybe the best we can do is to compare one instance of a device to > another instance of a device, without any capability to predict > compatibility prior to creating devices, in the case on mdev. The > string would need to include not only the device and vendor driver > compatibility, but also anything that has modified the state of the > device, such as creation time or post-creation time configuration. The > user is left on their own for creating a compatible device, or > filtering devices to determine which might be, or which might generate, > compatible devices. It's not much of a solution, I wonder if anyone > would even use it. > > > and what about adding another sysfs attribute for vendors to put > > recommended migration compatible device type. e.g. > > #cat > > /sys/bus/pci/devices/:00:02.0/mdev_supported_types/i915-GVTg_V5_8/migration_compatible_devices > > parent id: 8086 591d > > mdev_type: i915-GVTg_V5_8 > > > > vendors are free to define the format and conent of this > > migration_compatible_devices > > and it's even not to be a full list. > > > > before libvirt or user to do live migration, they have to read and test > > migration_version attributes of src/target devices to check migration > > compatibility. > > AFAICT, free-form, vendor defined attributes are useless to libvirt. > Vendors could already put this information in the description attribute > and have it ignored by userspace tools due to the lack of defined > format. It's also not clear what value this provides when it's > necessarily incomplete, a driver written today cannot know what future > drivers might be compatible with its migration data. Thanks, > hi Alex maybe the problem can be divided into two pieces: (1) how to create/locate two migration compatible devices. For normal users, the most common and safest way to do it is to find a exact duplication of the source device. so for mdev, it's probably to create a target mdev of the same parent pci id, mdev type and creation parameters as the source mdev; and for physical devices, it's to locate a target device of the same pci id as the source device, plus some extra constraints (e.g. the target NVMe device is configured to the same remote device as the source NVMe device; or the target QAT device is supporting equal encryption algorithm set as the source QAT device...). I think a possible solution for this piece is to let vendor drivers provide a creating/locating script to find such exact duplication of source device. Then before libvirt is about to do live migration, it can use this script to create a target vm of exactly duplicated configuration of the source vm. (2) how to identify two devices are migration compatible after they are created and even they are not exactly identical (e.g. their parent devices are of minor difference in hardware SKUs). This identification is necessary even after in step (1) when libvirt has created/located two identical devices and are about to start live migration. Also, users are free to create/configure target devices and use the read-and-test interfaces defined in this series to check if they are live migration compatible. The read and test behavior in this patch set can grant vendor drivers the freedom to decide whether to support migration between only exact identical devices or able to support migration
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Tue, 9 Jun 2020 20:37:31 -0400 Yan Zhao wrote: > On Fri, Jun 05, 2020 at 03:39:50PM +0100, Dr. David Alan Gilbert wrote: > > > > > I tried to simplify the problem a bit, but we keep going backwards. > > > > > If > > > > > the requirement is that potentially any source device can migrate to > > > > > any > > > > > target device and we cannot provide any means other than writing an > > > > > opaque source string into a version attribute on the target and > > > > > evaluating the result to determine compatibility, then we're requiring > > > > > userspace to do an exhaustive search to find a potential match. That > > > > > sucks. > > > > > hi Alex and Dave, > do you think it's good for us to put aside physical devices and mdev > aggregation > for the moment, and use Alex's original idea that > > + Userspace should regard two mdev devices compatible when ALL of below > + conditions are met: > + (0) The mdev devices are of the same type > + (1) success when reading from migration_version attribute of one mdev > device. > + (2) success when writing migration_version string of one mdev device to > + migration_version attribute of the other mdev device. I think Pandora's box is already opened, if we can't articulate how this solution would evolve to support features that we know are coming, why should we proceed with this approach? We've already seen interest in breaking rule (0) in this thread, so we can't focus the solution on mdev devices. Maybe the best we can do is to compare one instance of a device to another instance of a device, without any capability to predict compatibility prior to creating devices, in the case on mdev. The string would need to include not only the device and vendor driver compatibility, but also anything that has modified the state of the device, such as creation time or post-creation time configuration. The user is left on their own for creating a compatible device, or filtering devices to determine which might be, or which might generate, compatible devices. It's not much of a solution, I wonder if anyone would even use it. > and what about adding another sysfs attribute for vendors to put > recommended migration compatible device type. e.g. > #cat > /sys/bus/pci/devices/:00:02.0/mdev_supported_types/i915-GVTg_V5_8/migration_compatible_devices > parent id: 8086 591d > mdev_type: i915-GVTg_V5_8 > > vendors are free to define the format and conent of this > migration_compatible_devices > and it's even not to be a full list. > > before libvirt or user to do live migration, they have to read and test > migration_version attributes of src/target devices to check migration > compatibility. AFAICT, free-form, vendor defined attributes are useless to libvirt. Vendors could already put this information in the description attribute and have it ignored by userspace tools due to the lack of defined format. It's also not clear what value this provides when it's necessarily incomplete, a driver written today cannot know what future drivers might be compatible with its migration data. Thanks, Alex
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Fri, Jun 05, 2020 at 03:39:50PM +0100, Dr. David Alan Gilbert wrote:
> > > > I tried to simplify the problem a bit, but we keep going backwards. If
> > > > the requirement is that potentially any source device can migrate to any
> > > > target device and we cannot provide any means other than writing an
> > > > opaque source string into a version attribute on the target and
> > > > evaluating the result to determine compatibility, then we're requiring
> > > > userspace to do an exhaustive search to find a potential match. That
> > > > sucks.
> > >
hi Alex and Dave,
do you think it's good for us to put aside physical devices and mdev aggregation
for the moment, and use Alex's original idea that
+ Userspace should regard two mdev devices compatible when ALL of below
+ conditions are met:
+ (0) The mdev devices are of the same type
+ (1) success when reading from migration_version attribute of one mdev device.
+ (2) success when writing migration_version string of one mdev device to
+ migration_version attribute of the other mdev device.
and what about adding another sysfs attribute for vendors to put
recommended migration compatible device type. e.g.
#cat
/sys/bus/pci/devices/:00:02.0/mdev_supported_types/i915-GVTg_V5_8/migration_compatible_devices
parent id: 8086 591d
mdev_type: i915-GVTg_V5_8
vendors are free to define the format and conent of this
migration_compatible_devices
and it's even not to be a full list.
before libvirt or user to do live migration, they have to read and test
migration_version attributes of src/target devices to check migration
compatibility.
Thanks
Yan
> > > Why is the mechanism a 'write and test' why isn't it a 'write and ask'?
> > > i.e. the destination tells the driver what type it's received from the
> > > source, and the driver replies with a set of compatible configurations
> > > (in some preferred order).
> >
> > A 'write and ask' interface would imply some sort of session in order
> > to not be racy with concurrent users. More likely this would imply an
> > ioctl interface, which I don't think we have in sysfs. Where do we
> > host this ioctl?
>
> Or one fd?
> f=open()
> write(f, "The ID I want")
> do {
> read(f, ...) -> The IDs we're offering that are compatible
> } while (!eof)
>
> > > It's also not clear to me why the name has to be that opaque;
> > > I agree it's only got to be understood by the driver but that doesn't
> > > seem to be a reason for the driver to make it purposely obfuscated.
> > > I wouldn't expect a user to be able to parse it necessarily; but would
> > > expect something that would be useful for an error message.
> >
> > If the name is not opaque, then we're going to rat hole on the format
> > and the fields and evolving that format for every feature a vendor
> > decides they want the user to be able to parse out of the version
> > string. Then we require a full specification of the string in order
> > that it be parsed according to a standard such that we don't break
> > users inferring features in subtly different ways.
> >
> > This is a lot like the problems with mdev description attributes,
> > libvirt complains they can't use description because there's no
> > standard formatting, but even with two vendors describing the same class
> > of device we don't have an agreed set of things to expose in the
> > description attribute. Thanks,
>
> I'm not suggesting anything in anyway machine parsable; just something
> human readable that you can present in a menu/choice/configuration/error
> message. The text would be down to the vendor, and I'd suggest it start
> with the vendor name just as a disambiguator and to make it obvious when
> we get it grossly wrong.
>
> Dave
>
> > Alex
> --
> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
>
> ___
> intel-gvt-dev mailing list
> intel-gvt-...@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
* Alex Williamson (alex.william...@redhat.com) wrote: > On Fri, 5 Jun 2020 11:22:24 +0100 > "Dr. David Alan Gilbert" wrote: > > > * Alex Williamson (alex.william...@redhat.com) wrote: > > > On Wed, 3 Jun 2020 01:24:43 -0400 > > > Yan Zhao wrote: > > > > > > > On Tue, Jun 02, 2020 at 09:55:28PM -0600, Alex Williamson wrote: > > > > > On Tue, 2 Jun 2020 23:19:48 -0400 > > > > > Yan Zhao wrote: > > > > > > > > > > > On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > > > > > > > On Wed, 29 Apr 2020 20:39:50 -0400 > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan > > > > > > > > Gilbert wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a > > > > > > > > > > > > > > > > > > > > > software compatible interface, so in > > > > > > > > > > > > > > > > > > > > > the case of mdev->mdev migration, > > > > > > > > > > > > > > > > > > > > > doesn't migrating to a different type > > > > > > > > > > > > > > > > > > > > > fail the most basic of compatibility > > > > > > > > > > > > > > > > > > > > > tests that we expect userspace to > > > > > > > > > > > > > > > > > > > > > perform? IOW, if two mdev types are > > > > > > > > > > > > > > > > > > > > > migration compatible, it seems a > > > > > > > > > > > > > > > > > > > > > prerequisite to that is that they > > > > > > > > > > > > > > > > > > > > > provide the same software interface, > > > > > > > > > > > > > > > > > > > > > which means they should be the same > > > > > > > > > > > > > > > > > > > > > mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or > > > > > > > > > > > > > > > > > > > > > phys->mdev, how does a > > > > > > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > > > > > tool begin to even guess what might > > > > > > > > > > > > > > > > > > > > > be compatible? Are we expecting > > > > > > > > > > > > > > > > > > > > > libvirt to probe ever device with > > > > > > > > > > > > > > > > > > > > > this attribute in the system? Is > > > > > > > > > > > > > > > > > > > > > there going to be a new class > > > > > > > > > > > > > > > > > > > > > hierarchy created to enumerate all > > > > > > > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and > > > > > > > > > > > > > > > > > > > > test migration compatible > > > > > > > > > > > > > > > > > > > > between two devices. But I think it's > > > > > > > > > > > > > > > > > > > > not the problem only for > > > > > > > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for > > > > > > > > > > > > > > > > > > > > mdev->mdev, management tool needs > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > first assume that the two mdevs have > > > > > > > > > > > > > > > > > > > > the same type of parent devices > > > > > > > > > > > > > > > > > > > > (e.g.their pciids are equal). > > > > > > > > > > > > > > > > > > > > otherwise, it's still enumerating > > > > > > > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 > > > > > > > > > > > > > > > > > > > > of pdev1; > > > > > > > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 > > > > > > > > > > > > > > > > > > > > of pdev2; > > > > > > > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, > > > > > > > > > > > > > > > > > > > > why not allow migration between > > > > > > > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that > > > > > > > > > > > > > > > > > > > 1/2 of pdev1 is equivalent > > > > > > > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to > > > > > > > > > > > > > > > > > > > allow such thing happen, the best > > > > > > > > > > > > > > > > > > > choice is to report the same mdev type on > > > > > > > > > > > > > > > > > > > both pdev1 and pdev2. > > > > > > > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > > > > > > > the management tool can take advantage of > > > > > > > > > > > > > > > > > > this interface to know if two > > > > > > > > > > > > > > > > > > devices are migration compatible, no matter > > > > > > > > > > > > > > > > > > they are mdevs, non-mdevs, > > > > > > > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not > > > > > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Fri, 5 Jun 2020 11:22:24 +0100 "Dr. David Alan Gilbert" wrote: > * Alex Williamson (alex.william...@redhat.com) wrote: > > On Wed, 3 Jun 2020 01:24:43 -0400 > > Yan Zhao wrote: > > > > > On Tue, Jun 02, 2020 at 09:55:28PM -0600, Alex Williamson wrote: > > > > On Tue, 2 Jun 2020 23:19:48 -0400 > > > > Yan Zhao wrote: > > > > > > > > > On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > > > > > > On Wed, 29 Apr 2020 20:39:50 -0400 > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a > > > > > > > > > > > > > > > > > > > > software compatible interface, so in > > > > > > > > > > > > > > > > > > > > the case of mdev->mdev migration, > > > > > > > > > > > > > > > > > > > > doesn't migrating to a different type > > > > > > > > > > > > > > > > > > > > fail the most basic of compatibility > > > > > > > > > > > > > > > > > > > > tests that we expect userspace to > > > > > > > > > > > > > > > > > > > > perform? IOW, if two mdev types are > > > > > > > > > > > > > > > > > > > > migration compatible, it seems a > > > > > > > > > > > > > > > > > > > > prerequisite to that is that they > > > > > > > > > > > > > > > > > > > > provide the same software interface, > > > > > > > > > > > > > > > > > > > > which means they should be the same > > > > > > > > > > > > > > > > > > > > mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or > > > > > > > > > > > > > > > > > > > > phys->mdev, how does a > > > > > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > > > > tool begin to even guess what might be > > > > > > > > > > > > > > > > > > > > compatible? Are we expecting > > > > > > > > > > > > > > > > > > > > libvirt to probe ever device with this > > > > > > > > > > > > > > > > > > > > attribute in the system? Is > > > > > > > > > > > > > > > > > > > > there going to be a new class hierarchy > > > > > > > > > > > > > > > > > > > > created to enumerate all > > > > > > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and > > > > > > > > > > > > > > > > > > > test migration compatible > > > > > > > > > > > > > > > > > > > between two devices. But I think it's not > > > > > > > > > > > > > > > > > > > the problem only for > > > > > > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for > > > > > > > > > > > > > > > > > > > mdev->mdev, management tool needs > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > first assume that the two mdevs have the > > > > > > > > > > > > > > > > > > > same type of parent devices > > > > > > > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, > > > > > > > > > > > > > > > > > > > it's still enumerating > > > > > > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of > > > > > > > > > > > > > > > > > > > pdev1; > > > > > > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of > > > > > > > > > > > > > > > > > > > pdev2; > > > > > > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why > > > > > > > > > > > > > > > > > > > not allow migration between > > > > > > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that > > > > > > > > > > > > > > > > > > 1/2 of pdev1 is equivalent > > > > > > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow > > > > > > > > > > > > > > > > > > such thing happen, the best > > > > > > > > > > > > > > > > > > choice is to report the same mdev type on > > > > > > > > > > > > > > > > > > both pdev1 and pdev2. > > > > > > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > > > > > > the management tool can take advantage of > > > > > > > > > > > > > > > > > this interface to know if two > > > > > > > > > > > > > > > > > devices are migration compatible, no matter > > > > > > > > > > > > > > > > > they are mdevs, non-mdevs, > > > > > > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), > > > > > > > > > > > > > > > > > current libvirt still > > > > > > > > > > > > > > > > > requires manually generating mdev devices, > > > > > > > > > > > > > > > > > and it just duplicates src vm > > > > > > > > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
* Alex Williamson (alex.william...@redhat.com) wrote: > On Wed, 3 Jun 2020 01:24:43 -0400 > Yan Zhao wrote: > > > On Tue, Jun 02, 2020 at 09:55:28PM -0600, Alex Williamson wrote: > > > On Tue, 2 Jun 2020 23:19:48 -0400 > > > Yan Zhao wrote: > > > > > > > On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > > > > > On Wed, 29 Apr 2020 20:39:50 -0400 > > > > > Yan Zhao wrote: > > > > > > > > > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a > > > > > > > > > > > > > > > > > > > software compatible interface, so in > > > > > > > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't > > > > > > > > > > > > > > > > > > > migrating to a different type > > > > > > > > > > > > > > > > > > > fail the most basic of compatibility > > > > > > > > > > > > > > > > > > > tests that we expect userspace to > > > > > > > > > > > > > > > > > > > perform? IOW, if two mdev types are > > > > > > > > > > > > > > > > > > > migration compatible, it seems a > > > > > > > > > > > > > > > > > > > prerequisite to that is that they provide > > > > > > > > > > > > > > > > > > > the same software interface, > > > > > > > > > > > > > > > > > > > which means they should be the same mdev > > > > > > > > > > > > > > > > > > > type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or > > > > > > > > > > > > > > > > > > > phys->mdev, how does a > > > > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > > > tool begin to even guess what might be > > > > > > > > > > > > > > > > > > > compatible? Are we expecting > > > > > > > > > > > > > > > > > > > libvirt to probe ever device with this > > > > > > > > > > > > > > > > > > > attribute in the system? Is > > > > > > > > > > > > > > > > > > > there going to be a new class hierarchy > > > > > > > > > > > > > > > > > > > created to enumerate all > > > > > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and > > > > > > > > > > > > > > > > > > test migration compatible > > > > > > > > > > > > > > > > > > between two devices. But I think it's not > > > > > > > > > > > > > > > > > > the problem only for > > > > > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for > > > > > > > > > > > > > > > > > > mdev->mdev, management tool needs > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > first assume that the two mdevs have the > > > > > > > > > > > > > > > > > > same type of parent devices > > > > > > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, > > > > > > > > > > > > > > > > > > it's still enumerating > > > > > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of > > > > > > > > > > > > > > > > > > pdev1; > > > > > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of > > > > > > > > > > > > > > > > > > pdev2; > > > > > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why > > > > > > > > > > > > > > > > > > not allow migration between > > > > > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 > > > > > > > > > > > > > > > > > of pdev1 is equivalent > > > > > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow > > > > > > > > > > > > > > > > > such thing happen, the best > > > > > > > > > > > > > > > > > choice is to report the same mdev type on > > > > > > > > > > > > > > > > > both pdev1 and pdev2. > > > > > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > > > > > the management tool can take advantage of this > > > > > > > > > > > > > > > > interface to know if two > > > > > > > > > > > > > > > > devices are migration compatible, no matter > > > > > > > > > > > > > > > > they are mdevs, non-mdevs, > > > > > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), > > > > > > > > > > > > > > > > current libvirt still > > > > > > > > > > > > > > > > requires manually generating mdev devices, and > > > > > > > > > > > > > > > > it just duplicates src vm > > > > > > > > > > > > > > > > configuration to the target vm. > > > > > > > > > > > > > > > > for libvirt, currently it's always phys->phys > > > > > > > > > > > > > > > > and mdev->mdev (and of the > > > > > > > > > > > > > > > > same mdev type). > > > > > > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Wed, 3 Jun 2020 01:24:43 -0400 Yan Zhao wrote: > On Tue, Jun 02, 2020 at 09:55:28PM -0600, Alex Williamson wrote: > > On Tue, 2 Jun 2020 23:19:48 -0400 > > Yan Zhao wrote: > > > > > On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > > > > On Wed, 29 Apr 2020 20:39:50 -0400 > > > > Yan Zhao wrote: > > > > > > > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a software > > > > > > > > > > > > > > > > > > compatible interface, so in > > > > > > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't > > > > > > > > > > > > > > > > > > migrating to a different type > > > > > > > > > > > > > > > > > > fail the most basic of compatibility tests > > > > > > > > > > > > > > > > > > that we expect userspace to > > > > > > > > > > > > > > > > > > perform? IOW, if two mdev types are > > > > > > > > > > > > > > > > > > migration compatible, it seems a > > > > > > > > > > > > > > > > > > prerequisite to that is that they provide > > > > > > > > > > > > > > > > > > the same software interface, > > > > > > > > > > > > > > > > > > which means they should be the same mdev > > > > > > > > > > > > > > > > > > type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or > > > > > > > > > > > > > > > > > > phys->mdev, how does a > > > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > > tool begin to even guess what might be > > > > > > > > > > > > > > > > > > compatible? Are we expecting > > > > > > > > > > > > > > > > > > libvirt to probe ever device with this > > > > > > > > > > > > > > > > > > attribute in the system? Is > > > > > > > > > > > > > > > > > > there going to be a new class hierarchy > > > > > > > > > > > > > > > > > > created to enumerate all > > > > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and test > > > > > > > > > > > > > > > > > migration compatible > > > > > > > > > > > > > > > > > between two devices. But I think it's not the > > > > > > > > > > > > > > > > > problem only for > > > > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for > > > > > > > > > > > > > > > > > mdev->mdev, management tool needs > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > first assume that the two mdevs have the same > > > > > > > > > > > > > > > > > type of parent devices > > > > > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, it's > > > > > > > > > > > > > > > > > still enumerating > > > > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of > > > > > > > > > > > > > > > > > pdev1; > > > > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of > > > > > > > > > > > > > > > > > pdev2; > > > > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why not > > > > > > > > > > > > > > > > > allow migration between > > > > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 > > > > > > > > > > > > > > > > of pdev1 is equivalent > > > > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow > > > > > > > > > > > > > > > > such thing happen, the best > > > > > > > > > > > > > > > > choice is to report the same mdev type on both > > > > > > > > > > > > > > > > pdev1 and pdev2. > > > > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > > > > the management tool can take advantage of this > > > > > > > > > > > > > > > interface to know if two > > > > > > > > > > > > > > > devices are migration compatible, no matter they > > > > > > > > > > > > > > > are mdevs, non-mdevs, > > > > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), > > > > > > > > > > > > > > > current libvirt still > > > > > > > > > > > > > > > requires manually generating mdev devices, and it > > > > > > > > > > > > > > > just duplicates src vm > > > > > > > > > > > > > > > configuration to the target vm. > > > > > > > > > > > > > > > for libvirt, currently it's always phys->phys and > > > > > > > > > > > > > > > mdev->mdev (and of the > > > > > > > > > > > > > > > same mdev type). > > > > > > > > > > > > > > > But it does not justify that hybrid cases should > > > > > > > > > > > > > > > not be allowed. otherwise, > > > > > > > > > > > > > > > why do we need to introduce this > > > > > > > > > > > > > > > migration_version
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Tue, Jun 02, 2020 at 09:55:28PM -0600, Alex Williamson wrote: > On Tue, 2 Jun 2020 23:19:48 -0400 > Yan Zhao wrote: > > > On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > > > On Wed, 29 Apr 2020 20:39:50 -0400 > > > Yan Zhao wrote: > > > > > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert wrote: > > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a software > > > > > > > > > > > > > > > > > compatible interface, so in > > > > > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't > > > > > > > > > > > > > > > > > migrating to a different type > > > > > > > > > > > > > > > > > fail the most basic of compatibility tests > > > > > > > > > > > > > > > > > that we expect userspace to > > > > > > > > > > > > > > > > > perform? IOW, if two mdev types are > > > > > > > > > > > > > > > > > migration compatible, it seems a > > > > > > > > > > > > > > > > > prerequisite to that is that they provide the > > > > > > > > > > > > > > > > > same software interface, > > > > > > > > > > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or > > > > > > > > > > > > > > > > > phys->mdev, how does a > > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > > tool begin to even guess what might be > > > > > > > > > > > > > > > > > compatible? Are we expecting > > > > > > > > > > > > > > > > > libvirt to probe ever device with this > > > > > > > > > > > > > > > > > attribute in the system? Is > > > > > > > > > > > > > > > > > there going to be a new class hierarchy > > > > > > > > > > > > > > > > > created to enumerate all > > > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and test > > > > > > > > > > > > > > > > migration compatible > > > > > > > > > > > > > > > > between two devices. But I think it's not the > > > > > > > > > > > > > > > > problem only for > > > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for mdev->mdev, > > > > > > > > > > > > > > > > management tool needs > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > first assume that the two mdevs have the same > > > > > > > > > > > > > > > > type of parent devices > > > > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, it's > > > > > > > > > > > > > > > > still enumerating > > > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of pdev1; > > > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of pdev2; > > > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why not > > > > > > > > > > > > > > > > allow migration between > > > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 of > > > > > > > > > > > > > > > pdev1 is equivalent > > > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow such > > > > > > > > > > > > > > > thing happen, the best > > > > > > > > > > > > > > > choice is to report the same mdev type on both > > > > > > > > > > > > > > > pdev1 and pdev2. > > > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > > > the management tool can take advantage of this > > > > > > > > > > > > > > interface to know if two > > > > > > > > > > > > > > devices are migration compatible, no matter they > > > > > > > > > > > > > > are mdevs, non-mdevs, > > > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), > > > > > > > > > > > > > > current libvirt still > > > > > > > > > > > > > > requires manually generating mdev devices, and it > > > > > > > > > > > > > > just duplicates src vm > > > > > > > > > > > > > > configuration to the target vm. > > > > > > > > > > > > > > for libvirt, currently it's always phys->phys and > > > > > > > > > > > > > > mdev->mdev (and of the > > > > > > > > > > > > > > same mdev type). > > > > > > > > > > > > > > But it does not justify that hybrid cases should > > > > > > > > > > > > > > not be allowed. otherwise, > > > > > > > > > > > > > > why do we need to introduce this migration_version > > > > > > > > > > > > > > interface and leave > > > > > > > > > > > > > > the judgement of migration compatibility to vendor > > > > > > > > > > > > > > driver? why not simply > > > > > > > > > > > > > > set the criteria to something like "pciids of > > > > > > > > > > > > > > parent devices are equal, > > > > > > > > > > > > > > and mdev types are equal" ? > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Tue, 2 Jun 2020 23:19:48 -0400 Yan Zhao wrote: > On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > > On Wed, 29 Apr 2020 20:39:50 -0400 > > Yan Zhao wrote: > > > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert wrote: > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a software > > > > > > > > > > > > > > > > compatible interface, so in > > > > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't > > > > > > > > > > > > > > > > migrating to a different type > > > > > > > > > > > > > > > > fail the most basic of compatibility tests that > > > > > > > > > > > > > > > > we expect userspace to > > > > > > > > > > > > > > > > perform? IOW, if two mdev types are migration > > > > > > > > > > > > > > > > compatible, it seems a > > > > > > > > > > > > > > > > prerequisite to that is that they provide the > > > > > > > > > > > > > > > > same software interface, > > > > > > > > > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or > > > > > > > > > > > > > > > > phys->mdev, how does a > > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > > tool begin to even guess what might be > > > > > > > > > > > > > > > > compatible? Are we expecting > > > > > > > > > > > > > > > > libvirt to probe ever device with this > > > > > > > > > > > > > > > > attribute in the system? Is > > > > > > > > > > > > > > > > there going to be a new class hierarchy created > > > > > > > > > > > > > > > > to enumerate all > > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and test > > > > > > > > > > > > > > > migration compatible > > > > > > > > > > > > > > > between two devices. But I think it's not the > > > > > > > > > > > > > > > problem only for > > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for mdev->mdev, > > > > > > > > > > > > > > > management tool needs > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > first assume that the two mdevs have the same > > > > > > > > > > > > > > > type of parent devices > > > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, it's > > > > > > > > > > > > > > > still enumerating > > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of pdev1; > > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of pdev2; > > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why not > > > > > > > > > > > > > > > allow migration between > > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 of > > > > > > > > > > > > > > pdev1 is equivalent > > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow such > > > > > > > > > > > > > > thing happen, the best > > > > > > > > > > > > > > choice is to report the same mdev type on both > > > > > > > > > > > > > > pdev1 and pdev2. > > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > > the management tool can take advantage of this > > > > > > > > > > > > > interface to know if two > > > > > > > > > > > > > devices are migration compatible, no matter they are > > > > > > > > > > > > > mdevs, non-mdevs, > > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), current > > > > > > > > > > > > > libvirt still > > > > > > > > > > > > > requires manually generating mdev devices, and it > > > > > > > > > > > > > just duplicates src vm > > > > > > > > > > > > > configuration to the target vm. > > > > > > > > > > > > > for libvirt, currently it's always phys->phys and > > > > > > > > > > > > > mdev->mdev (and of the > > > > > > > > > > > > > same mdev type). > > > > > > > > > > > > > But it does not justify that hybrid cases should not > > > > > > > > > > > > > be allowed. otherwise, > > > > > > > > > > > > > why do we need to introduce this migration_version > > > > > > > > > > > > > interface and leave > > > > > > > > > > > > > the judgement of migration compatibility to vendor > > > > > > > > > > > > > driver? why not simply > > > > > > > > > > > > > set the criteria to something like "pciids of parent > > > > > > > > > > > > > devices are equal, > > > > > > > > > > > > > and mdev types are equal" ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > btw mdev<->phys just brings trouble to upper stack > > > > > > > > > > > > > > as Alex pointed out. > > > > > > > > > > > > > could you help me
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Tue, Jun 02, 2020 at 04:55:27PM -0600, Alex Williamson wrote: > On Wed, 29 Apr 2020 20:39:50 -0400 > Yan Zhao wrote: > > > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert wrote: > > > > > > > > > > > > > > > > > An mdev type is meant to define a software > > > > > > > > > > > > > > > compatible interface, so in > > > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't > > > > > > > > > > > > > > > migrating to a different type > > > > > > > > > > > > > > > fail the most basic of compatibility tests that > > > > > > > > > > > > > > > we expect userspace to > > > > > > > > > > > > > > > perform? IOW, if two mdev types are migration > > > > > > > > > > > > > > > compatible, it seems a > > > > > > > > > > > > > > > prerequisite to that is that they provide the > > > > > > > > > > > > > > > same software interface, > > > > > > > > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or phys->mdev, > > > > > > > > > > > > > > > how does a > > > > > > > > > > > > > > management > > > > > > > > > > > > > > > tool begin to even guess what might be > > > > > > > > > > > > > > > compatible? Are we expecting > > > > > > > > > > > > > > > libvirt to probe ever device with this attribute > > > > > > > > > > > > > > > in the system? Is > > > > > > > > > > > > > > > there going to be a new class hierarchy created > > > > > > > > > > > > > > > to enumerate all > > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and test > > > > > > > > > > > > > > migration compatible > > > > > > > > > > > > > > between two devices. But I think it's not the > > > > > > > > > > > > > > problem only for > > > > > > > > > > > > > > mdev->phys or phys->mdev. even for mdev->mdev, > > > > > > > > > > > > > > management tool needs > > > > > > > > > > > > > > to > > > > > > > > > > > > > > first assume that the two mdevs have the same type > > > > > > > > > > > > > > of parent devices > > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, it's still > > > > > > > > > > > > > > enumerating > > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of pdev1; > > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of pdev2; > > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why not allow > > > > > > > > > > > > > > migration between > > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 of > > > > > > > > > > > > > pdev1 is equivalent > > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow such > > > > > > > > > > > > > thing happen, the best > > > > > > > > > > > > > choice is to report the same mdev type on both pdev1 > > > > > > > > > > > > > and pdev2. > > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > > migration_version interface. > > > > > > > > > > > > the management tool can take advantage of this > > > > > > > > > > > > interface to know if two > > > > > > > > > > > > devices are migration compatible, no matter they are > > > > > > > > > > > > mdevs, non-mdevs, > > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), current > > > > > > > > > > > > libvirt still > > > > > > > > > > > > requires manually generating mdev devices, and it just > > > > > > > > > > > > duplicates src vm > > > > > > > > > > > > configuration to the target vm. > > > > > > > > > > > > for libvirt, currently it's always phys->phys and > > > > > > > > > > > > mdev->mdev (and of the > > > > > > > > > > > > same mdev type). > > > > > > > > > > > > But it does not justify that hybrid cases should not be > > > > > > > > > > > > allowed. otherwise, > > > > > > > > > > > > why do we need to introduce this migration_version > > > > > > > > > > > > interface and leave > > > > > > > > > > > > the judgement of migration compatibility to vendor > > > > > > > > > > > > driver? why not simply > > > > > > > > > > > > set the criteria to something like "pciids of parent > > > > > > > > > > > > devices are equal, > > > > > > > > > > > > and mdev types are equal" ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > btw mdev<->phys just brings trouble to upper stack as > > > > > > > > > > > > > Alex pointed out. > > > > > > > > > > > > could you help me understand why it will bring trouble > > > > > > > > > > > > to upper stack? > > > > > > > > > > > > > > > > > > > > > > > > I think it just needs to read src migration_version > > > > > > > > > > > > under src dev node, > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Wed, 29 Apr 2020 20:39:50 -0400 Yan Zhao wrote: > On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert wrote: > > > > > > > > > > > > > > > An mdev type is meant to define a software > > > > > > > > > > > > > > compatible interface, so in > > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't migrating > > > > > > > > > > > > > > to a different type > > > > > > > > > > > > > > fail the most basic of compatibility tests that we > > > > > > > > > > > > > > expect userspace to > > > > > > > > > > > > > > perform? IOW, if two mdev types are migration > > > > > > > > > > > > > > compatible, it seems a > > > > > > > > > > > > > > prerequisite to that is that they provide the same > > > > > > > > > > > > > > software interface, > > > > > > > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or phys->mdev, > > > > > > > > > > > > > > how does a > > > > > > > > > > > > > management > > > > > > > > > > > > > > tool begin to even guess what might be compatible? > > > > > > > > > > > > > > Are we expecting > > > > > > > > > > > > > > libvirt to probe ever device with this attribute in > > > > > > > > > > > > > > the system? Is > > > > > > > > > > > > > > there going to be a new class hierarchy created to > > > > > > > > > > > > > > enumerate all > > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and test > > > > > > > > > > > > > migration compatible > > > > > > > > > > > > > between two devices. But I think it's not the problem > > > > > > > > > > > > > only for > > > > > > > > > > > > > mdev->phys or phys->mdev. even for mdev->mdev, > > > > > > > > > > > > > management tool needs > > > > > > > > > > > > > to > > > > > > > > > > > > > first assume that the two mdevs have the same type of > > > > > > > > > > > > > parent devices > > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, it's still > > > > > > > > > > > > > enumerating > > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of pdev1; > > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of pdev2; > > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why not allow > > > > > > > > > > > > > migration between > > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 of pdev1 > > > > > > > > > > > > is equivalent > > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow such thing > > > > > > > > > > > > happen, the best > > > > > > > > > > > > choice is to report the same mdev type on both pdev1 > > > > > > > > > > > > and pdev2. > > > > > > > > > > > I think that's exactly the value of this > > > > > > > > > > > migration_version interface. > > > > > > > > > > > the management tool can take advantage of this interface > > > > > > > > > > > to know if two > > > > > > > > > > > devices are migration compatible, no matter they are > > > > > > > > > > > mdevs, non-mdevs, > > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), current > > > > > > > > > > > libvirt still > > > > > > > > > > > requires manually generating mdev devices, and it just > > > > > > > > > > > duplicates src vm > > > > > > > > > > > configuration to the target vm. > > > > > > > > > > > for libvirt, currently it's always phys->phys and > > > > > > > > > > > mdev->mdev (and of the > > > > > > > > > > > same mdev type). > > > > > > > > > > > But it does not justify that hybrid cases should not be > > > > > > > > > > > allowed. otherwise, > > > > > > > > > > > why do we need to introduce this migration_version > > > > > > > > > > > interface and leave > > > > > > > > > > > the judgement of migration compatibility to vendor > > > > > > > > > > > driver? why not simply > > > > > > > > > > > set the criteria to something like "pciids of parent > > > > > > > > > > > devices are equal, > > > > > > > > > > > and mdev types are equal" ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > btw mdev<->phys just brings trouble to upper stack as > > > > > > > > > > > > Alex pointed out. > > > > > > > > > > > could you help me understand why it will bring trouble to > > > > > > > > > > > upper stack? > > > > > > > > > > > > > > > > > > > > > > I think it just needs to read src migration_version under > > > > > > > > > > > src dev node, > > > > > > > > > > > and test it in target migration version under target dev > > > > > > > > > > > node. > > > > > > > > > > > > > > > > > > > > > > after all, through this interface we just help the upper > > > > > > > > > > > layer > > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Wed, Apr 29, 2020 at 10:13:01PM +0800, Eric Blake wrote: > [meta-comment] > > On 4/29/20 4:35 AM, Yan Zhao wrote: > > On Wed, Apr 29, 2020 at 04:22:01PM +0800, Dr. David Alan Gilbert wrote: > [...] > > This patchset introduces a migration_version attribute > > under sysfs > >>> of VFIO > > Mediated devices. > > Hmm, several pages with up to 16 levels of quoting, with editors making > the lines ragged, all before I get to the real meat of the email. > Remember, it's okay to trim content,... > > >> So why don't we split the difference; lets say that it should start with > >> the hex PCI Vendor ID. > >> > > The problem is for mdev devices, if the parent devices are not PCI devices, > > they don't have PCI vendor IDs. > > ...to just what you are replying to. > sorry for that. next time I'll try to make a better balance between keeping conversation background and leaving the real meat of the email. Thanks for reminding. Yan
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Wed, Apr 29, 2020 at 05:48:44PM +0800, Dr. David Alan Gilbert wrote: > > > > > > > > > > > > > An mdev type is meant to define a software compatible > > > > > > > > > > > > > interface, so in > > > > > > > > > > > > > the case of mdev->mdev migration, doesn't migrating > > > > > > > > > > > > > to a different type > > > > > > > > > > > > > fail the most basic of compatibility tests that we > > > > > > > > > > > > > expect userspace to > > > > > > > > > > > > > perform? IOW, if two mdev types are migration > > > > > > > > > > > > > compatible, it seems a > > > > > > > > > > > > > prerequisite to that is that they provide the same > > > > > > > > > > > > > software interface, > > > > > > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or phys->mdev, how > > > > > > > > > > > > > does a > > > > > > > > > > > > management > > > > > > > > > > > > > tool begin to even guess what might be compatible? > > > > > > > > > > > > > Are we expecting > > > > > > > > > > > > > libvirt to probe ever device with this attribute in > > > > > > > > > > > > > the system? Is > > > > > > > > > > > > > there going to be a new class hierarchy created to > > > > > > > > > > > > > enumerate all > > > > > > > > > > > > > possible migrate-able devices? > > > > > > > > > > > > > > > > > > > > > > > > > yes, management tool needs to guess and test migration > > > > > > > > > > > > compatible > > > > > > > > > > > > between two devices. But I think it's not the problem > > > > > > > > > > > > only for > > > > > > > > > > > > mdev->phys or phys->mdev. even for mdev->mdev, > > > > > > > > > > > > management tool needs > > > > > > > > > > > > to > > > > > > > > > > > > first assume that the two mdevs have the same type of > > > > > > > > > > > > parent devices > > > > > > > > > > > > (e.g.their pciids are equal). otherwise, it's still > > > > > > > > > > > > enumerating > > > > > > > > > > > > possibilities. > > > > > > > > > > > > > > > > > > > > > > > > on the other hand, for two mdevs, > > > > > > > > > > > > mdev1 from pdev1, its mdev_type is 1/2 of pdev1; > > > > > > > > > > > > mdev2 from pdev2, its mdev_type is 1/4 of pdev2; > > > > > > > > > > > > if pdev2 is exactly 2 times of pdev1, why not allow > > > > > > > > > > > > migration between > > > > > > > > > > > > mdev1 <-> mdev2. > > > > > > > > > > > > > > > > > > > > > > How could the manage tool figure out that 1/2 of pdev1 is > > > > > > > > > > > equivalent > > > > > > > > > > > to 1/4 of pdev2? If we really want to allow such thing > > > > > > > > > > > happen, the best > > > > > > > > > > > choice is to report the same mdev type on both pdev1 and > > > > > > > > > > > pdev2. > > > > > > > > > > I think that's exactly the value of this migration_version > > > > > > > > > > interface. > > > > > > > > > > the management tool can take advantage of this interface to > > > > > > > > > > know if two > > > > > > > > > > devices are migration compatible, no matter they are mdevs, > > > > > > > > > > non-mdevs, > > > > > > > > > > or mix. > > > > > > > > > > > > > > > > > > > > as I know, (please correct me if not right), current > > > > > > > > > > libvirt still > > > > > > > > > > requires manually generating mdev devices, and it just > > > > > > > > > > duplicates src vm > > > > > > > > > > configuration to the target vm. > > > > > > > > > > for libvirt, currently it's always phys->phys and > > > > > > > > > > mdev->mdev (and of the > > > > > > > > > > same mdev type). > > > > > > > > > > But it does not justify that hybrid cases should not be > > > > > > > > > > allowed. otherwise, > > > > > > > > > > why do we need to introduce this migration_version > > > > > > > > > > interface and leave > > > > > > > > > > the judgement of migration compatibility to vendor driver? > > > > > > > > > > why not simply > > > > > > > > > > set the criteria to something like "pciids of parent > > > > > > > > > > devices are equal, > > > > > > > > > > and mdev types are equal" ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > btw mdev<->phys just brings trouble to upper stack as > > > > > > > > > > > Alex pointed out. > > > > > > > > > > could you help me understand why it will bring trouble to > > > > > > > > > > upper stack? > > > > > > > > > > > > > > > > > > > > I think it just needs to read src migration_version under > > > > > > > > > > src dev node, > > > > > > > > > > and test it in target migration version under target dev > > > > > > > > > > node. > > > > > > > > > > > > > > > > > > > > after all, through this interface we just help the upper > > > > > > > > > > layer > > > > > > > > > > knowing available options through reading and testing, and > > > > > > > > > > they decide > > > > > > > > > > to use it or not. > > > > > > > > > > > > > > > > > > > > > Can we simplify the requirement by allowing only > > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
[meta-comment] On 4/29/20 4:35 AM, Yan Zhao wrote: On Wed, Apr 29, 2020 at 04:22:01PM +0800, Dr. David Alan Gilbert wrote: [...] This patchset introduces a migration_version attribute under sysfs of VFIO Mediated devices. Hmm, several pages with up to 16 levels of quoting, with editors making the lines ragged, all before I get to the real meat of the email. Remember, it's okay to trim content,... So why don't we split the difference; lets say that it should start with the hex PCI Vendor ID. The problem is for mdev devices, if the parent devices are not PCI devices, they don't have PCI vendor IDs. ...to just what you are replying to. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
* Yan Zhao (yan.y.z...@intel.com) wrote: > On Wed, Apr 29, 2020 at 04:22:01PM +0800, Dr. David Alan Gilbert wrote: > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > On Tue, Apr 28, 2020 at 10:14:37PM +0800, Dr. David Alan Gilbert wrote: > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > On Mon, Apr 27, 2020 at 11:37:43PM +0800, Dr. David Alan Gilbert > > > > > wrote: > > > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > > > On Sat, Apr 25, 2020 at 03:10:49AM +0800, Dr. David Alan Gilbert > > > > > > > wrote: > > > > > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > > > > > On Tue, Apr 21, 2020 at 08:08:49PM +0800, Tian, Kevin wrote: > > > > > > > > > > > From: Yan Zhao > > > > > > > > > > > Sent: Tuesday, April 21, 2020 10:37 AM > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 21, 2020 at 06:56:00AM +0800, Alex Williamson > > > > > > > > > > > wrote: > > > > > > > > > > > > On Sun, 19 Apr 2020 21:24:57 -0400 > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 07:24:57PM +0800, Cornelia > > > > > > > > > > > > > Huck wrote: > > > > > > > > > > > > > > On Fri, 17 Apr 2020 05:52:02 -0400 > > > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 04:44:50PM +0800, > > > > > > > > > > > > > > > Cornelia Huck wrote: > > > > > > > > > > > > > > > > On Mon, 13 Apr 2020 01:52:01 -0400 > > > > > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This patchset introduces a migration_version > > > > > > > > > > > > > > > > > attribute under sysfs > > > > > > > > > > > of VFIO > > > > > > > > > > > > > > > > > Mediated devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This migration_version attribute is used to > > > > > > > > > > > > > > > > > check migration > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > between two mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, it has two locations: > > > > > > > > > > > > > > > > > (1) under mdev_type node, > > > > > > > > > > > > > > > > > which can be used even before device > > > > > > > > > > > > > > > > > creation, but only for > > > > > > > > > > > mdev > > > > > > > > > > > > > > > > > devices of the same mdev type. > > > > > > > > > > > > > > > > > (2) under mdev device node, > > > > > > > > > > > > > > > > > which can only be used after the mdev > > > > > > > > > > > > > > > > > devices are created, but > > > > > > > > > > > the src > > > > > > > > > > > > > > > > > and target mdev devices are not > > > > > > > > > > > > > > > > > necessarily be of the same > > > > > > > > > > > mdev type > > > > > > > > > > > > > > > > > (The second location is newly added in v5, in > > > > > > > > > > > > > > > > > order to keep > > > > > > > > > > > consistent > > > > > > > > > > > > > > > > > with the migration_version node for > > > > > > > > > > > > > > > > > migratable pass-though > > > > > > > > > > > devices) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is the relationship between those two > > > > > > > > > > > > > > > > attributes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (1) is for mdev devices specifically, and (2) is > > > > > > > > > > > > > > > provided to keep the > > > > > > > > > > > same > > > > > > > > > > > > > > > sysfs interface as with non-mdev cases. so (2) is > > > > > > > > > > > > > > > for both mdev > > > > > > > > > > > devices and > > > > > > > > > > > > > > > non-mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in future, if we enable vfio-pci vendor ops, > > > > > > > > > > > > > > > (i.e. a non-mdev device > > > > > > > > > > > > > > > is binding to vfio-pci, but is able to register > > > > > > > > > > > > > > > migration region and do > > > > > > > > > > > > > > > migration transactions from a vendor provided > > > > > > > > > > > > > > > affiliate driver), > > > > > > > > > > > > > > > the vendor driver would export (2) directly, > > > > > > > > > > > > > > > under device node. > > > > > > > > > > > > > > > It is not able to provide (1) as there're no mdev > > > > > > > > > > > > > > > devices involved. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, creating an alternate attribute for non-mdev > > > > > > > > > > > > > > devices makes sense. > > > > > > > > > > > > > > However, wouldn't that rather be a case (3)? The > > > > > > > > > > > > > > change here only > > > > > > > > > > > > > > refers to mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > as you pointed below, (3) and (2) serve the same > > > > > > > > > > > > > purpose. > > > > > > > > > > > > > and I think a possible usage is to migrate between a > > > > > > > > > > > > > non-mdev device and > > > > > > > > > > > > > an mdev device. so I think
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Wed, Apr 29, 2020 at 04:22:01PM +0800, Dr. David Alan Gilbert wrote: > * Yan Zhao (yan.y.z...@intel.com) wrote: > > On Tue, Apr 28, 2020 at 10:14:37PM +0800, Dr. David Alan Gilbert wrote: > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > On Mon, Apr 27, 2020 at 11:37:43PM +0800, Dr. David Alan Gilbert wrote: > > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > > On Sat, Apr 25, 2020 at 03:10:49AM +0800, Dr. David Alan Gilbert > > > > > > wrote: > > > > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > > > > On Tue, Apr 21, 2020 at 08:08:49PM +0800, Tian, Kevin wrote: > > > > > > > > > > From: Yan Zhao > > > > > > > > > > Sent: Tuesday, April 21, 2020 10:37 AM > > > > > > > > > > > > > > > > > > > > On Tue, Apr 21, 2020 at 06:56:00AM +0800, Alex Williamson > > > > > > > > > > wrote: > > > > > > > > > > > On Sun, 19 Apr 2020 21:24:57 -0400 > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 07:24:57PM +0800, Cornelia Huck > > > > > > > > > > > > wrote: > > > > > > > > > > > > > On Fri, 17 Apr 2020 05:52:02 -0400 > > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 04:44:50PM +0800, Cornelia > > > > > > > > > > > > > > Huck wrote: > > > > > > > > > > > > > > > On Mon, 13 Apr 2020 01:52:01 -0400 > > > > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This patchset introduces a migration_version > > > > > > > > > > > > > > > > attribute under sysfs > > > > > > > > > > of VFIO > > > > > > > > > > > > > > > > Mediated devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This migration_version attribute is used to > > > > > > > > > > > > > > > > check migration > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > between two mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, it has two locations: > > > > > > > > > > > > > > > > (1) under mdev_type node, > > > > > > > > > > > > > > > > which can be used even before device > > > > > > > > > > > > > > > > creation, but only for > > > > > > > > > > mdev > > > > > > > > > > > > > > > > devices of the same mdev type. > > > > > > > > > > > > > > > > (2) under mdev device node, > > > > > > > > > > > > > > > > which can only be used after the mdev > > > > > > > > > > > > > > > > devices are created, but > > > > > > > > > > the src > > > > > > > > > > > > > > > > and target mdev devices are not necessarily > > > > > > > > > > > > > > > > be of the same > > > > > > > > > > mdev type > > > > > > > > > > > > > > > > (The second location is newly added in v5, in > > > > > > > > > > > > > > > > order to keep > > > > > > > > > > consistent > > > > > > > > > > > > > > > > with the migration_version node for migratable > > > > > > > > > > > > > > > > pass-though > > > > > > > > > > devices) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is the relationship between those two > > > > > > > > > > > > > > > attributes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (1) is for mdev devices specifically, and (2) is > > > > > > > > > > > > > > provided to keep the > > > > > > > > > > same > > > > > > > > > > > > > > sysfs interface as with non-mdev cases. so (2) is > > > > > > > > > > > > > > for both mdev > > > > > > > > > > devices and > > > > > > > > > > > > > > non-mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > in future, if we enable vfio-pci vendor ops, (i.e. > > > > > > > > > > > > > > a non-mdev device > > > > > > > > > > > > > > is binding to vfio-pci, but is able to register > > > > > > > > > > > > > > migration region and do > > > > > > > > > > > > > > migration transactions from a vendor provided > > > > > > > > > > > > > > affiliate driver), > > > > > > > > > > > > > > the vendor driver would export (2) directly, under > > > > > > > > > > > > > > device node. > > > > > > > > > > > > > > It is not able to provide (1) as there're no mdev > > > > > > > > > > > > > > devices involved. > > > > > > > > > > > > > > > > > > > > > > > > > > Ok, creating an alternate attribute for non-mdev > > > > > > > > > > > > > devices makes sense. > > > > > > > > > > > > > However, wouldn't that rather be a case (3)? The > > > > > > > > > > > > > change here only > > > > > > > > > > > > > refers to mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > as you pointed below, (3) and (2) serve the same > > > > > > > > > > > > purpose. > > > > > > > > > > > > and I think a possible usage is to migrate between a > > > > > > > > > > > > non-mdev device and > > > > > > > > > > > > an mdev device. so I think it's better for them both to > > > > > > > > > > > > use (2) rather > > > > > > > > > > > > than creating (3). > > > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a software compatible > > > > > > > > > > >
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
* Yan Zhao (yan.y.z...@intel.com) wrote: > On Tue, Apr 28, 2020 at 10:14:37PM +0800, Dr. David Alan Gilbert wrote: > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > On Mon, Apr 27, 2020 at 11:37:43PM +0800, Dr. David Alan Gilbert wrote: > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > On Sat, Apr 25, 2020 at 03:10:49AM +0800, Dr. David Alan Gilbert > > > > > wrote: > > > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > > > On Tue, Apr 21, 2020 at 08:08:49PM +0800, Tian, Kevin wrote: > > > > > > > > > From: Yan Zhao > > > > > > > > > Sent: Tuesday, April 21, 2020 10:37 AM > > > > > > > > > > > > > > > > > > On Tue, Apr 21, 2020 at 06:56:00AM +0800, Alex Williamson > > > > > > > > > wrote: > > > > > > > > > > On Sun, 19 Apr 2020 21:24:57 -0400 > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 07:24:57PM +0800, Cornelia Huck > > > > > > > > > > > wrote: > > > > > > > > > > > > On Fri, 17 Apr 2020 05:52:02 -0400 > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 04:44:50PM +0800, Cornelia > > > > > > > > > > > > > Huck wrote: > > > > > > > > > > > > > > On Mon, 13 Apr 2020 01:52:01 -0400 > > > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This patchset introduces a migration_version > > > > > > > > > > > > > > > attribute under sysfs > > > > > > > > > of VFIO > > > > > > > > > > > > > > > Mediated devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This migration_version attribute is used to check > > > > > > > > > > > > > > > migration > > > > > > > > > compatibility > > > > > > > > > > > > > > > between two mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, it has two locations: > > > > > > > > > > > > > > > (1) under mdev_type node, > > > > > > > > > > > > > > > which can be used even before device > > > > > > > > > > > > > > > creation, but only for > > > > > > > > > mdev > > > > > > > > > > > > > > > devices of the same mdev type. > > > > > > > > > > > > > > > (2) under mdev device node, > > > > > > > > > > > > > > > which can only be used after the mdev devices > > > > > > > > > > > > > > > are created, but > > > > > > > > > the src > > > > > > > > > > > > > > > and target mdev devices are not necessarily > > > > > > > > > > > > > > > be of the same > > > > > > > > > mdev type > > > > > > > > > > > > > > > (The second location is newly added in v5, in > > > > > > > > > > > > > > > order to keep > > > > > > > > > consistent > > > > > > > > > > > > > > > with the migration_version node for migratable > > > > > > > > > > > > > > > pass-though > > > > > > > > > devices) > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is the relationship between those two > > > > > > > > > > > > > > attributes? > > > > > > > > > > > > > > > > > > > > > > > > > > > (1) is for mdev devices specifically, and (2) is > > > > > > > > > > > > > provided to keep the > > > > > > > > > same > > > > > > > > > > > > > sysfs interface as with non-mdev cases. so (2) is for > > > > > > > > > > > > > both mdev > > > > > > > > > devices and > > > > > > > > > > > > > non-mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > in future, if we enable vfio-pci vendor ops, (i.e. a > > > > > > > > > > > > > non-mdev device > > > > > > > > > > > > > is binding to vfio-pci, but is able to register > > > > > > > > > > > > > migration region and do > > > > > > > > > > > > > migration transactions from a vendor provided > > > > > > > > > > > > > affiliate driver), > > > > > > > > > > > > > the vendor driver would export (2) directly, under > > > > > > > > > > > > > device node. > > > > > > > > > > > > > It is not able to provide (1) as there're no mdev > > > > > > > > > > > > > devices involved. > > > > > > > > > > > > > > > > > > > > > > > > Ok, creating an alternate attribute for non-mdev > > > > > > > > > > > > devices makes sense. > > > > > > > > > > > > However, wouldn't that rather be a case (3)? The change > > > > > > > > > > > > here only > > > > > > > > > > > > refers to mdev devices. > > > > > > > > > > > > > > > > > > > > > > > as you pointed below, (3) and (2) serve the same purpose. > > > > > > > > > > > and I think a possible usage is to migrate between a > > > > > > > > > > > non-mdev device and > > > > > > > > > > > an mdev device. so I think it's better for them both to > > > > > > > > > > > use (2) rather > > > > > > > > > > > than creating (3). > > > > > > > > > > > > > > > > > > > > An mdev type is meant to define a software compatible > > > > > > > > > > interface, so in > > > > > > > > > > the case of mdev->mdev migration, doesn't migrating to a > > > > > > > > > > different type > > > > > > > > > > fail the most basic of compatibility tests that we expect > > > > > > > > > > userspace to > > > > > > > > > > perform? IOW, if two
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
On Tue, Apr 28, 2020 at 10:14:37PM +0800, Dr. David Alan Gilbert wrote: > * Yan Zhao (yan.y.z...@intel.com) wrote: > > On Mon, Apr 27, 2020 at 11:37:43PM +0800, Dr. David Alan Gilbert wrote: > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > On Sat, Apr 25, 2020 at 03:10:49AM +0800, Dr. David Alan Gilbert wrote: > > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > > On Tue, Apr 21, 2020 at 08:08:49PM +0800, Tian, Kevin wrote: > > > > > > > > From: Yan Zhao > > > > > > > > Sent: Tuesday, April 21, 2020 10:37 AM > > > > > > > > > > > > > > > > On Tue, Apr 21, 2020 at 06:56:00AM +0800, Alex Williamson wrote: > > > > > > > > > On Sun, 19 Apr 2020 21:24:57 -0400 > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 07:24:57PM +0800, Cornelia Huck > > > > > > > > > > wrote: > > > > > > > > > > > On Fri, 17 Apr 2020 05:52:02 -0400 > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 04:44:50PM +0800, Cornelia Huck > > > > > > > > > > > > wrote: > > > > > > > > > > > > > On Mon, 13 Apr 2020 01:52:01 -0400 > > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > This patchset introduces a migration_version > > > > > > > > > > > > > > attribute under sysfs > > > > > > > > of VFIO > > > > > > > > > > > > > > Mediated devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This migration_version attribute is used to check > > > > > > > > > > > > > > migration > > > > > > > > compatibility > > > > > > > > > > > > > > between two mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, it has two locations: > > > > > > > > > > > > > > (1) under mdev_type node, > > > > > > > > > > > > > > which can be used even before device creation, > > > > > > > > > > > > > > but only for > > > > > > > > mdev > > > > > > > > > > > > > > devices of the same mdev type. > > > > > > > > > > > > > > (2) under mdev device node, > > > > > > > > > > > > > > which can only be used after the mdev devices > > > > > > > > > > > > > > are created, but > > > > > > > > the src > > > > > > > > > > > > > > and target mdev devices are not necessarily be > > > > > > > > > > > > > > of the same > > > > > > > > mdev type > > > > > > > > > > > > > > (The second location is newly added in v5, in order > > > > > > > > > > > > > > to keep > > > > > > > > consistent > > > > > > > > > > > > > > with the migration_version node for migratable > > > > > > > > > > > > > > pass-though > > > > > > > > devices) > > > > > > > > > > > > > > > > > > > > > > > > > > What is the relationship between those two attributes? > > > > > > > > > > > > > > > > > > > > > > > > > (1) is for mdev devices specifically, and (2) is > > > > > > > > > > > > provided to keep the > > > > > > > > same > > > > > > > > > > > > sysfs interface as with non-mdev cases. so (2) is for > > > > > > > > > > > > both mdev > > > > > > > > devices and > > > > > > > > > > > > non-mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > in future, if we enable vfio-pci vendor ops, (i.e. a > > > > > > > > > > > > non-mdev device > > > > > > > > > > > > is binding to vfio-pci, but is able to register > > > > > > > > > > > > migration region and do > > > > > > > > > > > > migration transactions from a vendor provided affiliate > > > > > > > > > > > > driver), > > > > > > > > > > > > the vendor driver would export (2) directly, under > > > > > > > > > > > > device node. > > > > > > > > > > > > It is not able to provide (1) as there're no mdev > > > > > > > > > > > > devices involved. > > > > > > > > > > > > > > > > > > > > > > Ok, creating an alternate attribute for non-mdev devices > > > > > > > > > > > makes sense. > > > > > > > > > > > However, wouldn't that rather be a case (3)? The change > > > > > > > > > > > here only > > > > > > > > > > > refers to mdev devices. > > > > > > > > > > > > > > > > > > > > > as you pointed below, (3) and (2) serve the same purpose. > > > > > > > > > > and I think a possible usage is to migrate between a > > > > > > > > > > non-mdev device and > > > > > > > > > > an mdev device. so I think it's better for them both to use > > > > > > > > > > (2) rather > > > > > > > > > > than creating (3). > > > > > > > > > > > > > > > > > > An mdev type is meant to define a software compatible > > > > > > > > > interface, so in > > > > > > > > > the case of mdev->mdev migration, doesn't migrating to a > > > > > > > > > different type > > > > > > > > > fail the most basic of compatibility tests that we expect > > > > > > > > > userspace to > > > > > > > > > perform? IOW, if two mdev types are migration compatible, it > > > > > > > > > seems a > > > > > > > > > prerequisite to that is that they provide the same software > > > > > > > > > interface, > > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > > > In the hybrid
Re: [PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
* Yan Zhao (yan.y.z...@intel.com) wrote: > On Mon, Apr 27, 2020 at 11:37:43PM +0800, Dr. David Alan Gilbert wrote: > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > On Sat, Apr 25, 2020 at 03:10:49AM +0800, Dr. David Alan Gilbert wrote: > > > > * Yan Zhao (yan.y.z...@intel.com) wrote: > > > > > On Tue, Apr 21, 2020 at 08:08:49PM +0800, Tian, Kevin wrote: > > > > > > > From: Yan Zhao > > > > > > > Sent: Tuesday, April 21, 2020 10:37 AM > > > > > > > > > > > > > > On Tue, Apr 21, 2020 at 06:56:00AM +0800, Alex Williamson wrote: > > > > > > > > On Sun, 19 Apr 2020 21:24:57 -0400 > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 07:24:57PM +0800, Cornelia Huck wrote: > > > > > > > > > > On Fri, 17 Apr 2020 05:52:02 -0400 > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 04:44:50PM +0800, Cornelia Huck > > > > > > > > > > > wrote: > > > > > > > > > > > > On Mon, 13 Apr 2020 01:52:01 -0400 > > > > > > > > > > > > Yan Zhao wrote: > > > > > > > > > > > > > > > > > > > > > > > > > This patchset introduces a migration_version > > > > > > > > > > > > > attribute under sysfs > > > > > > > of VFIO > > > > > > > > > > > > > Mediated devices. > > > > > > > > > > > > > > > > > > > > > > > > > > This migration_version attribute is used to check > > > > > > > > > > > > > migration > > > > > > > compatibility > > > > > > > > > > > > > between two mdev devices. > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, it has two locations: > > > > > > > > > > > > > (1) under mdev_type node, > > > > > > > > > > > > > which can be used even before device creation, > > > > > > > > > > > > > but only for > > > > > > > mdev > > > > > > > > > > > > > devices of the same mdev type. > > > > > > > > > > > > > (2) under mdev device node, > > > > > > > > > > > > > which can only be used after the mdev devices are > > > > > > > > > > > > > created, but > > > > > > > the src > > > > > > > > > > > > > and target mdev devices are not necessarily be of > > > > > > > > > > > > > the same > > > > > > > mdev type > > > > > > > > > > > > > (The second location is newly added in v5, in order > > > > > > > > > > > > > to keep > > > > > > > consistent > > > > > > > > > > > > > with the migration_version node for migratable > > > > > > > > > > > > > pass-though > > > > > > > devices) > > > > > > > > > > > > > > > > > > > > > > > > What is the relationship between those two attributes? > > > > > > > > > > > > > > > > > > > > > > > (1) is for mdev devices specifically, and (2) is provided > > > > > > > > > > > to keep the > > > > > > > same > > > > > > > > > > > sysfs interface as with non-mdev cases. so (2) is for > > > > > > > > > > > both mdev > > > > > > > devices and > > > > > > > > > > > non-mdev devices. > > > > > > > > > > > > > > > > > > > > > > in future, if we enable vfio-pci vendor ops, (i.e. a > > > > > > > > > > > non-mdev device > > > > > > > > > > > is binding to vfio-pci, but is able to register migration > > > > > > > > > > > region and do > > > > > > > > > > > migration transactions from a vendor provided affiliate > > > > > > > > > > > driver), > > > > > > > > > > > the vendor driver would export (2) directly, under device > > > > > > > > > > > node. > > > > > > > > > > > It is not able to provide (1) as there're no mdev devices > > > > > > > > > > > involved. > > > > > > > > > > > > > > > > > > > > Ok, creating an alternate attribute for non-mdev devices > > > > > > > > > > makes sense. > > > > > > > > > > However, wouldn't that rather be a case (3)? The change > > > > > > > > > > here only > > > > > > > > > > refers to mdev devices. > > > > > > > > > > > > > > > > > > > as you pointed below, (3) and (2) serve the same purpose. > > > > > > > > > and I think a possible usage is to migrate between a non-mdev > > > > > > > > > device and > > > > > > > > > an mdev device. so I think it's better for them both to use > > > > > > > > > (2) rather > > > > > > > > > than creating (3). > > > > > > > > > > > > > > > > An mdev type is meant to define a software compatible > > > > > > > > interface, so in > > > > > > > > the case of mdev->mdev migration, doesn't migrating to a > > > > > > > > different type > > > > > > > > fail the most basic of compatibility tests that we expect > > > > > > > > userspace to > > > > > > > > perform? IOW, if two mdev types are migration compatible, it > > > > > > > > seems a > > > > > > > > prerequisite to that is that they provide the same software > > > > > > > > interface, > > > > > > > > which means they should be the same mdev type. > > > > > > > > > > > > > > > > In the hybrid cases of mdev->phys or phys->mdev, how does a > > > > > > > management > > > > > > > > tool begin to even guess what might be compatible? Are we > > > > > > > > expecting > > > > > > > > libvirt to probe ever device with this attribute in the system? > > > > > > > > Is > > > > > > > >
