[PATCH 3/8] vfio: add external user support
VFIO is designed to be used via ioctls on file descriptors
returned by VFIO.
However in some situations support for an external user is required.
The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
use the existing VFIO groups for exclusive access in real/virtual mode
in the host kernel to avoid passing map/unmap requests to the user
space which would made things pretty slow.
The proposed protocol includes:
1. do normal VFIO init stuff such as opening a new container, attaching
group(s) to it, setting an IOMMU driver for a container. When IOMMU is
set for a container, all groups in it are considered ready to use by
an external user.
2. pass a fd of the group we want to accelerate to KVM. KVM calls
vfio_group_iommu_id_from_file() to verify if the group is initialized
and IOMMU is set for it. The current TCE IOMMU driver marks the whole
IOMMU table as busy when IOMMU is set for a container what this prevents
other DMA users from allocating from it so it is safe to pass the group
to the user space.
3. KVM increases the container users counter via
vfio_group_add_external_user(). This prevents the VFIO group from
being disposed prior to exiting KVM.
4. When KVM is finished and doing cleanup, it releases the group file
and decrements the container users counter. Everything gets released.
5. KVM also keeps the group file as otherwise its fd might have been
closed at the moment of KVM finish so vfio_group_del_external_user()
call will not be possible.
The "vfio: Limit group opens" patch is also required for the consistency.
Signed-off-by: Alexey Kardashevskiy
---
drivers/vfio/vfio.c | 53 +++
1 file changed, 53 insertions(+)
diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index c488da5..54192b2 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -1370,6 +1370,59 @@ static const struct file_operations vfio_device_fops = {
};
/**
+ * External user API, exported by symbols to be linked dynamically.
+ */
+
+/* Allows an external user (for example, KVM) to lock an IOMMU group */
+static int vfio_group_add_external_user(struct file *filep)
+{
+ struct vfio_group *group = filep->private_data;
+
+ if (filep->f_op != &vfio_group_fops)
+ return -EINVAL;
+
+ if (!atomic_inc_not_zero(&group->container_users))
+ return -EINVAL;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_group_add_external_user);
+
+/* Allows an external user (for example, KVM) to unlock an IOMMU group */
+static void vfio_group_del_external_user(struct file *filep)
+{
+ struct vfio_group *group = filep->private_data;
+
+ BUG_ON(filep->f_op != &vfio_group_fops);
+
+ vfio_group_try_dissolve_container(group);
+}
+EXPORT_SYMBOL_GPL(vfio_group_del_external_user);
+
+/*
+ * Checks if a group for the specified file can be used by
+ * an external user and returns the IOMMU ID if external use is possible.
+ */
+static int vfio_group_iommu_id_from_file(struct file *filep)
+{
+ int ret;
+ struct vfio_group *group = filep->private_data;
+
+ if (WARN_ON(filep->f_op != &vfio_group_fops))
+ return -EINVAL;
+
+ if (0 == atomic_read(&group->container_users) ||
+ !group->container->iommu_driver ||
+ !vfio_group_viable(group))
+ return -EINVAL;
+
+ ret = iommu_group_id(group->iommu_group);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_group_iommu_id_from_file);
+
+/**
* Module/class support
*/
static char *vfio_devnode(struct device *dev, umode_t *mode)
--
1.7.10.4
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/8] vfio: add external user support
VFIO is designed to be used via ioctls on file descriptors
returned by VFIO.
However in some situations support for an external user is required.
The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
use the existing VFIO groups for exclusive access in real/virtual mode
on a host to avoid passing map/unmap requests to the user space which
would made things pretty slow.
The proposed protocol includes:
1. do normal VFIO init stuff such as opening a new container, attaching
group(s) to it, setting an IOMMU driver for a container. When IOMMU is
set for a container, all groups in it are considered ready to use by
an external user.
2. pass a fd of the group we want to accelerate to KVM. KVM calls
vfio_group_get_external_user() to verify if the group is initialized,
IOMMU is set for it and increment the container user counter to prevent
the VFIO group from disposal prior to KVM exit.
The current TCE IOMMU driver marks the whole IOMMU table as busy when
IOMMU is set for a container what prevents other DMA users from
allocating from it so it is safe to grant user space access to it.
3. KVM calls vfio_external_user_iommu_id() to obtian an IOMMU ID which
KVM uses to get an iommu_group struct for later use.
4. When KVM is finished, it calls vfio_group_put_external_user() to
release the VFIO group by decrementing the container user counter.
Everything gets released.
The "vfio: Limit group opens" patch is also required for the consistency.
Signed-off-by: Alexey Kardashevskiy
---
diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
index c488da5..57aa191 100644
--- a/drivers/vfio/vfio.c
+++ b/drivers/vfio/vfio.c
@@ -1370,6 +1370,62 @@ static const struct file_operations vfio_device_fops = {
};
/**
+ * External user API, exported by symbols to be linked dynamically.
+ *
+ * The protocol includes:
+ * 1. do normal VFIO init operation:
+ * - opening a new container;
+ * - attaching group(s) to it;
+ * - setting an IOMMU driver for a container.
+ * When IOMMU is set for a container, all groups in it are
+ * considered ready to use by an external user.
+ *
+ * 2. The user space passed a group fd which we want to accelerate in
+ * KVM. KVM uses vfio_group_get_external_user() to verify that:
+ * - the group is initialized;
+ * - IOMMU is set for it.
+ * Then vfio_group_get_external_user() increments the container user
+ * counter to prevent the VFIO group from disposal prior to KVM exit.
+ *
+ * 3. KVM calls vfio_external_user_iommu_id() to know an IOMMU ID which
+ * KVM uses to get an iommu_group struct for later use.
+ *
+ * 4. When KVM is finished, it calls vfio_group_put_external_user() to
+ * release the VFIO group by decrementing the container user counter.
+ */
+struct vfio_group *vfio_group_get_external_user(struct file *filep)
+{
+ struct vfio_group *group = filep->private_data;
+
+ if (filep->f_op != &vfio_group_fops)
+ return NULL;
+
+ if (!atomic_inc_not_zero(&group->container_users))
+ return NULL;
+
+ if (!group->container->iommu_driver ||
+ !vfio_group_viable(group)) {
+ atomic_dec(&group->container_users);
+ return NULL;
+ }
+
+ return group;
+}
+EXPORT_SYMBOL_GPL(vfio_group_get_external_user);
+
+void vfio_group_put_external_user(struct vfio_group *group)
+{
+ vfio_group_try_dissolve_container(group);
+}
+EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
+
+int vfio_external_user_iommu_id(struct vfio_group *group)
+{
+ return iommu_group_id(group->iommu_group);
+}
+EXPORT_SYMBOL_GPL(vfio_external_user_iommu_id);
+
+/**
* Module/class support
*/
static char *vfio_devnode(struct device *dev, umode_t *mode)
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index ac8d488..24579a0 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -90,4 +90,11 @@ extern void vfio_unregister_iommu_driver(
TYPE tmp; \
offsetof(TYPE, MEMBER) + sizeof(tmp.MEMBER); }) \
+/*
+ * External user API
+ */
+extern struct vfio_group *vfio_group_get_external_user(struct file *filep);
+extern void vfio_group_put_external_user(struct vfio_group *group);
+extern int vfio_external_user_iommu_id(struct vfio_group *group);
+
#endif /* VFIO_H */
--
1.8.3.2
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/8] vfio: add external user support
Hi Alexy,
On Thu, 27 Jun 2013 15:02:31 +1000 Alexey Kardashevskiy wrote:
>
> index c488da5..54192b2 100644
> --- a/drivers/vfio/vfio.c
> +++ b/drivers/vfio/vfio.c
> @@ -1370,6 +1370,59 @@ static const struct file_operations vfio_device_fops =
> {
> };
>
> /**
> + * External user API, exported by symbols to be linked dynamically.
> + */
> +
> +/* Allows an external user (for example, KVM) to lock an IOMMU group */
> +static int vfio_group_add_external_user(struct file *filep)
> +{
> + struct vfio_group *group = filep->private_data;
> +
> + if (filep->f_op != &vfio_group_fops)
> + return -EINVAL;
> +
> + if (!atomic_inc_not_zero(&group->container_users))
> + return -EINVAL;
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(vfio_group_add_external_user);
You cannot EXPORT a static symbol ... The same through the rest of the
file.
--
Cheers,
Stephen Rothwells...@canb.auug.org.au
pgpv4zCGfMNC4.pgp
Description: PGP signature
Re: [PATCH 3/8] vfio: add external user support
Hi Alexy,
On Thu, 27 Jun 2013 15:02:31 +1000 Alexey Kardashevskiy wrote:
>
> +/* Allows an external user (for example, KVM) to unlock an IOMMU group */
> +static void vfio_group_del_external_user(struct file *filep)
> +{
> + struct vfio_group *group = filep->private_data;
> +
> + BUG_ON(filep->f_op != &vfio_group_fops);
We usually reserve BUG_ON for situations where there is no way to
continue running or continuing will corrupt the running kernel. Maybe
WARN_ON() and return?
--
Cheers,
Stephen Rothwells...@canb.auug.org.au
pgpiQGEYh_pRZ.pgp
Description: PGP signature
Re: [PATCH 3/8] vfio: add external user support
On Thu, 2013-06-27 at 16:59 +1000, Stephen Rothwell wrote:
> > +/* Allows an external user (for example, KVM) to unlock an IOMMU
> group */
> > +static void vfio_group_del_external_user(struct file *filep)
> > +{
> > + struct vfio_group *group = filep->private_data;
> > +
> > + BUG_ON(filep->f_op != &vfio_group_fops);
>
> We usually reserve BUG_ON for situations where there is no way to
> continue running or continuing will corrupt the running kernel. Maybe
> WARN_ON() and return?
Not even that. This is a user space provided "fd", we shouldn't oops the
kernel because we passed a wrong argument, just return -EINVAL or
something like that (add a return code).
Ben.
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/8] vfio: add external user support
On 06/27/2013 07:42 PM, Benjamin Herrenschmidt wrote:
> On Thu, 2013-06-27 at 16:59 +1000, Stephen Rothwell wrote:
>>> +/* Allows an external user (for example, KVM) to unlock an IOMMU
>> group */
>>> +static void vfio_group_del_external_user(struct file *filep)
>>> +{
>>> + struct vfio_group *group = filep->private_data;
>>> +
>>> + BUG_ON(filep->f_op != &vfio_group_fops);
>>
>> We usually reserve BUG_ON for situations where there is no way to
>> continue running or continuing will corrupt the running kernel. Maybe
>> WARN_ON() and return?
>
> Not even that. This is a user space provided "fd", we shouldn't oops the
> kernel because we passed a wrong argument, just return -EINVAL or
> something like that (add a return code).
I'll change to WARN_ON but...
This is going to be called on KVM exit on a file pointer previously
verified for correctness. If it is a wrong file*, then something went
terribly wrong.
--
Alexey
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 3/8] vfio: add external user support
On Sun, 2013-07-07 at 01:07 +1000, Alexey Kardashevskiy wrote:
> VFIO is designed to be used via ioctls on file descriptors
> returned by VFIO.
>
> However in some situations support for an external user is required.
> The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
> use the existing VFIO groups for exclusive access in real/virtual mode
> on a host to avoid passing map/unmap requests to the user space which
> would made things pretty slow.
>
> The proposed protocol includes:
>
> 1. do normal VFIO init stuff such as opening a new container, attaching
> group(s) to it, setting an IOMMU driver for a container. When IOMMU is
> set for a container, all groups in it are considered ready to use by
> an external user.
>
> 2. pass a fd of the group we want to accelerate to KVM. KVM calls
> vfio_group_get_external_user() to verify if the group is initialized,
> IOMMU is set for it and increment the container user counter to prevent
> the VFIO group from disposal prior to KVM exit.
> The current TCE IOMMU driver marks the whole IOMMU table as busy when
> IOMMU is set for a container what prevents other DMA users from
> allocating from it so it is safe to grant user space access to it.
>
> 3. KVM calls vfio_external_user_iommu_id() to obtian an IOMMU ID which
> KVM uses to get an iommu_group struct for later use.
>
> 4. When KVM is finished, it calls vfio_group_put_external_user() to
> release the VFIO group by decrementing the container user counter.
> Everything gets released.
>
> The "vfio: Limit group opens" patch is also required for the consistency.
>
> Signed-off-by: Alexey Kardashevskiy
> ---
> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
> index c488da5..57aa191 100644
> --- a/drivers/vfio/vfio.c
> +++ b/drivers/vfio/vfio.c
> @@ -1370,6 +1370,62 @@ static const struct file_operations vfio_device_fops =
> {
> };
>
> /**
> + * External user API, exported by symbols to be linked dynamically.
> + *
> + * The protocol includes:
> + * 1. do normal VFIO init operation:
> + * - opening a new container;
> + * - attaching group(s) to it;
> + * - setting an IOMMU driver for a container.
> + * When IOMMU is set for a container, all groups in it are
> + * considered ready to use by an external user.
> + *
> + * 2. The user space passed a group fd which we want to accelerate in
> + * KVM. KVM uses vfio_group_get_external_user() to verify that:
> + * - the group is initialized;
> + * - IOMMU is set for it.
> + * Then vfio_group_get_external_user() increments the container user
> + * counter to prevent the VFIO group from disposal prior to KVM exit.
> + *
> + * 3. KVM calls vfio_external_user_iommu_id() to know an IOMMU ID which
> + * KVM uses to get an iommu_group struct for later use.
> + *
> + * 4. When KVM is finished, it calls vfio_group_put_external_user() to
> + * release the VFIO group by decrementing the container user counter.
nit, the interface is for any external user, not just kvm.
> + */
> +struct vfio_group *vfio_group_get_external_user(struct file *filep)
> +{
> + struct vfio_group *group = filep->private_data;
> +
> + if (filep->f_op != &vfio_group_fops)
> + return NULL;
ERR_PTR(-EINVAL)
There also needs to be a vfio_group_get(group) here and put in error
cases.
> +
> + if (!atomic_inc_not_zero(&group->container_users))
> + return NULL;
ERR_PTR(-EINVAL)
> +
> + if (!group->container->iommu_driver ||
> + !vfio_group_viable(group)) {
> + atomic_dec(&group->container_users);
> + return NULL;
ERR_PTR(-EINVAL)
> + }
> +
> + return group;
> +}
> +EXPORT_SYMBOL_GPL(vfio_group_get_external_user);
> +
> +void vfio_group_put_external_user(struct vfio_group *group)
> +{
> + vfio_group_try_dissolve_container(group);
And a vfio_group_put(group) here
> +}
> +EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
> +
> +int vfio_external_user_iommu_id(struct vfio_group *group)
> +{
> + return iommu_group_id(group->iommu_group);
> +}
> +EXPORT_SYMBOL_GPL(vfio_external_user_iommu_id);
> +
> +/**
> * Module/class support
> */
> static char *vfio_devnode(struct device *dev, umode_t *mode)
> diff --git a/include/linux/vfio.h b/include/linux/vfio.h
> index ac8d488..24579a0 100644
> --- a/include/linux/vfio.h
> +++ b/include/linux/vfio.h
> @@ -90,4 +90,11 @@ extern void vfio_unregister_iommu_driver(
> TYPE tmp; \
> offsetof(TYPE, MEMBER) + sizeof(tmp.MEMBER); }) \
>
> +/*
> + * External user API
> + */
> +extern struct vfio_group *vfio_group_get_external_user(struct file *filep);
> +extern void vfio_group_put_external_user(struct vfio_group *group);
> +extern int vfio_external_user_iommu_id(struct vfio_group *group);
> +
> #endif /* VFIO_H */
--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http:/
Re: [PATCH 3/8] vfio: add external user support
On 07/09/2013 07:52 AM, Alex Williamson wrote:
> On Sun, 2013-07-07 at 01:07 +1000, Alexey Kardashevskiy wrote:
>> VFIO is designed to be used via ioctls on file descriptors
>> returned by VFIO.
>>
>> However in some situations support for an external user is required.
>> The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
>> use the existing VFIO groups for exclusive access in real/virtual mode
>> on a host to avoid passing map/unmap requests to the user space which
>> would made things pretty slow.
>>
>> The proposed protocol includes:
>>
>> 1. do normal VFIO init stuff such as opening a new container, attaching
>> group(s) to it, setting an IOMMU driver for a container. When IOMMU is
>> set for a container, all groups in it are considered ready to use by
>> an external user.
>>
>> 2. pass a fd of the group we want to accelerate to KVM. KVM calls
>> vfio_group_get_external_user() to verify if the group is initialized,
>> IOMMU is set for it and increment the container user counter to prevent
>> the VFIO group from disposal prior to KVM exit.
>> The current TCE IOMMU driver marks the whole IOMMU table as busy when
>> IOMMU is set for a container what prevents other DMA users from
>> allocating from it so it is safe to grant user space access to it.
>>
>> 3. KVM calls vfio_external_user_iommu_id() to obtian an IOMMU ID which
>> KVM uses to get an iommu_group struct for later use.
>>
>> 4. When KVM is finished, it calls vfio_group_put_external_user() to
>> release the VFIO group by decrementing the container user counter.
>> Everything gets released.
>>
>> The "vfio: Limit group opens" patch is also required for the consistency.
>>
>> Signed-off-by: Alexey Kardashevskiy
>> ---
>> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
>> index c488da5..57aa191 100644
>> --- a/drivers/vfio/vfio.c
>> +++ b/drivers/vfio/vfio.c
>> @@ -1370,6 +1370,62 @@ static const struct file_operations vfio_device_fops
>> = {
>> };
>>
>> /**
>> + * External user API, exported by symbols to be linked dynamically.
>> + *
>> + * The protocol includes:
>> + * 1. do normal VFIO init operation:
>> + * - opening a new container;
>> + * - attaching group(s) to it;
>> + * - setting an IOMMU driver for a container.
>> + * When IOMMU is set for a container, all groups in it are
>> + * considered ready to use by an external user.
>> + *
>> + * 2. The user space passed a group fd which we want to accelerate in
>> + * KVM. KVM uses vfio_group_get_external_user() to verify that:
>> + * - the group is initialized;
>> + * - IOMMU is set for it.
>> + * Then vfio_group_get_external_user() increments the container user
>> + * counter to prevent the VFIO group from disposal prior to KVM exit.
>> + *
>> + * 3. KVM calls vfio_external_user_iommu_id() to know an IOMMU ID which
>> + * KVM uses to get an iommu_group struct for later use.
>> + *
>> + * 4. When KVM is finished, it calls vfio_group_put_external_user() to
>> + * release the VFIO group by decrementing the container user counter.
>
> nit, the interface is for any external user, not just kvm.
s/KVM/An external user/ ?
Or add "the description below uses KVM just as an example of an external user"?
>> + */
>> +struct vfio_group *vfio_group_get_external_user(struct file *filep)
>> +{
>> +struct vfio_group *group = filep->private_data;
>> +
>> +if (filep->f_op != &vfio_group_fops)
>> +return NULL;
>
> ERR_PTR(-EINVAL)
>
> There also needs to be a vfio_group_get(group) here and put in error
> cases.
Is that because I do not hold a reference to the file anymore?
>> +
>> +if (!atomic_inc_not_zero(&group->container_users))
>> +return NULL;
>
> ERR_PTR(-EINVAL)
>
>> +
>> +if (!group->container->iommu_driver ||
>> +!vfio_group_viable(group)) {
>> +atomic_dec(&group->container_users);
>> +return NULL;
>
> ERR_PTR(-EINVAL)
>
>> +}
>> +
>> +return group;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_group_get_external_user);
>> +
>> +void vfio_group_put_external_user(struct vfio_group *group)
>> +{
>> +vfio_group_try_dissolve_container(group);
>
> And a vfio_group_put(group) here
>
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
>> +
>> +int vfio_external_user_iommu_id(struct vfio_group *group)
>> +{
>> +return iommu_group_id(group->iommu_group);
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_external_user_iommu_id);
>> +
>> +/**
>> * Module/class support
>> */
>> static char *vfio_devnode(struct device *dev, umode_t *mode)
>> diff --git a/include/linux/vfio.h b/include/linux/vfio.h
>> index ac8d488..24579a0 100644
>> --- a/include/linux/vfio.h
>> +++ b/include/linux/vfio.h
>> @@ -90,4 +90,11 @@ extern void vfio_unregister_iommu_driver(
>> TYPE tmp; \
>> offsetof(TYPE, MEMBER) + sizeof(tmp.MEMBER); }) \
>>
>> +/*
>> + * External user API
>> + */
>> +extern struct vfio_group *vfio_group_get_external_user(str
Re: [PATCH 3/8] vfio: add external user support
On Tue, 2013-07-09 at 15:40 +1000, Alexey Kardashevskiy wrote:
> On 07/09/2013 07:52 AM, Alex Williamson wrote:
> > On Sun, 2013-07-07 at 01:07 +1000, Alexey Kardashevskiy wrote:
> >> VFIO is designed to be used via ioctls on file descriptors
> >> returned by VFIO.
> >>
> >> However in some situations support for an external user is required.
> >> The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
> >> use the existing VFIO groups for exclusive access in real/virtual mode
> >> on a host to avoid passing map/unmap requests to the user space which
> >> would made things pretty slow.
> >>
> >> The proposed protocol includes:
> >>
> >> 1. do normal VFIO init stuff such as opening a new container, attaching
> >> group(s) to it, setting an IOMMU driver for a container. When IOMMU is
> >> set for a container, all groups in it are considered ready to use by
> >> an external user.
> >>
> >> 2. pass a fd of the group we want to accelerate to KVM. KVM calls
> >> vfio_group_get_external_user() to verify if the group is initialized,
> >> IOMMU is set for it and increment the container user counter to prevent
> >> the VFIO group from disposal prior to KVM exit.
> >> The current TCE IOMMU driver marks the whole IOMMU table as busy when
> >> IOMMU is set for a container what prevents other DMA users from
> >> allocating from it so it is safe to grant user space access to it.
> >>
> >> 3. KVM calls vfio_external_user_iommu_id() to obtian an IOMMU ID which
> >> KVM uses to get an iommu_group struct for later use.
> >>
> >> 4. When KVM is finished, it calls vfio_group_put_external_user() to
> >> release the VFIO group by decrementing the container user counter.
> >> Everything gets released.
> >>
> >> The "vfio: Limit group opens" patch is also required for the consistency.
> >>
> >> Signed-off-by: Alexey Kardashevskiy
> >> ---
> >> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
> >> index c488da5..57aa191 100644
> >> --- a/drivers/vfio/vfio.c
> >> +++ b/drivers/vfio/vfio.c
> >> @@ -1370,6 +1370,62 @@ static const struct file_operations
> >> vfio_device_fops = {
> >> };
> >>
> >> /**
> >> + * External user API, exported by symbols to be linked dynamically.
> >> + *
> >> + * The protocol includes:
> >> + * 1. do normal VFIO init operation:
> >> + *- opening a new container;
> >> + *- attaching group(s) to it;
> >> + *- setting an IOMMU driver for a container.
> >> + * When IOMMU is set for a container, all groups in it are
> >> + * considered ready to use by an external user.
> >> + *
> >> + * 2. The user space passed a group fd which we want to accelerate in
> >> + * KVM. KVM uses vfio_group_get_external_user() to verify that:
> >> + *- the group is initialized;
> >> + *- IOMMU is set for it.
> >> + * Then vfio_group_get_external_user() increments the container user
> >> + * counter to prevent the VFIO group from disposal prior to KVM exit.
> >> + *
> >> + * 3. KVM calls vfio_external_user_iommu_id() to know an IOMMU ID which
> >> + * KVM uses to get an iommu_group struct for later use.
> >> + *
> >> + * 4. When KVM is finished, it calls vfio_group_put_external_user() to
> >> + * release the VFIO group by decrementing the container user counter.
> >
> > nit, the interface is for any external user, not just kvm.
>
> s/KVM/An external user/ ?
> Or add "the description below uses KVM just as an example of an external
> user"?
Give a generic API description, KVM is just an example.
> >> + */
> >> +struct vfio_group *vfio_group_get_external_user(struct file *filep)
> >> +{
> >> + struct vfio_group *group = filep->private_data;
> >> +
> >> + if (filep->f_op != &vfio_group_fops)
> >> + return NULL;
> >
> > ERR_PTR(-EINVAL)
> >
> > There also needs to be a vfio_group_get(group) here and put in error
> > cases.
>
>
> Is that because I do not hold a reference to the file anymore?
We were debating whether it was needed even with the file reference
because we weren't sure that we wanted to trust the user to hold the
reference. Since we're now passing an object, we absolutely must
increase the reference count on the object for this user. Thanks,
Alex
> >> +
> >> + if (!atomic_inc_not_zero(&group->container_users))
> >> + return NULL;
> >
> > ERR_PTR(-EINVAL)
> >
> >> +
> >> + if (!group->container->iommu_driver ||
> >> + !vfio_group_viable(group)) {
> >> + atomic_dec(&group->container_users);
> >> + return NULL;
> >
> > ERR_PTR(-EINVAL)
> >
> >> + }
> >> +
> >> + return group;
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_group_get_external_user);
> >> +
> >> +void vfio_group_put_external_user(struct vfio_group *group)
> >> +{
> >> + vfio_group_try_dissolve_container(group);
> >
> > And a vfio_group_put(group) here
> >
> >> +}
> >> +EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
> >> +
> >> +int vfio_external_user_iommu_id(struct vfio_group *group)
> >> +{
> >> + return iommu_group_id(group->
