Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 09:19 PM, Tony Krowiak wrote: > The following fixup attempts to clarify the bit ordering confusion, > hopefully this is acceptable. > looks good to me, I will fold in. > ---8<--- > > From: Tony Krowiak > Date: Thu, 27 Sep 2018 14:51:12 -0400 > Subject: [FIXUP v10] fixup! s390: doc: detailed specifications for AP > virtualization > > Better explains mask bit ordering. > > Signed-off-by: Tony Krowiak > --- > Documentation/s390/vfio-ap.txt | 127 +++-- > 1 file changed, 91 insertions(+), 36 deletions(-) > > diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt > index bec67eb7141c..599eb0f75c07 100644 > --- a/Documentation/s390/vfio-ap.txt > +++ b/Documentation/s390/vfio-ap.txt > @@ -123,21 +123,24 @@ to identify the adapters, usage domains and control > domains assigned to the KVM > guest: > > * The AP Mask (APM) field is a bit mask that identifies the AP adapters > assigned > - to the KVM guest. Each bit in the mask, from most significant to least > - significant bit, corresponds to an APID from 0-255. If a bit is set, the > - corresponding adapter is valid for use by the KVM guest. > + to the KVM guest. Each bit in the mask, from left to right (i.e. from most > + significant to least significant bit in big endian order), corresponds to > + an APID from 0-255. If a bit is set, the corresponding adapter is valid for > + use by the KVM guest. > > * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage > domains > - assigned to the KVM guest. Each bit in the mask, from most significant to > - least significant bit, corresponds to an AP queue index (APQI) from 0-255. > If > - a bit is set, the corresponding queue is valid for use by the KVM guest. > + assigned to the KVM guest. Each bit in the mask, from left to right (i.e. > from > + most significant to least significant bit in big endian order), > corresponds to > + an AP queue index (APQI) from 0-255. If a bit is set, the corresponding > queue > + is valid for use by the KVM guest. > > * The AP Domain Mask field is a bit mask that identifies the AP control > domains > assigned to the KVM guest. The ADM bit mask controls which domains can be > changed by an AP command-request message sent to a usage domain from the > - guest. Each bit in the mask, from least significant to most significant > bit, > - corresponds to a domain from 0-255. If a bit is set, the corresponding > domain > - can be modified by an AP command-request message sent to a usage domain. > + guest. Each bit in the mask, from left to right (i.e. from most > significant to > + least significant bit in big endian order), corresponds to a domain from > + 0-255. If a bit is set, the corresponding domain can be modified by an AP > + command-request message sent to a usage domain. > > If you recall from the description of an AP Queue, AP instructions include > an APQN to identify the AP queue to which an AP command-request message is > to be > @@ -503,23 +506,34 @@ These are the steps: > access them. To secure them, there are two sysfs files that specify > bitmasks marking a subset of the APQN range as 'usable by the default AP > queue device drivers' or 'not usable by the default device drivers' and > thus > - available for use by the vfio_ap device driver'. The sysfs files > containing > - the sysfs locations of the masks are: > + available for use by the vfio_ap device driver'. The location of the sysfs > + files containing the masks are: > > /sys/bus/ap/apmask > /sys/bus/ap/aqmask > > The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs > - (APID). Each bit in the mask, from most significant to least significant > bit, > - corresponds to an APID from 0-255. If a bit is set, the APID is marked as > - usable only by the default AP queue device drivers; otherwise, the APID is > - usable by the vfio_ap device driver. > + (APID). Each bit in the mask, from left to right (i.e., from most > significant > + to least significant bit in big endian order), corresponds to an APID from > + 0-255. If a bit is set, the APID is marked as usable only by the default > AP > + queue device drivers; otherwise, the APID is usable by the vfio_ap > + device driver. > > The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes > - (APQI). Each bit in the mask, from most significant to least significant > bit, > - corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as > - usable only by the default AP queue device drivers; otherwise, the APQI is > - usable by the vfio_ap device driver. > + (APQI). Each bit in the mask, from left to right (i.e., from most > significant > + to least significant bit in big endian order), corresponds to an APQI from > + 0-255. If a bit is set, the APQI is
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 09:19 PM, Tony Krowiak wrote: > The following fixup attempts to clarify the bit ordering confusion, > hopefully this is acceptable. > looks good to me, I will fold in. > ---8<--- > > From: Tony Krowiak > Date: Thu, 27 Sep 2018 14:51:12 -0400 > Subject: [FIXUP v10] fixup! s390: doc: detailed specifications for AP > virtualization > > Better explains mask bit ordering. > > Signed-off-by: Tony Krowiak > --- > Documentation/s390/vfio-ap.txt | 127 +++-- > 1 file changed, 91 insertions(+), 36 deletions(-) > > diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt > index bec67eb7141c..599eb0f75c07 100644 > --- a/Documentation/s390/vfio-ap.txt > +++ b/Documentation/s390/vfio-ap.txt > @@ -123,21 +123,24 @@ to identify the adapters, usage domains and control > domains assigned to the KVM > guest: > > * The AP Mask (APM) field is a bit mask that identifies the AP adapters > assigned > - to the KVM guest. Each bit in the mask, from most significant to least > - significant bit, corresponds to an APID from 0-255. If a bit is set, the > - corresponding adapter is valid for use by the KVM guest. > + to the KVM guest. Each bit in the mask, from left to right (i.e. from most > + significant to least significant bit in big endian order), corresponds to > + an APID from 0-255. If a bit is set, the corresponding adapter is valid for > + use by the KVM guest. > > * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage > domains > - assigned to the KVM guest. Each bit in the mask, from most significant to > - least significant bit, corresponds to an AP queue index (APQI) from 0-255. > If > - a bit is set, the corresponding queue is valid for use by the KVM guest. > + assigned to the KVM guest. Each bit in the mask, from left to right (i.e. > from > + most significant to least significant bit in big endian order), > corresponds to > + an AP queue index (APQI) from 0-255. If a bit is set, the corresponding > queue > + is valid for use by the KVM guest. > > * The AP Domain Mask field is a bit mask that identifies the AP control > domains > assigned to the KVM guest. The ADM bit mask controls which domains can be > changed by an AP command-request message sent to a usage domain from the > - guest. Each bit in the mask, from least significant to most significant > bit, > - corresponds to a domain from 0-255. If a bit is set, the corresponding > domain > - can be modified by an AP command-request message sent to a usage domain. > + guest. Each bit in the mask, from left to right (i.e. from most > significant to > + least significant bit in big endian order), corresponds to a domain from > + 0-255. If a bit is set, the corresponding domain can be modified by an AP > + command-request message sent to a usage domain. > > If you recall from the description of an AP Queue, AP instructions include > an APQN to identify the AP queue to which an AP command-request message is > to be > @@ -503,23 +506,34 @@ These are the steps: > access them. To secure them, there are two sysfs files that specify > bitmasks marking a subset of the APQN range as 'usable by the default AP > queue device drivers' or 'not usable by the default device drivers' and > thus > - available for use by the vfio_ap device driver'. The sysfs files > containing > - the sysfs locations of the masks are: > + available for use by the vfio_ap device driver'. The location of the sysfs > + files containing the masks are: > > /sys/bus/ap/apmask > /sys/bus/ap/aqmask > > The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs > - (APID). Each bit in the mask, from most significant to least significant > bit, > - corresponds to an APID from 0-255. If a bit is set, the APID is marked as > - usable only by the default AP queue device drivers; otherwise, the APID is > - usable by the vfio_ap device driver. > + (APID). Each bit in the mask, from left to right (i.e., from most > significant > + to least significant bit in big endian order), corresponds to an APID from > + 0-255. If a bit is set, the APID is marked as usable only by the default > AP > + queue device drivers; otherwise, the APID is usable by the vfio_ap > + device driver. > > The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes > - (APQI). Each bit in the mask, from most significant to least significant > bit, > - corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as > - usable only by the default AP queue device drivers; otherwise, the APQI is > - usable by the vfio_ap device driver. > + (APQI). Each bit in the mask, from left to right (i.e., from most > significant > + to least significant bit in big endian order), corresponds to an APQI from > + 0-255. If a bit is set, the APQI is
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 09:19 PM, Tony Krowiak wrote: > On 09/26/2018 06:42 PM, Alex Williamson wrote: >> On Tue, 25 Sep 2018 19:16:41 -0400 >> Tony Krowiak wrote: >> >>> From: Tony Krowiak >>> >>> This patch provides documentation describing the AP architecture and >>> design concepts behind the virtualization of AP devices. It also >>> includes an example of how to configure AP devices for exclusive >>> use of KVM guests. >>> >>> Signed-off-by: Tony Krowiak >>> Reviewed-by: Halil Pasic >>> --- >>> Documentation/s390/vfio-ap.txt | 782 + >>> MAINTAINERS | 1 + >>> 2 files changed, 783 insertions(+) >>> create mode 100644 Documentation/s390/vfio-ap.txt >> ... >>> +Example: >>> +=== >>> +Let's now provide an example to illustrate how KVM guests may be given >>> +access to AP facilities. For this example, we will show how to configure >>> +three guests such that executing the lszcrypt command on the guests would >>> +look like this: >>> + >>> +Guest1 >>> +-- >>> +CARD.DOMAIN TYPE MODE >>> +-- >>> +05 CEX5C CCA-Coproc >>> +05.0004 CEX5C CCA-Coproc >>> +05.00ab CEX5C CCA-Coproc >>> +06 CEX5A Accelerator >>> +06.0004 CEX5A Accelerator >>> +06.00ab CEX5C CCA-Coproc >>> + >>> +Guest2 >>> +-- >>> +CARD.DOMAIN TYPE MODE >>> +-- >>> +05 CEX5A Accelerator >>> +05.0047 CEX5A Accelerator >>> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), >> ^^^ >> Seems like an unfinished thought here. Can you submit another fixup that takes care of this comment?
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 09:19 PM, Tony Krowiak wrote: > On 09/26/2018 06:42 PM, Alex Williamson wrote: >> On Tue, 25 Sep 2018 19:16:41 -0400 >> Tony Krowiak wrote: >> >>> From: Tony Krowiak >>> >>> This patch provides documentation describing the AP architecture and >>> design concepts behind the virtualization of AP devices. It also >>> includes an example of how to configure AP devices for exclusive >>> use of KVM guests. >>> >>> Signed-off-by: Tony Krowiak >>> Reviewed-by: Halil Pasic >>> --- >>> Documentation/s390/vfio-ap.txt | 782 + >>> MAINTAINERS | 1 + >>> 2 files changed, 783 insertions(+) >>> create mode 100644 Documentation/s390/vfio-ap.txt >> ... >>> +Example: >>> +=== >>> +Let's now provide an example to illustrate how KVM guests may be given >>> +access to AP facilities. For this example, we will show how to configure >>> +three guests such that executing the lszcrypt command on the guests would >>> +look like this: >>> + >>> +Guest1 >>> +-- >>> +CARD.DOMAIN TYPE MODE >>> +-- >>> +05 CEX5C CCA-Coproc >>> +05.0004 CEX5C CCA-Coproc >>> +05.00ab CEX5C CCA-Coproc >>> +06 CEX5A Accelerator >>> +06.0004 CEX5A Accelerator >>> +06.00ab CEX5C CCA-Coproc >>> + >>> +Guest2 >>> +-- >>> +CARD.DOMAIN TYPE MODE >>> +-- >>> +05 CEX5A Accelerator >>> +05.0047 CEX5A Accelerator >>> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), >> ^^^ >> Seems like an unfinished thought here. Can you submit another fixup that takes care of this comment?
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/26/2018 06:42 PM, Alex Williamson wrote: On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: From: Tony Krowiak This patch provides documentation describing the AP architecture and design concepts behind the virtualization of AP devices. It also includes an example of how to configure AP devices for exclusive use of KVM guests. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- Documentation/s390/vfio-ap.txt | 782 + MAINTAINERS| 1 + 2 files changed, 783 insertions(+) create mode 100644 Documentation/s390/vfio-ap.txt ... +Example: +=== +Let's now provide an example to illustrate how KVM guests may be given +access to AP facilities. For this example, we will show how to configure +three guests such that executing the lszcrypt command on the guests would +look like this: + +Guest1 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5C CCA-Coproc +05.0004 CEX5C CCA-Coproc +05.00ab CEX5C CCA-Coproc +06 CEX5A Accelerator +06.0004 CEX5A Accelerator +06.00ab CEX5C CCA-Coproc + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5A Accelerator +05.0047 CEX5A Accelerator +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^ Seems like an unfinished thought here. + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +06 CEX5A Accelerator +06.0047 CEX5A Accelerator +06.00ff CEX5A Accelerator + +These are the steps: + +1. Install the vfio_ap module on the linux host. The dependency chain for the + vfio_ap module is: + * iommu + * s390 + * zcrypt + * vfio + * vfio_mdev + * vfio_mdev_device + * KVM + + To build the vfio_ap module, the kernel build must be configured with the + following Kconfig elements selected: + * IOMMU_SUPPORT + * S390 + * ZCRYPT + * S390_AP_IOMMU + * VFIO + * VFIO_MDEV + * VFIO_MDEV_DEVICE + * KVM + + If using make menuconfig select the following to build the vfio_ap module: + -> Device Drivers + -> IOMMU Hardware Support + select S390 AP IOMMU Support + -> VFIO Non-Privileged userspace driver framework + -> Mediated device driver frramework +-> VFIO driver for Mediated devices + -> I/O subsystem + -> VFIO support for AP devices + +2. Secure the AP queues to be used by the three guests so that the host can not + access them. To secure them, there are two sysfs files that specify + bitmasks marking a subset of the APQN range as 'usable by the default AP + queue device drivers' or 'not usable by the default device drivers' and thus + available for use by the vfio_ap device driver'. The sysfs files containing + the sysfs locations of the masks are: + + /sys/bus/ap/apmask + /sys/bus/ap/aqmask + + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs + (APID). Each bit in the mask, from most significant to least significant bit, + corresponds to an APID from 0-255. If a bit is set, the APID is marked as + usable only by the default AP queue device drivers; otherwise, the APID is + usable by the vfio_ap device driver. + + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes + (APQI). Each bit in the mask, from most significant to least significant bit, + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as + usable only by the default AP queue device drivers; otherwise, the APQI is + usable by the vfio_ap device driver. + + The APQN of each AP queue device assigned to the linux host is checked by the + AP bus against the set of APQNs derived from the cross product of APIDs + and APQIs marked as usable only by the default AP queue device drivers. If a + match is detected, only the default AP queue device drivers will be probed; + otherwise, the vfio_ap device driver will be probed. + + By default, the two masks are set to reserve all APQNs for use by the default + AP queue device drivers. There are two ways the default masks can be changed: + + 1. The masks can be changed at boot time with the kernel command line + like this: + + ap.apmask=0x ap.aqmask=0x40 + + This would give these two pools: + +default drivers pool:adapter 0-15, domain 1 +alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? + + 2. The sysfs mask files can also be edited by echoing a string into the + respective file in one of two formats: + + * An absolute hex string starting with 0x - like "0x12345678" - sets +the mask. If the given string is shorter than the mask, it
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/26/2018 06:42 PM, Alex Williamson wrote: On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: From: Tony Krowiak This patch provides documentation describing the AP architecture and design concepts behind the virtualization of AP devices. It also includes an example of how to configure AP devices for exclusive use of KVM guests. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- Documentation/s390/vfio-ap.txt | 782 + MAINTAINERS| 1 + 2 files changed, 783 insertions(+) create mode 100644 Documentation/s390/vfio-ap.txt ... +Example: +=== +Let's now provide an example to illustrate how KVM guests may be given +access to AP facilities. For this example, we will show how to configure +three guests such that executing the lszcrypt command on the guests would +look like this: + +Guest1 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5C CCA-Coproc +05.0004 CEX5C CCA-Coproc +05.00ab CEX5C CCA-Coproc +06 CEX5A Accelerator +06.0004 CEX5A Accelerator +06.00ab CEX5C CCA-Coproc + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5A Accelerator +05.0047 CEX5A Accelerator +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^ Seems like an unfinished thought here. + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +06 CEX5A Accelerator +06.0047 CEX5A Accelerator +06.00ff CEX5A Accelerator + +These are the steps: + +1. Install the vfio_ap module on the linux host. The dependency chain for the + vfio_ap module is: + * iommu + * s390 + * zcrypt + * vfio + * vfio_mdev + * vfio_mdev_device + * KVM + + To build the vfio_ap module, the kernel build must be configured with the + following Kconfig elements selected: + * IOMMU_SUPPORT + * S390 + * ZCRYPT + * S390_AP_IOMMU + * VFIO + * VFIO_MDEV + * VFIO_MDEV_DEVICE + * KVM + + If using make menuconfig select the following to build the vfio_ap module: + -> Device Drivers + -> IOMMU Hardware Support + select S390 AP IOMMU Support + -> VFIO Non-Privileged userspace driver framework + -> Mediated device driver frramework +-> VFIO driver for Mediated devices + -> I/O subsystem + -> VFIO support for AP devices + +2. Secure the AP queues to be used by the three guests so that the host can not + access them. To secure them, there are two sysfs files that specify + bitmasks marking a subset of the APQN range as 'usable by the default AP + queue device drivers' or 'not usable by the default device drivers' and thus + available for use by the vfio_ap device driver'. The sysfs files containing + the sysfs locations of the masks are: + + /sys/bus/ap/apmask + /sys/bus/ap/aqmask + + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs + (APID). Each bit in the mask, from most significant to least significant bit, + corresponds to an APID from 0-255. If a bit is set, the APID is marked as + usable only by the default AP queue device drivers; otherwise, the APID is + usable by the vfio_ap device driver. + + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes + (APQI). Each bit in the mask, from most significant to least significant bit, + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as + usable only by the default AP queue device drivers; otherwise, the APQI is + usable by the vfio_ap device driver. + + The APQN of each AP queue device assigned to the linux host is checked by the + AP bus against the set of APQNs derived from the cross product of APIDs + and APQIs marked as usable only by the default AP queue device drivers. If a + match is detected, only the default AP queue device drivers will be probed; + otherwise, the vfio_ap device driver will be probed. + + By default, the two masks are set to reserve all APQNs for use by the default + AP queue device drivers. There are two ways the default masks can be changed: + + 1. The masks can be changed at boot time with the kernel command line + like this: + + ap.apmask=0x ap.aqmask=0x40 + + This would give these two pools: + +default drivers pool:adapter 0-15, domain 1 +alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? + + 2. The sysfs mask files can also be edited by echoing a string into the + respective file in one of two formats: + + * An absolute hex string starting with 0x - like "0x12345678" - sets +the mask. If the given string is shorter than the mask, it
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/26/2018 06:42 PM, Alex Williamson wrote: On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: From: Tony Krowiak This patch provides documentation describing the AP architecture and design concepts behind the virtualization of AP devices. It also includes an example of how to configure AP devices for exclusive use of KVM guests. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- Documentation/s390/vfio-ap.txt | 782 + MAINTAINERS| 1 + 2 files changed, 783 insertions(+) create mode 100644 Documentation/s390/vfio-ap.txt ... +Example: +=== +Let's now provide an example to illustrate how KVM guests may be given +access to AP facilities. For this example, we will show how to configure +three guests such that executing the lszcrypt command on the guests would +look like this: + +Guest1 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5C CCA-Coproc +05.0004 CEX5C CCA-Coproc +05.00ab CEX5C CCA-Coproc +06 CEX5A Accelerator +06.0004 CEX5A Accelerator +06.00ab CEX5C CCA-Coproc + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5A Accelerator +05.0047 CEX5A Accelerator +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^ Seems like an unfinished thought here. I have no idea how that got in there, there must be a gremlin in my laptop. It does not belong. + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +06 CEX5A Accelerator +06.0047 CEX5A Accelerator +06.00ff CEX5A Accelerator + +These are the steps: + +1. Install the vfio_ap module on the linux host. The dependency chain for the + vfio_ap module is: + * iommu + * s390 + * zcrypt + * vfio + * vfio_mdev + * vfio_mdev_device + * KVM + + To build the vfio_ap module, the kernel build must be configured with the + following Kconfig elements selected: + * IOMMU_SUPPORT + * S390 + * ZCRYPT + * S390_AP_IOMMU + * VFIO + * VFIO_MDEV + * VFIO_MDEV_DEVICE + * KVM + + If using make menuconfig select the following to build the vfio_ap module: + -> Device Drivers + -> IOMMU Hardware Support + select S390 AP IOMMU Support + -> VFIO Non-Privileged userspace driver framework + -> Mediated device driver frramework +-> VFIO driver for Mediated devices + -> I/O subsystem + -> VFIO support for AP devices + +2. Secure the AP queues to be used by the three guests so that the host can not + access them. To secure them, there are two sysfs files that specify + bitmasks marking a subset of the APQN range as 'usable by the default AP + queue device drivers' or 'not usable by the default device drivers' and thus + available for use by the vfio_ap device driver'. The sysfs files containing + the sysfs locations of the masks are: + + /sys/bus/ap/apmask + /sys/bus/ap/aqmask + + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs + (APID). Each bit in the mask, from most significant to least significant bit, + corresponds to an APID from 0-255. If a bit is set, the APID is marked as + usable only by the default AP queue device drivers; otherwise, the APID is + usable by the vfio_ap device driver. + + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes + (APQI). Each bit in the mask, from most significant to least significant bit, + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as + usable only by the default AP queue device drivers; otherwise, the APQI is + usable by the vfio_ap device driver. + + The APQN of each AP queue device assigned to the linux host is checked by the + AP bus against the set of APQNs derived from the cross product of APIDs + and APQIs marked as usable only by the default AP queue device drivers. If a + match is detected, only the default AP queue device drivers will be probed; + otherwise, the vfio_ap device driver will be probed. + + By default, the two masks are set to reserve all APQNs for use by the default + AP queue device drivers. There are two ways the default masks can be changed: + + 1. The masks can be changed at boot time with the kernel command line + like this: + + ap.apmask=0x ap.aqmask=0x40 + + This would give these two pools: + +default drivers pool:adapter 0-15, domain 1 +alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? As Halil stated, it should be: alternate drivers pool: adapter 16-255, domains 0, 2-255 I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? Up above, the apmask and aqmask are described. Both descriptions state,
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/26/2018 06:42 PM, Alex Williamson wrote: On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: From: Tony Krowiak This patch provides documentation describing the AP architecture and design concepts behind the virtualization of AP devices. It also includes an example of how to configure AP devices for exclusive use of KVM guests. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- Documentation/s390/vfio-ap.txt | 782 + MAINTAINERS| 1 + 2 files changed, 783 insertions(+) create mode 100644 Documentation/s390/vfio-ap.txt ... +Example: +=== +Let's now provide an example to illustrate how KVM guests may be given +access to AP facilities. For this example, we will show how to configure +three guests such that executing the lszcrypt command on the guests would +look like this: + +Guest1 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5C CCA-Coproc +05.0004 CEX5C CCA-Coproc +05.00ab CEX5C CCA-Coproc +06 CEX5A Accelerator +06.0004 CEX5A Accelerator +06.00ab CEX5C CCA-Coproc + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +05 CEX5A Accelerator +05.0047 CEX5A Accelerator +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^ Seems like an unfinished thought here. I have no idea how that got in there, there must be a gremlin in my laptop. It does not belong. + +Guest2 +-- +CARD.DOMAIN TYPE MODE +-- +06 CEX5A Accelerator +06.0047 CEX5A Accelerator +06.00ff CEX5A Accelerator + +These are the steps: + +1. Install the vfio_ap module on the linux host. The dependency chain for the + vfio_ap module is: + * iommu + * s390 + * zcrypt + * vfio + * vfio_mdev + * vfio_mdev_device + * KVM + + To build the vfio_ap module, the kernel build must be configured with the + following Kconfig elements selected: + * IOMMU_SUPPORT + * S390 + * ZCRYPT + * S390_AP_IOMMU + * VFIO + * VFIO_MDEV + * VFIO_MDEV_DEVICE + * KVM + + If using make menuconfig select the following to build the vfio_ap module: + -> Device Drivers + -> IOMMU Hardware Support + select S390 AP IOMMU Support + -> VFIO Non-Privileged userspace driver framework + -> Mediated device driver frramework +-> VFIO driver for Mediated devices + -> I/O subsystem + -> VFIO support for AP devices + +2. Secure the AP queues to be used by the three guests so that the host can not + access them. To secure them, there are two sysfs files that specify + bitmasks marking a subset of the APQN range as 'usable by the default AP + queue device drivers' or 'not usable by the default device drivers' and thus + available for use by the vfio_ap device driver'. The sysfs files containing + the sysfs locations of the masks are: + + /sys/bus/ap/apmask + /sys/bus/ap/aqmask + + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs + (APID). Each bit in the mask, from most significant to least significant bit, + corresponds to an APID from 0-255. If a bit is set, the APID is marked as + usable only by the default AP queue device drivers; otherwise, the APID is + usable by the vfio_ap device driver. + + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes + (APQI). Each bit in the mask, from most significant to least significant bit, + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as + usable only by the default AP queue device drivers; otherwise, the APQI is + usable by the vfio_ap device driver. + + The APQN of each AP queue device assigned to the linux host is checked by the + AP bus against the set of APQNs derived from the cross product of APIDs + and APQIs marked as usable only by the default AP queue device drivers. If a + match is detected, only the default AP queue device drivers will be probed; + otherwise, the vfio_ap device driver will be probed. + + By default, the two masks are set to reserve all APQNs for use by the default + AP queue device drivers. There are two ways the default masks can be changed: + + 1. The masks can be changed at boot time with the kernel command line + like this: + + ap.apmask=0x ap.aqmask=0x40 + + This would give these two pools: + +default drivers pool:adapter 0-15, domain 1 +alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? As Halil stated, it should be: alternate drivers pool: adapter 16-255, domains 0, 2-255 I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? Up above, the apmask and aqmask are described. Both descriptions state,
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 07:29 AM, Halil Pasic wrote:
On 09/27/2018 12:42 AM, Alex Williamson wrote:
On Tue, 25 Sep 2018 19:16:41 -0400
Tony Krowiak wrote:
From: Tony Krowiak
[..]
+
+2. Secure the AP queues to be used by the three guests so that the host can not
+ access them. To secure them, there are two sysfs files that specify
+ bitmasks marking a subset of the APQN range as 'usable by the default AP
+ queue device drivers' or 'not usable by the default device drivers' and thus
+ available for use by the vfio_ap device driver'. The sysfs files containing
+ the sysfs locations of the masks are:
+
+ /sys/bus/ap/apmask
+ /sys/bus/ap/aqmask
+
+ The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
+ (APID). Each bit in the mask, from most significant to least significant
bit,
+ corresponds to an APID from 0-255. If a bit is set, the APID is marked as
+ usable only by the default AP queue device drivers; otherwise, the APID is
+ usable by the vfio_ap device driver.
+
+ The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
+ (APQI). Each bit in the mask, from most significant to least significant
bit,
+ corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
+ usable only by the default AP queue device drivers; otherwise, the APQI is
+ usable by the vfio_ap device driver.
+
+ The APQN of each AP queue device assigned to the linux host is checked by
the
+ AP bus against the set of APQNs derived from the cross product of APIDs
+ and APQIs marked as usable only by the default AP queue device drivers. If a
+ match is detected, only the default AP queue device drivers will be probed;
+ otherwise, the vfio_ap device driver will be probed.
+
+ By default, the two masks are set to reserve all APQNs for use by the
default
+ AP queue device drivers. There are two ways the default masks can be
changed:
+
+ 1. The masks can be changed at boot time with the kernel command line
+ like this:
+
+ ap.apmask=0x ap.aqmask=0x40
+
+ This would give these two pools:
+
+default drivers pool:adapter 0-15, domain 1
+alternate drivers pool: adapter 16-255, domains 2-255
What happened to domain 0?
Right, domain 0 is also 'alternate'. So it should have been
alternate drivers pool: adapter 16-255, domains 0,2-255
My mistake.
I'm also a little confused by the bit
ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least
significant bit is furthest left? Did I miss documentation of that?
Harald already tried to explain this, let me give it a try too.
Yes it is a bit confusing. I would try to describe it like this: the big endian
mask,
which is of fixed length of 256 bytes is specified byte-wise using hexadecimal
notation. If only a prefix of the whole mask is specified, the not explicitly
specified bytes are specified are as if they were specified as zero.
I didn't quite get this thing with 'the least significant bit is furthest left'.
I think it is to the right if we assume we are reading left-to-right. It is big
endian, so we consider the most significant bit of a byte to be the first bit,
and the byte with the lowest address to be the first byte of the mask (that
holds the
first 8 bits of the mask).
I'm not quite sure to what you are referring, but the description of the
apqmask and aqmask above states: "Each bit in the mask, from most
significant to least significant bit, corresponds to an APID from
0-255."
It should probably mention that the ordering is big endian, or
say something like "each bit in the mask, from left to right ...".
+
+ 2. The sysfs mask files can also be edited by echoing a string into the
+ respective file in one of two formats:
+
+ * An absolute hex string starting with 0x - like "0x12345678" - sets
+the mask. If the given string is shorter than the mask, it is padded
+with 0s on the right. If the string is longer than the mask, the
+operation is terminated with an error (EINVAL).
And this does say zero padding on the right, but then in the next
bullet our hex digits use normal least significant bit right notation,
ie. 0x41 is 65, not 82, correct?
The zero padding on the right is about the non specified bytes of the mask.
While this bullet is about specifying a whole mask, the next butlet is about
changing a mask by setting the value of bits at a certain position. So in the
context of the next bullet point, the hex string here specifies an integer
value -- plainly a number written in hexadecimal notation (pure math with no
significant bits whatsoever) - in the range 0-256: the index of the bit to be
set ('+') or cleared ('-').
I hope that makes some sense. As I said it's indeed a bit confusing.
+
+ * A plus ('+') or minus ('-') followed by a numerical value. Valid
+examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only
+
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 07:29 AM, Halil Pasic wrote:
On 09/27/2018 12:42 AM, Alex Williamson wrote:
On Tue, 25 Sep 2018 19:16:41 -0400
Tony Krowiak wrote:
From: Tony Krowiak
[..]
+
+2. Secure the AP queues to be used by the three guests so that the host can not
+ access them. To secure them, there are two sysfs files that specify
+ bitmasks marking a subset of the APQN range as 'usable by the default AP
+ queue device drivers' or 'not usable by the default device drivers' and thus
+ available for use by the vfio_ap device driver'. The sysfs files containing
+ the sysfs locations of the masks are:
+
+ /sys/bus/ap/apmask
+ /sys/bus/ap/aqmask
+
+ The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
+ (APID). Each bit in the mask, from most significant to least significant
bit,
+ corresponds to an APID from 0-255. If a bit is set, the APID is marked as
+ usable only by the default AP queue device drivers; otherwise, the APID is
+ usable by the vfio_ap device driver.
+
+ The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
+ (APQI). Each bit in the mask, from most significant to least significant
bit,
+ corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
+ usable only by the default AP queue device drivers; otherwise, the APQI is
+ usable by the vfio_ap device driver.
+
+ The APQN of each AP queue device assigned to the linux host is checked by
the
+ AP bus against the set of APQNs derived from the cross product of APIDs
+ and APQIs marked as usable only by the default AP queue device drivers. If a
+ match is detected, only the default AP queue device drivers will be probed;
+ otherwise, the vfio_ap device driver will be probed.
+
+ By default, the two masks are set to reserve all APQNs for use by the
default
+ AP queue device drivers. There are two ways the default masks can be
changed:
+
+ 1. The masks can be changed at boot time with the kernel command line
+ like this:
+
+ ap.apmask=0x ap.aqmask=0x40
+
+ This would give these two pools:
+
+default drivers pool:adapter 0-15, domain 1
+alternate drivers pool: adapter 16-255, domains 2-255
What happened to domain 0?
Right, domain 0 is also 'alternate'. So it should have been
alternate drivers pool: adapter 16-255, domains 0,2-255
My mistake.
I'm also a little confused by the bit
ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least
significant bit is furthest left? Did I miss documentation of that?
Harald already tried to explain this, let me give it a try too.
Yes it is a bit confusing. I would try to describe it like this: the big endian
mask,
which is of fixed length of 256 bytes is specified byte-wise using hexadecimal
notation. If only a prefix of the whole mask is specified, the not explicitly
specified bytes are specified are as if they were specified as zero.
I didn't quite get this thing with 'the least significant bit is furthest left'.
I think it is to the right if we assume we are reading left-to-right. It is big
endian, so we consider the most significant bit of a byte to be the first bit,
and the byte with the lowest address to be the first byte of the mask (that
holds the
first 8 bits of the mask).
I'm not quite sure to what you are referring, but the description of the
apqmask and aqmask above states: "Each bit in the mask, from most
significant to least significant bit, corresponds to an APID from
0-255."
It should probably mention that the ordering is big endian, or
say something like "each bit in the mask, from left to right ...".
+
+ 2. The sysfs mask files can also be edited by echoing a string into the
+ respective file in one of two formats:
+
+ * An absolute hex string starting with 0x - like "0x12345678" - sets
+the mask. If the given string is shorter than the mask, it is padded
+with 0s on the right. If the string is longer than the mask, the
+operation is terminated with an error (EINVAL).
And this does say zero padding on the right, but then in the next
bullet our hex digits use normal least significant bit right notation,
ie. 0x41 is 65, not 82, correct?
The zero padding on the right is about the non specified bytes of the mask.
While this bullet is about specifying a whole mask, the next butlet is about
changing a mask by setting the value of bits at a certain position. So in the
context of the next bullet point, the hex string here specifies an integer
value -- plainly a number written in hexadecimal notation (pure math with no
significant bits whatsoever) - in the range 0-256: the index of the bit to be
set ('+') or cleared ('-').
I hope that makes some sense. As I said it's indeed a bit confusing.
+
+ * A plus ('+') or minus ('-') followed by a numerical value. Valid
+examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only
+
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 07:59 AM, Christian Borntraeger wrote: On 09/27/2018 01:51 PM, Cornelia Huck wrote: On Thu, 27 Sep 2018 13:29:43 +0200 Halil Pasic wrote: On 09/27/2018 12:42 AM, Alex Williamson wrote: On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: + This is how the matrix is configured for Guest2: + + echo 5 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + + This is how the matrix is configured for Guest3: + + echo 6 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + I'm curious why this interface didn't adopt the +/- notation invented above for consistency. Too difficult to do rollbacks with a string on entries? I remember that we did discuss that possibility around v9, but I can't tell why did we decide to not implement it. Maybe Tony has an answer. IIRC, that was a discussion on the base ap driver interfaces rather than vfio-ap. Anyway, if we were to do that, we would use different attribute names (e.g. just domain_mask, or something similar instead of (assign|unassign)_xxx). So I think such an interface can still be added on top of the existing one. Having that said having multiple interfaces for the very same thing is usually not so nice IMHO. Nod to all of your points. As we do the configuration while the guest is not running anyway, the different interfaces probably do not make that much difference in practice. It should be fine to stick to the current interface for now and only add a new one if we really think it is significantly better. Tony, can you maybe provide a quick on-top patch that clarifies Alex comments regarding the documentation? (State that is is big endian, fixup the small things etc). I can then either fold it in or provide it as an on top patch depending on how much has changed. Will do.
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 07:59 AM, Christian Borntraeger wrote: On 09/27/2018 01:51 PM, Cornelia Huck wrote: On Thu, 27 Sep 2018 13:29:43 +0200 Halil Pasic wrote: On 09/27/2018 12:42 AM, Alex Williamson wrote: On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: + This is how the matrix is configured for Guest2: + + echo 5 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + + This is how the matrix is configured for Guest3: + + echo 6 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + I'm curious why this interface didn't adopt the +/- notation invented above for consistency. Too difficult to do rollbacks with a string on entries? I remember that we did discuss that possibility around v9, but I can't tell why did we decide to not implement it. Maybe Tony has an answer. IIRC, that was a discussion on the base ap driver interfaces rather than vfio-ap. Anyway, if we were to do that, we would use different attribute names (e.g. just domain_mask, or something similar instead of (assign|unassign)_xxx). So I think such an interface can still be added on top of the existing one. Having that said having multiple interfaces for the very same thing is usually not so nice IMHO. Nod to all of your points. As we do the configuration while the guest is not running anyway, the different interfaces probably do not make that much difference in practice. It should be fine to stick to the current interface for now and only add a new one if we really think it is significantly better. Tony, can you maybe provide a quick on-top patch that clarifies Alex comments regarding the documentation? (State that is is big endian, fixup the small things etc). I can then either fold it in or provide it as an on top patch depending on how much has changed. Will do.
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 01:51 PM, Cornelia Huck wrote: > On Thu, 27 Sep 2018 13:29:43 +0200 > Halil Pasic wrote: > >> On 09/27/2018 12:42 AM, Alex Williamson wrote: >>> On Tue, 25 Sep 2018 19:16:41 -0400 >>> Tony Krowiak wrote: + This is how the matrix is configured for Guest2: + + echo 5 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + + This is how the matrix is configured for Guest3: + + echo 6 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + >>> >>> I'm curious why this interface didn't adopt the +/- notation invented >>> above for consistency. Too difficult to do rollbacks with a string on >>> entries? >>> >> >> I remember that we did discuss that possibility around v9, but I can't >> tell why did we decide to not implement it. Maybe Tony has an answer. > > IIRC, that was a discussion on the base ap driver interfaces rather > than vfio-ap. > >> >> Anyway, if we were to do that, we would use different attribute names >> (e.g. just domain_mask, or something similar instead of >> (assign|unassign)_xxx). So I think such an interface can still be added >> on top of the existing one. Having that said having multiple interfaces >> for the very same thing is usually not so nice IMHO. > > Nod to all of your points. > > As we do the configuration while the guest is not running anyway, the > different interfaces probably do not make that much difference in > practice. It should be fine to stick to the current interface for now > and only add a new one if we really think it is significantly better. Tony, can you maybe provide a quick on-top patch that clarifies Alex comments regarding the documentation? (State that is is big endian, fixup the small things etc). I can then either fold it in or provide it as an on top patch depending on how much has changed.
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 01:51 PM, Cornelia Huck wrote: > On Thu, 27 Sep 2018 13:29:43 +0200 > Halil Pasic wrote: > >> On 09/27/2018 12:42 AM, Alex Williamson wrote: >>> On Tue, 25 Sep 2018 19:16:41 -0400 >>> Tony Krowiak wrote: + This is how the matrix is configured for Guest2: + + echo 5 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + + This is how the matrix is configured for Guest3: + + echo 6 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + >>> >>> I'm curious why this interface didn't adopt the +/- notation invented >>> above for consistency. Too difficult to do rollbacks with a string on >>> entries? >>> >> >> I remember that we did discuss that possibility around v9, but I can't >> tell why did we decide to not implement it. Maybe Tony has an answer. > > IIRC, that was a discussion on the base ap driver interfaces rather > than vfio-ap. > >> >> Anyway, if we were to do that, we would use different attribute names >> (e.g. just domain_mask, or something similar instead of >> (assign|unassign)_xxx). So I think such an interface can still be added >> on top of the existing one. Having that said having multiple interfaces >> for the very same thing is usually not so nice IMHO. > > Nod to all of your points. > > As we do the configuration while the guest is not running anyway, the > different interfaces probably do not make that much difference in > practice. It should be fine to stick to the current interface for now > and only add a new one if we really think it is significantly better. Tony, can you maybe provide a quick on-top patch that clarifies Alex comments regarding the documentation? (State that is is big endian, fixup the small things etc). I can then either fold it in or provide it as an on top patch depending on how much has changed.
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On Thu, 27 Sep 2018 13:29:43 +0200 Halil Pasic wrote: > On 09/27/2018 12:42 AM, Alex Williamson wrote: > > On Tue, 25 Sep 2018 19:16:41 -0400 > > Tony Krowiak wrote: > >> + This is how the matrix is configured for Guest2: > >> + > >> + echo 5 > assign_adapter > >> + echo 0x47 > assign_domain > >> + echo 0xff > assign_domain > >> + > >> + This is how the matrix is configured for Guest3: > >> + > >> + echo 6 > assign_adapter > >> + echo 0x47 > assign_domain > >> + echo 0xff > assign_domain > >> + > > > > I'm curious why this interface didn't adopt the +/- notation invented > > above for consistency. Too difficult to do rollbacks with a string on > > entries? > > > > I remember that we did discuss that possibility around v9, but I can't > tell why did we decide to not implement it. Maybe Tony has an answer. IIRC, that was a discussion on the base ap driver interfaces rather than vfio-ap. > > Anyway, if we were to do that, we would use different attribute names > (e.g. just domain_mask, or something similar instead of > (assign|unassign)_xxx). So I think such an interface can still be added > on top of the existing one. Having that said having multiple interfaces > for the very same thing is usually not so nice IMHO. Nod to all of your points. As we do the configuration while the guest is not running anyway, the different interfaces probably do not make that much difference in practice. It should be fine to stick to the current interface for now and only add a new one if we really think it is significantly better.
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On Thu, 27 Sep 2018 13:29:43 +0200 Halil Pasic wrote: > On 09/27/2018 12:42 AM, Alex Williamson wrote: > > On Tue, 25 Sep 2018 19:16:41 -0400 > > Tony Krowiak wrote: > >> + This is how the matrix is configured for Guest2: > >> + > >> + echo 5 > assign_adapter > >> + echo 0x47 > assign_domain > >> + echo 0xff > assign_domain > >> + > >> + This is how the matrix is configured for Guest3: > >> + > >> + echo 6 > assign_adapter > >> + echo 0x47 > assign_domain > >> + echo 0xff > assign_domain > >> + > > > > I'm curious why this interface didn't adopt the +/- notation invented > > above for consistency. Too difficult to do rollbacks with a string on > > entries? > > > > I remember that we did discuss that possibility around v9, but I can't > tell why did we decide to not implement it. Maybe Tony has an answer. IIRC, that was a discussion on the base ap driver interfaces rather than vfio-ap. > > Anyway, if we were to do that, we would use different attribute names > (e.g. just domain_mask, or something similar instead of > (assign|unassign)_xxx). So I think such an interface can still be added > on top of the existing one. Having that said having multiple interfaces > for the very same thing is usually not so nice IMHO. Nod to all of your points. As we do the configuration while the guest is not running anyway, the different interfaces probably do not make that much difference in practice. It should be fine to stick to the current interface for now and only add a new one if we really think it is significantly better.
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 12:42 AM, Alex Williamson wrote:
> On Tue, 25 Sep 2018 19:16:41 -0400
> Tony Krowiak wrote:
>
>> From: Tony Krowiak
[..]
>> +
>> +2. Secure the AP queues to be used by the three guests so that the host can
>> not
>> + access them. To secure them, there are two sysfs files that specify
>> + bitmasks marking a subset of the APQN range as 'usable by the default AP
>> + queue device drivers' or 'not usable by the default device drivers' and
>> thus
>> + available for use by the vfio_ap device driver'. The sysfs files
>> containing
>> + the sysfs locations of the masks are:
>> +
>> + /sys/bus/ap/apmask
>> + /sys/bus/ap/aqmask
>> +
>> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
>> + (APID). Each bit in the mask, from most significant to least significant
>> bit,
>> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as
>> + usable only by the default AP queue device drivers; otherwise, the APID
>> is
>> + usable by the vfio_ap device driver.
>> +
>> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
>> + (APQI). Each bit in the mask, from most significant to least significant
>> bit,
>> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
>> + usable only by the default AP queue device drivers; otherwise, the APQI
>> is
>> + usable by the vfio_ap device driver.
>> +
>> + The APQN of each AP queue device assigned to the linux host is checked
>> by the
>> + AP bus against the set of APQNs derived from the cross product of APIDs
>> + and APQIs marked as usable only by the default AP queue device drivers.
>> If a
>> + match is detected, only the default AP queue device drivers will be
>> probed;
>> + otherwise, the vfio_ap device driver will be probed.
>> +
>> + By default, the two masks are set to reserve all APQNs for use by the
>> default
>> + AP queue device drivers. There are two ways the default masks can be
>> changed:
>> +
>> + 1. The masks can be changed at boot time with the kernel command line
>> + like this:
>> +
>> + ap.apmask=0x ap.aqmask=0x40
>> +
>> + This would give these two pools:
>> +
>> +default drivers pool:adapter 0-15, domain 1
>> +alternate drivers pool: adapter 16-255, domains 2-255
>
> What happened to domain 0?
Right, domain 0 is also 'alternate'. So it should have been
alternate drivers pool: adapter 16-255, domains 0,2-255
> I'm also a little confused by the bit
> ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least
> significant bit is furthest left? Did I miss documentation of that?
>
Harald already tried to explain this, let me give it a try too.
Yes it is a bit confusing. I would try to describe it like this: the big endian
mask,
which is of fixed length of 256 bytes is specified byte-wise using hexadecimal
notation. If only a prefix of the whole mask is specified, the not explicitly
specified bytes are specified are as if they were specified as zero.
I didn't quite get this thing with 'the least significant bit is furthest left'.
I think it is to the right if we assume we are reading left-to-right. It is big
endian, so we consider the most significant bit of a byte to be the first bit,
and the byte with the lowest address to be the first byte of the mask (that
holds the
first 8 bits of the mask).
>> +
>> + 2. The sysfs mask files can also be edited by echoing a string into the
>> + respective file in one of two formats:
>> +
>> + * An absolute hex string starting with 0x - like "0x12345678" - sets
>> +the mask. If the given string is shorter than the mask, it is padded
>> +with 0s on the right. If the string is longer than the mask, the
>> +operation is terminated with an error (EINVAL).
>
> And this does say zero padding on the right, but then in the next
> bullet our hex digits use normal least significant bit right notation,
> ie. 0x41 is 65, not 82, correct?
The zero padding on the right is about the non specified bytes of the mask.
While this bullet is about specifying a whole mask, the next butlet is about
changing a mask by setting the value of bits at a certain position. So in the
context of the next bullet point, the hex string here specifies an integer
value -- plainly a number written in hexadecimal notation (pure math with no
significant bits whatsoever) - in the range 0-256: the index of the bit to be
set ('+') or cleared ('-').
I hope that makes some sense. As I said it's indeed a bit confusing.
>> +
>> + * A plus ('+') or minus ('-') followed by a numerical value. Valid
>> +examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0".
>> Only
>> +the corresponding bit in the mask is switched on ('+') or off
>> ('-'). The
>> +values may also be specified in a comma-separated list to switch
>> more
>> +than one
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 09/27/2018 12:42 AM, Alex Williamson wrote:
> On Tue, 25 Sep 2018 19:16:41 -0400
> Tony Krowiak wrote:
>
>> From: Tony Krowiak
[..]
>> +
>> +2. Secure the AP queues to be used by the three guests so that the host can
>> not
>> + access them. To secure them, there are two sysfs files that specify
>> + bitmasks marking a subset of the APQN range as 'usable by the default AP
>> + queue device drivers' or 'not usable by the default device drivers' and
>> thus
>> + available for use by the vfio_ap device driver'. The sysfs files
>> containing
>> + the sysfs locations of the masks are:
>> +
>> + /sys/bus/ap/apmask
>> + /sys/bus/ap/aqmask
>> +
>> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
>> + (APID). Each bit in the mask, from most significant to least significant
>> bit,
>> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as
>> + usable only by the default AP queue device drivers; otherwise, the APID
>> is
>> + usable by the vfio_ap device driver.
>> +
>> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
>> + (APQI). Each bit in the mask, from most significant to least significant
>> bit,
>> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
>> + usable only by the default AP queue device drivers; otherwise, the APQI
>> is
>> + usable by the vfio_ap device driver.
>> +
>> + The APQN of each AP queue device assigned to the linux host is checked
>> by the
>> + AP bus against the set of APQNs derived from the cross product of APIDs
>> + and APQIs marked as usable only by the default AP queue device drivers.
>> If a
>> + match is detected, only the default AP queue device drivers will be
>> probed;
>> + otherwise, the vfio_ap device driver will be probed.
>> +
>> + By default, the two masks are set to reserve all APQNs for use by the
>> default
>> + AP queue device drivers. There are two ways the default masks can be
>> changed:
>> +
>> + 1. The masks can be changed at boot time with the kernel command line
>> + like this:
>> +
>> + ap.apmask=0x ap.aqmask=0x40
>> +
>> + This would give these two pools:
>> +
>> +default drivers pool:adapter 0-15, domain 1
>> +alternate drivers pool: adapter 16-255, domains 2-255
>
> What happened to domain 0?
Right, domain 0 is also 'alternate'. So it should have been
alternate drivers pool: adapter 16-255, domains 0,2-255
> I'm also a little confused by the bit
> ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least
> significant bit is furthest left? Did I miss documentation of that?
>
Harald already tried to explain this, let me give it a try too.
Yes it is a bit confusing. I would try to describe it like this: the big endian
mask,
which is of fixed length of 256 bytes is specified byte-wise using hexadecimal
notation. If only a prefix of the whole mask is specified, the not explicitly
specified bytes are specified are as if they were specified as zero.
I didn't quite get this thing with 'the least significant bit is furthest left'.
I think it is to the right if we assume we are reading left-to-right. It is big
endian, so we consider the most significant bit of a byte to be the first bit,
and the byte with the lowest address to be the first byte of the mask (that
holds the
first 8 bits of the mask).
>> +
>> + 2. The sysfs mask files can also be edited by echoing a string into the
>> + respective file in one of two formats:
>> +
>> + * An absolute hex string starting with 0x - like "0x12345678" - sets
>> +the mask. If the given string is shorter than the mask, it is padded
>> +with 0s on the right. If the string is longer than the mask, the
>> +operation is terminated with an error (EINVAL).
>
> And this does say zero padding on the right, but then in the next
> bullet our hex digits use normal least significant bit right notation,
> ie. 0x41 is 65, not 82, correct?
The zero padding on the right is about the non specified bytes of the mask.
While this bullet is about specifying a whole mask, the next butlet is about
changing a mask by setting the value of bits at a certain position. So in the
context of the next bullet point, the hex string here specifies an integer
value -- plainly a number written in hexadecimal notation (pure math with no
significant bits whatsoever) - in the range 0-256: the index of the bit to be
set ('+') or cleared ('-').
I hope that makes some sense. As I said it's indeed a bit confusing.
>> +
>> + * A plus ('+') or minus ('-') followed by a numerical value. Valid
>> +examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0".
>> Only
>> +the corresponding bit in the mask is switched on ('+') or off
>> ('-'). The
>> +values may also be specified in a comma-separated list to switch
>> more
>> +than one
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 27.09.2018 00:42, Alex Williamson wrote: > On Tue, 25 Sep 2018 19:16:41 -0400 > Tony Krowiak wrote: > >> From: Tony Krowiak >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak >> Reviewed-by: Halil Pasic >> --- >> Documentation/s390/vfio-ap.txt | 782 + >> MAINTAINERS| 1 + >> 2 files changed, 783 insertions(+) >> create mode 100644 Documentation/s390/vfio-ap.txt > ... >> +Example: >> +=== >> +Let's now provide an example to illustrate how KVM guests may be given >> +access to AP facilities. For this example, we will show how to configure >> +three guests such that executing the lszcrypt command on the guests would >> +look like this: >> + >> +Guest1 >> +-- >> +CARD.DOMAIN TYPE MODE >> +-- >> +05 CEX5C CCA-Coproc >> +05.0004 CEX5C CCA-Coproc >> +05.00ab CEX5C CCA-Coproc >> +06 CEX5A Accelerator >> +06.0004 CEX5A Accelerator >> +06.00ab CEX5C CCA-Coproc >> + >> +Guest2 >> +-- >> +CARD.DOMAIN TYPE MODE >> +-- >> +05 CEX5A Accelerator >> +05.0047 CEX5A Accelerator >> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), > ^^^ > Seems like an unfinished thought here. > >> + >> +Guest2 >> +-- >> +CARD.DOMAIN TYPE MODE >> +-- >> +06 CEX5A Accelerator >> +06.0047 CEX5A Accelerator >> +06.00ff CEX5A Accelerator >> + >> +These are the steps: >> + >> +1. Install the vfio_ap module on the linux host. The dependency chain for >> the >> + vfio_ap module is: >> + * iommu >> + * s390 >> + * zcrypt >> + * vfio >> + * vfio_mdev >> + * vfio_mdev_device >> + * KVM >> + >> + To build the vfio_ap module, the kernel build must be configured with the >> + following Kconfig elements selected: >> + * IOMMU_SUPPORT >> + * S390 >> + * ZCRYPT >> + * S390_AP_IOMMU >> + * VFIO >> + * VFIO_MDEV >> + * VFIO_MDEV_DEVICE >> + * KVM >> + >> + If using make menuconfig select the following to build the vfio_ap >> module: >> + -> Device Drivers >> + -> IOMMU Hardware Support >> + select S390 AP IOMMU Support >> + -> VFIO Non-Privileged userspace driver framework >> + -> Mediated device driver frramework >> +-> VFIO driver for Mediated devices >> + -> I/O subsystem >> + -> VFIO support for AP devices >> + >> +2. Secure the AP queues to be used by the three guests so that the host can >> not >> + access them. To secure them, there are two sysfs files that specify >> + bitmasks marking a subset of the APQN range as 'usable by the default AP >> + queue device drivers' or 'not usable by the default device drivers' and >> thus >> + available for use by the vfio_ap device driver'. The sysfs files >> containing >> + the sysfs locations of the masks are: >> + >> + /sys/bus/ap/apmask >> + /sys/bus/ap/aqmask >> + >> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >> + (APID). Each bit in the mask, from most significant to least significant >> bit, >> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >> + usable only by the default AP queue device drivers; otherwise, the APID >> is >> + usable by the vfio_ap device driver. >> + >> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >> + (APQI). Each bit in the mask, from most significant to least significant >> bit, >> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >> + usable only by the default AP queue device drivers; otherwise, the APQI >> is >> + usable by the vfio_ap device driver. >> + >> + The APQN of each AP queue device assigned to the linux host is checked >> by the >> + AP bus against the set of APQNs derived from the cross product of APIDs >> + and APQIs marked as usable only by the default AP queue device drivers. >> If a >> + match is detected, only the default AP queue device drivers will be >> probed; >> + otherwise, the vfio_ap device driver will be probed. >> + >> + By default, the two masks are set to reserve all APQNs for use by the >> default >> + AP queue device drivers. There are two ways the default masks can be >> changed: >> + >> + 1. The masks can be changed at boot time with the kernel command line >> + like this: >> + >> + ap.apmask=0x ap.aqmask=0x40 >> + >> + This would give these two pools: >> + >> +default drivers pool:adapter 0-15, domain 1 >> +alternate drivers pool: adapter 16-255, domains 2-255 > What happened to domain 0? I'm also a little confused by the bit
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On 27.09.2018 00:42, Alex Williamson wrote: > On Tue, 25 Sep 2018 19:16:41 -0400 > Tony Krowiak wrote: > >> From: Tony Krowiak >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak >> Reviewed-by: Halil Pasic >> --- >> Documentation/s390/vfio-ap.txt | 782 + >> MAINTAINERS| 1 + >> 2 files changed, 783 insertions(+) >> create mode 100644 Documentation/s390/vfio-ap.txt > ... >> +Example: >> +=== >> +Let's now provide an example to illustrate how KVM guests may be given >> +access to AP facilities. For this example, we will show how to configure >> +three guests such that executing the lszcrypt command on the guests would >> +look like this: >> + >> +Guest1 >> +-- >> +CARD.DOMAIN TYPE MODE >> +-- >> +05 CEX5C CCA-Coproc >> +05.0004 CEX5C CCA-Coproc >> +05.00ab CEX5C CCA-Coproc >> +06 CEX5A Accelerator >> +06.0004 CEX5A Accelerator >> +06.00ab CEX5C CCA-Coproc >> + >> +Guest2 >> +-- >> +CARD.DOMAIN TYPE MODE >> +-- >> +05 CEX5A Accelerator >> +05.0047 CEX5A Accelerator >> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), > ^^^ > Seems like an unfinished thought here. > >> + >> +Guest2 >> +-- >> +CARD.DOMAIN TYPE MODE >> +-- >> +06 CEX5A Accelerator >> +06.0047 CEX5A Accelerator >> +06.00ff CEX5A Accelerator >> + >> +These are the steps: >> + >> +1. Install the vfio_ap module on the linux host. The dependency chain for >> the >> + vfio_ap module is: >> + * iommu >> + * s390 >> + * zcrypt >> + * vfio >> + * vfio_mdev >> + * vfio_mdev_device >> + * KVM >> + >> + To build the vfio_ap module, the kernel build must be configured with the >> + following Kconfig elements selected: >> + * IOMMU_SUPPORT >> + * S390 >> + * ZCRYPT >> + * S390_AP_IOMMU >> + * VFIO >> + * VFIO_MDEV >> + * VFIO_MDEV_DEVICE >> + * KVM >> + >> + If using make menuconfig select the following to build the vfio_ap >> module: >> + -> Device Drivers >> + -> IOMMU Hardware Support >> + select S390 AP IOMMU Support >> + -> VFIO Non-Privileged userspace driver framework >> + -> Mediated device driver frramework >> +-> VFIO driver for Mediated devices >> + -> I/O subsystem >> + -> VFIO support for AP devices >> + >> +2. Secure the AP queues to be used by the three guests so that the host can >> not >> + access them. To secure them, there are two sysfs files that specify >> + bitmasks marking a subset of the APQN range as 'usable by the default AP >> + queue device drivers' or 'not usable by the default device drivers' and >> thus >> + available for use by the vfio_ap device driver'. The sysfs files >> containing >> + the sysfs locations of the masks are: >> + >> + /sys/bus/ap/apmask >> + /sys/bus/ap/aqmask >> + >> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >> + (APID). Each bit in the mask, from most significant to least significant >> bit, >> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >> + usable only by the default AP queue device drivers; otherwise, the APID >> is >> + usable by the vfio_ap device driver. >> + >> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >> + (APQI). Each bit in the mask, from most significant to least significant >> bit, >> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >> + usable only by the default AP queue device drivers; otherwise, the APQI >> is >> + usable by the vfio_ap device driver. >> + >> + The APQN of each AP queue device assigned to the linux host is checked >> by the >> + AP bus against the set of APQNs derived from the cross product of APIDs >> + and APQIs marked as usable only by the default AP queue device drivers. >> If a >> + match is detected, only the default AP queue device drivers will be >> probed; >> + otherwise, the vfio_ap device driver will be probed. >> + >> + By default, the two masks are set to reserve all APQNs for use by the >> default >> + AP queue device drivers. There are two ways the default masks can be >> changed: >> + >> + 1. The masks can be changed at boot time with the kernel command line >> + like this: >> + >> + ap.apmask=0x ap.aqmask=0x40 >> + >> + This would give these two pools: >> + >> +default drivers pool:adapter 0-15, domain 1 >> +alternate drivers pool: adapter 16-255, domains 2-255 > What happened to domain 0? I'm also a little confused by the bit
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: > From: Tony Krowiak > > This patch provides documentation describing the AP architecture and > design concepts behind the virtualization of AP devices. It also > includes an example of how to configure AP devices for exclusive > use of KVM guests. > > Signed-off-by: Tony Krowiak > Reviewed-by: Halil Pasic > --- > Documentation/s390/vfio-ap.txt | 782 + > MAINTAINERS| 1 + > 2 files changed, 783 insertions(+) > create mode 100644 Documentation/s390/vfio-ap.txt ... > +Example: > +=== > +Let's now provide an example to illustrate how KVM guests may be given > +access to AP facilities. For this example, we will show how to configure > +three guests such that executing the lszcrypt command on the guests would > +look like this: > + > +Guest1 > +-- > +CARD.DOMAIN TYPE MODE > +-- > +05 CEX5C CCA-Coproc > +05.0004 CEX5C CCA-Coproc > +05.00ab CEX5C CCA-Coproc > +06 CEX5A Accelerator > +06.0004 CEX5A Accelerator > +06.00ab CEX5C CCA-Coproc > + > +Guest2 > +-- > +CARD.DOMAIN TYPE MODE > +-- > +05 CEX5A Accelerator > +05.0047 CEX5A Accelerator > +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^ Seems like an unfinished thought here. > + > +Guest2 > +-- > +CARD.DOMAIN TYPE MODE > +-- > +06 CEX5A Accelerator > +06.0047 CEX5A Accelerator > +06.00ff CEX5A Accelerator > + > +These are the steps: > + > +1. Install the vfio_ap module on the linux host. The dependency chain for the > + vfio_ap module is: > + * iommu > + * s390 > + * zcrypt > + * vfio > + * vfio_mdev > + * vfio_mdev_device > + * KVM > + > + To build the vfio_ap module, the kernel build must be configured with the > + following Kconfig elements selected: > + * IOMMU_SUPPORT > + * S390 > + * ZCRYPT > + * S390_AP_IOMMU > + * VFIO > + * VFIO_MDEV > + * VFIO_MDEV_DEVICE > + * KVM > + > + If using make menuconfig select the following to build the vfio_ap module: > + -> Device Drivers > + -> IOMMU Hardware Support > + select S390 AP IOMMU Support > + -> VFIO Non-Privileged userspace driver framework > + -> Mediated device driver frramework > +-> VFIO driver for Mediated devices > + -> I/O subsystem > + -> VFIO support for AP devices > + > +2. Secure the AP queues to be used by the three guests so that the host can > not > + access them. To secure them, there are two sysfs files that specify > + bitmasks marking a subset of the APQN range as 'usable by the default AP > + queue device drivers' or 'not usable by the default device drivers' and > thus > + available for use by the vfio_ap device driver'. The sysfs files > containing > + the sysfs locations of the masks are: > + > + /sys/bus/ap/apmask > + /sys/bus/ap/aqmask > + > + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs > + (APID). Each bit in the mask, from most significant to least significant > bit, > + corresponds to an APID from 0-255. If a bit is set, the APID is marked as > + usable only by the default AP queue device drivers; otherwise, the APID is > + usable by the vfio_ap device driver. > + > + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes > + (APQI). Each bit in the mask, from most significant to least significant > bit, > + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as > + usable only by the default AP queue device drivers; otherwise, the APQI is > + usable by the vfio_ap device driver. > + > + The APQN of each AP queue device assigned to the linux host is checked by > the > + AP bus against the set of APQNs derived from the cross product of APIDs > + and APQIs marked as usable only by the default AP queue device drivers. > If a > + match is detected, only the default AP queue device drivers will be > probed; > + otherwise, the vfio_ap device driver will be probed. > + > + By default, the two masks are set to reserve all APQNs for use by the > default > + AP queue device drivers. There are two ways the default masks can be > changed: > + > + 1. The masks can be changed at boot time with the kernel command line > + like this: > + > + ap.apmask=0x ap.aqmask=0x40 > + > + This would give these two pools: > + > +default drivers pool:adapter 0-15, domain 1 > +alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? > + > + 2. The sysfs mask files can also be edited by echoing
Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization
On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak wrote: > From: Tony Krowiak > > This patch provides documentation describing the AP architecture and > design concepts behind the virtualization of AP devices. It also > includes an example of how to configure AP devices for exclusive > use of KVM guests. > > Signed-off-by: Tony Krowiak > Reviewed-by: Halil Pasic > --- > Documentation/s390/vfio-ap.txt | 782 + > MAINTAINERS| 1 + > 2 files changed, 783 insertions(+) > create mode 100644 Documentation/s390/vfio-ap.txt ... > +Example: > +=== > +Let's now provide an example to illustrate how KVM guests may be given > +access to AP facilities. For this example, we will show how to configure > +three guests such that executing the lszcrypt command on the guests would > +look like this: > + > +Guest1 > +-- > +CARD.DOMAIN TYPE MODE > +-- > +05 CEX5C CCA-Coproc > +05.0004 CEX5C CCA-Coproc > +05.00ab CEX5C CCA-Coproc > +06 CEX5A Accelerator > +06.0004 CEX5A Accelerator > +06.00ab CEX5C CCA-Coproc > + > +Guest2 > +-- > +CARD.DOMAIN TYPE MODE > +-- > +05 CEX5A Accelerator > +05.0047 CEX5A Accelerator > +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^ Seems like an unfinished thought here. > + > +Guest2 > +-- > +CARD.DOMAIN TYPE MODE > +-- > +06 CEX5A Accelerator > +06.0047 CEX5A Accelerator > +06.00ff CEX5A Accelerator > + > +These are the steps: > + > +1. Install the vfio_ap module on the linux host. The dependency chain for the > + vfio_ap module is: > + * iommu > + * s390 > + * zcrypt > + * vfio > + * vfio_mdev > + * vfio_mdev_device > + * KVM > + > + To build the vfio_ap module, the kernel build must be configured with the > + following Kconfig elements selected: > + * IOMMU_SUPPORT > + * S390 > + * ZCRYPT > + * S390_AP_IOMMU > + * VFIO > + * VFIO_MDEV > + * VFIO_MDEV_DEVICE > + * KVM > + > + If using make menuconfig select the following to build the vfio_ap module: > + -> Device Drivers > + -> IOMMU Hardware Support > + select S390 AP IOMMU Support > + -> VFIO Non-Privileged userspace driver framework > + -> Mediated device driver frramework > +-> VFIO driver for Mediated devices > + -> I/O subsystem > + -> VFIO support for AP devices > + > +2. Secure the AP queues to be used by the three guests so that the host can > not > + access them. To secure them, there are two sysfs files that specify > + bitmasks marking a subset of the APQN range as 'usable by the default AP > + queue device drivers' or 'not usable by the default device drivers' and > thus > + available for use by the vfio_ap device driver'. The sysfs files > containing > + the sysfs locations of the masks are: > + > + /sys/bus/ap/apmask > + /sys/bus/ap/aqmask > + > + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs > + (APID). Each bit in the mask, from most significant to least significant > bit, > + corresponds to an APID from 0-255. If a bit is set, the APID is marked as > + usable only by the default AP queue device drivers; otherwise, the APID is > + usable by the vfio_ap device driver. > + > + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes > + (APQI). Each bit in the mask, from most significant to least significant > bit, > + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as > + usable only by the default AP queue device drivers; otherwise, the APQI is > + usable by the vfio_ap device driver. > + > + The APQN of each AP queue device assigned to the linux host is checked by > the > + AP bus against the set of APQNs derived from the cross product of APIDs > + and APQIs marked as usable only by the default AP queue device drivers. > If a > + match is detected, only the default AP queue device drivers will be > probed; > + otherwise, the vfio_ap device driver will be probed. > + > + By default, the two masks are set to reserve all APQNs for use by the > default > + AP queue device drivers. There are two ways the default masks can be > changed: > + > + 1. The masks can be changed at boot time with the kernel command line > + like this: > + > + ap.apmask=0x ap.aqmask=0x40 > + > + This would give these two pools: > + > +default drivers pool:adapter 0-15, domain 1 > +alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0x is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? > + > + 2. The sysfs mask files can also be edited by echoing
