Re: [Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
On Fri, 2015-09-25 at 18:16 +0800, He Chen wrote: > [...] > > And if cdp is not enabled: > > [...] > Right above. > > > xl psr-cat-cbm-set -c -d 0xd00dfeed > > > > *ERRROR* > > [now: cbm=] > > > > In current code, it is valid since -c & -d have the same behaviour as > neither of them. > So, it will not show error, and cbm=0xd00dfeed I think with cdp disabled this seems like surprising behaviour, but you might want to argue that when CDP is disabled the single CBM acts like a unified code and data CBM and it therefore makes sense to act this way. If you agree this behaviour is surprising then the simplest answer would be to disallow -c and -d together in both cdp enabled and disabled configurations (since otherwise you would have to ask Xen if it was on to validate the options, which is too much faff). If you think it makes sense then you could leave it as is. > What do you think of this? Is it proper to do so? Like I say I found it surprising, but I'm not an expert in CDP/CBM or what their semantics should be, so my surprise might be misplaced. Please do feel free to disagree if you think the correct behaviour is as the current behaviour (or indeed something else). Ultimately if you have considered these corner cases and have an opinion on what the right behaviour is then I'm happy to go with what you choose. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
On Fri, Sep 25, 2015 at 10:58:58AM +0100, Ian Campbell wrote: > On Fri, 2015-09-25 at 17:29 +0800, He Chen wrote: > > On Thu, Sep 24, 2015 at 12:22:02PM +0100, Ian Campbell wrote: > > > On Thu, 2015-09-17 at 17:35 +0800, He Chen wrote: > > > > Add new CDP options with CAT commands in xl interface man page. > > > > Add description of CDP in xl-psr.markdown. > > > > > > It would have been fine to include this in the previous patch by the > > > way. > > > > > > > You mean include both xl man page and xl-psr.markdown in the previous > > patch? Or just only xl man page? > > All of the docs could validly be included with the patch which introduces > the feature. i.e. you could fold patch #4 into #3 completely. > > You don't have to if you don't want to. > Thanks for your reminding, I will merge patch #3 and #4 in next version. > > Maybe I make words a little confused here. > > With CDP enabled, user can specify code cache (or data cache) only, and > > the other would remain previous value (if no previous value, it would > > keep as default value). > > The value is just enabled or disabled, so do I understand correctly that > this is what you can do if cdp is enabled: > > [now: code cbm=; data cbm=] > > xl psr-cat-cbm-set -c 0xdeadbeed > [now: code cbm=0xdeadbeef; data cbm=] > > xl psr-cat-cbm-set -d 0xf00fb00f > [now: code cbm=0xdeadbeef; data cbm=0xf00fb00f] > > xl psr-cat-cbm-set -c -d 0xd00dfeed > > [now: code cbm=0xd00dfeed; data cbm=0xd00dfeed] > > xl psr-cat-cbm-set 0xee11ee11 > > [now: code cbm=0xee11ee11; data cbm=0xee11ee11] > > Is that right? Yes. > > And if cdp is not enabled: > > [now: cbm=] > > xl psr-cat-cbm-set -c 0xdeadbeed > *ERRROR* > [now: cbm=] > > xl psr-cat-cbm-set -d 0xf00fb00f > *ERRROR* > [now: cbm=] > Right above. > xl psr-cat-cbm-set -c -d 0xd00dfeed > > *ERRROR* > [now: cbm=] > In current code, it is valid since -c & -d have the same behaviour as neither of them. So, it will not show error, and cbm=0xd00dfeed What do you think of this? Is it proper to do so? > xl psr-cat-cbm-set 0xee11ee11 > > [now: cbm=0xee11ee11] > Right. > > So, I think independently would be fine here. > > If I'm correct in all the above then yes. > > Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
On Fri, 2015-09-25 at 17:29 +0800, He Chen wrote: > On Thu, Sep 24, 2015 at 12:22:02PM +0100, Ian Campbell wrote: > > On Thu, 2015-09-17 at 17:35 +0800, He Chen wrote: > > > Add new CDP options with CAT commands in xl interface man page. > > > Add description of CDP in xl-psr.markdown. > > > > It would have been fine to include this in the previous patch by the > > way. > > > > You mean include both xl man page and xl-psr.markdown in the previous > patch? Or just only xl man page? All of the docs could validly be included with the patch which introduces the feature. i.e. you could fold patch #4 into #3 completely. You don't have to if you don't want to. > Maybe I make words a little confused here. > With CDP enabled, user can specify code cache (or data cache) only, and > the other would remain previous value (if no previous value, it would > keep as default value). The value is just enabled or disabled, so do I understand correctly that this is what you can do if cdp is enabled: [now: code cbm=; data cbm=] xl psr-cat-cbm-set -c 0xdeadbeed [now: code cbm=0xdeadbeef; data cbm=] xl psr-cat-cbm-set -d 0xf00fb00f [now: code cbm=0xdeadbeef; data cbm=0xf00fb00f] xl psr-cat-cbm-set -c -d 0xd00dfeed [now: code cbm=0xd00dfeed; data cbm=0xd00dfeed] xl psr-cat-cbm-set 0xee11ee11 [now: code cbm=0xee11ee11; data cbm=0xee11ee11] Is that right? And if cdp is not enabled: [now: cbm=] xl psr-cat-cbm-set -c 0xdeadbeed *ERRROR* [now: cbm=] xl psr-cat-cbm-set -d 0xf00fb00f *ERRROR* [now: cbm=] xl psr-cat-cbm-set -c -d 0xd00dfeed *ERRROR* [now: cbm=] xl psr-cat-cbm-set 0xee11ee11 [now: cbm=0xee11ee11] > So, I think independently would be fine here. If I'm correct in all the above then yes. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
On Thu, Sep 24, 2015 at 12:22:02PM +0100, Ian Campbell wrote: > On Thu, 2015-09-17 at 17:35 +0800, He Chen wrote: > > Add new CDP options with CAT commands in xl interface man page. > > Add description of CDP in xl-psr.markdown. > > It would have been fine to include this in the previous patch by the way. > You mean include both xl man page and xl-psr.markdown in the previous patch? Or just only xl man page? > > +code or data CBM for the domain, CDP feature must be enabled and CBM type > > +options need to be specified when setting CBM. > > I asked on patch 3 whether these options were mutually exclusive or not, > the answer should be reflected in the documentation too please. > Agreed. Both code and data options can be specified at the same time. I will make it clear in the documentation as possible as I can. > > + > > +When CDP is enabled, > > + > > + * the CAT masks are re-mapped into interleaved pairs of masks for data or > > code > > + fetches. > > + > > + * the range of COS for CAT is re-indexed, with the lower-half of the COS > > + range available for CDP. > > + > > +CDP allows OS or Hypervisor to partition cache allocation more > > fine-grained, > > ^the ^in a > > > +code cache and data cache can be specified respectively. With CDP enabled, > > End fine-grained with a full-stop and then: > > The code cache and data cache can be specified separately > > or s/separately/independently/? > > (I think, I'm not 100% sure what you meant by "respectively", so maybe the > suggestion is wrong) > Maybe I make words a little confused here. With CDP enabled, user can specify code cache (or data cache) only, and the other would remain previous value (if no previous value, it would keep as default value). So, I think independently would be fine here. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
>>> On 24.09.15 at 13:22, wrote: > On Thu, 2015-09-17 at 17:35 +0800, He Chen wrote: >> Add new CDP options with CAT commands in xl interface man page. >> --- a/docs/man/xl.pod.1 >> +++ b/docs/man/xl.pod.1 >> @@ -1530,6 +1530,12 @@ applications. In the Xen implementation, CAT is >> used to control cache allocation >> on VM basis. To enforce cache on a specific domain, just set capacity >> bitmasks >> (CBM) for the domain. >> >> +Intel Broadwell and later server platforms also offer Code/Data > Prioritization >> +(CDP) for cache allocation, which support > >allocations, > >> specifying code or data cache for > > It seems like a word is missing, perhaps "size" as in "code or data case > size"? Or perhaps "priority"? No, talk is really about the kind of cache. >> --- a/docs/misc/xl-psr.markdown >> +++ b/docs/misc/xl-psr.markdown >> @@ -14,7 +14,7 @@ tracks cache utilization of memory accesses according >> to the RMID and reports >> monitored data via a counter register. >> >> For more detailed information please refer to Intel SDM chapter >> -"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology". >> +"17.15 - Platform Shared Resource Monitoring: Cache Monitoring Technology". > > Was the chapter number just wrong, or are these things prone to changing? > > If the latter then we should either omit them or we need to refer to a > specific revision of the SDM as well. They change all the time; I previously suggested to Intel folks to omit the numbers and just make sure the titles get quoted fully and correctly. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
On Thu, 2015-09-17 at 17:35 +0800, He Chen wrote: > Add new CDP options with CAT commands in xl interface man page. > Add description of CDP in xl-psr.markdown. It would have been fine to include this in the previous patch by the way. > > Signed-off-by: He Chen > --- > docs/man/xl.pod.1 | 14 ++ > docs/misc/xl-psr.markdown | 44 +++-- > --- > 2 files changed, 53 insertions(+), 5 deletions(-) > > diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1 > index f22c3f3..3c7107d 100644 > --- a/docs/man/xl.pod.1 > +++ b/docs/man/xl.pod.1 > @@ -1530,6 +1530,12 @@ applications. In the Xen implementation, CAT is > used to control cache allocation > on VM basis. To enforce cache on a specific domain, just set capacity > bitmasks > (CBM) for the domain. > > +Intel Broadwell and later server platforms also offer Code/Data > Prioritization > +(CDP) for cache allocation, which support allocations, > specifying code or data cache for It seems like a word is missing, perhaps "size" as in "code or data case size"? Or perhaps "priority"? > +applications. CDP is used on VM basis in the Xen implementation. To specify "used on a per VM basis" > +code or data CBM for the domain, CDP feature must be enabled and CBM type > +options need to be specified when setting CBM. I asked on patch 3 whether these options were mutually exclusive or not, the answer should be reflected in the documentation too please. > diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown > index 3545912..5eb97cc 100644 > --- a/docs/misc/xl-psr.markdown > +++ b/docs/misc/xl-psr.markdown > @@ -14,7 +14,7 @@ tracks cache utilization of memory accesses according > to the RMID and reports > monitored data via a counter register. > > For more detailed information please refer to Intel SDM chapter > -"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology". > +"17.15 - Platform Shared Resource Monitoring: Cache Monitoring Technology". Was the chapter number just wrong, or are these things prone to changing? If the latter then we should either omit them or we need to refer to a specific revision of the SDM as well. > > In Xen's implementation, each domain in the system can be assigned a > RMID > independently, while RMID=0 is reserved for monitoring domains that > don't > @@ -91,17 +91,42 @@ For example, assuming a system with 8 portions and 3 > domains: > first domain exclusive access to half the cache, and the other two > exclusive > access to one quarter each. > > -For more detailed information please refer to Intel SDM chapter > -"17.15 - Platform Shared Resource Control: Cache Allocation Technology". > - > In Xen's implementation, CBM can be configured with libxl/xl interfaces but > COS is maintained in hypervisor only. The cache partition granularity is per > domain, each domain has COS=0 assigned by default, the corresponding CBM is > all-ones, which means all the cache resource can be used by default. > > +Code/Data Prioritization (CDP) Technology is an extension of CAT, which is > +available on Intel Broadwell and later server platforms. CDP enables > isolation > +and separate prioritization of code and data fetches to the L3 cache in a > +software configurable manner, which can enable workload prioritization and > +tuning of cache capacity to the characteristics of the workload. CDP extends > +Cache Allocation Technology (CAT) by providing separate code and data masks > +per Class of Service (COS). Looks good. > + > +CDP can be enabled by adding `psr=cdp` to Xen bootline. "bootline" => "command line" > + > +When CDP is enabled, > + > + * the CAT masks are re-mapped into interleaved pairs of masks for data or > code > + fetches. > + > + * the range of COS for CAT is re-indexed, with the lower-half of the COS > + range available for CDP. > + > +CDP allows OS or Hypervisor to partition cache allocation more fine-grained, ^the ^in a > +code cache and data cache can be specified respectively. With CDP enabled, End fine-grained with a full-stop and then: The code cache and data cache can be specified separately or s/separately/independently/? (I think, I'm not 100% sure what you meant by "respectively", so maybe the suggestion is wrong) > +one COS corresponds to two CBMs (code CBM & data CBM), since the sum of CBMs > +is fixed, that means the number of available COSes will reduce to half when reduce by half Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [PATCH v4 4/4] docs: add document to introduce CDP command
Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown. Signed-off-by: He Chen --- docs/man/xl.pod.1 | 14 ++ docs/misc/xl-psr.markdown | 44 +++- 2 files changed, 53 insertions(+), 5 deletions(-) diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1 index f22c3f3..3c7107d 100644 --- a/docs/man/xl.pod.1 +++ b/docs/man/xl.pod.1 @@ -1530,6 +1530,12 @@ applications. In the Xen implementation, CAT is used to control cache allocation on VM basis. To enforce cache on a specific domain, just set capacity bitmasks (CBM) for the domain. +Intel Broadwell and later server platforms also offer Code/Data Prioritization +(CDP) for cache allocation, which support specifying code or data cache for +applications. CDP is used on VM basis in the Xen implementation. To specify +code or data CBM for the domain, CDP feature must be enabled and CBM type +options need to be specified when setting CBM. + =over 4 =item B [I] I I @@ -1545,6 +1551,14 @@ B Specify the socket to process, otherwise all sockets are processed. +=item B<-c>, B<--code> + +Set code CBM when CDP is enabled. + +=item B<-d>, B<--data> + +Set data CBM when CDP is enabled. + =back =item B [I] diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown index 3545912..5eb97cc 100644 --- a/docs/misc/xl-psr.markdown +++ b/docs/misc/xl-psr.markdown @@ -14,7 +14,7 @@ tracks cache utilization of memory accesses according to the RMID and reports monitored data via a counter register. For more detailed information please refer to Intel SDM chapter -"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology". +"17.15 - Platform Shared Resource Monitoring: Cache Monitoring Technology". In Xen's implementation, each domain in the system can be assigned a RMID independently, while RMID=0 is reserved for monitoring domains that don't @@ -91,17 +91,42 @@ For example, assuming a system with 8 portions and 3 domains: first domain exclusive access to half the cache, and the other two exclusive access to one quarter each. -For more detailed information please refer to Intel SDM chapter -"17.15 - Platform Shared Resource Control: Cache Allocation Technology". - In Xen's implementation, CBM can be configured with libxl/xl interfaces but COS is maintained in hypervisor only. The cache partition granularity is per domain, each domain has COS=0 assigned by default, the corresponding CBM is all-ones, which means all the cache resource can be used by default. +Code/Data Prioritization (CDP) Technology is an extension of CAT, which is +available on Intel Broadwell and later server platforms. CDP enables isolation +and separate prioritization of code and data fetches to the L3 cache in a +software configurable manner, which can enable workload prioritization and +tuning of cache capacity to the characteristics of the workload. CDP extends +Cache Allocation Technology (CAT) by providing separate code and data masks +per Class of Service (COS). + +CDP can be enabled by adding `psr=cdp` to Xen bootline. + +When CDP is enabled, + + * the CAT masks are re-mapped into interleaved pairs of masks for data or code + fetches. + + * the range of COS for CAT is re-indexed, with the lower-half of the COS + range available for CDP. + +CDP allows OS or Hypervisor to partition cache allocation more fine-grained, +code cache and data cache can be specified respectively. With CDP enabled, +one COS corresponds to two CBMs (code CBM & data CBM), since the sum of CBMs +is fixed, that means the number of available COSes will reduce to half when +CDP is on. + +For more detailed information please refer to Intel SDM chapter +"17.16 - Platform Shared Resource Control: Cache Allocation Technology". + ### xl interfaces -System CAT information such as maximum COS and CBM length can be obtained by: +System CAT information such as maximum COS, CBM length and CDP status can be +obtained by: `xl psr-hwinfo --cat` @@ -116,6 +141,15 @@ A cbm is valid only when: obtained with `xl psr-hwinfo --cat`. * All the set bits are contiguous. +When CDP is enabled, `-c` or `--code` option is available to set code CBM for +the domain. + +When CDP is enabled, `-d` or `--data` option is available to set data CBM for +the domain. + +If neither `-c` nor `-d` option is specified when CDP is on, the same code CBM +and data CBM will be set for the domain. + In a multi-socket system, the same cbm will be set on each socket by default. Per socket cbm can be specified with the `--socket SOCKET` option. -- 1.9.1 ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
