Re: [Xen-devel] [v2 0/3] Enable L2 Cache Allocation Technology
Hi, Many thanks for reviewing the patches and provide your comments! Please check my comments below. Thanks! On 16-09-30 16:32:33, Konrad Rzeszutek Wilk wrote: > On Thu, Sep 22, 2016 at 10:14:00AM +0800, Yi Sun wrote: > > Design document is below: > > Could the design document be a patch itself please? > Yes, I will do it. Thanks! > Meaning it will be added in docs/misc/ ? > > Also is there a git tree for your patches? > No. Because the number of patches is not much, I do not create git tree. If you strongly recommend it, I can create it. Thanks! > Thanks. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [v2 0/3] Enable L2 Cache Allocation Technology
On 16-09-23 10:41:45, Wei Liu wrote: > On Fri, Sep 23, 2016 at 10:06:54AM +0800, Yi Sun wrote: > > On 16-09-22 11:18:12, Wei Liu wrote: > > > Hi Yi > > > > > > Thanks for submitting this series. I see that all the actual patches > > > are not chained to 0/3. It would be better if they show up as replies to > > > 0/3. > > > > > > The workflow for sending patch series differs from person to person, so > > > we don't want to dictate how you use your tool. But we have written a > > > wiki page on how to submit patches to xen-devel. That contains some tips > > > about git. > > > > > > https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list > > > > > > Feel free to ask questions here. I would be happy to share how I do it > > > if you're interested. > > > > > > Wei. > > > > Hi, Wei, > > > > Thanks a lot for your kindness! I just went through the wiki and found I > > should use '--in-reply-to=', right? If you can share your steps that would > > be great. > > Assuming you send out your cover letter separately from your patches, > yes, --in-reply-to= is the option that you need to add to `git > send-email' invocation. The (see `git help send-email') > part is the message id of your cover letter. You should be able to get > that after sending out the cover letter. > > My current preferred workflow is to have git handle everything for me. > I have a separate file for the *content* of cover letter. And to generate > everything including the cover letter *email*: > > $ git format-patch --reroll-count=$N --cover-letter > > Then you will have v$N--cover-letter.patch plus a few v$N-*.patch. > Use you favourite text editor to edit the cover letter, import the > content, edit it as you wish. > > To send out all you patches including the cover letter: > > $ git send-email *.patch --to xen-devel@... --cc maintainers@... --dry-run > > When you're confident the list of emails looks correct, delete --dry-run > to actually send them. > > This way git send-email will automatically chain all actual patches to > cover letter. > > Some people have built other tools on top of porcelain git to track more > stuff. I'm experimenting with a tool called git-series, but it is too > early to say if it is the right tool for me. > > Anyway, read git manpage. It has a lot of information in it. > Many thanks for the details! They are very helpful. I will read manual and test the flow. > > > > Can I send the patches again after addressing Ian's comments? > > > > It's better to wait a bit for hypervisor maintainers to comment on the > first three patches. > Sure. Thank you! > Wei. > > > Thanks, > > Yi ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [v2 0/3] Enable L2 Cache Allocation Technology
On Fri, Sep 23, 2016 at 10:06:54AM +0800, Yi Sun wrote: > On 16-09-22 11:18:12, Wei Liu wrote: > > Hi Yi > > > > Thanks for submitting this series. I see that all the actual patches > > are not chained to 0/3. It would be better if they show up as replies to > > 0/3. > > > > The workflow for sending patch series differs from person to person, so > > we don't want to dictate how you use your tool. But we have written a > > wiki page on how to submit patches to xen-devel. That contains some tips > > about git. > > > > https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list > > > > Feel free to ask questions here. I would be happy to share how I do it > > if you're interested. > > > > Wei. > > Hi, Wei, > > Thanks a lot for your kindness! I just went through the wiki and found I > should use '--in-reply-to=', right? If you can share your steps that would > be great. Assuming you send out your cover letter separately from your patches, yes, --in-reply-to= is the option that you need to add to `git send-email' invocation. The (see `git help send-email') part is the message id of your cover letter. You should be able to get that after sending out the cover letter. My current preferred workflow is to have git handle everything for me. I have a separate file for the *content* of cover letter. And to generate everything including the cover letter *email*: $ git format-patch --reroll-count=$N --cover-letter Then you will have v$N--cover-letter.patch plus a few v$N-*.patch. Use you favourite text editor to edit the cover letter, import the content, edit it as you wish. To send out all you patches including the cover letter: $ git send-email *.patch --to xen-devel@... --cc maintainers@... --dry-run When you're confident the list of emails looks correct, delete --dry-run to actually send them. This way git send-email will automatically chain all actual patches to cover letter. Some people have built other tools on top of porcelain git to track more stuff. I'm experimenting with a tool called git-series, but it is too early to say if it is the right tool for me. Anyway, read git manpage. It has a lot of information in it. > > Can I send the patches again after addressing Ian's comments? > It's better to wait a bit for hypervisor maintainers to comment on the first three patches. Wei. > Thanks, > Yi ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [v2 0/3] Enable L2 Cache Allocation Technology
On 16-09-22 11:18:12, Wei Liu wrote: > Hi Yi > > Thanks for submitting this series. I see that all the actual patches > are not chained to 0/3. It would be better if they show up as replies to > 0/3. > > The workflow for sending patch series differs from person to person, so > we don't want to dictate how you use your tool. But we have written a > wiki page on how to submit patches to xen-devel. That contains some tips > about git. > > https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list > > Feel free to ask questions here. I would be happy to share how I do it > if you're interested. > > Wei. Hi, Wei, Thanks a lot for your kindness! I just went through the wiki and found I should use '--in-reply-to=', right? If you can share your steps that would be great. Can I send the patches again after addressing Ian's comments? Thanks, Yi ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [v2 0/3] Enable L2 Cache Allocation Technology
Hi Yi Thanks for submitting this series. I see that all the actual patches are not chained to 0/3. It would be better if they show up as replies to 0/3. The workflow for sending patch series differs from person to person, so we don't want to dictate how you use your tool. But we have written a wiki page on how to submit patches to xen-devel. That contains some tips about git. https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches#Sending_the_patches_to_the_list Feel free to ask questions here. I would be happy to share how I do it if you're interested. Wei. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
[Xen-devel] [v2 0/3] Enable L2 Cache Allocation Technology
Design document is below: === % Intel L2 Cache Allocation Technology (L2 CAT) Feature % Revision 2.0 \clearpage Hi all, We plan to bring a new PSR (Platform Shared Resource) feature called Intel L2 Cache Allocation Technology (L2 CAT) to Xen. This is the design of L2 CAT. It might be a little long and detailed, hope it doesn't matter. Besides the L2 CAT implementaion, we refactor the psr.c to make it more flexible to add new features and fulfill the principle, open for extension but closed for modification. Comments and suggestions are welcome :-) # Basics Status: **Tech Preview** Architecture(s): Intel x86 Component(s): Hypervisor, toolstack Hardware: Atom codename Goldmont and beyond # Overview L2 CAT allows an OS or Hypervisor/VMM to control allocation of a CPU's shared L2 cache based on application priority or Class of Service (COS). Each CLOS is configured using capacity bitmasks (CBM) which represent cache capacity and indicate the degree of overlap and isolation between classes. Once L2 CAT is configured, the processor allows access to portions of L2 cache according to the established class of service (COS). # Technical information L2 CAT is a member of Intel PSR features and part of CAT, it shares some base PSR infrastructure in Xen. ## Hardware perspective L2 CAT defines a new range MSRs to assign different L2 cache access patterns which are known as CBMs (Capacity BitMask), each CBM is associated with a COS. ``` +++ IA32_PQR_ASSOC | MSR (per socket) |Address | ++---+---+ +++ ||COS| | | IA32_L2_QOS_MASK_0 | 0xD10 | ++---+---+ +++ └-> | ...| ... | +++ | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) | +++ ``` When context switch happens, the COS of VCPU is written to per-thread MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation according to the corresponding CBM. ## The relationship between L2 CAT and L3 CAT/CDP L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled. L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of L2 CAT must be continuous too. N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same associate register `IA32_PQR_ASSOC`, which means one COS associate to a pair of L2 CBM and L3 CBM. Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or other PSR features in future). In some cases, a VM is permitted to have a COS that is beyond one (or more) of PSR features but within the others. For instance, let's assume the max COS of L2 CAT is 8 but the max COS of L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no limit) since COS 9 is beyond the max COS (8) of L2 CAT. ## Design Overview * Core COS/CBM association When enforcing L2 CAT, all cores of domains have the same default COS (COS0) which associated to the fully open CBM (all ones bitmask) to access all L2 cache. The default COS is used only in hypervisor and is transparent to tool stack and user. System administrator can change PSR allocation policy at runtime by tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP is enabled, one COS corresponds to a 3-tuple, like [L2 CBM, L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled, things would be easier, one COS corresponds to one L2 CBM. * VCPU schedule This part reuses L3 CAT COS infrastructure. * Multi-sockets Different sockets may have different L2 CAT capability (e.g. max COS) although it is consistent on the same socket. So the capability of per-socket L2 CAT is specified. ## Implementation Description * Hypervisor interfaces: 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and L3 CAT if hardware supported. 2. New: SYSCTL: - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information. 3. New: DOMCTL: - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain. - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain. * xl
