> From: Borislav Petkov [mailto:[email protected]] > Sent: Friday, July 01, 2016 10:28 AM > To: Luck, Tony <[email protected]> > Cc: Yu, Fenghua <[email protected]>; Thomas Gleixner > <[email protected]>; Ingo Molnar <[email protected]>; Anvin, H Peter > <[email protected]>; Peter Zijlstra <[email protected]>; > Stephane Eranian <[email protected]>; Shankar, Ravi V > <[email protected]>; Vikas Shivappa > <[email protected]>; linux-kernel <linux- > [email protected]>; x86 <[email protected]> > Subject: Re: [PATCH] cacheinfo: Introduce cache id > > On Fri, Jul 01, 2016 at 09:50:41AM -0700, Luck, Tony wrote: > > Here's the situation. We have lots of (mostly) independent caches on a > system. > > The L3 cache (also called LLC - Last Level Cache - in some > > documentation) is usually shared by all cpus on a physical socket. The > > L1 and L2 caches are typically local to a core, so only shared by the > hyperthreads on a core. > > But I say "usually" and "typically" because the architecture doesn't > > require that. We could have multiple separate L3 caches on a socket > > with a subset of cpus assigned to each of them. We could have an L2 > > cache that is shared by two or more cores. > > Right, so I'm presuming we don't want to make it special to the LLC but be > generic and have any cache level have an ID. > > And, since we simply call it cache, how about we drop the "node" thing and > simply talk about cache and it having an ID. We say it is unique on the system > and the cache with ID X is local to only thread X and the cache with ID Y is > shared by threads Y_0, ... Y_k. And so on... > > How does that sound?
That's good for me. > > Struct doc will have then: > > /** > * struct cacheinfo - represent a cache leaf node > + * @id: This cache's ID. ID is unique on the platform. > > ? > > which begs the question, does get_cache_id() give unique IDs for all caches > in the hierarchy on the system? Cache id is unique on the same level of cache across platform. > > Because I tried this patch quickly in kvm and it was of course wrong but it > hints at that important question: > > cpu0/cache/index0/id:0b > cpu0/cache/index1/id:0 > cpu0/cache/index2/id:0 > cpu1/cache/index0/id:1 > cpu1/cache/index1/id:1 > cpu1/cache/index2/id:1 > cpu2/cache/index0/id:2 > cpu2/cache/index1/id:2 > cpu2/cache/index2/id:2 > cpu3/cache/index0/id:3 > cpu3/cache/index1/id:3 > cpu3/cache/index2/id:3 > cpu4/cache/index0/id:4 > cpu4/cache/index1/id:4 > cpu4/cache/index2/id:4 > cpu5/cache/index0/id:5 > cpu5/cache/index1/id:5 > cpu5/cache/index2/id:5 > cpu6/cache/index0/id:6 > cpu6/cache/index1/id:6 > cpu6/cache/index2/id:6 > cpu7/cache/index0/id:7 > cpu7/cache/index1/id:7 > cpu7/cache/index2/id:7 > > Basically all cache indices carry the APIC ID of the core, so L1D on > CPU0 has ID 0 and then L1I has ID 0 too and then L2 has also the same ID. The cache id should represent shared cpus on one cache level according to SDM. Could you check cpu#/cache/index#/shared_cpu_map or shared_cpu_list. According to the cache id, there is NO shared cpu on each level. The shared_cpu_map In your KVM should tell that as well, I think. I would guess all shared_cpu_map in each cache level on each cpu has itself on your KVM. > > How does that look on a CAT system? Do all the different cache levels get > different IDs? On my Broadwell EP, the cache id info is as follows: From index3/id, there are 2 cache ids for L3. L3 cache id 0 has shared_cpu_map: 00,003ffff0,0003ffff L3 cache id 1 has shared_cpu_map: ff,ffc0000f,fffc0000 Similar for L2 and L1. Cache id is unique on the same level across platform. #find /sys/device/system/cpu/. -name id|xargs cat 0 0 0 0 1 1 1 0 2 2 2 0 3 3 3 0 4 4 4 0 8 8 8 0 9 9 9 0 10 10 10 0 11 11 11 0 16 16 16 0 17 17 17 0 18 18 18 0 19 19 19 0 20 20 20 0 24 24 24 0 25 25 25 0 26 26 26 0 27 27 27 0 32 32 32 1 33 33 33 1 34 34 34 1 35 35 35 1 36 36 36 1 40 40 40 1 41 41 41 1 42 42 42 1 43 43 43 1 48 48 48 1 49 49 49 1 50 50 50 1 51 51 51 1 52 52 52 1 56 56 56 1 57 57 57 1 58 58 58 1 59 59 59 1 0 0 0 0 1 1 1 0 2 2 2 0 3 3 3 0 4 4 4 0 8 8 8 0 9 9 9 0 10 10 10 0 11 11 11 0 16 16 16 0 17 17 17 0 18 18 18 0 19 19 19 0 20 20 20 0 24 24 24 0 25 25 25 0 26 26 26 0 27 27 27 0 32 32 32 1 33 33 33 1 34 34 34 1 35 35 35 1 36 36 36 1 40 40 40 1 41 41 41 1 42 42 42 1 43 43 43 1 48 48 48 1 49 49 49 1 50 50 50 1 51 51 51 1 52 52 52 1 56 56 56 1 57 57 57 1 58 58 58 1 59 59 59 1
