> > Jack Steiner is trying to determine how much more expensive the off-node > page tables are than on node. Once we know that, we will know if a > per-cpu or per-node quicklist type arrangement is truely beneficial or > if a general use slab without node awareness will be adequate. > > Thanks, > Robin
Here is the results of a test that measures the benefit of
node-local page tables vs page tables located on a remote
node. This was run on a 64p Altix system using 900MHz/1.5M L3
cpus.
The test is very likely a ~worse-case test. It was contrived to
maximize the number of TLB misses to page table entries that
are not in the cache of the processor. The test references
memory with a 32M+128 byte stride. The maximizes data cache
hits & minimizes PT cache hits. The page table pages were all
allocated on 256K boundaries to ensure maximum cache conflicts
for PT references by the VHPT.
Granted, this is a worse case test but I wanted to see
if I could measure the effect. Although this test is atypical of
most apps, it is similar to some very large pointer chasing
applications in the real world.
The test essentially follows a linked list of pointers. When the number of
entries in the list exceed the size of the TLB, TLB misses occur &
the processor must make references to the in-memory page table.
Memory Reference Time (ns/reference) for Pointer Chasing Test
POINTERS PT_LOCAL PT_REMOTE
1 5.621 5.621
10 4.226 4.226
20 3.224 3.224
30 2.890 2.890
40 11.418 11.418
50 11.357 11.357
60 11.316 11.316
70 11.287 11.286
80 11.329 11.302
90 11.292 12.082
100 11.272 12.763
110 11.974 16.075
120 51.758 132.389 << exceeds TLB capacity here
130 119.311 382.279
140 143.726 466.469
150 158.851 495.946
160 164.415 515.392
170 168.822 525.382
180 168.057 537.250
190 173.043 536.515
300 192.804 632.459
400 204.916 666.561
500 230.489 693.896
600 256.716 725.338
700 286.068 731.309
800 293.407 740.035
900 306.296 747.635
I've also attach a graph. It is prettier but may not be friendly
to all mail readers.
--
Thanks
Jack Steiner ([EMAIL PROTECTED]) 651-683-5302
Principal Engineer SGI - Silicon Graphics, Inc.
<<attachment: z.png>>
