On 5 November 2014 00:38, David Gwynne <da...@gwynne.id.au> wrote: > >> On 30 Oct 2014, at 07:52, Ted Unangst <t...@tedunangst.com> wrote: >> >> On Wed, Oct 29, 2014 at 07:25, David Gwynne wrote: >> >> >>> i dunno. im fine with either removing colouring altogether or setting it >>> from something else completely. i just want a decision to be made cos >>> right now ph_color isnt set, which is a bug. >> >> there. i fixed it. > > looks like we were both ignorant and wrong. mikeb@ points out this from the > original slab paper: > > 4.1. Impact of Buffer Address Distribution on Cache > Utilization > > The address distribution of mid-size buffers can > affect the system’s overall cache utilization. In par- > ticular, power-of-two allocators - where all buffers > are 2 n bytes and are 2 n -byte aligned - are pes- > simal.* Suppose, for example, that every inode > (∼ 300 bytes) is assigned a 512-byte buffer, 512-byte > aligned, and that only the first dozen fields of an > inode (48 bytes) are frequently referenced. Then > the majority of inode-related memory traffic will be > at addresses between 0 and 47 modulo 512. Thus > the cache lines near 512-byte boundaries will be > heavily loaded while the rest lie fallow. In effect > only 9% (48/512) of the cache will be usable by > inodes. Fully-associative caches would not suffer > this problem, but current hardware trends are toward > simpler rather than more complex caches. > > 4.3. Slab Coloring > > The slab allocator incorporates a simple coloring > scheme that distributes buffers evenly throughout > the cache, resulting in excellent cache utilization > and bus balance. The concept is simple: each time > a new slab is created, the buffer addresses start at a > slightly different offset (color) from the slab base > (which is always page-aligned). For example, for a > cache of 200-byte objects with 8-byte alignment, the > first slab’s buffers would be at addresses 0, 200, > 400, ... relative to the slab base. The next slab’s > buffers would be at offsets 8, 208, 408, ... and so > on. The maximum slab color is determined by the > amount of unused space in the slab. > > > we run on enough different machines that i think we should consider this. >
well, first of all, right now this is a rather theoretical gain. we need to test it to understand if it makes things easier. to see cache statistics we can use performance counters, however current pctr code might be a bit out of date. > so the question is if we do bring colouring back, how do we calculate it? > arc4random? mask bits off ph_magic? atomic_inc something in the pool? > read a counter from the pool? shift bits off the page address? the way i read it is that you have a per-pool running value pr_color that you increment by the item alignment or native cache line size modulo space available for every page you are getting from uvm. however i can see that it might entail a problem by locating a page header (or was it page boundary? don't have the code at hand) using simple math.
