Hey,
On Fri, 10 Jul 2015, Scott Mansfield wrote:
> We've seen issues recently where we run a cluster that typically has the
> majority of items overwritten in the same slab every day and a sudden change
> in data size evicts a ton of data, affecting downstream systems. To be clear
> that is our problem, but I think there's a tweak in memcached that might be
> useful and another possible feature that would be even
> better.
> The data that is written to this cache is overwritten every day, though the
> TTL is 7 days. One slab takes up the majority of the space in the cache. The
> application wrote e.g. 10KB (slab 21) every day for each key consistently.
> One day, a change occurred where it started writing 15KB (slab 23), causing a
> migration of data from one slab to another. We had -o
> slab_reassign,slab_automove=1 set on the server, causing large numbers of
> evictions on the initial slab. Let's say the cache could hold the data at
> 15KB per key, but the old data was not technically TTL'd out in it's old
> slab. This means that memory was not being freed by the lru crawler thread (I
> think) because its expiry had not come around.
>
> lines 1199 and 1200 in items.c:
> if ((search->exptime != 0 && search->exptime < current_time) ||
> is_flushed(search)) {
>
> If there was a check to see if this data was "orphaned," i.e. that the key,
> if accessed, would map to a different slab than the current one, then these
> orphans could be reclaimed as free memory. I am working on a patch to do
> this, though I have reservations about performing a hash on the key on the
> lru crawler thread (if the hash is not already available).
> I have very little experience in the memcached codebase so I don't know the
> most efficient way to do this. Any help would be appreciated.
There seems to be a misconception about how the slab classes work. A key,
if already existing in a slab, will always map to the slab class it
currently fits into. The slab classes always exist, but the amount of
memory reserved for each of them will shift with the slab_reassign. ie: 10
pages in slab class 21, then memory pressure on 23 causes it to move over.
So if you examine a key that still exists in slab class 21, it has no
reason to move up or down the slab classes.
> Alternatively, and possibly more beneficial is compaction of data in a slab
> using the same set of criteria as lru crawling. Understandably, compaction is
> a very difficult problem to solve since moving the data would be a pain in
> the ass. I saw a couple of discussions about this in the mailing list, though
> I didn't see any firm thoughts about it. I think it
> can probably be done in O(1) like the lru crawler by limiting the number of
> items it touches each time. Writing and reading are doable in O(1) so moving
> should be as well. Has anyone given more thought on compaction?
I'd be interested in hacking this up for you folks if you can provide me
testing and some data to work with. With all of the LRU work I did in
1.4.24, the next things I wanted to do is a big improvement on the slab
reassignment code.
Currently it picks essentially a random slab page, empties it, and moves
the slab page into the class under pressure.
One thing we can do is first examine for free memory in the existing slab,
IE:
- Take a page from slab 21
- Scan the page for valid items which need to be moved
- Pull free memory from slab 21, migrate the item (moderately complicated)
- When the page is empty, move it (or give up if you run out of free
chunks).
The next step is to pull from the LRU on slab 21:
- Take page from slab 21
- Scan page for valid items
- Pull free memory from slab 21, migrate the item
- If no memory free, evict tail of slab 21. use that chunk.
- When the page is empty, move it.
Then, when you hit this condition your least-recently-used data gets
culled as new data migrates your page class. This should match a natural
occurrance if you would already be evicting valid (but old) items to make
room for new items.
A bonus to using the free memory trick, is that I can use the amount of
free space in a slab class as a heuristic to more quickly move slab pages
around.
If it's still necessary from there, we can explore "upgrading" items to a
new slab class, but that is much much more complicated since the item has
to shift LRU's. Do you put it at the head, the tail, the middle, etc? It
might be impossible to make a good generic decision there.
What version are you currently on? If 1.4.24, have you seen any
instability? I'm currently torn between fighting a few bugs and start on
improving the slab rebalancer.
-Dormando
