Re: memcached with access controls

[Trond Norbye]

On 07/22/10 02:38 AM, KaiGai Kohei wrote:
> BTW, I have noticed a few symbols are not available unexpectedly when
> I've been under development.
>
> 1. create_instance() of the default_engine.c is not available, even if
>--enable-default-engine was provided.
> 2. safe_strtoull() of the util.c is not available from the engine module.
>
> It seems to me lack of the -rdynamic option drops these unreferenced
> symbols (at the build time) from the binary.
>
> The gcc man page says as follows:
> | -rdynamic
> |  Pass the flag -export-dynamic to the ELF linker, on targets that
> |  support it. This instructs the linker to add all symbols, not only
> |  used ones, to the dynamic symbol table. This option is needed
> |  for some uses of "dlopen" or to allow obtaining backtraces from
> |  within a program.
> and
> | -dynamic
> |  These options are passed to the Darwin linker.  The Darwin linker
> |  man page describes them in detail.
>
> How about the attached patch?
>
>   

The patch failed to build on my Solaris machine (not using gcc), but I
just took a quick look in the libtool manual and pushed another fix for
the problem in rev 456a9afd5ae007b9b3f8679b89a16f16ffe01689


> And, I have a question. Do we have an explicit release schedule of
> the memcached with engine interfaces?
>
>   

I'm currently working on cleaning up the builds on all of the platforms
we got builders for...

Cheers,

Trond

Re: Get multi error - too many keys returned

[Guille -bisho-]

On Thu, Jul 22, 2010 at 08:07, David Morel  wrote:
> On 21 juil, 10:59, Guille -bisho-  wrote:
>> On Wed, Jul 21, 2010 at 10:37, dormando  wrote:
>> >> I'm having an issue with my memcached farm. I'm using 3 memcached
>> >> servers (v1.2.6) and python-memcached client (v1.43).
>>
>> >> When using get_multi function, memcached returns sometimes keys that I
>> >> wasn't asking for. Memcached can return less data than expected (if
>> >> key-value does not exist), but documentation says nothing about
>> >> returning more data than it should. What can cause this?
>>
>> We have experienced similar behaviours with php memcache under heavy
>> load. Sometimes the clients seems to mix requests from different
>> replies.
>
> This happens when your client forks and forgets to reopen a
> connection, but re-uses the one that was open before the fork.
> As a results, several clients use the same socket which ends up
> interleaving the traffic.
> This is completely evil, not about memcache, really (it will happen
> just the same with any network protocol), and is solved by
> disconnecting and reconnecting.
> Also, it is only visible under heavy load because this is when you
> have a chance of seeing actually concurrent requests despite the very
> short roundtrip of a typical memcached query.
> Solutions: #1 make very sure you open the connection after the fork,
> or #2 have a simple check in your memcached layer that checks the PID
> hasn't changed. If it has, do a disconnect_all() or whatever it's
> called in your language, and then run your queries. There is some
> overhead when you adopt solution #2, but #1 is not always applicable
> if you have spaghetti code and/or lax coding policies.

Yeah, I know. PHP Memcache module is not really good. PECL Memcache
has his own implementation, and while more complete (supports UDP)
shows some problems like this, not too much fortunately. Memcached
(that uses libmemcached) seems more robust, but UDP support in
libmemcached is useless (has not implemented the wait for replies in
UDP).

It would be nice to give a bit more love to UDP implementation in
libmemcached (after all it's the official library and it's a pity his
implementation is not complete), so many drivers get advantage of a
well written client code.

-- 
Guille -ℬḭṩḩø- 
:wq

Re: memcached with access controls

[KaiGai Kohei]

(2010/07/22 17:09), Trond Norbye wrote:
> On 07/22/10 02:38 AM, KaiGai Kohei wrote:
>> BTW, I have noticed a few symbols are not available unexpectedly when
>> I've been under development.
>>
>> 1. create_instance() of the default_engine.c is not available, even if
>> --enable-default-engine was provided.
>> 2. safe_strtoull() of the util.c is not available from the engine module.
>>
>> It seems to me lack of the -rdynamic option drops these unreferenced
>> symbols (at the build time) from the binary.
>>
>> The gcc man page says as follows:
>> | -rdynamic
>> |  Pass the flag -export-dynamic to the ELF linker, on targets that
>> |  support it. This instructs the linker to add all symbols, not only
>> |  used ones, to the dynamic symbol table. This option is needed
>> |  for some uses of "dlopen" or to allow obtaining backtraces from
>> |  within a program.
>> and
>> | -dynamic
>> |  These options are passed to the Darwin linker.  The Darwin linker
>> |  man page describes them in detail.
>>
>> How about the attached patch?
>>
>>
> 
> The patch failed to build on my Solaris machine (not using gcc), but I
> just took a quick look in the libtool manual and pushed another fix for
> the problem in rev 456a9afd5ae007b9b3f8679b89a16f16ffe01689
> 
It works for me, on the Fedora rawhide.

>> And, I have a question. Do we have an explicit release schedule of
>> the memcached with engine interfaces?
> 
> I'm currently working on cleaning up the builds on all of the platforms
> we got builders for...
> 
Thanks for your efforts.
-- 
KaiGai Kohei 

Deserialization issue when cache object from WCF Service

[Adeel Nasim]

Hi,
We are using MemCache for caching in our ASP.NET Website for performance
improvements.



We use WCF Service at backend and our website and other Silverlight
applications call these WCF services.



Now, whenever a service call is made, service retrieves data from the
database and sets it in the MemCache server.



At client (application) end, before making service call, we first want to
check the memCache server, if it already has that particular object, but
unfortunately, this is not happening, we always get NULL if we want to check
data from the cache server.


On further digging it down, we came to know that when MemCache de-serialize
our object at client end (website), it doesn’t find the DLL of the service
and throws the exception at de-serialization and returns NULL object.

When i put the WCF service DLL in Bin folder of my website (that i don't
want to do) than the de-serialzation working fine but i stuck into another
problem when i tried cast the object into WCF service reference object, it
gives error of invalide cast because the deserialized object (deserialize by
service DLL) cannot converted into service reference object due to different
namespace.

 My question is, does .NET framework has any support or workaround by which
we can de-serialize byte array into real objects without having the DLL and
using only the service references.

Thanks
Adeel

Re: Deserialization issue when cache object from WCF Service

[a.]

you. cannot deserialize real clr objects without having their code around. 
(when using the default .net serializer)

what you can do in this instance is to use an intermediary format, for example 
serialize all the objects into a json string and store this string in the 
cache. the other end can deserialize that string indo a dictionay or another 
object which looks the sqme as the source.

On Jul 22, 2010, at 4:58 PM, Adeel Nasim  wrote:

> Hi,
> We are using MemCache for caching in our ASP.NET Website for performance 
> improvements.
>  
> We use WCF Service at backend and our website and other Silverlight 
> applications call these WCF services.
>  
> Now, whenever a service call is made, service retrieves data from the 
> database and sets it in the MemCache server.
>  
> At client (application) end, before making service call, we first want to 
> check the memCache server, if it already has that particular object, but 
> unfortunately, this is not happening, we always get NULL if we want to check 
> data from the cache server.
>  
> On further digging it down, we came to know that when MemCache de-serialize 
> our object at client end (website), it doesn’t find the DLL of the service 
> and throws the exception at de-serialization and returns NULL object.
>  
> When i put the WCF service DLL in Bin folder of my website (that i don't want 
> to do) than the de-serialzation working fine but i stuck into another problem 
> when i tried cast the object into WCF service reference object, it gives 
> error of invalide cast because the deserialized object (deserialize by 
> service DLL) cannot converted into service reference object due to different 
> namespace.
>  
>  My question is, does .NET framework has any support or workaround by which 
> we can de-serialize byte array into real objects without having the DLL and 
> using only the service references.
>  
> Thanks
> Adeel

Re: Deserialization issue when cache object from WCF Service

[a.]

or maybe you could the datacontractserializer could help... it depends on
the client you're using

if you're using the "Enyim" client, please send me an email (let's take this
off the list) and i'll try to help


a.


2010/7/22 a. 

> you. cannot deserialize real clr objects without having their code around.
> (when using the default .net serializer)
>
> what you can do in this instance is to use an intermediary format, for
> example serialize all the objects into a json string and store this string
> in the cache. the other end can deserialize that string indo a dictionay or
> another object which looks the sqme as the source.
>
>
> On Jul 22, 2010, at 4:58 PM, Adeel Nasim  wrote:
>
> Hi,
> We are using MemCache for caching in our ASP.NET Website
> for performance improvements.
>
>
>
> We use WCF Service at backend and our website and other Silverlight
> applications call these WCF services.
>
>
>
> Now, whenever a service call is made, service retrieves data from the
> database and sets it in the MemCache server.
>
>
>
> At client (application) end, before making service call, we first want to
> check the memCache server, if it already has that particular object, but
> unfortunately, this is not happening, we always get NULL if we want to check
> data from the cache server.
>
>
> On further digging it down, we came to know that when MemCache de-serialize
> our object at client end (website), it doesn’t find the DLL of the service
> and throws the exception at de-serialization and returns NULL object.
>
> When i put the WCF service DLL in Bin folder of my website (that i don't
> want to do) than the de-serialzation working fine but i stuck into another
> problem when i tried cast the object into WCF service reference object, it
> gives error of invalide cast because the deserialized object (deserialize by
> service DLL) cannot converted into service reference object due to different
> namespace.
>
>  My question is, does .NET framework has any support or workaround by
> which we can de-serialize byte array into real objects without having the
> DLL and using only the service references.
>
> Thanks
> Adeel
>
>

Re: Using PCIe SSDs instead of RAM

[Jakub Łopuszański]

I see that my patch for garbage collection is still being ignored, and your
post gives me some idea about why it is so.
I think that RAM is a real problem, because currently (without GC) you have
no clue about how much RAM you really need. So you can end up blindly buying
more and more machines, which effectively means that multiget works worse
and worse (client issues one big multiget but it gets split into many
packets to many servers).
Currently we try to get number of servers in the cluster smaller based on
the reall consumption to get more from multiget feature.

So I believe that there is an important connection between RAM and speed,
and this connection is number of servers in the cluster.

On Wed, Jul 21, 2010 at 1:51 PM, Guille -bisho-  wrote:

> Many memcache users are more interested in latency than in huge
> amounts of memory to cache. The drive you mention is 26µs [1] compared
> with ~22.5 ns [2], 3 orders of magnitude more.
>
> If your application is accesing a lot of small memcache data to
> process a page, the increased latency will be noticed. If it's just
> for caching just full content (full pages) might be interesting, but
> then you might be more interested in something like varnish that uses
> regular disks and memory as cache.
>
> Whats is your use case?
>
> [1] http://www.fusionio.com/products/iodrive/?tab=specs
> [2]
> http://en.wikipedia.org/wiki/Dynamic_random_access_memory#Memory_timing
>
> --
> Guille -ℬḭṩḩø- 
> :wq
>

Re: Using PCIe SSDs instead of RAM

[Brian Moon]

On 7/22/10 5:46 AM, Jakub Łopuszański wrote:

I see that my patch for garbage collection is still being ignored, and
your post gives me some idea about why it is so.
I think that RAM is a real problem, because currently (without GC) you
have no clue about how much RAM you really need. So you can end up
blindly buying more and more machines, which effectively means that
multiget works worse and worse (client issues one big multiget but it
gets split into many packets to many servers).
Currently we try to get number of servers in the cluster smaller based
on the reall consumption to get more from multiget feature.


I would never, never, never want my memcached daemon ram usage to 
fluctuate wildly. Eviction rate is a much better determination of how 
well your cache is being used.


--

Brian.

http://brian.moonspot.net/

Re: Using PCIe SSDs instead of RAM

[Jakub Łopuszański]

Well, I beg to differ.
We used to have evictions > 0, actually around 200 (per whatever munin
counts them), so we used to think, that we have too small number of
machines, and kept adding them.
After using the patch, the memory usage dropped by 80%, and we have no
evictions since a long time, which means, that evictions where misleading,
and happened just because LRU sometimes kills fresh items, even though there
are lots of outdated keys.

Moreover it's not like RAM usage "fluctuates wildly". It's kind of constant,
or at least periodic, so you can very accurately say if something bad
happened, as it would be instantly visible as a deviation from yesterday's
charts. Before applying the patch, you could as well not look at the chart
at all, as it was more than sure that it always shows 100% usage, which in
my opinion gives no clue about what is actually going on.

Even if you are afraid of "wildly fluctuating" charts, you will not solve
the problem by hiding it, and this is what actually happens if you don't
have GC -- the traffic, the number of outdated keys, they all fluctuate, but
you just don't see it, if the chart always shows 100% usage...

2010/7/22 Brian Moon 

> On 7/22/10 5:46 AM, Jakub Łopuszański wrote:
>
>> I see that my patch for garbage collection is still being ignored, and
>> your post gives me some idea about why it is so.
>> I think that RAM is a real problem, because currently (without GC) you
>> have no clue about how much RAM you really need. So you can end up
>> blindly buying more and more machines, which effectively means that
>> multiget works worse and worse (client issues one big multiget but it
>> gets split into many packets to many servers).
>> Currently we try to get number of servers in the cluster smaller based
>> on the reall consumption to get more from multiget feature.
>>
>
> I would never, never, never want my memcached daemon ram usage to fluctuate
> wildly. Eviction rate is a much better determination of how well your cache
> is being used.
>
> --
>
> Brian.
> 
> http://brian.moonspot.net/
>

Re: Using PCIe SSDs instead of RAM

[Brian Moon]

On 7/22/10 2:02 PM, Jakub Łopuszański wrote:

Well, I beg to differ.
We used to have evictions > 0, actually around 200 (per whatever munin
counts them), so we used to think, that we have too small number of
machines, and kept adding them.
After using the patch, the memory usage dropped by 80%, and we have no
evictions since a long time, which means, that evictions where
misleading, and happened just because LRU sometimes kills fresh items,
even though there are lots of outdated keys.


Let me make sure I understand your claim here. You are claiming that the 
LRU is evicting things even though there are expired items in the slabs? 
And that expired items are left in the slabs and non-expired items are 
removed from the slab by the LRU? That is your claim? I just want to be 
clear.



Moreover it's not like RAM usage "fluctuates wildly". It's kind of
constant, or at least periodic, so you can very accurately say if
something bad happened, as it would be instantly visible as a deviation
from yesterday's charts. Before applying the patch, you could as well
not look at the chart at all, as it was more than sure that it always
shows 100% usage, which in my opinion gives no clue about what is
actually going on.

Even if you are afraid of "wildly fluctuating" charts, you will not
solve the problem by hiding it, and this is what actually happens if you
don't have GC -- the traffic, the number of outdated keys, they all
fluctuate, but you just don't see it, if the chart always shows 100%
usage...


It has nothing to do with fear. It has to do with managing resources. A 
sudden peak in evictions is much better than a sudden lack of memory on 
all my memcached servers.  Evictions > OOM.


--

Brian.

http://brian.moonspot.net/

Re: Using PCIe SSDs instead of RAM

[dormando]

http://code.google.com/p/memcached/wiki/NewServerMaint#Looks_Can_be_Deceiving

Think I'll write a separate page about managing memory, based off of the
slides from my mysqlconf presentation about monitoring memcached...

We're not ignoring you, the patch is against what the LRU is designed for.
Several people have argued to put garbage collection back into memcached,
but it just doesn't mix.

In the interest of being constructive, you should look back through the
mailing list for details on the storage engine branch, and if you really
want it to work, it'd be a good exercise to implement this as a custom
storage engine.

In the interest of being thorough; you proved your own patch unnecessary
by noting that the hitrate did not change. It just confirmed you weren't
having a problem.

The short notes of my slides are just:

- Note evictions over time
- Note hitrate over time
- Investigate changes to either via a traffic snapshot from maatkit,
either on your memcached server or from an app server. Or setup one app
server to log its memcached traffic. whatever you need to do.
- Note your DB load as well, and correlate *all* of these numbers.

You'll get way more useful information out of the *flow* through memcached
than from *what's inside it*. What's inside it doesn't matter, at all!

Keep your hitrate stable, investigate what your app is doing when it
changes. If there's nothing for you to fix and the hitrate is dropping, db
load is increasing, add more memcached servers. It's really really simple.
Honestly! Looking at just one stat and making that decision is pretty
weird.

In your case, you were seeing evictions despite 50% of your memory being
loaded with expired items. Neither of these things are a problem or even
matter, because:

- expired items are freed when they're fetched
- evicted items are picked off of the tail of the LRU

which means that *neither* the expired items or the evicted items are
being accessed at all. You have unexpired items which are being accessed
less frequently than stuff that's being expired!

It *could* indicate a problem, but simply garbage collecting will actually
*hide* it from you! You'll find it by analyzing your miss's and set's. You
might then see that your app is uselessly setting hundreds of keys every
time a user loads their profile, or frontpage, or whatever. Those keys
then expire without ever being used again.

That should lead you into a *real* benefit of not wasting time setting
extraneous keys, or fetching keys that never exist, or finding places to
combine data or issue multigets more correctly.

With respect to your multiget note, I went over this in quite a bit of
detail: http://dormando.livejournal.com/521163.html

If you're multiget'ing related data, there's zero reason for it to hit
more than one memcached instance. Except maybe you're fetching mass
numbers of huge keys and it makes more sense for the TCP sessions to be
split up in parallel. I dunno.

In one final note, I'd really really appreciate it if you could stop
hijacking threads to promote your patch. It's pretty rude, as your garbage
collector issue has been discussed on the list several times.

On Thu, 22 Jul 2010, Jakub Łopuszański wrote:

> Well, I beg to differ.
> We used to have evictions > 0, actually around 200 (per whatever munin counts 
> them), so we used to think, that we have too small number of machines, and 
> kept adding them.
> After using the patch, the memory usage dropped by 80%, and we have no 
> evictions since a long time, which means, that evictions where misleading, 
> and happened just because LRU sometimes kills fresh items,
> even though there are lots of outdated keys.
>
> Moreover it's not like RAM usage "fluctuates wildly". It's kind of constant, 
> or at least periodic, so you can very accurately say if something bad 
> happened, as it would be instantly visible as a deviation
> from yesterday's charts. Before applying the patch, you could as well not 
> look at the chart at all, as it was more than sure that it always shows 100% 
> usage, which in my opinion gives no clue about what is
> actually going on.
>
> Even if you are afraid of "wildly fluctuating" charts, you will not solve the 
> problem by hiding it, and this is what actually happens if you don't have GC 
> -- the traffic, the number of outdated keys, they
> all fluctuate, but you just don't see it, if the chart always shows 100% 
> usage...
>
> 2010/7/22 Brian Moon 
>   On 7/22/10 5:46 AM, Jakub Łopuszański wrote:
> I see that my patch for garbage collection is still being 
> ignored, and
> your post gives me some idea about why it is so.
> I think that RAM is a real problem, because currently (without 
> GC) you
> have no clue about how much RAM you really need. So you can end up
> blindly buying more and more machines, which effectively means 
> that
> multiget works worse and worse (client issues one big multiget 
> but it

Re: Issue 122 in memcached: failed to write, and not due blocking: No error

[memcached]

Comment #1 on issue 122 by billyle...@suddenlink.net: failed to write, and  
not due blocking: No error

http://code.google.com/p/memcached/issues/detail?id=122

I have just installed win32-1.4.4-54 server along with  
memcacheddotnet_clientlib-1.1.5 and am getting 'Failed to write, and not  
due to blocking: No error' every time my client calls the server(Get, Set,  
Stats, i.e.).  I have tried tweaking the command line args with no  
success.  Any ideas.  I've run memcached on linux for years without any  
type of problems but as usual there is probably an issue with something in  
windoze7.


thx.

Re: Using PCIe SSDs instead of RAM

[Jakub Łopuszański]

While I agree with most of your thesis, I can't see how GC is against the
LRU.

I agree, that often accessed keys with short TTL seem strange, and so do
rarely accessed keys with long TTL. But there are lots of perfect reasons to
have such situation, and we do.
GC does not work against the LRU (at least I can't see it), it cooperates.
Apparently LRU is never used, because you have smaller chances to run out of
memory, but I'd like to answer doubts of Brian Moon: in case whole memory is
occupied you will not get "sudden lack of memory", but just the usuall
thing: LRU will start to evict oldest items.
I agree that monitoring hitrates and evictions makes sens, but you can
forcast problems much sooner if you monitor number of unexpired items, as
well.
The point is: GC does not forbid you from using your regular monitoring
tools, skills and procedures. It just gives you another tool: live
monitoring of unexpired items.
I see nothing bad about it:)

Scenario 1. You are releasing new feature, and you want to scale the number
of servers accordingly to the load. You can monitor memory usage as the
users join, extrapolate, and order new machines much sooner, than by
monitoring evictions, as evictions indicate that you already have a problem.
Scenario 2. You need to steal machines from one cluster to help build
another one, and you have to decide if you can do so safely without risking
that the old cluster will "run of memory". Again monitoring evictions can
not reliably tell you how many machines can you remove from the cluster,
while monitoring memory gives you perfectly accurate info.


On Fri, Jul 23, 2010 at 12:12 AM, dormando  wrote:

>
> http://code.google.com/p/memcached/wiki/NewServerMaint#Looks_Can_be_Deceiving
>
> Think I'll write a separate page about managing memory, based off of the
> slides from my mysqlconf presentation about monitoring memcached...
>
> We're not ignoring you, the patch is against what the LRU is designed for.
> Several people have argued to put garbage collection back into memcached,
> but it just doesn't mix.
>
> In the interest of being constructive, you should look back through the
> mailing list for details on the storage engine branch, and if you really
> want it to work, it'd be a good exercise to implement this as a custom
> storage engine.
>
> In the interest of being thorough; you proved your own patch unnecessary
> by noting that the hitrate did not change. It just confirmed you weren't
> having a problem.
>
> The short notes of my slides are just:
>
> - Note evictions over time
> - Note hitrate over time
> - Investigate changes to either via a traffic snapshot from maatkit,
> either on your memcached server or from an app server. Or setup one app
> server to log its memcached traffic. whatever you need to do.
> - Note your DB load as well, and correlate *all* of these numbers.
>
> You'll get way more useful information out of the *flow* through memcached
> than from *what's inside it*. What's inside it doesn't matter, at all!
>
> Keep your hitrate stable, investigate what your app is doing when it
> changes. If there's nothing for you to fix and the hitrate is dropping, db
> load is increasing, add more memcached servers. It's really really simple.
> Honestly! Looking at just one stat and making that decision is pretty
> weird.
>
> In your case, you were seeing evictions despite 50% of your memory being
> loaded with expired items. Neither of these things are a problem or even
> matter, because:
>
> - expired items are freed when they're fetched
> - evicted items are picked off of the tail of the LRU
>
> which means that *neither* the expired items or the evicted items are
> being accessed at all. You have unexpired items which are being accessed
> less frequently than stuff that's being expired!
>
> It *could* indicate a problem, but simply garbage collecting will actually
> *hide* it from you! You'll find it by analyzing your miss's and set's. You
> might then see that your app is uselessly setting hundreds of keys every
> time a user loads their profile, or frontpage, or whatever. Those keys
> then expire without ever being used again.
>
> That should lead you into a *real* benefit of not wasting time setting
> extraneous keys, or fetching keys that never exist, or finding places to
> combine data or issue multigets more correctly.
>
> With respect to your multiget note, I went over this in quite a bit of
> detail: http://dormando.livejournal.com/521163.html
>
> If you're multiget'ing related data, there's zero reason for it to hit
> more than one memcached instance. Except maybe you're fetching mass
> numbers of huge keys and it makes more sense for the TCP sessions to be
> split up in parallel. I dunno.
>
> In one final note, I'd really really appreciate it if you could stop
> hijacking threads to promote your patch. It's pretty rude, as your garbage
> collector issue has been discussed on the list several times.
>
> On Thu, 22 Jul 2010, Jakub Łopuszański wrote:
>
> >

Re: Using PCIe SSDs instead of RAM

[dormando]

I tried.

Try the engine branch?

On Fri, 23 Jul 2010, Jakub Łopuszański wrote:

> While I agree with most of your thesis, I can't see how GC is against the LRU.
>
> I agree, that often accessed keys with short TTL seem strange, and so do 
> rarely accessed keys with long TTL. But there are lots of perfect reasons to 
> have such situation, and we do.
> GC does not work against the LRU (at least I can't see it), it cooperates. 
> Apparently LRU is never used, because you have smaller chances to run out of 
> memory, but I'd like to answer doubts of Brian Moon:
> in case whole memory is occupied you will not get "sudden lack of memory", 
> but just the usuall thing: LRU will start to evict oldest items.
> I agree that monitoring hitrates and evictions makes sens, but you can 
> forcast problems much sooner if you monitor number of unexpired items, as 
> well.
> The point is: GC does not forbid you from using your regular monitoring 
> tools, skills and procedures. It just gives you another tool: live monitoring 
> of unexpired items.
> I see nothing bad about it:)
>
> Scenario 1. You are releasing new feature, and you want to scale the number 
> of servers accordingly to the load. You can monitor memory usage as the users 
> join, extrapolate, and order new machines much
> sooner, than by monitoring evictions, as evictions indicate that you already 
> have a problem.
> Scenario 2. You need to steal machines from one cluster to help build another 
> one, and you have to decide if you can do so safely without risking that the 
> old cluster will "run of memory". Again monitoring
> evictions can not reliably tell you how many machines can you remove from the 
> cluster, while monitoring memory gives you perfectly accurate info.
>
>
> On Fri, Jul 23, 2010 at 12:12 AM, dormando  wrote:
>   
> http://code.google.com/p/memcached/wiki/NewServerMaint#Looks_Can_be_Deceiving
>
>   Think I'll write a separate page about managing memory, based off of the
>   slides from my mysqlconf presentation about monitoring memcached...
>
>   We're not ignoring you, the patch is against what the LRU is designed 
> for.
>   Several people have argued to put garbage collection back into 
> memcached,
>   but it just doesn't mix.
>
>   In the interest of being constructive, you should look back through the
>   mailing list for details on the storage engine branch, and if you really
>   want it to work, it'd be a good exercise to implement this as a custom
>   storage engine.
>
>   In the interest of being thorough; you proved your own patch unnecessary
>   by noting that the hitrate did not change. It just confirmed you weren't
>   having a problem.
>
>   The short notes of my slides are just:
>
>   - Note evictions over time
>   - Note hitrate over time
>   - Investigate changes to either via a traffic snapshot from maatkit,
>   either on your memcached server or from an app server. Or setup one app
>   server to log its memcached traffic. whatever you need to do.
>   - Note your DB load as well, and correlate *all* of these numbers.
>
>   You'll get way more useful information out of the *flow* through 
> memcached
>   than from *what's inside it*. What's inside it doesn't matter, at all!
>
>   Keep your hitrate stable, investigate what your app is doing when it
>   changes. If there's nothing for you to fix and the hitrate is dropping, 
> db
>   load is increasing, add more memcached servers. It's really really 
> simple.
>   Honestly! Looking at just one stat and making that decision is pretty
>   weird.
>
>   In your case, you were seeing evictions despite 50% of your memory being
>   loaded with expired items. Neither of these things are a problem or even
>   matter, because:
>
>   - expired items are freed when they're fetched
>   - evicted items are picked off of the tail of the LRU
>
>   which means that *neither* the expired items or the evicted items are
>   being accessed at all. You have unexpired items which are being accessed
>   less frequently than stuff that's being expired!
>
>   It *could* indicate a problem, but simply garbage collecting will 
> actually
>   *hide* it from you! You'll find it by analyzing your miss's and set's. 
> You
>   might then see that your app is uselessly setting hundreds of keys every
>   time a user loads their profile, or frontpage, or whatever. Those keys
>   then expire without ever being used again.
>
>   That should lead you into a *real* benefit of not wasting time setting
>   extraneous keys, or fetching keys that never exist, or finding places to
>   combine data or issue multigets more correctly.
>
>   With respect to your multiget note, I went over this in quite a bit of
>   detail: http://dormando.livejournal.com/521163.html
>
>   If you're multiget'ing related data, there's zero reason for it to