Diagnosing Corruption

2014-11-18 Thread labnextddx
Hi There, I'm trying to diagnose a new problem with Memcache that seems to be happening with greater frequency. The issue has to do with memcache get requests returning incorrect responses (data from from other keys returned). Restarting or flushing the servers seems to resolve the issue. D

Re: Diagnosing Corruption

2014-11-19 Thread labnextddx
For what it is worth, I'm hesitant to upgrade memcached to the latest version as a step to try and solve this issue. It seems to me that since our installs have been running without issue for quite some time (close to a year), that there are other variables at play here. I just don't understa

Re: Diagnosing Corruption

2014-11-19 Thread labnextddx
I just had another failure. After pulling down my apache web servers, and before restarting memcached I grabbed stats to see if they showed anything of interest: - All 3 servers were reporting for duty following a getServerStatus (PHP client call) - curr_connections were listed as 8 across a

Re: Diagnosing Corruption

2014-11-19 Thread Boris Partensky
Hi Mike, this sounds to me more like a client/coding error rather than memcached server. That's where I would focus first. Boris On Wed, Nov 19, 2014 at 11:41 AM, wrote: > I just had another failure. After pulling down my apache web servers, and > before restarting memcached I grabbed stats to

Re: Diagnosing Corruption

2014-11-19 Thread labnextddx
Thanks Boris, I haven't really given that much thought. Out of curiosity, why do you think the issue might be on the client end? I ask, cause I really don't have a sense of what to look for on that end and wonder if you might have some suggestions. Best, Mike On Wednesday, November 19, 20

Re: Diagnosing Corruption

2014-11-19 Thread Boris Partensky
I can think of many ways to screw up an application in a way that you describe. Simple programmer error can lead to this sort of behavior. I'd just log every time you do a set for that key with value type you are setting. On Wed, Nov 19, 2014 at 1:00 PM, wrote: > Thanks Boris, > > I haven't real

Re: Diagnosing Corruption

2014-11-19 Thread labnextddx
Hi Boris, I think I may have mislead you. It is not one or two keys that get corrupted, it seems that most (if not all) keys fetched return incorrect data. For example during one of these failures (just this morning), a session key (prefixed with session_) returned an array related to a cust

Re: Diagnosing Corruption

2014-11-19 Thread dormando
You're probably getting spaces or newlines into your keys, which can cause the client protocol to desync with the server. Then you'll get all sorts of junk into random keys (or random responses from keys which're fine). Either filtering those or using the binary protocol should fix that for you.

RE: Diagnosing Corruption

2014-11-19 Thread Jason CHAN
ando Sent: Thursday, November 20, 2014 9:09 AM To: memcached@googlegroups.com Subject: Re: Diagnosing Corruption You're probably getting spaces or newlines into your keys, which can cause the client protocol to desync with the server. Then you'll get all sorts of junk into random

Re: Diagnosing Corruption

2014-11-20 Thread labnextddx
Thanks Dormando, I just ran some tests to see if I can reproduce the issue by introducing a space in a key. I'm using Zend_Cache which under the hood is using PHP Memcache. Zend_Cache does a regex on each key before hitting the actual memcache client code. The regex tosses an exception when

Re: Diagnosing Corruption

2014-11-20 Thread labnextddx
use 1.4.15 > and 1.4.20 in a big cache cluster). > Good luck. > > -Original Message- > From: memc...@googlegroups.com [mailto: > memc...@googlegroups.com ] On Behalf Of dormando > Sent: Thursday, November 20, 2014 9:09 AM > To: memc...@googlegroups.com