You're probably getting spaces or newlines into your keys, which can cause the client protocol to desync with the server. Then you'll get all sorts of junk into random keys (or random responses from keys which're fine).
Either filtering those or using the binary protocol should fix that for you. On Wed, 19 Nov 2014, labnext...@gmail.com wrote: > Hi Boris, > I think I may have mislead you. It is not one or two keys that get > corrupted, it seems that most (if not all) keys fetched return incorrect data. > For example during one of these failures (just this morning), a session key > (prefixed with session_) returned an array related to a customer > record (prefixed with lab_), a key related to a customer return a string > related to a translation, and a key related to a translation returned.... > > All heck breaks loose (seemingly) across all keys. A flush brings things > back into the fold. > > Make sense? > > Thanks, > > Mike > > > On Wednesday, November 19, 2014 2:22:50 PM UTC-4, Boris wrote: > I can think of many ways to screw up an application in a way that you > describe. Simple programmer error can lead to this sort of > behavior. I'd just log every time you do a set for that key with value > type you are setting. > > On Wed, Nov 19, 2014 at 1:00 PM, <labne...@gmail.com> wrote: > Thanks Boris, > I haven't really given that much thought. Out of curiosity, why do you think > the issue might be on the client end? I ask, cause I > really don't have a sense of what to look for on that end and wonder if you > might have some suggestions. > > Best, > > Mike > > > On Wednesday, November 19, 2014 12:46:16 PM UTC-4, Boris wrote: > Hi Mike, this sounds to me more like a client/coding error rather than > memcached server. That's where I would focus first. > Boris > > On Wed, Nov 19, 2014 at 11:41 AM, <labne...@gmail.com> wrote: > I just had another failure. After pulling down my apache web servers, > and before restarting memcached I grabbed > stats to see if they showed anything of interest: > - All 3 servers were reporting for duty following a getServerStatus (PHP > client call) > - curr_connections were listed as 8 across all the instances (apache was > down but cron jobs up, so that would have dropped > things down considerably) > - listen_disabled_num was listed as 0 across all the instances > - accepting_conns was listed as 1 across all the instances > - evictions listed as 0 > - All items across all instances had an evicted and evicted_nonzero and > evicted_time value of 0 > - All slabs across all instances had a total_pages value of 1 > - tailrepairs and outofmemory is listed with a value of 0 across all items > in each instance > - global hit rate is 0.9937 > - get_hits is always* greater than cmd_set on a per slab basis. *One slab > reported both values as equal > > > As far as I can tell, memcache is reporting that the world is fine and dandy. > Should I be enlarging scope of the search to > look at OS related factors that could result in the client receiving bad > data? None of the machines are dipping into swap. > > Thanks, > > Mike > > > > On Wednesday, November 19, 2014 9:35:19 AM UTC-4, labne...@gmail.com wrote: > For what it is worth, I'm hesitant to upgrade memcached to the latest > version as a step to try and solve this > issue. It seems to me that since our installs have been running > without issue for quite some time (close to a > year), that there are other variables at play here. I just don't > understand the variables. ;) > Thanks, > > Mike > > > On Tuesday, November 18, 2014 2:00:46 PM UTC-4, labne...@gmail.com wrote: > Hi There, > I'm trying to diagnose a new problem with Memcache that seems to be happening > with greater frequency. The > issue has to do with memcache get requests returning incorrect responses > (data from from other keys returned). > Restarting or flushing the servers seems to resolve the issue. > > Do any memcache veterans have any suggestions of how I might dig into this > issue? Stats that I might want to > trace, log files to look at, etc? Does maybe this symptom fit the > description of any known issues? > > I'm keeping a casual eye on on curr_connections, listen_disabled_num, > accepting_conns, bytes, and > limit_maxbytes (all show nothing unusual). I've verified that all servers > and clients are set up in a > consistent fashion. I'm not sure where to go from here to better understand > the problem. > > > If it helps, I'm running 1.4.13 (ubuntu 12.04 LTS) across 3 servers, > connecting in with PHP Memcache 3.0.6 > > > Tips? > > Mike > > > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > >
