Re: Multi-get getting stuck
Hi Brian, Sorry, I didn't have net access for a few days. They keys were actually pretty simple since I was just doing some basic testing. they were of the form: key[i] = xtoa(i)+foo2; Where i was just a loop variable ranging from 0 to 5000 Their lengths wouldn't exceed 10 characters. Thanks, Prasad On Sep 29, 8:22 pm, Brian Aker br...@tangent.org wrote: Hi! On Sep 29, 2011, at 2:30 PM, Prasad wrote: And so with this fix does the client call a fetch after a small chunk of multigets are reached inherently? Since I don't call the fetch till I return from the mget API call. There is a water mark that causes the mget to go multi if it is too long. The specific watermark is not something I recall (i.e. I would need to pull up the code). Could you provide for me a sample set of keys? I'd like to create a test case for this. Cheers, -Brian
Re: Multi-get getting stuck
I am trying to use mget() to get multiple data pieces using the C++ API. When it works the performance difference is great. However, I seem to hit some sort of a communication hang when I have about 5000 or higher key requests together. What client is that, exactly? The C++ API isn't specific enough. Has anyone seen this issue earlier? Is there a size limit to how much combined data can be requested using multiget?(I read its 32 MB somewhere but this happens for even smaller data sizes). The hanging is during the call to mget() (even before calling fetch) and it seems the server gets stuck trying to handle too many requests. Any idea if I might be overrunning the server buffer/ TCPIP buffer? If you're using the binary protocol, the server starts sending back responses as soon as you send in requests. Eventually your read buffer fills and you won't be able to send any packets until you first read some. iirc libmemcached has a workaround for this by starting to call the read handler callbacks while sending requests still. -Dormando
Re: Multi-get getting stuck
Hi! On Thu, Sep 29, 2011 at 7:47 AM, Prasad pspha...@gmail.com wrote: I am trying to use mget() to get multiple data pieces using the C++ API. When it works the performance difference is great. Is this the one in libmemcached? If so use the C API. The C++ API is just there for a couple of folks who wanted to have one. Its a wrapper around the C api and it wouldn't surprise me if it was slower. In version 2.0 we are likely to completely rewrite it. Cheers, -Brian
Re: Multi-get getting stuck
Hi, Thanks for your responses. Sorry, I should not have said C++ API. This was my wrapper over the libmemcached C API, 0.43 Dormando, yes it is the binary protocol and I think what you mention is exactly what might be happening. The client seems to get stuck on send(). The read buffer you mention is the socket revc buffer right? Its just that the key number that the client is currently sending is much higher than the one the server has processed. So can't figure out why the disjoin. The iirc solution sounds seems the right way to go to get around this. Thanks, Prasad On Sep 29, 3:44 pm, Brian Aker br...@tangent.org wrote: Hi! On Thu, Sep 29, 2011 at 7:47 AM, Prasad pspha...@gmail.com wrote: I am trying to use mget() to get multiple data pieces using the C++ API. When it works the performance difference is great. Is this the one in libmemcached? If so use the C API. The C++ API is just there for a couple of folks who wanted to have one. Its a wrapper around the C api and it wouldn't surprise me if it was slower. In version 2.0 we are likely to completely rewrite it. Cheers, -Brian
Re: Multi-get getting stuck
Hi! On Sep 29, 2011, at 2:07 PM, dormando dorma...@rydia.net wrote: Still, I thought this was fixed already. As a workaround you can chunk your multigets into smaller sections, but I thought that the client did support it It does, so I would wonder how many total key requests are being sent. Cheers, -Brian
Re: Multi-get getting stuck
Hi, It starts getting stuck when I have a list of about 5000 keys in the mget. If I try to step and check which key is currently being sent ( when the process is stuck ) its somewhere around the 2500 mark. Is this fixed in 0.43 itself? And so with this fix does the client call a fetch after a small chunk of multigets are reached inherently? Since I don't call the fetch till I return from the mget API call. Thanks, Prasad On Sep 29, 5:14 pm, Brian Aker br...@tangent.org wrote: Hi! On Sep 29, 2011, at 2:07 PM, dormando dorma...@rydia.net wrote: Still, I thought this was fixed already. As a workaround you can chunk your multigets into smaller sections, but I thought that the client did support it It does, so I would wonder how many total key requests are being sent. Cheers, -Brian
Re: Multi-get getting stuck
Hi! On Sep 29, 2011, at 2:30 PM, Prasad wrote: And so with this fix does the client call a fetch after a small chunk of multigets are reached inherently? Since I don't call the fetch till I return from the mget API call. There is a water mark that causes the mget to go multi if it is too long. The specific watermark is not something I recall (i.e. I would need to pull up the code). Could you provide for me a sample set of keys? I'd like to create a test case for this. Cheers, -Brian
