Expiring session... timeout of 600000ms exceeded
Hi all, I am seeing a lot of my clients being kicked out after the 10 minute negotiated timeout is exceeded. My clients are each a JVM (around 100 running on a machine) which are doing web crawling of specific endpoints and handling the response XML - so they do wait around for 3-4 minutes on HTTP timeouts, but certainly not 10 mins. I am just prototyping right now on a 2xquad core mac pro with 12GB memory, and the 100 child processes only get -Xmx64m and I don't see my machine exhausted. Do my clients need to do anything in order to initiate keep alive heart beats or should this be automatic (I thought the ticktime would dictate this)? # my conf is: tickTime=2000 dataDir=/Volumes/Data/zookeeper clientPort=2181 maxClientCnxns=1 minSessionTimeout=4000 maxSessionTimeout=80 Thanks for any pointers to this newbie, Tim
Re: Expiring session... timeout of 600000ms exceeded
To answer your last question first, no you don't have to do anything explicit to keep the ZK connection alive. It is maintained by a dedicated thread. You do have to keep your java program responsive and ZK problems like this almost always indicate that you have a problem with your program checking out for extended periods of time. My strong guess is that you have something evil happening with your java process that is actually causing this delay. Since you have tiny memory, it probably isn't GC. Since you have a bunch of processes, swap and process wakeup delays seem plausible. What is the load average on your box? On the topic of your application, why you are using processes instead of threads? With threads, you can get your memory overhead down to 10's of kilobytes as opposed to 10's of megabytes. Also, why not use something like Bixo so you don't have to prototype a threaded crawler? On Tue, Sep 21, 2010 at 8:24 AM, Tim Robertson wrote: > Hi all, > > I am seeing a lot of my clients being kicked out after the 10 minute > negotiated timeout is exceeded. > My clients are each a JVM (around 100 running on a machine) which are > doing web crawling of specific endpoints and handling the response XML > - so they do wait around for 3-4 minutes on HTTP timeouts, but > certainly not 10 mins. > I am just prototyping right now on a 2xquad core mac pro with 12GB > memory, and the 100 child processes only get -Xmx64m and I don't see > my machine exhausted. > > Do my clients need to do anything in order to initiate keep alive > heart beats or should this be automatic (I thought the ticktime would > dictate this)? > > # my conf is: > tickTime=2000 > dataDir=/Volumes/Data/zookeeper > clientPort=2181 > maxClientCnxns=1 > minSessionTimeout=4000 > maxSessionTimeout=80 > > Thanks for any pointers to this newbie, > Tim >
Re: Expiring session... timeout of 600000ms exceeded
Thanks Ted, > To answer your last question first, no you don't have to do anything > explicit to keep the ZK connection alive. It is maintained by a dedicated > thread. You do have to keep your java program responsive and ZK problems > like this almost always indicate that you have a problem with your program > checking out for extended periods of time. > > My strong guess is that you have something evil happening with your java > process that is actually causing this delay. > > Since you have tiny memory, it probably isn't GC. Since you have a bunch of > processes, swap and process wakeup delays seem plausible. What is the load > average on your box? CPU spikes when responses come in, but mostly it's IO wait on the endpoints (timeout of 3 minutes). I suspect HTTP client 4 is dropping into a retry mechanism though, but have not investigated this yet. > On the topic of your application, why you are using processes instead of > threads? With threads, you can get your memory overhead down to 10's of > kilobytes as opposed to 10's of megabytes. I am just prototyping scaling out many processes and potentially across multiple machines. Our live crawler runs in a single JVM, but some of these crawlers take 4-6 weeks, so long running processes block others, so I was looking at alternatives - our live crawler also uses DOM based XML parsing so hitting memory limits - SAX would address this. Also we want to be able to deploy patches to the crawlers without interrupting those long running jobs if possible. > Also, why not use something like Bixo so you don't have to prototype a > threaded crawler? It is not a web crawler but more of a custom web service client that issues queries for pages of data. A second query is assembled based on the response of the first. These are Biodiversity domain specific protocols DiGIR, TAPIR and BioCASe which are closer to SOAP based requests / response. I'll look at Bixo. Thanks again, Tim > > On Tue, Sep 21, 2010 at 8:24 AM, Tim Robertson > wrote: > >> Hi all, >> >> I am seeing a lot of my clients being kicked out after the 10 minute >> negotiated timeout is exceeded. >> My clients are each a JVM (around 100 running on a machine) which are >> doing web crawling of specific endpoints and handling the response XML >> - so they do wait around for 3-4 minutes on HTTP timeouts, but >> certainly not 10 mins. >> I am just prototyping right now on a 2xquad core mac pro with 12GB >> memory, and the 100 child processes only get -Xmx64m and I don't see >> my machine exhausted. >> >> Do my clients need to do anything in order to initiate keep alive >> heart beats or should this be automatic (I thought the ticktime would >> dictate this)? >> >> # my conf is: >> tickTime=2000 >> dataDir=/Volumes/Data/zookeeper >> clientPort=2181 >> maxClientCnxns=1 >> minSessionTimeout=4000 >> maxSessionTimeout=80 >> >> Thanks for any pointers to this newbie, >> Tim >> >
Re: Expiring session... timeout of 600000ms exceeded
Generally best practices for crawlers is that no process runs more than an hour or five. All crawler processes update a central state store with their progress, but they exit when they reach a time limit knowing that somebody else will take up the work where they leave off. This avoids a multitude of ills. On Tue, Sep 21, 2010 at 11:53 AM, Tim Robertson wrote: > > On the topic of your application, why you are using processes instead of > > threads? With threads, you can get your memory overhead down to 10's of > > kilobytes as opposed to 10's of megabytes. > > I am just prototyping scaling out many processes and potentially > across multiple machines. Our live crawler runs in a single JVM, but > some of these crawlers take 4-6 weeks, so long running processes block > others, so I was looking at alternatives - our live crawler also uses > DOM based XML parsing so hitting memory limits - SAX would address > this. Also we want to be able to deploy patches to the crawlers > without interrupting those long running jobs if possible.
