er, make that JVMFLAGS=-Xmx<mem in gig>g bin/zkServer.sh (no -D)
Patrick On Wed, Nov 9, 2011 at 2:31 PM, Patrick Hunt <[email protected]> wrote: > On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti > <[email protected]> wrote: >> I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance. >> Each element of the matrix is a ZNODE and value of each element is a >> digit(0-9). >> >> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an >> error after that. Mostly it is throwing a "-5" error code which is a >> marshalling/unmarshalling error. I'm using the perl interface of the >> zookeeper. >> >> My question is: Is there a limit to the maximum number of ZNODES a zookeeper >> instance can hold or this is limited by the system memory? >> >> Any pointers on how to avoid the error would be very helpful. > > Available heap memory is really the only limit. Try > > $ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh > > also > > $ sudo jmap -heap <jvm pid> > > will give you some insight into whether it was set correctly or not > (ie MaxHeapSize) > > The most I've tried is 5 million znodes with 25 million watches (using > zkpython, but zkperl should be fine). iirc that was 8gig heap, but > ymmv depending on data size. > > You may also need to tune the GC at some point (would suggest turning > on cms and parallel collector) to limit stop the world pauses. > > Regards, > > Patrick >
