VM Settings are -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
error stack was containing 2 threads for the same key, stalling on digest query The below bytes which I referred is the actual value of "_body" variable in org.apache.cassandra.net.Message object got from the heap dump. As I understand from the code, ReadVerbHandler will deserialize this "_body" variable into a SliceByNamesReadCommand object. When I manually inspected this byte array, it seems hold all details correctly, except the super-column name, causing it to fetch the entire wide row. -- Ravi On Thu, Mar 28, 2013 at 8:36 AM, aaron morton <aa...@thelastpickle.com>wrote: > We started receiving OOMs in our cassandra grid and took a heap dump > > What are the JVM settings ? > What was the error stack? > > I am pasting the serialized byte array of SliceByNamesReadCommand, which > seems to be corrupt on issuing certain digest queries. > > Sorry I don't follow what you are saying here. > Can you can you enable DEBUG logging and identify the behaviour you think > is incorrect ? > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan < > ravikumar.govindara...@gmail.com> wrote: > > We started receiving OOMs in our cassandra grid and took a heap dump. We > are running version 1.0.7 with LOCAL_QUORUM from both reads/writes. > > After some analysis, we kind of identified the problem, with > SliceByNamesReadCommand, involving a single Super-Column. This seems to be > happening only in digest query and not during actual reads. > > I am pasting the serialized byte array of SliceByNamesReadCommand, which > seems to be corrupt on issuing certain digest queries. > > //Type is SliceByNamesReadCommand > body[0] = (byte)1; > //This is a digest query here. > body[1] = (byte)1; > > //Table-Name from 2-8 bytes > > //Key-Name from 9-18 bytes > > //QueryPath deserialization here > > //CF-Name from 19-30 bytes > > //Super-Col-Name from 31st byte onwards, but gets > corrupt as found in heap dump > > //body[32-37] = 0, body[38] = 1, body[39] = 0. This > causes the SliceByNamesDeserializer to mark both ColName=NULL and > SuperColName=NULL, fetching entire wide-row!!! > > //Actual super-col-name starts only from byte 40, > whereas it should have started from 31st byte itself > > Has someone already encountered such an issue? Why is the super-col-name > not correctly de-serialized during digest query. > > -- > Ravi > > >
