> When I manually inspected this byte array, it seems hold all details > correctly, except the super-column name, causing it to fetch the entire wide > row. What is the CF definition and what is the exact query you are sending? There does not appear to be anything obvious in the QueryPath serde for 1.0.7
Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 28/03/2013, at 10:54 AM, Ravikumar Govindarajan <ravikumar.govindara...@gmail.com> wrote: > VM Settings are > -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn800M > -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > > error stack was containing 2 threads for the same key, stalling on digest > query > > The below bytes which I referred is the actual value of "_body" variable in > org.apache.cassandra.net.Message object got from the heap dump. > > As I understand from the code, ReadVerbHandler will deserialize this "_body" > variable into a SliceByNamesReadCommand object. > > When I manually inspected this byte array, it seems hold all details > correctly, except the super-column name, causing it to fetch the entire wide > row. > > -- > Ravi > > On Thu, Mar 28, 2013 at 8:36 AM, aaron morton <aa...@thelastpickle.com> wrote: >> We started receiving OOMs in our cassandra grid and took a heap dump > What are the JVM settings ? > What was the error stack? > >> I am pasting the serialized byte array of SliceByNamesReadCommand, which >> seems to be corrupt on issuing certain digest queries. > > Sorry I don't follow what you are saying here. > Can you can you enable DEBUG logging and identify the behaviour you think is > incorrect ? > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan > <ravikumar.govindara...@gmail.com> wrote: > >> We started receiving OOMs in our cassandra grid and took a heap dump. We are >> running version 1.0.7 with LOCAL_QUORUM from both reads/writes. >> >> After some analysis, we kind of identified the problem, with >> SliceByNamesReadCommand, involving a single Super-Column. This seems to be >> happening only in digest query and not during actual reads. >> >> I am pasting the serialized byte array of SliceByNamesReadCommand, which >> seems to be corrupt on issuing certain digest queries. >> >> //Type is SliceByNamesReadCommand >> body[0] = (byte)1; >> >> //This is a digest query here. >> body[1] = (byte)1; >> >> //Table-Name from 2-8 bytes >> >> //Key-Name from 9-18 bytes >> >> //QueryPath deserialization here >> >> //CF-Name from 19-30 bytes >> >> //Super-Col-Name from 31st byte onwards, but gets >> corrupt as found in heap dump >> >> //body[32-37] = 0, body[38] = 1, body[39] = 0. This >> causes the SliceByNamesDeserializer to mark both ColName=NULL and >> SuperColName=NULL, fetching entire wide-row!!! >> >> //Actual super-col-name starts only from byte 40, whereas >> it should have started from 31st byte itself >> >> Has someone already encountered such an issue? Why is the super-col-name not >> correctly de-serialized during digest query. >> >> -- >> Ravi >> > >
