Re: Digest Query Seems to be corrupt on certain cases
When I manually inspected this byte array, it seems hold all details correctly, except the super-column name, causing it to fetch the entire wide row. What is the CF definition and what is the exact query you are sending? There does not appear to be anything obvious in the QueryPath serde for 1.0.7 Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 28/03/2013, at 10:54 AM, Ravikumar Govindarajan ravikumar.govindara...@gmail.com wrote: VM Settings are -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly error stack was containing 2 threads for the same key, stalling on digest query The below bytes which I referred is the actual value of _body variable in org.apache.cassandra.net.Message object got from the heap dump. As I understand from the code, ReadVerbHandler will deserialize this _body variable into a SliceByNamesReadCommand object. When I manually inspected this byte array, it seems hold all details correctly, except the super-column name, causing it to fetch the entire wide row. -- Ravi On Thu, Mar 28, 2013 at 8:36 AM, aaron morton aa...@thelastpickle.com wrote: We started receiving OOMs in our cassandra grid and took a heap dump What are the JVM settings ? What was the error stack? I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to be corrupt on issuing certain digest queries. Sorry I don't follow what you are saying here. Can you can you enable DEBUG logging and identify the behaviour you think is incorrect ? Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan ravikumar.govindara...@gmail.com wrote: We started receiving OOMs in our cassandra grid and took a heap dump. We are running version 1.0.7 with LOCAL_QUORUM from both reads/writes. After some analysis, we kind of identified the problem, with SliceByNamesReadCommand, involving a single Super-Column. This seems to be happening only in digest query and not during actual reads. I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to be corrupt on issuing certain digest queries. //Type is SliceByNamesReadCommand body[0] = (byte)1; //This is a digest query here. body[1] = (byte)1; //Table-Name from 2-8 bytes //Key-Name from 9-18 bytes //QueryPath deserialization here //CF-Name from 19-30 bytes //Super-Col-Name from 31st byte onwards, but gets corrupt as found in heap dump //body[32-37] = 0, body[38] = 1, body[39] = 0. This causes the SliceByNamesDeserializer to mark both ColName=NULL and SuperColName=NULL, fetching entire wide-row!!! //Actual super-col-name starts only from byte 40, whereas it should have started from 31st byte itself Has someone already encountered such an issue? Why is the super-col-name not correctly de-serialized during digest query. -- Ravi
Re: Digest Query Seems to be corrupt on certain cases
We started receiving OOMs in our cassandra grid and took a heap dump What are the JVM settings ? What was the error stack? I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to be corrupt on issuing certain digest queries. Sorry I don't follow what you are saying here. Can you can you enable DEBUG logging and identify the behaviour you think is incorrect ? Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan ravikumar.govindara...@gmail.com wrote: We started receiving OOMs in our cassandra grid and took a heap dump. We are running version 1.0.7 with LOCAL_QUORUM from both reads/writes. After some analysis, we kind of identified the problem, with SliceByNamesReadCommand, involving a single Super-Column. This seems to be happening only in digest query and not during actual reads. I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to be corrupt on issuing certain digest queries. //Type is SliceByNamesReadCommand body[0] = (byte)1; //This is a digest query here. body[1] = (byte)1; //Table-Name from 2-8 bytes //Key-Name from 9-18 bytes //QueryPath deserialization here //CF-Name from 19-30 bytes //Super-Col-Name from 31st byte onwards, but gets corrupt as found in heap dump //body[32-37] = 0, body[38] = 1, body[39] = 0. This causes the SliceByNamesDeserializer to mark both ColName=NULL and SuperColName=NULL, fetching entire wide-row!!! //Actual super-col-name starts only from byte 40, whereas it should have started from 31st byte itself Has someone already encountered such an issue? Why is the super-col-name not correctly de-serialized during digest query. -- Ravi
Re: Digest Query Seems to be corrupt on certain cases
VM Settings are -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly error stack was containing 2 threads for the same key, stalling on digest query The below bytes which I referred is the actual value of _body variable in org.apache.cassandra.net.Message object got from the heap dump. As I understand from the code, ReadVerbHandler will deserialize this _body variable into a SliceByNamesReadCommand object. When I manually inspected this byte array, it seems hold all details correctly, except the super-column name, causing it to fetch the entire wide row. -- Ravi On Thu, Mar 28, 2013 at 8:36 AM, aaron morton aa...@thelastpickle.comwrote: We started receiving OOMs in our cassandra grid and took a heap dump What are the JVM settings ? What was the error stack? I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to be corrupt on issuing certain digest queries. Sorry I don't follow what you are saying here. Can you can you enable DEBUG logging and identify the behaviour you think is incorrect ? Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 28/03/2013, at 4:15 AM, Ravikumar Govindarajan ravikumar.govindara...@gmail.com wrote: We started receiving OOMs in our cassandra grid and took a heap dump. We are running version 1.0.7 with LOCAL_QUORUM from both reads/writes. After some analysis, we kind of identified the problem, with SliceByNamesReadCommand, involving a single Super-Column. This seems to be happening only in digest query and not during actual reads. I am pasting the serialized byte array of SliceByNamesReadCommand, which seems to be corrupt on issuing certain digest queries. //Type is SliceByNamesReadCommand body[0] = (byte)1; //This is a digest query here. body[1] = (byte)1; //Table-Name from 2-8 bytes //Key-Name from 9-18 bytes //QueryPath deserialization here //CF-Name from 19-30 bytes //Super-Col-Name from 31st byte onwards, but gets corrupt as found in heap dump //body[32-37] = 0, body[38] = 1, body[39] = 0. This causes the SliceByNamesDeserializer to mark both ColName=NULL and SuperColName=NULL, fetching entire wide-row!!! //Actual super-col-name starts only from byte 40, whereas it should have started from 31st byte itself Has someone already encountered such an issue? Why is the super-col-name not correctly de-serialized during digest query. -- Ravi
