"How can I detect wide rows?" --> nodetool cfhistograms <keyspace> <suspected column family>
Look at column "Column count" (last column) and identify a line in this column with very high value of "Offset". In a well designed application you should have a gaussian distribution where 80% of your row have a similar number of columns. "Anyone know what debug level I can set so that I can see what reads the hot node is handling? " --> "nodetool settraceprobability <value>", where value is a small number (0.001) on the node where you encounter the issue. Activate the tracing for a while (5 mins) and deactivate it (value = 0). Then look into system_traces tables "events" & "sessions". It may help or not since the tracing is done once every 1000. "Any way to get the server to blacklist these wide rows automatically?" --> No On Thu, Jul 24, 2014 at 8:48 PM, Keith Wright <kwri...@nanigans.com> wrote: > Hi all, > > We are seeing an issue where basically daily one of our nodes spikes in > load and is churning in CMS heap pressure. It appears that reads are > backing up and my guess is that our application is reading a large row > repeatedly. Our write structure can lead itself to wide rows very > infrequently (<0.001%) and we do our best to detect and delete them but > obviously we’re missing a case. Hoping for assistance on the following > questions: > > - How can I detect wide rows? > - Anyone know what debug level I can set so that I can see what reads > the hot node is handling? I’m hoping to see the “bad” row > - Any way to get the server to blacklist these wide rows automatically? > > We’re using C* 2.0.6 with Vnodes. > > Thanks >
