Hello Kenneth, Apologies for the late reply.
1) On production the value of x was 67 MB and y was 16 MV as value of commitlog_segment_size_in_mb is 32. 2) On Dev the value of x was 18 MB and y was 16 MV as value of commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of commitlog_segment_size_in_mb to 128 when the node eventually crashed. 3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did try "nodetool stop" and "nodetool stop VIEW_BUILD". Thanks, Dipan Shah ________________________________ From: Kenneth Brotman <kenbrot...@yahoo.com.INVALID> Sent: Friday, March 1, 2019 8:19 PM To: user@cassandra.apache.org Subject: RE: MV's stuck in build state Dipan, On your production cluster, when you were first getting the “Mutation of <x> bytes …” message, what was the value of x and y? How about when you got the message on the Dev Cluster, what was the value of x and y in that message? On the Dev cluster, did you try going into JMX and directly hitting the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation? From: Dipan Shah [mailto:dipan....@hotmail.com] Sent: Friday, March 01, 2019 12:56 AM To: Kenneth Brotman; user@cassandra.apache.org Subject: Re: MV's stuck in build state Hello Kenneth, Thanks for replying. I had actually tried this on a Dev environment earlier and it caused the node to spin out of control. I'll explain what I did over there: 1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>" and thus increased the value of "commitlog_segment_size_in_mb" to 64 2) This worked for a few minutes and again the view started failing when it hit the new limits and the messages now were "Mutation of <x> bytes is too large for the maxiumum size of 2*<y>" 3) So just to try I increased the value to 128 4) Now after this change the node started crashing as soon as I brought the service online. I was not able to recover even after restoring the value of "commitlog_segment_size_in_mb" to 32 Now there is a key differences to that issue and what I am facing currently: The views were not dropped on the earlier environment whereas I have already dropped the view on the current environment (and cant experiment much as the current environment is in production). I know this is a bit tricky but I'm pretty much stuck over here and thinking of finding a non-problem creating solution over here. Thanks, Dipan Shah ________________________________ From: Kenneth Brotman <kenbrot...@yahoo.com.INVALID> Sent: Friday, March 1, 2019 12:26 AM To: user@cassandra.apache.org Subject: RE: MV's stuck in build state Hi Dipan, Did you try following the advice in the referenced DataStax article called Mutation of <x> bytes is too large for the maximum size of <y><https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y-> as suggested in the stackoverflow.com post you cited? Kenneth Brotman From: Dipan Shah [mailto:dipan....@hotmail.com] Sent: Thursday, February 28, 2019 2:23 AM To: Dipan Shah; user@cassandra.apache.org Subject: Re: MV's stuck in build state Forgot to add version info. This is on 3.7. [cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4] Thanks, Dipan Shah ________________________________ From: Dipan Shah <dipan....@hotmail.com> Sent: Thursday, February 28, 2019 3:38 PM To: user@cassandra.apache.org Subject: MV's stuck in build state Hello All, I have a few MV's that are stuck in build state because of a bad schema design and thus getting a lot of messages like this "Mutation xxx is too large for maximum size of 16.000MiB". [cid:image001.png@01D4CFFA.539C41B0] I have dropped those MV's and I can no longer see their schema in the keyspace. But they are visible under "system.views_build_in_progress" and "nodetool viewbuildstatus". I have tried "nodetool stop VIEW_BUILD" as suggested here: https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build and have also reboot a few nodes in the cluster. This has also not helped. Is there anything else that can be done over here? [https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build> Stop Cassandra Materialized View Build - Stack Overflow<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-view-build> Its not documented, but nodetool stop actually takes any compaction type, not just the ones listed (which the view build is one of). So you can simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation.. All thats really gonna do is set a flag for the view builder to stop on its next loop. stackoverflow.com Thanks, Dipan Shah
