Thanks Ryan... We will keep ur valuable suggestion in resolving this issue.. But what is your take on Cassandra patch for issue 8248 to resolve this.
On Tuesday, January 6, 2015, Ryan Svihla <r...@foundev.pro> wrote: > Btw side note here, you're using GIANT Batches, and the logs are > indicating such, this will cause a signficant amount of heap pressure. > > The root cause fix is not to use giant batches in the first place. > > On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj < > rahul.bhard...@indiamart.com > <javascript:_e(%7B%7D,'cvml','rahul.bhard...@indiamart.com');>> wrote: > >> Hi Joe.. >> >> Thanks for your valuable solution.. it worked. >> >> But for this problem >> >> *The processes are killed by kernel, coz they are eating all memory >> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) >> because we have 64 GB RAM.* >> >> Should I apply patch given for issue >> https://issues.apache.org/jira/browse/CASSANDRA-8248 ?? >> >> >> regards: >> Rahul Bhardwaj >> >> On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <joe.ram...@mac.com >> <javascript:_e(%7B%7D,'cvml','joe.ram...@mac.com');>> wrote: >> >>> Thanks Rahul and good luck! I’m really curious to hear what the result >>> is. >>> >>> >>> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <rahul.bhard...@indiamart.com >>> <javascript:_e(%7B%7D,'cvml','rahul.bhard...@indiamart.com');>> wrote: >>> >>> Thanks for your response.. i will get back to you with my findings. >>> >>> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <joe.ram...@mac.com >>> <javascript:_e(%7B%7D,'cvml','joe.ram...@mac.com');>> wrote: >>> >>>> That should be “writing too many bytes” not “waiting too many bytes” >>>> just for clarity’s sake. >>>> >>>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <joe.ram...@mac.com >>>> <javascript:_e(%7B%7D,'cvml','joe.ram...@mac.com');>> wrote: >>>> >>>> I’m not an expert. Really just learning this myself but it looks like >>>> according to the stack you’re getting an exception waiting too many bytes >>>> to the commit log. >>>> >>>> That’s controlled by commit_log_segment_size_in_mb setting. The >>>> maximum write size that C* will allow is half of the value set for this >>>> parameter so if it’s set for 32 (default) that means your max write would >>>> be 16MB. (And that’s what’s getting reported in the stack trace. You’re >>>> writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing >>>> the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and >>>> see if the error goes away. It really should get tuned for the amount of >>>> writes but that’ll tell you at least if this is the issue. >>>> >>>> Let me know how it goes! >>>> >>>> Joe >>>> >>>> >>>> >>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj < >>>> rahul.bhard...@indiamart.com >>>> <javascript:_e(%7B%7D,'cvml','rahul.bhard...@indiamart.com');>> wrote: >>>> >>>> Hi Joe, >>>> >>>> PFB output of system.log >>>> >>>> tail -n 100 system.log >>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 >>>> CompactionTask.java:251 - Compacted 4 sstables to >>>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. >>>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = >>>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge >>>> counts were {1:246186, 2:11302, 3:2259, 4:538, } >>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 >>>> CompactionTask.java:251 - Compacted 4 sstables to >>>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. >>>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total >>>> partitions merged to 1. Partition merge counts were {1:1, 2:3, } >>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.business_feed] is of size >>>> 433053, exceeding specified threshold of 5120 by 427933. >>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')] >>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.business_feed] is of size >>>> 433819, exceeding specified threshold of 5120 by 428699. >>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')] >>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.business_feed] is of size >>>> 436755, exceeding specified threshold of 5120 by 431635. >>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')] >>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 >>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>>> Thread[SharedPool-Worker-1,5,main]: {} >>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too >>>> large for the maxiumum size of 16777216 >>>> at >>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>>> Source) ~[na:1.7.0_71] >>>> at >>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')] >>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - >>>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: >>>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936 >>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - >>>> Batch of prepared statements for >>>> [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding >>>> specified threshold of 5120 by 54901. >>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')] >>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.business_feed] is of size >>>> 517769, exceeding specified threshold of 5120 by 512649. >>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')] >>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.im_csl_log] is of size >>>> 13220136, exceeding specified threshold of 5120 by 13215016. >>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 >>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>>> Thread[SharedPool-Worker-1,5,main]: {} >>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: >>>> Mutation of 16800887 bytes is too large for the maxiumum size of 16777216 >>>> at >>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>>> Source) ~[na:1.7.0_71] >>>> at >>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 >>>> bytes is too large for the maxiumum size of 16777216 >>>> at >>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> ... 4 common frames omitted >>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')] >>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.im_csl_log] is of size >>>> 13136950, exceeding specified threshold of 5120 by 13131830. >>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 >>>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) >>>> on-heap, 0 (0%) off-heap >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 >>>> Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 >>>> serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 >>>> Memtable.java:364 - Completed flushing >>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db >>>> (21516225 bytes) for commitlog position >>>> ReplayPosition(segmentId=1420516895898, position=16708232) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 >>>> Memtable.java:325 - Writing >>>> Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized >>>> bytes, 52771 ops, 1%/0% of on/off-heap limit) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 >>>> Memtable.java:364 - Completed flushing >>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db >>>> (361277 bytes) for commitlog position >>>> ReplayPosition(segmentId=1420516895898, position=16708232) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 >>>> Memtable.java:325 - Writing >>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 >>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 >>>> Memtable.java:364 - Completed flushing >>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db >>>> (543861 bytes) for commitlog position >>>> ReplayPosition(segmentId=1420516895898, position=16708232) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 >>>> Memtable.java:325 - Writing >>>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 >>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 >>>> Memtable.java:364 - Completed flushing >>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db >>>> (481902 bytes) for commitlog position >>>> ReplayPosition(segmentId=1420516895898, position=16708232) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 >>>> Memtable.java:325 - Writing >>>> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 >>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 >>>> Memtable.java:364 - Completed flushing >>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db >>>> (460053 bytes) for commitlog position >>>> ReplayPosition(segmentId=1420516895898, position=16708232) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 >>>> Memtable.java:325 - Writing >>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 >>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) >>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 >>>> Memtable.java:364 - Completed flushing >>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db >>>> (395084 bytes) for commitlog position >>>> ReplayPosition(segmentId=1420516895898, position=16708232) >>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')] >>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')] >>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - >>>> Batch of prepared statements for [clickstream.im_csl_log] is of size >>>> 13480721, exceeding specified threshold of 5120 by 13475601. >>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 >>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>>> Thread[SharedPool-Worker-1,5,main]: {} >>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: >>>> Mutation of 17138032 bytes is too large for the maxiumum size of 16777216 >>>> at >>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>>> Source) ~[na:1.7.0_71] >>>> at >>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 >>>> bytes is too large for the maxiumum size of 16777216 >>>> at >>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> ... 4 common frames omitted >>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')] >>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 >>>> CompactionTask.java:136 - Compacting >>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')] >>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 >>>> CompactionTask.java:251 - Compacted 1 sstables to >>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. >>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. >>>> 26 total partitions merged to 26. Partition merge counts were {1:26, } >>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 >>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread >>>> Thread[SharedPool-Worker-1,5,main]: {} >>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too >>>> large for the maxiumum size of 16777216 >>>> at >>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>>> Source) ~[na:1.7.0_71] >>>> at >>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) >>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at >>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] >>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] >>>> >>>> >>>> >>>> Regards: >>>> Rahul Bhardwaj >>>> >>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj < >>>> rahul.bhard...@indiamart.com >>>> <javascript:_e(%7B%7D,'cvml','rahul.bhard...@indiamart.com');>> wrote: >>>> >>>>> Hi Joe, >>>>> >>>>> PFA heap dump >>>>> >>>>> >>>>> regards: >>>>> Rahul Bhardwaj >>>>> >>>>> >>>>> >>>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ram...@mac.com >>>>> <javascript:_e(%7B%7D,'cvml','joe.ram...@mac.com');>> wrote: >>>>> >>>>>> Did you try generating a heap dump so you can look through it to see >>>>>> what’s actually happened? >>>>>> >>>>>> >>>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj < >>>>>> rahul.bhard...@indiamart.com >>>>>> <javascript:_e(%7B%7D,'cvml','rahul.bhard...@indiamart.com');>> >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> We are using cassandra 2.1 version in a cluster of three machines >>>>>> each with 64 GB RAM >>>>>> >>>>>> The processes are killed by kernel, coz they are eating all memory >>>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) >>>>>> because we have 64 GB RAM. >>>>>> >>>>>> Please help. >>>>>> >>>>>> >>>>>> Regards: >>>>>> Rahul Bhardwaj >>>>>> >>>>>> >>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates >>>>>> on this and more: <https://plus.google.com/+indiamart> >>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>>>>> Mobile Channel: >>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>>>>> <http://m.indiamart.com/> >>>>>> >>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, >>>>>> kyunki Kaam Yahin Banta Hai >>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on >>>> this and more: <https://plus.google.com/+indiamart> >>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>>> Mobile Channel: >>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>>> <http://m.indiamart.com/> >>>> >>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, >>>> kyunki Kaam Yahin Banta Hai >>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >>>> >>>> >>>> >>>> >>> >>> >>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on >>> this and more: <https://plus.google.com/+indiamart> >>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >>> Mobile Channel: >>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >>> <https://play.google.com/store/apps/details?id=com.indiamart.m> >>> <http://m.indiamart.com/> >>> >>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki >>> Kaam >>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >>> >>> >>> >> >> >> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on >> this and more: <https://plus.google.com/+indiamart> >> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> >> Mobile Channel: >> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> >> <https://play.google.com/store/apps/details?id=com.indiamart.m> >> <http://m.indiamart.com/> >> >> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> >> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam >> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!! >> > > > > -- > > Thanks, > Ryan Svihla > > -- Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
