[ https://issues.apache.org/jira/browse/CASSANDRA-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604512#comment-15604512 ]
Tom van der Woerdt commented on CASSANDRA-12125: ------------------------------------------------ This issue still exists in 3.0.9. {noformat} DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [10..11]: 11 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [12..13]: 38 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [14..15]: 23 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [16..17]: 17 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [18..19]: 12 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [20..21]: 9 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [22..29]: 6 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 Validator.java:235 - Validated 1868 partitions for bb1f9dd0-98db-11e6-ad5a-89fa523ea316. Partition sizes are: INFO [AntiEntropyStage:1] 2016-10-23 04:46:59,703 Validator.java:274 - [repair #bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Sending completed merkle tree to /10.0.0.6 for myschema.mytable2 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [17232..387341]: 99 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [387342..1097582]: 16 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [1097583..1807823]: 3 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [1807824..2518064]: 7 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [2518065..3959516]: 3 DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 CompactionManager.java:1135 - Validation finished in 793 msec, for [repair #bb1f9dd0-98db-11e6-ad5a-89fa523ea316 on myschema/mytable2, [(-6682001123294953015,-6681421039287311765]]] DEBUG [CompactionExecutor:51358] 2016-10-23 04:47:00,499 CompactionTask.java:218 - Compacted (bb4d3d80-98db-11e6-b4a3-2b509f67851f) 2 sstables to [/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big,] to level=0. 31,664,366 bytes to 31,653,685 (~99% of original) in 1,592ms = 18.961876MB/s. 0 total partitions merged to 1,888. Partition merge counts were {1:2063, } DEBUG [GossipStage:1] 2016-10-23 04:47:00,705 FailureDetector.java:456 - Ignoring interval time of 2065449328 for /10.0.0.1 DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 - Ignoring interval time of 2065577694 for /10.0.0.2 DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 - Ignoring interval time of 2065549822 for /10.0.0.3 DEBUG [Thread-6] 2016-10-23 04:47:01,201 StorageProxy.java:2246 - Schemas are in agreement. DEBUG [GossipStage:1] 2016-10-23 04:47:01,635 FailureDetector.java:456 - Ignoring interval time of 2543137972 for /10.0.0.4 DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 - Ignoring interval time of 2218246254 for /10.0.0.4 DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 - Ignoring interval time of 2218388771 for /10.0.0.5 DEBUG [GossipStage:1] 2016-10-23 04:47:04,549 FailureDetector.java:456 - Ignoring interval time of 2000269752 for /10.0.0.2 DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,281 RepairMessageVerbHandler.java:116 - Validating ValidationRequest{gcBefore=1474606025} org.apache.cassandra.repair.messages.ValidationRequest@57e4b3c9 DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,281 StorageService.java:2955 - Forcing flush on keyspace myschema, CF mytable1 DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,282 ColumnFamilyStore.java:850 - Enqueuing flush of mytable1: 28142 (0%) on-heap, 16292220 (0%) off-heap DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,282 Memtable.java:368 - Writing Memtable-mytable1@1772797855(3.616MiB serialized bytes, 47 ops, 0%/0% of on/off-heap limit) DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,285 RepairMessageVerbHandler.java:133 - Syncing SyncRequest{initiator=/10.0.0.6, src=/10.0.0.7, dst=/10.0.0.1, ranges=[(-6681996591388643318,-6681992059482333621], (-6681946740419236648,-6681942208512926951], (-6681933144700307557,-6681928612793997859], (-6681915017075068768,-6681910485168759070], (-6681896889449829979,-6681892357543520281], (-6681887825637210584,-6681883293730900887], (-6681842506574113612,-6681837974667803914], (-6681837974667803914,-6681833442761494217], (-6681815315136255428,-6681810783229945731], (-6681810783229945731,-6681801719417326336], (-6681783591792087547,-6681779059885777850], (-6681769996073158456,-6681765464166848758], (-6681747336541609969,-6681742804635300272], (-6681720145103751785,-6681715613197442088], (-6681634038883867537,-6681629506977557840], (-6681629506977557840,-6681620443164938445], (-6681593251727080262,-6681584187914460867], (-6681570592195531776,-6681566060289222078], (-6681552464570292987,-6681547932663983289], (-6681547932663983289,-6681543400757673592], (-6681520741226125106,-6681516209319815409], (-6681493549788266922,-6681484485975647528], (-6681448230725169950,-6681443698818860253]]} org.apache.cassandra.repair.messages.SyncRequest@5fbf8b2c INFO [AntiEntropyStage:1] 2016-10-23 04:47:05,286 StreamingRepairTask.java:58 - [streaming task #bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Performing streaming repair of 23 ranges with /10.0.0.1 DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,286 ColumnFamilyStore.java:850 - Enqueuing flush of mytable2: 1695 (0%) on-heap, 19377 (0%) off-heap DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,286 Memtable.java:368 - Writing Memtable-mytable2@1296803980(19.105KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,322 Memtable.java:401 - Completed flushing /var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db (19.019KiB) for commitlog position ReplayPosition(segmentId=1474469138946, position=165305985) DEBUG [CompactionExecutor:51352] 2016-10-23 04:47:05,366 CompactionTask.java:146 - Compacting (bf274360-98db-11e6-b4a3-2b509f67851f) [/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db:level=0, /var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big-Data.db:level=0, ] DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,533 Memtable.java:401 - Completed flushing /var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/mc-422108-big-Data.db (3.615MiB) for commitlog position ReplayPosition(segmentId=1474469138946, position=165305985) DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,695 Memtable.java:368 - Writing Memtable-mytable1.mytable1_my_idx@185942079(1.187KiB serialized bytes, 28 ops, 0%/0% of on/off-heap limit) ERROR [MemtableFlushWriter:5503] 2016-10-23 04:47:05,757 CassandraDaemon.java:205 - Exception in thread Thread[MemtableFlushWriter:5503,5,main] java.lang.RuntimeException: Last written key DecoratedKey(1byAcm-002Hsb-EL, 31627941636d2d3030324873622d454c) >= current key DecoratedKey(^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@, 31627941656e2d3030304171552d554b) writing into /var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/.mytable1_my_idx/mc-16240-big-Data.db at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:394) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.Memtable.flush(Memtable.java:332) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-3.0.9.jar:3.0.9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] {noformat} After this, the commitlog directory grew a lot (files in the directory weren't cleaned up anymore, because MemtableFlushWriter died), and the node ran out of space :) > ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137 > CassandraDaemon.java:185 - Exception in thread > Thread[MemtableFlushWriter:4,5,main] java.lang.RuntimeException: Last > written key DecoratedKey(XXXX.XX, XXXXX) >= current key DecoratedKey > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > Key: CASSANDRA-12125 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12125 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: RHEL-6.5 64-bit Apache Cassandra 2.2.5v > Reporter: Relish Chackochan > Fix For: 2.2.x > > > We are running on RHEL-6.5 64-bit with Apache Cassandra 2.2.5v on 4 node > cluster and getting the following error on multiple node while running the > repair job and when getting the error repair job is hang. > Can some one help to identify the issue. > {code} > ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137 > CassandraDaemon.java:185 - Exception in thread > Thread[MemtableFlushWriter:4,5,main] > java.lang.RuntimeException: Last written key DecoratedKey(1467371986.8870, > 313436373337313938362e38383730) >= current key DecoratedKey(, > 313436373337323030312e38383730) writing into > /opt/cassandra/data/proddb/log_data1-0a5092a0a4fa11e5872fc1ce0a46dc27/.maxdatetimeindex_idx/tmp-la-470-big-Data.db > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)