[
https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thibaut updated CASSANDRA-2408:
-------------------------------
Description:
Hi,
We had to replace a node with faulty ram. Besides the cassandra cluster getting
unresponsive, we also observerd a "keys must be written in ascending order"
exception on all adjacent quorum nodes, causing their compactation to fail.
We had another node with faulty ram a week earlier, and it was causing the same
errors on it's neighbours.
A faulty node shouldn't affect the key ordering of adjacent nodes?
INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java
(line 396) Compacting
[SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line
108) Last written key :
DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
109) Current key :
DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
110) Writing into file
/cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
was:
Hi,
We had to replace a node with faulty ram. Besides the cassandra cluster getting
unresponsive, we also observerd a "keys must be written in ascending order"
exception on all adjacent quorum nodes, causing the compactation to fail.
We had another node with faulty ram a week earlier, and it was causing the same
errors on it's neighbours.
A faulty node shouldn't affect the key ordering of adjacent nodes?
INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java
(line 396) Compacting
[SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line
108) Last written key :
DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
109) Current key :
DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
110) Writing into file
/cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> Faulty memory causes adjacent nodes to have invalid data and fail
> compactation (java.io.IOException: Keys must be written in ascending order.)
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Thibaut
> Fix For: 0.7.5
>
>
> Hi,
> We had to replace a node with faulty ram. Besides the cassandra cluster
> getting unresponsive, we also observerd a "keys must be written in ascending
> order" exception on all adjacent quorum nodes, causing their compactation to
> fail.
> We had another node with faulty ram a week earlier, and it was causing the
> same errors on it's neighbours.
> A faulty node shouldn't affect the key ordering of adjacent nodes?
> INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java
> (line 396) Compacting
> [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
> INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line
> 108) Last written key :
> DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88,
>
> 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
> INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
> 109) Current key :
> DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4,
>
> 62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
> INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
> 110) Writing into file
> /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
> ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516
> AbstractCassandraDaemon.java (line 112) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.io.IOException: Keys must be written in ascending order.
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
