unsubscribe
unsubscribe
RE: live nodes list in ring
When this happens to me I have to do a full cluster restart. Even doing a rolling restart across the cluster doesn’t seem to fix them, all of the nodes need to be stopped at the same time. After bringing everything back up the ring is correct. Does anyone know how a cluster gets into this state? Stephen From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, June 30, 2010 1:42 PM To: user@cassandra.apache.org Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' Subject: Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: IO errors after upgrading from 0.5.1 to 0.6
I found out what was wrong. The schema file had gotten changed but not deployed to the cluster recently. During the migration the new schema was used. A column family got switched from a normal column family to a super column family. Stephen Hamer On Mon, May 17, 2010 at 6:16 PM, Stephen Hamer wrote: > After upgrading my cluster from 0.5.1 to the 0.6 branch (commit 1206bcf in > git). I am seeing lots of IO errors in the log output. > > Two questions: > >1. Is this a sign that I have corrupt data? Is there some way for me to >recover it or at the very least remove the bad data? >2. If this is an issue with 0.6 can I safely downgrading back to 0.5.1? > > There are three different error's that I am seeing. > Exception 1: > > ERROR 00:21:23,419 Error in ThreadPoolExecutor > java.lang.RuntimeException: java.io.IOException: Corrupt (negative) value > length encountered > at > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.pollColumn(SSTableSliceIterator.java:159) > at > org.apache.cassandra.db.filter.SSTableSliceIterator.computeNext(SSTableSliceIterator.java:92) > at > org.apache.cassandra.db.filter.SSTableSliceIterator.computeNext(SSTableSliceIterator.java:39) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) > at > org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(CollatingIterator.java:364) > at > org.apache.commons.collections.iterators.CollatingIterator.hasNext(CollatingIterator.java:217) > at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:55) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:109) > at > org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:112) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:869) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:770) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739) > at org.apache.cassandra.db.Table.getRow(Table.java:381) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Corrupt (negative) value length encountered > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:335) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:284) > at > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.getNextBlock(SSTableSliceIterator.java:194) > at > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.pollColumn(SSTableSliceIterator.java:154) > ... 21 more > > > Exception 2: > > ERROR 00:30:27,752 Fatal exception in thread > Thread[ROW-READ-STAGE:85,5,main] > java.lang.RuntimeException: java.io.IOException: Corrupt (negative) value > length encountered > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:106) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Corrupt (negative) value length encountered > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:335) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:284) > at > org.ap
IO errors after upgrading from 0.5.1 to 0.6
After upgrading my cluster from 0.5.1 to the 0.6 branch (commit 1206bcf in git). I am seeing lots of IO errors in the log output. Two questions: 1. Is this a sign that I have corrupt data? Is there some way for me to recover it or at the very least remove the bad data? 2. If this is an issue with 0.6 can I safely downgrading back to 0.5.1? There are three different error's that I am seeing. Exception 1: ERROR 00:21:23,419 Error in ThreadPoolExecutor java.lang.RuntimeException: java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.pollColumn(SSTableSliceIterator.java:159) at org.apache.cassandra.db.filter.SSTableSliceIterator.computeNext(SSTableSliceIterator.java:92) at org.apache.cassandra.db.filter.SSTableSliceIterator.computeNext(SSTableSliceIterator.java:39) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) at org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(CollatingIterator.java:364) at org.apache.commons.collections.iterators.CollatingIterator.hasNext(CollatingIterator.java:217) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:55) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:109) at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:112) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:869) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:770) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739) at org.apache.cassandra.db.Table.getRow(Table.java:381) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:335) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:284) at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.getNextBlock(SSTableSliceIterator.java:194) at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.pollColumn(SSTableSliceIterator.java:154) ... 21 more Exception 2: ERROR 00:30:27,752 Fatal exception in thread Thread[ROW-READ-STAGE:85,5,main] java.lang.RuntimeException: java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:106) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Corrupt (negative) value length encountered at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:335) at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:284) at org.apache.cassandra.db.filter.SSTableNamesIterator.(SSTableNamesIterator.java:104) at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:770) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739) at org.apache.cassandra.db.Table.getRow(Table.java:381) at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70) ... 4 more Exception 3: ERROR 00:08:27,909 E
