Hi,
I am running a small 2 DC cluster of 3 nodes (each DC). I use 3 replicas in
both DCs (all 6 nodes have everything) on Cassandra 1.2.11. I populated the
cluster via cqlsh pipelined with a series of inserts. I use the cluster for
tests, the dataset is pretty small (hundreds of thousands of records max).
The cluster was completely up during inserts. Inserts were done serially on
one of the nodes.
The resulting load is uneven:
Datacenter: xxx
==================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN ip 1.63 GB 256 100.0%
83ecd32a-3f2b-4cf6-b3c7-b316cb1986cc default-rackUN ip 1.5 GB
256 100.0% 091ca530-2e95-4954-92c4-76f51fab0b66
default-rack
UN ip 1.44 GB 256 100.0%
d94d335e-08bf-4a30-ad58-4c5acdc2ef45 default-rack
Datacenter: yyy
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN ip 2.27 GB 256 100.0%
e2584981-71f7-45b0-82f4-e08942c47585 1c
UN ip 2.27 GB 256 100.0%
e5c6de9a-819e-4757-a420-55ec3ffaf131 1c
UN ip 2.27 GB 256 100.0%
fa53f391-2dd3-4ec8-885d-8db6d453a708 1c
And 4 out of 6 nodes report corrupted sstables:
java.lang.RuntimeException:
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.IOException: mmap segment underflow; remaining is 239882945
but 1349280116 requested
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1618)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.IOException: mmap segment underflow; remaining is 239882945
but 1349280116 requested
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:119)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:44)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:104)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1062)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1614)
... 3 more
Caused by: java.io.IOException: mmap segment underflow; remaining is
239882945 but 1349280116 requested
at
org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:135)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:108)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$SimpleBlockFetcher.<init>(IndexedSliceReader.java:477)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:94)
repair -pr hangs, rebuild from the less corrupted dc hangs.
The only interesting exception (besides the java.io.EOFException during
repair) is the following:
org.apache.cassandra.db.marshal.MarshalException: invalid UTF8 bytes 52f2665b
at org.apache.cassandra.db.marshal.UTF8Type.getString(UTF8Type.java:54)
at
org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.insert(AbstractSimplePerColumnSecondaryIndex.java:102)
at
org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(SecondaryIndexManager.java:448)
at org.apache.cassandra.db.Table.indexRow(Table.java:431) at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:803)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
during rebuild and some repair -pr sessions.
Does anyone know what might have caused this?
regards,
ondrej cernos
