Hi All,
I'm experiencing the following errors while bulk loading data into a cluster
ERROR [Thread-23] 2012-04-05 09:58:12,252 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-23,5,main]
java.lang.RuntimeException: Insufficient disk space to flush
7813594056494754913 bytes
at
org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:635)
at
org.apache.cassandra.streaming.StreamIn.getContextMapping(StreamIn.java:92)
at
org.apache.cassandra.streaming.IncomingStreamReader.<init>(IncomingStreamReader.java:68)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
Here I'm not really sure I was able to generate 7 exa bytes of data ;)
ERROR [Thread-46] 2012-04-05 09:58:14,453 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[Thread-46,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.io.sstable.SSTable.getMinimalKey(SSTable.java:156)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:302)
at
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:155)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:89)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:185)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
This one sounds like a null key added to the SSTable at some point,
but I'm rather confident I'm checking for key nullity.
Errors are seen in different nodes. Some nodes succeeded. All 5
cluster nodes run 1.0.8 with JNA enabled.
Basically I'm generating SSTables in a Hadoop Reducer, storing them in
HDFS. After the job finished, I download them back into a single node
and streaming the files into Cassandra (Yes, I definitely need to try
1.1).
Do someone have a hint or a pointer where to start looking ?
Thanks,
Benoit.
