The malformed input bug was fixed after beta1 and should be in a nightly build by now. (I introduced a regression where it couldn't handle the last entry in the commitlog being incomplete. So upgrading should be able to restart on the existing commitlogs.)
The OOM puzzles me a little; I'm not sure how it could be unable to replay a mutation that it was able to write to the commitlog in the first place. You could try setting the memtable object and memory thresholds lower temporarily and see if that leaves enough extra free to do the replay. -Jonathan On Wed, Aug 19, 2009 at 7:12 PM, Brian Frank Cooper<coop...@yahoo-inc.com> wrote: > Hi folks, > > I'm using 0.4 beta1 and had six servers loaded with 20 GB of data per server. > (In this test, 10 KB per record, and 2 GB heap space allocated to the JVM.) I > stopped the servers (using what I think is the recommended method, the kill > command). Upon trying to restart, some servers threw a > UTFDataFormatException, while others threw an OutOfMemoryError exception. > None of them started. > > Is this a known issue? > > ERROR - Fatal exception in thread Thread[main,5,main] > java.lang.OutOfMemoryError: Java heap space > at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274) > at > org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171) > > ERROR - Exception encountered during startup. > java.io.UTFDataFormatException: malformed input around byte 5497 > at java.io.DataInputStream.readUTF(DataInputStream.java:639) > at java.io.DataInputStream.readUTF(DataInputStream.java:547) > at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218) > at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285) > at > org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171) > Exception encountered during startup. > java.io.UTFDataFormatException: malformed input around byte 5497 > at java.io.DataInputStream.readUTF(DataInputStream.java:639) > at java.io.DataInputStream.readUTF(DataInputStream.java:547) > at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218) > at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285) > at > org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171) > > Thanks for the help! > > Brian
