[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049271#comment-13049271 ] Hudson commented on CASSANDRA-2752: --- Integrated in Cassandra-0.8 #170 (See [https://builds.apache.org/job/Cassandra-0.8/170/]) > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen >Assignee: Jonathan Ellis >Priority: Critical > Fix For: 0.8.1 > > Attachments: 2752.txt > > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at j
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049045#comment-13049045 ] Sylvain Lebresne commented on CASSANDRA-2752: - Good catch. +1 (committed). Thanks Terje. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen >Assignee: Terje Marthinussen >Priority: Critical > Fix For: 0.8.1 > > Attachments: 2752.txt > > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at java.util.concurrent.FutureTask
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048655#comment-13048655 ] Terje Marthinussen commented on CASSANDRA-2752: --- SSTableWriter.java: ColumnFamily.serializer().deserializeFromSSTableNoColumns(ColumnFamily.create(cfs.metadata), dfile); // don't move that statement around, it expects the dfile to be before the columns updateCache(key, dataSize, null); rowSizes.add(dataSize); columnCounts.add(dfile.readInt()); I believe the problem is in updateCache. If rowcache is enabled (and it is in this case) and the row needs to be updated in cache, this will read (deserialize) the row. However, after all the columns is read, the offset in the file is not reset back to the location where the column count is stored and things go bad. I haven't actually tried to change the code to test, but I tried to disable the row cache, and so far, repair seems to work fine when it is disabled. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048637#comment-13048637 ] Muga Nishizawa commented on CASSANDRA-2752: --- It seems like timing issue. The exception doesn't always occur always even with use of same data. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.ge
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047266#comment-13047266 ] Terje Marthinussen commented on CASSANDRA-2752: --- Also struck me on the way home from work today that the CF with secondary indexes also happen to be the only CF in this system which, I think, on a regular basis may actually update all columns for a key. That is, sstables will on a regular basis have keys where no columns is valid anymore. Not sure if that could for instance trigger something odd in the streaming? > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > or
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047211#comment-13047211 ] Terje Marthinussen commented on CASSANDRA-2752: --- Chris: Same as the "compactionexecutor" or the infinite loop? (made CASSANDRA-2758 for that just now). Seems like the sstables here has been truncated for some reason. Rowindexer iterates through a bunch of rows just fine. Then it reaches the problem row. For this row, it can get the key, it jumps pass the bloomfilter etc and when it is about to read the column count, it fails and is trying to read at an offset which equals the length of the file.. So far, all the stacktraces I have seen are all are on the column count read. A wild guess may be that it has failed to write the actual content (columns) of the last row that was stream? Unfortunately it does not to happen all the time, but it does only happen on the CF with secondary indexes. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.u
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046984#comment-13046984 ] Muga Nishizawa commented on CASSANDRA-2752: --- It seems tmp files (e.g. XXX-tmp-XXX-Data.db) that receiver node creates during repair process are broken. EOFException occurs while RowIndexer is reading the broken files. According to result of scrub command, data files on sender nodes are not broken. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.Executi
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046897#comment-13046897 ] Chris Goffinet commented on CASSANDRA-2752: --- I am trying to track down a similar issue. Instead was bootstrapping a new node in my case. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureT
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046628#comment-13046628 ] Terje Marthinussen commented on CASSANDRA-2752: --- Fresh 0.8 data. Just as a side test since the "compactionexecutor" is involved, we issued a full compaction of that CF and it completed without any error so the source SSTables seems good. I was trying to reproduce this locally on my desktop before leaving office to get it in a debugger. Quickly generated 10k random inserts into a CF with secondary index, but then I experienced that repair got stuck eating 100% on both nodes instead... I did not have time to figure out if it was due to some config issue or related to the same issue though. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.strea
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046606#comment-13046606 ] Jonathan Ellis commented on CASSANDRA-2752: --- Is this data from 0.7 that got upgraded? > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.st
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046586#comment-13046586 ] Terje Marthinussen commented on CASSANDRA-2752: --- Seems confirmed, only able to reproduce this on a CF with secondary indexes. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java
[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
[ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046509#comment-13046509 ] Terje Marthinussen commented on CASSANDRA-2752: --- Need to test a bit more, but quite likely this is related to repair on CFs with secondary indexes. > repair fails with java.io.EOFException > -- > > Key: CASSANDRA-2752 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2752 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Terje Marthinussen > > Issuing repair on node 1 (1.10.42.81) in a cluster quickly fails with > INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java > (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1 > .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out > of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])] > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java > (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306 > fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]> > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) > Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db > sections=1 progress=0/36592 - 0%], 1 sstables. > INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java > (line 174) Streaming to /1.10.42.82 > ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:9,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > On .82 > ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:12,1,main] > java.io.EOFException > at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line > 113) Fatal exception in thread Thread[Thread-132,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.EOFException > at > org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) > at > org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) > Caused by: java.util.concurrent.ExecutionException: java.io.EOFException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureT