[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-14 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049271#comment-13049271
 ] 

Hudson commented on CASSANDRA-2752:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])


> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>Assignee: Jonathan Ellis
>Priority: Critical
> Fix For: 0.8.1
>
> Attachments: 2752.txt
>
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at j

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-14 Thread Sylvain Lebresne (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049045#comment-13049045
 ] 

Sylvain Lebresne commented on CASSANDRA-2752:
-

Good catch. +1 (committed).
Thanks Terje.

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>Assignee: Terje Marthinussen
>Priority: Critical
> Fix For: 0.8.1
>
> Attachments: 2752.txt
>
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at java.util.concurrent.FutureTask

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-13 Thread Terje Marthinussen (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048655#comment-13048655
 ] 

Terje Marthinussen commented on CASSANDRA-2752:
---

SSTableWriter.java:

ColumnFamily.serializer().deserializeFromSSTableNoColumns(ColumnFamily.create(cfs.metadata),
 dfile);

// don't move that statement around, it expects the dfile to be 
before the columns
updateCache(key, dataSize, null);

rowSizes.add(dataSize);
columnCounts.add(dfile.readInt());


I believe the problem is in updateCache.
If rowcache is enabled (and it is in this case) and the row needs to be updated 
in cache, this will read (deserialize) the row.

However, after all the columns is read, the offset in the file is not reset 
back to the location where the  column count is stored and things go bad.

I haven't actually tried to change the code to test, but I tried to disable the 
row cache, and so far, repair seems to work fine when it is disabled.

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-13 Thread Muga Nishizawa (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048637#comment-13048637
 ] 

Muga Nishizawa commented on CASSANDRA-2752:
---

It seems like timing issue.  The exception doesn't always occur always even 
with use of same data.  

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.ge

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-10 Thread Terje Marthinussen (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047266#comment-13047266
 ] 

Terje Marthinussen commented on CASSANDRA-2752:
---

Also struck me on the way home from work today that the CF with secondary 
indexes also happen to be the only CF in this system which, I think, on a 
regular basis may actually update all columns for a key.

That is, sstables will on a regular basis have keys where no columns is valid 
anymore.

Not sure if that could for instance trigger something odd in the streaming?




> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> or

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-10 Thread Terje Marthinussen (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047211#comment-13047211
 ] 

Terje Marthinussen commented on CASSANDRA-2752:
---

Chris: Same as the "compactionexecutor" or the infinite loop? (made 
CASSANDRA-2758 for that just now).

Seems like the sstables here has been truncated for some reason.
Rowindexer iterates through a bunch of rows just fine.
Then it reaches the problem row.

For this row, it can get the key, it jumps pass the bloomfilter etc and when it 
is about to read the column count, it fails and is trying to read at an offset 
which equals the length of the file..
So far, all the stacktraces I have seen are all are on the column count read. 

A wild guess may be that it has failed to write the actual content (columns) of 
the last row that was stream?

Unfortunately it does not to happen all the time, but it does only happen on 
the CF with secondary indexes.





> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.u

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-09 Thread Muga Nishizawa (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046984#comment-13046984
 ] 

Muga Nishizawa commented on CASSANDRA-2752:
---

It seems tmp files (e.g. XXX-tmp-XXX-Data.db) that receiver node creates during 
repair process are broken.  EOFException occurs while RowIndexer is reading the 
broken files.  

According to result of scrub command, data files on sender nodes are not 
broken.  

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.Executi

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-09 Thread Chris Goffinet (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046897#comment-13046897
 ] 

Chris Goffinet commented on CASSANDRA-2752:
---

I am trying to track down a similar issue. Instead was bootstrapping a new node 
in my case.

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureT

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-09 Thread Terje Marthinussen (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046628#comment-13046628
 ] 

Terje Marthinussen commented on CASSANDRA-2752:
---

Fresh 0.8 data.

Just as a side test since the "compactionexecutor" is involved, we issued a 
full compaction of that CF and it completed without any error so the source 
SSTables seems good.

I was trying to reproduce this locally on my desktop before leaving office to 
get it in a debugger. Quickly generated 10k random inserts into a CF with 
secondary index, but then I experienced that repair got stuck eating 100% on 
both nodes instead...

I did not have time to figure out if it was due to some config issue or related 
to the same issue though.



> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.strea

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-09 Thread Jonathan Ellis (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046606#comment-13046606
 ] 

Jonathan Ellis commented on CASSANDRA-2752:
---

Is this data from 0.7 that got upgraded?

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at 
> org.apache.cassandra.st

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-09 Thread Terje Marthinussen (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046586#comment-13046586
 ] 

Terje Marthinussen commented on CASSANDRA-2752:
---

Seems confirmed, only able to reproduce this on a CF with secondary indexes.

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-09 Thread Terje Marthinussen (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046509#comment-13046509
 ] 

Terje Marthinussen commented on CASSANDRA-2752:
---

Need to test a bit more, but quite likely this is related to repair on CFs with 
secondary indexes.

> repair fails with java.io.EOFException
> --
>
> Key: CASSANDRA-2752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 0.8.0
>Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
> (line 234) Queueing comparison # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
> of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
> (line 485) Performing streaming repair of 2 ranges for # manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
> Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
> sections=1 progress=0/36592 - 0%], 1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
> (line 174) Streaming to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
> Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
> at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
> at 
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
> at 
> org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
> 113) Fatal exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.io.EOFException
> at 
> org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
> at 
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at 
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureT