Hi all,
I ran decommission on a node in my 32 node cluster. After about an hour of
streaming files to another node, I got this error on the node being
decommissioned:
INFO [MiscStage:1] 2011-05-03 21:49:00,235 StreamReplyVerbHandler.java (line
58) Need to re-stream file /raiddrive/MDR/MeterRecords-f-2283-Data.db to
/10.206.63.208
ERROR [Streaming:1] 2011-05-03 21:49:01,580 DebuggableThreadPoolExecutor.java
(line 103) Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.IOException: Broken pipe
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
at
org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:105)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:67)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
ERROR [Streaming:1] 2011-05-03 21:49:01,581 AbstractCassandraDaemon.java (line
112) Fatal exception in thread Thread[Streaming:1,1,main]
java.lang.RuntimeException: java.io.IOException: Broken pipe
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
at
org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:105)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:67)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
And this message on the node that it was streaming to:
INFO [Thread-333] 2011-05-03 21:49:00,234 StreamInSession.java (line 121)
Streaming of file
/raiddrive/MDR/MeterRecords-f-2283-Data.db/(98605680685,197932763967)
progress=49016107008/99327083282 - 49% from
org.apache.cassandra.streaming.StreamInSession@33721219 failed: requesting a
retry.
I tried running decommission again (and running scrub + decommission), but I
keep getting this error on the same file.
I checked out the file and saw that it is a lot bigger than all the other
sstables... 184GB instead of about 74MB. I haven't run a major compaction for a
bit, so I'm trying to stream 658 sstables.
I'm using Cassandra 0.7.4, I have two data directories (I know that's not good
practice...), and all my nodes are on Amazon EC2.
Any thoughts on what could be going on or how to prevent this?
Thanks!
Tamara
________________________________
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise private information. If you have received it in
error, please notify the sender immediately and delete the original. Any other
use of the email by you is prohibited.
