Hi,
I just reran the command and collected following. Any suggestions would be
appreciated.
Thanks,
Jason
from 192.168.10.8
ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10
failed stream session.
INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Session with /192.168.10.10 is complete
WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858
StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Stream failed
from 192.168.10.8 debug
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received Received (79256340-bbbb-11e5-9f70-7d76a8de8480, #0)
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0, version:
ma, format: BIG, estimated keys: 128, transfer size: 4653, compressed?: true,
repairedAt: 0, level: 0), file:
/home/cassandra/data/sensordb/sensor/ma-76-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Start streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to
/192.168.10.10, repairedAt = 0, totalSize = 4653
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Finished streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db
to /192.168.10.10, bytesTransferred = 4653, totalSize = 4653
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0, version:
ma, format: BIG, estimated keys: 128, transfer size: 705, compressed?: true,
repairedAt: 0, level: 0), file:
/home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856
CompressedStreamWriter.java:63 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Start streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db
to /192.168.10.10, repairedAt = 0, totalSize = 705
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856
CompressedStreamWriter.java:94 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Finished streaming file
/home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10,
bytesTransferred = 705, totalSize = 705
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received Session Failed
ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10
failed stream session.
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
ConnectionHandler.java:110 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Closing stream connection handler on /192.168.10.10
INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Session with /192.168.10.10 is complete
WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858
StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Stream failed
from 192.168.10.10
[2016-02-27 20:37:53,413] received file
/home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db
(progress: 365%)
[2016-02-27 20:37:53,414] received file
/home/cassandra/data/sensordb/liestedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db
(progress: 369%)
[2016-02-27 20:37:53,865] session with /192.168.10.8 complete (progress: 369%)
[2016-02-27 20:37:53,866] Stream failed
from 192.168.10.10 debug
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,201
CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 166627, ks =
'sensordb', table = 'listAttributes'.
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412
CompressedStreamReader.java:110 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Finished receiving file #0 from
/192.168.10.8 readBytes = 166627, totalSize = 166627
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received File (Header (cfId: 79256340-bbbb-11e5-9f70-7d76a8de8480, #0, version:
ma, format: BIG, estimated keys: 128, transfer size: 166627, compressed?: true,
repairedAt: 0, level: 0), file:
/home/cassandra/data/sensordb/listAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,412
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Sending Received (79256340-bbbb-11e5-9f70-7d76a8de8480, #0)
DEBUG [CompactionExecutor:3] 2016-02-27 20:37:53,833 CompactionTask.java:217 -
Compacted (e224bef0-ddbb-11e5-80c0-89f591237aca) 4 sstables to
[/home/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/ma-5-big,]
to level=0. 2,743,164 bytes to 685,791 (~25% of original) in 1,096ms =
0.596735MB/s. 0 total partitions merged to 57. Partition merge counts were
{4:57, }
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,850
CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 4653, ks =
'sensordb', table = 'sensor'.
WARN [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,851 StreamSession.java:641
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error
java.lang.RuntimeException: Unknown column lastEvaluation during deserialization
at
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-3.0.3.jar:3.0.3]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,852
ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Sending Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,852
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received null
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,853
CompressedStreamReader.java:80 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Start receiving file #0 from /192.168.10.8, repairedAt = 0, size = 705, ks =
'sensordb', table = 'sensorUnit'.
WARN [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 StreamSession.java:641
- [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error
java.lang.RuntimeException: Unknown column lastCheckTime during deserialization
at
org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-3.0.3.jar:3.0.3]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854
ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
Received null
From: Sebastian Estevez <sebastian.este...@datastax.com>
To: user@cassandra.apache.org; Jason Kania <jason.ka...@ymail.com>
Sent: Saturday, February 27, 2016 8:24 PM
Subject: Re: How to complete bootstrap with exception due to stream failure?
progress: 361% does not look right (probably a bug).
Can you check the corresponding messages on the other side of the stream? I.E.
the system log for 192.168.10.8 around 18:02:06?
All the best,
Sebastián EstévezSolutions Architect | 954 905 8615 |
sebastian.este...@datastax.com
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any size.
With more than 500 customers in 45 countries, DataStax is the database
technology and transactional backbone of choice for the worlds most innovative
companies such as Netflix, Adobe, Intuit, and eBay.
On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania <jason.ka...@ymail.com> wrote:
Hello,
I am trying to get a node bootstrapped in 3.0.3, but just at the point where
the bootstrap process is to complete, a broken pipe exception occurs so the
bootstrap process hangs. Once I kill the bootstrap process, I can execute
"nodetool bootstrap resume" again and the same problem will occur just at the
end of the bootstrap exercise. Here is the tail of the log:
[2016-02-27 18:02:05,898] received file
/home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-30-big-Data.db
(progress: 357%)
[2016-02-27 18:02:06,479] received file
/home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
(progress: 361%)
[2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress: 361%)
[2016-02-27 18:02:06,886] Stream failed
I attempted to run nodetool repair, but get the following which I have been
told indicates that the replication factor is 1:
root@bull:~# nodetool repair
[2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'
Thanks,
Jason
