[
https://issues.apache.org/jira/browse/CASSANDRA-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089267#comment-15089267
]
Paulo Motta commented on CASSANDRA-10963:
-----------------------------------------
If scrubbing doesn't help could you try the patch on CASSANDRA-10961?
> Can join cluster java.lang.InterruptedException
> ------------------------------------------------
>
> Key: CASSANDRA-10963
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10963
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: [cqlsh 5.0.1 | Cassandra 2.2.4 | CQL spec 3.3.1 | Native
> protocol v4]
> java version "1.8.0_65"
> Reporter: Jack Money
> Assignee: Paulo Motta
>
> hello
> I got 2 nodes in 2 DC.
> Each node own 100% data of keyspace hugespace.
> Keyspace have 21 tables with 2TB data
> Biggest table have 1.6 TB of data.
> Biggest sstable 1,3 TB.
> Schemats:
> {noformat}
> KEYSPACE hugespace WITH replication = {'class': 'NetworkTopologyStrategy',
> 'DC1': '3', 'DC2': '1'};
> CREATE TABLE hugespace.content (
> y int,
> m int,
> d int,
> ts bigint,
> ha text,
> co text,
> he text,
> ids bigint,
> ifr text,
> js text,
> PRIMARY KEY ((y, m, d), ts, ha)
> ) WITH CLUSTERING ORDER BY (ts ASC, ha ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> CREATE INDEX content_ids_idx ON hugespace.content (ids);
> {noformat}
> I tried to add one node (target 6 node in DC1) to DC1.
> Names:
> Existing node in DC1 = nodeDC1
> Existing node in DC2 = nodeDC2
> New node joining DC1 = joiningDC1
> joiningDC1
> {noformat}
> INFO [main] 2016-01-04 12:17:55,535 StorageService.java:1176 - JOINING:
> Starting to bootstrap...
> INFO [main] 2016-01-04 12:17:55,802 StreamResultFuture.java:86 - [Stream
> #2f473320-b2dd-11e5-8353-b5506ad414a4] Executing streaming plan for Bootstrap
> INFO [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,803
> StreamSession.java:232 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Starting streaming to /nodeDC1
> INFO [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,803
> StreamSession.java:232 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Starting streaming to /nodeDC2
> DEBUG [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,803
> ConnectionHandler.java:82 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending stream init for incoming stream
> DEBUG [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,803
> ConnectionHandler.java:82 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending stream init for incoming stream
> DEBUG [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,806
> ConnectionHandler.java:87 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending stream init for outgoing stream
> DEBUG [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,806
> ConnectionHandler.java:87 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending stream init for outgoing stream
> DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:17:55,810
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Prepare (5 requests, 0 files}
> DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:55,810
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Prepare (2 requests, 0 files}
> INFO [StreamConnectionEstablisher:2] 2016-01-04 12:17:55,810
> StreamCoordinator.java:213 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4,
> ID#0] Beginning stream session with /nodeDC2
> INFO [StreamConnectionEstablisher:1] 2016-01-04 12:17:55,810
> StreamCoordinator.java:213 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4,
> ID#0] Beginning stream session with /nodeDC1
> DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,821 ConnectionHandler.java:266
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (0
> requests, 1 files}
> INFO [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,822
> StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4
> ID#0] Prepare completed. Receiving 1 files(161 bytes), sending 0 files(0
> bytes)
> DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,828
> CompressedStreamReader.java:67 - reading file from /nodeDC2, repairedAt =
> 1451483586917
> DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,831 ConnectionHandler.java:266
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId:
> 5bc52802-de25-35ed-aeab-188eecebb090, #0, version: la, format: BIG, estimated
> keys: 128, transfer size: 161, compressed?: true, repairedAt: 1451483586917,
> level: 0), file:
> /cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/tmp-la-1-big-Data.db)
> DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:55,831
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Received (5bc52802-de25-35ed-aeab-188eecebb090, #0)
> DEBUG [STREAM-IN-/nodeDC2] 2016-01-04 12:17:55,832 ConnectionHandler.java:266
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Complete
> DEBUG [STREAM-OUT-/nodeDC2] 2016-01-04 12:17:56,049
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Complete
> DEBUG [StreamReceiveTask:1] 2016-01-04 12:17:56,049
> ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Closing stream connection handler on /nodeDC2
> INFO [StreamReceiveTask:1] 2016-01-04 12:17:56,049
> StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Session with /nodeDC2 is complete
> DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,848 ConnectionHandler.java:266
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received Prepare (0
> requests, 194 files}
> INFO [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,849
> StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4
> ID#0] Prepare completed. Receiving 194 files(2130405704708 bytes), sending 0
> files(0 bytes)
> DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:17:56,882
> CompressedStreamReader.java:67 - reading file from /nodeDC1, repairedAt =
> 1451476116723
> DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:01,418 ConnectionHandler.java:266
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId:
> 99e93a30-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated
> keys: 128, transfer size: 56333243, compressed?: true, repairedAt:
> 1451476116723, level: 0), file:
> /cassandra/data/hugespace/other_data-99e93a30814a11e5a573a94b3a1b0e16/tmp-la-1-big-Data.db)
> DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:01,419
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Received (99e93a30-814a-11e5-a573-a94b3a1b0e16, #0)
> DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:01,596
> CompressedStreamReader.java:67 - reading file from /nodeDC1, repairedAt = 0
> DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,371 ConnectionHandler.java:266
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Received File (Header (cfId:
> a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2, version: la, format: BIG, estimated
> keys: 128, transfer size: 1693486699, compressed?: true, repairedAt: 0,
> level: 0), file:
> /cassandra/data/hugespace/content_demo-a5f465809e5e11e59e43a94b3a1b0e16/tmp-la-1-big-Data.db)
> DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:55,374
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Received (a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2)
> ERROR [Thread-4] 2016-01-04 12:18:55,375 CassandraDaemon.java:185 - Exception
> in thread Thread[Thread-4,5,main]
> java.lang.RuntimeException: java.lang.InterruptedException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
> Caused by: java.lang.InterruptedException: null
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 1 common frames omitted
> ERROR [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,375 StreamSession.java:524 -
> [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Streaming error occurred
> java.nio.channels.ClosedChannelException: null
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
> ~[na:1.8.0_65]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> DEBUG [STREAM-OUT-/nodeDC1] 2016-01-04 12:18:55,376
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Session Failed
> DEBUG [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,488 ConnectionHandler.java:110
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Closing stream connection
> handler on /nodeDC1
> INFO [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,489
> StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Session with /nodeDC1 is complete
> WARN [STREAM-IN-/nodeDC1] 2016-01-04 12:18:55,489
> StreamResultFuture.java:209 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Stream failed
> ERROR [main] 2016-01-04 12:18:55,490 StorageService.java:1245 - Error while
> waiting on bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
> [apache-cassandra-2.2.4.jar:2.2.4]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
> WARN [main] 2016-01-04 12:18:55,496 StorageService.java:975 - Some data
> streaming failed. Use nodetool to check bootstrap state and resume. For more,
> see `nodetool help bootstrap`. IN_PROGRESS
> {noformat}
> nodeDC1
> {noformat}
> DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:56,848
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Prepare (0 requests, 194 files}
> INFO [STREAM-IN-/joiningDC1] 2016-01-04 12:17:56,848
> StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 194 files
> 2130405704708 bytes)
> DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:56,851
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending File (Header (cfId: 99e93a30-814a-11e5-a573-a94b3a1b0e16, #0,
> version: la, format: BIG, estimated keys: 128, transfer size: 56333243,
> compressed?: true, repairedAt: 1451476116723, level: 0), file:
> /cassandra/data/hugespace/other_data-99e93a30814a11e5a573a94b3a1b0e16/la-202-big-Data.db)
> DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 12:17:59,596
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending File (Header (cfId: a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2,
> version: la, format: BIG, estimated keys: 128, transfer size: 1693486699,
> compressed?: true, repairedAt: 0, level: 0), file:
> /cassandra/data/hugespace/content_demo-a5f465809e5e11e59e43a94b3a1b0e16/la-1280-big-Data.db)
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:01,422
> ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Received Received (99e93a30-814a-11e5-a573-a94b3a1b0e16, #0)
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,375
> ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Received Received (a5f46580-9e5e-11e5-9e43-a94b3a1b0e16, #2)
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,920
> ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Received Session Failed
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,920
> ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Closing stream connection handler on /joiningDC1
> INFO [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,921
> StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Session with /joiningDC1 is complete
> WARN [STREAM-IN-/joiningDC1] 2016-01-04 12:18:55,923
> StreamResultFuture.java:209 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Stream failed
> ERROR [STREAM-OUT-/joiningDC1] 2016-01-04 12:18:55,925 StreamSession.java:524
> - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4] Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
> ~[na:1.8.0_65]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_65]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_65]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 11 common frames omitted
> {noformat}
> nodeDC2 _other timezone +1_ no error
> {noformat}
> INFO [STREAM-INIT-/joiningDC1:46377] 2016-01-04 13:17:55,807
> StreamResultFuture.java:111 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4
> ID#0] Creating new streaming plan for Bootstrap
> INFO [STREAM-INIT-/joiningDC1:46377] 2016-01-04 13:17:55,808
> StreamResultFuture.java:118 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4,
> ID#0] Received streaming plan for Bootstrap
> INFO [STREAM-INIT-/joiningDC1:46380] 2016-01-04 13:17:55,809
> StreamResultFuture.java:118 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4,
> ID#0] Received streaming plan for Bootstrap
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,819
> ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Received Prepare (2 requests, 0 files}
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,819 StreamSession.java:347
> - ViewFilter for 1/1 sstables
> DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,820
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Prepare (0 requests, 1 files}
> INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,820
> StreamResultFuture.java:168 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 1 files(161
> bytes)
> DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,820
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending File (Header (cfId: 5bc52802-de25-35ed-aeab-188eecebb090, #0,
> version: la, format: BIG, estimated keys: 128, transfer size: 161,
> compressed?: true, repairedAt: 1451483586917, level: 0), file:
> /srv/cassandra/data/system_auth/roles-5bc52802de2535edaeab188eecebb090/la-1-big-Data.db)
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:55,831
> ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Received Received (5bc52802-de25-35ed-aeab-188eecebb090, #0)
> DEBUG [STREAM-OUT-/joiningDC1] 2016-01-04 13:17:55,832
> ConnectionHandler.java:334 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Sending Complete
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,049
> ConnectionHandler.java:266 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Received Complete
> DEBUG [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050
> ConnectionHandler.java:110 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Closing stream connection handler on /joiningDC1
> INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050
> StreamResultFuture.java:182 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> Session with /joiningDC1 is complete
> INFO [STREAM-IN-/joiningDC1] 2016-01-04 13:17:56,050
> StreamResultFuture.java:214 - [Stream #2f473320-b2dd-11e5-8353-b5506ad414a4]
> All sessions completed
> {noformat}
> I also ran sstableverify two times on nodeDC1 and i get this error (2 times
> for the same sstables)
> {noformat}
> ERROR 23:13:50 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@799ed08e) to class
> org.apache.cassandra.io.util.ChannelProxy$Cleanup@546942821:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-12252-big-Data.db
> was not released before the reference was garbage collected
> ERROR 23:13:50 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@2c747998) to class
> org.apache.cassandra.io.util.ChannelProxy$Cleanup@218678140:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13487-big-Data.db
> was not released before the reference was garbage collected
> ERROR 23:13:50 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@63a80e20) to class
> org.apache.cassandra.io.util.ChannelProxy$Cleanup@652932560:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13499-big-Data.db
> was not released before the reference was garbage collected
> ERROR 23:13:50 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@295df02c) to class
> org.apache.cassandra.io.util.ChannelProxy$Cleanup@1644545368:/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/la-13497-big-Data.db
> was not released before the reference was garbage collected
> {noformat}
> sstable size 176 GB, 94GB, 3,6GB, 3,6GB all in the same table as biggest one
> - 1,3 TB
> I albo ran nodetool repair on nodeDC1 and i get the same error
> nodeDC1
> {noformat}
> ERROR [Thread-10] 2015-12-31 05:09:27,960 CassandraDaemon.java:185 -
> Exception in thread Thread[Thread-10,5,main]
> java.lang.RuntimeException: java.lang.InterruptedException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
> Caused by: java.lang.InterruptedException: null
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 1 common frames omitted
> DEBUG [STREAM-IN-/nodeDC2] 2015-12-31 05:09:27,960 ConnectionHandler.java:266
> - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Received File (Header (cfId:
> 94be9870-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated
> keys: 1, transfer size: 5190031760, compressed?: true, repairedAt:
> 1451517863052, level: 0), file:
> /cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/tmp-la-13501-big-Data.db)
> ERROR [STREAM-IN-/nodeDC2] 2015-12-31 05:09:27,967 StreamSession.java:524 -
> [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred
> java.nio.channels.ClosedChannelException: null
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
> ~[na:1.8.0_65]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> DEBUG [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,235 ConnectionHandler.java:110
> - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Closing stream connection
> handler on /nodeDC2
> INFO [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,235
> StreamResultFuture.java:182 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16]
> Session with /nodeDC2 is complete
> WARN [STREAM-IN-/nodeDC2] 2015-12-31 05:09:34,236
> StreamResultFuture.java:209 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16]
> Stream failed
> ERROR [STREAM-OUT-/nodeDC2] 2015-12-31 05:09:34,236 StreamSession.java:524 -
> [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
> ~[na:1.8.0_65]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_65]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_65]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 11 common frames omitted
> WARN [RepairJobTask:17] 2015-12-31 05:09:34,236 RepairJob.java:162 - [repair
> #75cfdc00-af4c-11e5-8429-a94b3a1b0e16] content sync failed
> ERROR [RepairJobTask:17] 2015-12-31 05:09:34,238 RepairSession.java:290 -
> [repair #75cfdc00-af4c-11e5-8429-a94b3a1b0e16] Session completed with the
> following error
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> ERROR [RepairJobTask:17] 2015-12-31 05:09:34,239 RepairRunnable.java:243 -
> Repair session 75cfdc00-af4c-11e5-8429-a94b3a1b0e16 for range
> (-308435053110486424,-206175645991596694] failed with error Stream failed
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:434)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:529)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> {noformat}
> nodeDC2 _timezone +1_
> {noformat}
> ERROR [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,590 StreamSession.java:524 -
> [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred
> org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
> ~[na:1.8.0_65]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427)
> ~[na:1.8.0_65]
> at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492)
> ~[na:1.8.0_65]
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 11 common frames omitted
> DEBUG [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591
> ConnectionHandler.java:110 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16]
> Closing stream connection handler on /nodeDC1
> INFO [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591
> StreamResultFuture.java:182 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16]
> Session with /nodeDC1 is complete
> WARN [STREAM-OUT-/nodeDC1] 2015-12-31 06:09:28,591
> StreamResultFuture.java:209 - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16]
> Stream failed
> ERROR [Thread-25] 2015-12-31 06:09:33,932 CassandraDaemon.java:185 -
> Exception in thread Thread[Thread-25,5,main]
> java.lang.RuntimeException: java.lang.InterruptedException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
> Caused by: java.lang.InterruptedException: null
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350)
> ~[na:1.8.0_65]
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:176)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 1 common frames omitted
> DEBUG [STREAM-IN-/nodeDC1] 2015-12-31 06:09:33,931 ConnectionHandler.java:266
> - [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Received File (Header (cfId:
> 94be9870-814a-11e5-a573-a94b3a1b0e16, #0, version: la, format: BIG, estimated
> keys: 1, transfer size: 5344434128, compressed?: true, repairedAt:
> 1451517863052, level: 0), file:
> /srv/cassandra/data/hugespace/content-94be9870814a11e5a573a94b3a1b0e16/tmp-la-13949-big-Data.db)
> ERROR [STREAM-IN-/nodeDC1] 2015-12-31 06:09:33,935 StreamSession.java:524 -
> [Stream #8b40bf70-af7c-11e5-8429-a94b3a1b0e16] Streaming error occurred
> java.lang.RuntimeException: Outgoing stream handler has been closed
> at
> org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:132)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:587)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:476)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:267)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> {noformat}
> Please advice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
