[
https://issues.apache.org/jira/browse/CASSANDRA-9624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646776#comment-14646776
]
Eric Evans commented on CASSANDRA-9624:
---------------------------------------
Since we had previously bootstrapped with 2.1.3 when the cluster had less data
and lower throughput, we scaled the cluster down to see if it would help, and
in so doing, just managed to successfully bootstrap a new node.
I don't even.
> unable to bootstrap; streaming fails with NullPointerException
> --------------------------------------------------------------
>
> Key: CASSANDRA-9624
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9624
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
> Reporter: Eric Evans
> Assignee: Yuki Morishita
> Fix For: 2.1.x
>
>
> When attempting to bootstrap a new node into a 2.1.3 cluster, the stream
> source fails with a {{NullPointerException}}:
> {noformat}
> ERROR [STREAM-IN-/10.xx.x.xxx] 2015-06-13 00:02:01,264 StreamSession.java:477
> - [Stream #60e8c120-
> 115f-11e5-9fee-xxxxxxxxxxxx] Streaming error occurred
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:1277)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.streaming.StreamSession.getSSTableSectionsForRanges(StreamSession.java:313)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.streaming.StreamSession.addTransferRanges(StreamSession.java:266)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:493)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:425)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:251)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> INFO [STREAM-IN-/10.xx.x.xxx] 2015-06-13 00:02:01,265
> StreamResultFuture.java:180 - [Stream #60e8c120-115f-11e5-9fee-xxxxxxxxxxxx]
> Session with /10.xx.x.xx1 is complete
> {noformat}
> _Update (2015-06-26):_
> I can also reproduce this on 2.1.7, though without the NPE on the stream-from
> side.
> Stream source / existing node:
> {noformat}
> INFO [STREAM-IN-/10.64.32.178] 2015-06-26 06:48:53,060
> StreamResultFuture.java:180 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9]
> Session with /10.64.32.178 is complete
> INFO [STREAM-IN-/10.64.32.178] 2015-06-26 06:48:53,064
> StreamResultFuture.java:212 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9]
> All sessions completed
> {noformat}
> Stream sink / bootstrapping node:
> {noformat}
> INFO [StreamReceiveTask:57] 2015-06-26 06:48:53,061
> StreamResultFuture.java:180 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9]
> Session with /10.64.32.160 is complete
> WARN [StreamReceiveTask:57] 2015-06-26 06:48:53,062
> StreamResultFuture.java:207 - [Stream #8bdeb1b0-1ad2-11e5-abd8-3fcfb96209d9]
> Stream failed
> INFO [CompactionExecutor:2885] 2015-06-26 06:48:53,062
> ColumnFamilyStore.java:906 - Enqueuing flush of compactions_in_progress: 428
> (0%) on-heap, 379 (0%) off-heap
> INFO [MemtableFlushWriter:959] 2015-06-26 06:48:53,063 Memtable.java:346 -
> Writing Memtable-compactions_in_progress@1203013482(294 serialized bytes, 12
> ops, 0%/0% of on/off-heap limit)
> ERROR [main] 2015-06-26 06:48:53,063 CassandraDaemon.java:541 - Exception
> encountered during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1137)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:927)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:723)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:605)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
> [apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524)
> [apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613)
> [apache-cassandra-2.1.7.jar:2.1.7]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:409)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:682)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:645)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:139)
> ~[apache-cassandra-2.1.7.jar:2.1.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_79]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_79]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
