[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916263#comment-13916263
]
sankalp kohli commented on CASSANDRA-6636:
--
The StreamPlan.handlers is not thread safe but here it is set by same thread so
it is good.
So looks good.
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>Assignee: Yuki Morishita
>Priority: Minor
> Fix For: 2.0.6
>
> Attachments: 6636-2.0.txt
>
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13910577#comment-13910577
]
sankalp kohli commented on CASSANDRA-6636:
--
Sure.
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>Assignee: Yuki Morishita
>Priority: Minor
> Fix For: 2.0.6
>
> Attachments: 6636-2.0.txt
>
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900888#comment-13900888
]
Jonathan Ellis commented on CASSANDRA-6636:
---
How involved would fixing this be?
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>Assignee: Yuki Morishita
>Priority: Minor
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13887088#comment-13887088
]
Yuki Morishita commented on CASSANDRA-6636:
---
So the root cause I tracked down is actually NPE when showing progress.
BulkLoader expects to receive StreamEvent PREPARE to setup progress indicator.
But when you get the exception, event listener is registered after streaming
emits PREPARE event. I think this is more likely to happen in dev environment
like ccm.
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>Assignee: Yuki Morishita
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885941#comment-13885941
]
Yuki Morishita commented on CASSANDRA-6636:
---
I can reproduced in my box. I will investigate.
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>Assignee: Yuki Morishita
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885931#comment-13885931
]
Russ Hatch commented on CASSANDRA-6636:
---
One other strange thing to note: I was able to successfully load data into
node2 and node3. After that, the load also worked on node1.
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6636) sstableloader fails when attempting to load data from a single node into a multi-node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885883#comment-13885883
]
Russ Hatch commented on CASSANDRA-6636:
---
I reproduced this with the latest code on the cassandra-2.0 branch.
I should also note that the sstableloader command succeeds if I attempt
importing into a single node cluster (instead of into a 3 node cluster).
> sstableloader fails when attempting to load data from a single node into a
> multi-node cluster
> -
>
> Key: CASSANDRA-6636
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6636
> Project: Cassandra
> Issue Type: Bug
> Environment: java version "1.7.0_51"
> cassandra from cassandra-2.0 branch (0be424...)
>Reporter: Russ Hatch
>
> I'm running into this exception when trying to use sstableloader to bring in
> data from another cluster:
> {noformat}
> rhatch@whatup:~/.ccm/test_cluster_1391031988/node1$ bin/sstableloader -d
> 127.0.0.1 ~/tmp/Keyspace1/Standard1
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-5-Data.db
> /home/rhatch/tmp/Keyspace1/Standard1/Keyspace1-Standard1-jb-6-Data.db to
> [/127.0.0.1, /127.0.0.2, /127.0.0.3]
> Exception in thread "STREAM-OUT-/127.0.0.1" java.lang.NullPointerException
> at
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> This is what I see in the node system.log:
> {noformat}
> ==> ./test_cluster_1391031988/node1/logs/system.log <==
> INFO [STREAM-INIT-/127.0.0.1:60971] 2014-01-29 14:57:25,375
> StreamResultFuture.java (line 116) [Stream
> #564ded70-8930-11e3-84e9-2766c3cc4197] Received streaming plan for Bulk Load
> INFO [STREAM-IN-/127.0.1.1] 2014-01-29 14:57:25,375 StreamResultFuture.java
> (line 168) [Stream #564ded70-8930-11e3-84e9-2766c3cc4197] Prepare completed.
> Receiving 2 files(91047224 bytes), sending 0 files(0 bytes)
> {noformat}
> steps to reproduce:
> create a 3 node cluster with ccm
> run stress on one node with 'ccm node1 stress'
> copy the node's data from the data/Keyspace1/Standard1 directory to save it
> for re-loading (preserve the directory structure for sstableloader)
> remove the cluster, and create a new 3 node cluster
> pick a node and run bin/nodetool -d localhost
> ~/saved_data_location/Keyspace1/Standard1
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
