[ https://issues.apache.org/jira/browse/CASSANDRA-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852313#comment-13852313 ]
Russell Alexander Spitzer commented on CASSANDRA-6210: ------------------------------------------------------ Repair running on this node {code} INFO [AntiEntropyStage:1] 2013-12-18 22:39:28,209 StreamResultFuture.java (line 82) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Executing streaming plan for Repair INFO [AntiEntropyStage:1] 2013-12-18 22:39:28,209 StreamResultFuture.java (line 86) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Beginning stream session with /10.171.81.22 DEBUG [StreamConnectionEstablisher:2] 2013-12-18 22:39:28,210 ConnectionHandler.java (line 78) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Sending stream init for incoming stream DEBUG [StreamConnectionEstablisher:2] 2013-12-18 22:39:28,211 ConnectionHandler.java (line 84) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Sending stream init for outgoing stream DEBUG [STREAM-OUT-/10.171.81.22] 2013-12-18 22:39:28,212 ConnectionHandler.java (line 356) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Sending Prepare (1 requests, 2 files} {code} On requested node {code} DEBUG [STREAM-IN-/10.172.27.174] 2013-12-18 22:39:28,296 ConnectionHandler.java (line 292) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Received Prepare (1 requests, 2 files} ERROR [STREAM-IN-/10.172.27.174] 2013-12-18 22:39:28,314 StreamSession.java (line 410) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Streaming error occurred java.lang.NullPointerException at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174) at org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436) at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293) at java.lang.Thread.run(Thread.java:724) DEBUG [STREAM-IN-/10.172.27.174] 2013-12-18 22:39:28,316 ConnectionHandler.java (line 153) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Closing stream connection handler on /10.172.27.174 INFO [STREAM-IN-/10.172.27.174] 2013-12-18 22:39:28,317 StreamResultFuture.java (line 181) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Session with /10.172.27.174 is complete WARN [STREAM-IN-/10.172.27.174] 2013-12-18 22:39:28,317 StreamResultFuture.java (line 210) [Stream #40d875d0-6835-11e3-a172-3729e500a0e7] Stream failed {code} > Repair hangs when a new datacenter is added to a cluster > -------------------------------------------------------- > > Key: CASSANDRA-6210 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6210 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Amazon Ec2 > 2 M1.large nodes > Reporter: Russell Alexander Spitzer > Assignee: Yuki Morishita > > Attempting to add a new datacenter to a cluster seems to cause repair > operations to break. I've been reproducing this with 20~ node clusters but > can get it to reliably occur on 2 node setups. > {code} > ##Basic Steps to reproduce > #Node 1 is started using GossipingPropertyFileSnitch as dc1 > #Cassandra-stress is used to insert a minimal amount of data > $CASSANDRA_STRESS -t 100 -R > org.apache.cassandra.locator.NetworkTopologyStrategy --num-keys=1000 > --columns=10 --consistency-level=LOCAL_QUORUM --average-size-values - > -compaction-strategy='LeveledCompactionStrategy' -O dc1:1 > --operation=COUNTER_ADD > #Alter "Keyspace1" > ALTER KEYSPACE "Keyspace1" WITH replication = {'class': > 'NetworkTopologyStrategy', 'dc1': 1 , 'dc2': 1 }; > #Add node 2 using GossipingPropertyFileSnitch as dc2 > run repair on node 1 > run repair on node 2 > {code} > The repair task on node 1 never completes and while there are no exceptions > in the logs of node1, netstat reports the following repair tasks > {code} > Mode: NORMAL > Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab > Repair 6c64ded0-36b4-11e3-bedc-1d1bb5c9abab > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed > Commands n/a 0 10239 > Responses n/a 0 3839 > {code} > Checking on node 2 we see the following exceptions > {code} > ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:42:58,961 StreamSession.java > (line 410) [Stream #4e71a250-36b4-11e3-bedc-1d1bb5c9abab] Streaming error > occurred > java.lang.NullPointerException > at > org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174) > at > org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293) > at java.lang.Thread.run(Thread.java:724) > ... > ERROR [STREAM-IN-/10.171.122.130] 2013-10-16 22:43:49,214 StreamSession.java > (line 410) [Stream #6c64ded0-36b4-11e3-bedc-1d1bb5c9abab] Streaming error > occurred > java.lang.NullPointerException > at > org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174) > at > org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293) > at java.lang.Thread.run(Thread.java:724) > {code} > Netstats on node 2 reports > {code} > automaton@ip-10-171-15-234:~$ nodetool netstats > Mode: NORMAL > Repair 4e71a250-36b4-11e3-bedc-1d1bb5c9abab > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed > Commands n/a 0 2562 > Responses n/a 0 4284 > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)