Thanks, Jonathan. I'll try the workaround and see if that gets the streams flowing properly.
As I mentioned before, we did not run scrub yet. What is the consequence of letting the streams from the hinted handoffs complete if scrub hasn't been run on these nodes? I'm currently running scrub on one node to get a sense of the time frame. Thanks again. - Ethan On Thu, Sep 15, 2011 at 9:09 AM, Jonathan Ellis <jbel...@gmail.com> wrote: > That means we missed a place we needed to special-case for backwards > compatibility -- the workaround is, add an empty encryption_options section > to cassandra.yaml: > > encryption_options: > internode_encryption: none > keystore: conf/.keystore > keystore_password: cassandra > truststore: conf/.truststore > truststore_password: cassandra > > Created https://issues.apache.org/jira/browse/CASSANDRA-3212 to fix this. > > On Thu, Sep 15, 2011 at 7:13 AM, Ethan Rowe <et...@the-rowes.com> wrote: > > Here's a typical log slice (not terribly informative, I fear): > >> > >> INFO [AntiEntropyStage:2] 2011-09-15 05:41:36,106 > AntiEntropyService.java > >> (l > >> ine 884) Performing streaming repair of 1003 ranges with /10.34.90.8for > >> (299 > >> > >> > 90798416657667504332586989223299634,54296681768153272037430773234349600451] > >> INFO [AntiEntropyStage:2] 2011-09-15 05:41:36,427 StreamOut.java (line > >> 181) > >> Stream context metadata > >> [/mnt/cassandra/data/events_production/FitsByShip-g-1 > >> 0-Data.db sections=88 progress=0/11707163 - 0%, > >> /mnt/cassandra/data/events_pr > >> oduction/FitsByShip-g-11-Data.db sections=169 progress=0/6133240 - 0%, > >> /mnt/c > >> assandra/data/events_production/FitsByShip-g-6-Data.db sections=1 > >> progress=0/ > >> 6918814 - 0%, > >> /mnt/cassandra/data/events_production/FitsByShip-g-12-Data.db s > >> ections=260 progress=0/9091780 - 0%], 4 sstables. > >> INFO [AntiEntropyStage:2] 2011-09-15 05:41:36,428 StreamOutSession.java > >> (lin > >> e 174) Streaming to /10.34.90.8 > >> ERROR [Thread-56] 2011-09-15 05:41:38,515 AbstractCassandraDaemon.java > >> (line > >> 139) Fatal exception in thread Thread[Thread-56,5,main] > >> java.lang.NullPointerException > >> at > >> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpC > >> onnection.java:174) > >> at > >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConn > >> ection.java:114) > > > > Not sure if the exception is related to the outbound streaming above; > other > > nodes are actively trying to stream to this node, so perhaps it comes > from > > those and temporal adjacency to the outbound stream is just coincidental. > I > > have other snippets that look basically identical to the above, except if > I > > look at the logs to which this node is trying to stream, I see that it > has > > concurrently opened a stream in the other direction, which could be the > one > > that the exception pertains to. > > > > On Thu, Sep 15, 2011 at 7:41 AM, Sylvain Lebresne <sylv...@datastax.com> > > wrote: > >> > >> On Thu, Sep 15, 2011 at 1:16 PM, Ethan Rowe <et...@the-rowes.com> > wrote: > >> > Hi. > >> > > >> > We've been running a 7-node cluster with RF 3, QUORUM reads/writes in > >> > our > >> > production environment for a few months. It's been consistently > stable > >> > during this period, particularly once we got out maintenance strategy > >> > fully > >> > worked out (per node, one repair a week, one major compaction a week, > >> > the > >> > latter due to the nature of our data model and usage). While this > >> > cluster > >> > started, back in June or so, on the 0.7 series, it's been running > 0.8.3 > >> > for > >> > a while now with no issues. We upgraded to 0.8.5 two days ago, having > >> > tested the upgrade in our staging cluster (with an otherwise identical > >> > configuration) previously and verified that our application's various > >> > use > >> > cases appeared successful. > >> > > >> > One of our nodes suffered a disk failure yesterday. We attempted to > >> > replace > >> > the dead node by placing a new node at OldNode.initial_token - 1 with > >> > auto_bootstrap on. A few things went awry from there: > >> > > >> > 1. We never saw the new node in bootstrap mode; it became available > >> > pretty > >> > much immediately upon joining the ring, and never reported a "joining" > >> > state. I did verify that auto_bootstrap was on. > >> > > >> > 2. I mistakenly ran repair on the new node rather than removetoken on > >> > the > >> > old node, due to a delightful mental error. The repair got nowhere > >> > fast, as > >> > it attempts to repair against the down node which throws an exception. > >> > So I > >> > interrupted the repair, restarted the node to clear any pending > >> > validation > >> > compactions, and... > >> > > >> > 3. Ran removetoken for the old node. > >> > > >> > 4. We let this run for some time and saw eventually that all the nodes > >> > appeared to be done various compactions and were stuck at streaming. > >> > Many > >> > streams listed as open, none making any progress. > >> > > >> > 5. I observed an Rpc-related exception on the new node (where the > >> > removetoken was launched) and concluded that the streams were broken > so > >> > the > >> > process wouldn't ever finish. > >> > > >> > 6. Ran a "removetoken force" to get the dead node out of the mix. No > >> > problems. > >> > > >> > 7. Ran a repair on the new node. > >> > > >> > 8. Validations ran, streams opened up, and again things got stuck in > >> > streaming, hanging for over an hour with no progress. > >> > > >> > 9. Musing that lingering tasks from the removetoken could be a factor, > I > >> > performed a rolling restart and attempted a repair again. > >> > > >> > 10. Same problem. Did another rolling restart and attempted a fresh > >> > repair > >> > on the most important column family alone. > >> > > >> > 11. Same problem. Streams included CFs not specified, so I guess they > >> > must > >> > be for hinted handoff. > >> > > >> > In concluding that streaming is stuck, I've observed: > >> > - streams will be open to the new node from other nodes, but the new > >> > node > >> > doesn't list them > >> > - streams will be open to the other nodes from the new node, but the > >> > other > >> > nodes don't list them > >> > - the streams reported may make some initial progress, but then they > >> > hang at > >> > a particular point and do not move on for an hour or more. > >> > - The logs report repair-related activity, until NPEs on incoming TCP > >> > connections show up, which appear likely to be the culprit. > >> > >> Can you send the stack trace from those NPE. > >> > >> > > >> > I can provide more exact details when I'm done commuting. > >> > > >> > With streaming broken on this node, I'm unable to run repairs, which > is > >> > obviously problematic. The application didn't suffer any operational > >> > issues > >> > as a consequence of this, but I need to review the overnight results > to > >> > verify we're not suffering data loss (I doubt we are). > >> > > >> > At this point, I'm considering a couple options: > >> > 1. Remove the new node and let the adjacent node take over its range > >> > 2. Bring the new node down, add a new one in front of it, and properly > >> > removetoken the problematic one. > >> > 3. Bring the new node down, remove all its data except for the system > >> > keyspace, then bring it back up and repair it. > >> > 4. Revert to 0.8.3 and see if that helps. > >> > > >> > Recommendations? > >> > > >> > Thanks. > >> > - Ethan > >> > > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
