Hi Tom, "The idea of join_ring=false is that other nodes are not aware of the new node, and therefore never send requests to it. The new node can then be repaired" Nicely explained, but I still see the issue that this node would not receive writes during that time. So after the repair the node would still miss data. Again, what is needed is either some joining-state or write-survey that allows disabling reads, but still accepts writes.
"To set up a new DC, I was hoping that you could also rebuild (instead of a repair) a new node while join_ring=false, but that seems not to work." Correct. The node does not get any tokens with join_ring=false. And again, your node won't receive any writes while you are rebuilding. Therefore you will have outdated data at the point when you are done rebuilding. kind regards, Christian On Tue, Sep 8, 2015 at 10:00 AM, Tom van den Berge < tom.vandenbe...@gmail.com> wrote: > "one drawback: the node joins the cluster as soon as the bootstrapping >> begins." >> I am not sure I understand this correctly. It will get tokens, but not >> load data if you combine it with autobootstrap=false. >> > Joining the cluster means that all other nodes become aware of the new > node, and therefore it might receive reads. And yes, it will not have any > data, because auto_bootstrap=false. > > > >> How I see it: You should be able to start all the new nodes in the new DC >> with autobootstrap=false and survey-mode=true. Then you should have a new >> DC with nodes that have tokens but no data. Then you can start rebuild on >> all new nodes. During this process, the new nodes should get writes, but >> not serve reads. >> > Maybe you're right. > > >> >> "It turns out that join_ring=false in this scenario does not solve this >> problem" >> I also don't see how joing_ring would help here. (Actually I have no clue >> where you would ever need that option) >> > The idea of join_ring=false is that other nodes are not aware of the new > node, and therefore never send requests to it. The new node can then be > repaired (see https://issues.apache.org/jira/browse/CASSANDRA-6961). To > set up a new DC, I was hoping that you could also rebuild (instead of a > repair) a new node while join_ring=false, but that seems not to work. > >> >> >> "Currently I'm trying to auto_bootstrap my new DC. The good thing is that >> it doesn't accept reads from other DCs." >> The joining-state actually works perfectly. The joining state is a state >> where node take writes, but not serve ready. It would be really cool if you >> could boot a node into the joining state. Actually, write_survey should >> basically be the same. >> > It would be great if you could select the DC from where it's bootstrapped, > similar to nodetool rebuild. I'm currently bootstrapping a node in > San-Jose. It decides to stream all data from another DC in Amsterdam, while > we also have another DC in San-Jose, right next to it. Streaming data > across the Atlantic takes a lot more time :( > > > >> >> kind regards, >> Christian >> >> PS: I would love to see the results, if you perform any tests on the >> write-survey. Please share it here on the mailing list :-) >> >> >> >> On Mon, Sep 7, 2015 at 11:10 PM, Tom van den Berge < >> tom.vandenbe...@gmail.com> wrote: >> >>> Hi Christian, >>> >>> No, I never tried survey mode. I didn't know it until now, but form the >>> info I was able to find it looks like it is meant for a different purpose. >>> Maybe it can be used to bootstrap a new DC, though. >>> >>> On the other hand, the auto_bootstrap=false + rebuild scenario seems to >>> be designed to do exactly what I need, except that it has one drawback: the >>> node joins the cluster as soon as the bootstrapping begins. >>> >>> It turns out that join_ring=false in this scenario does not solve this >>> problem, since nodetool rebuild does not do anything if C* is started with >>> this option. >>> >>> A workaround could be to ensure that only LOCAL_* CL is used by all >>> clients, but even then I'm seeing failed queries, because they're >>> mysteriously routed to the new DC every now and then. >>> >>> Currently I'm trying to auto_bootstrap my new DC. The good thing is that >>> it doesn't accept reads from other DCs. The bad thing is that a) I can't >>> choose where it streams its data from, and b) the two nodes I've been >>> trying to bootstrap crashed when they were almost finished... >>> >>> >>> >>> On Mon, Sep 7, 2015 at 10:22 PM, horschi <hors...@gmail.com> wrote: >>> >>>> Hi Tom, >>>> >>>> this sounds very much like my thread: "auto_bootstrap=false broken?" >>>> >>>> Did you try booting the new node with survey-mode? I wanted to try >>>> this, but I am waiting for 2.0.17 to come out (survey mode is broken in >>>> earlier versions). Imho survey mode is what you (and me too) want: start a >>>> node, accepting writes, but not serving reads. I have not tested it yet, >>>> but I think it should work. >>>> >>>> Also the manual join mentioned in CASSANDRA-9667 sounds very >>>> interesting. >>>> >>>> kind regards, >>>> Christian >>>> >>>> On Mon, Sep 7, 2015 at 10:11 PM, Tom van den Berge <t...@drillster.com> >>>> wrote: >>>> >>>>> Running nodetool rebuild on a node that was started with >>>>> join_ring=false does not work, unfortunately. The nodetool command returns >>>>> immediately, after a message appears in the log that the streaming of data >>>>> has started. After that, nothing happens. >>>>> >>>>> Tom >>>>> >>>>> >>>>> On Fri, Sep 12, 2014 at 5:47 PM, Robert Coli <rc...@eventbrite.com> >>>>> wrote: >>>>> >>>>>> On Fri, Sep 12, 2014 at 6:57 AM, Tom van den Berge <t...@drillster.com >>>>>> > wrote: >>>>>> >>>>>>> Wouldn't it be far more efficient if a node that is rebuilding >>>>>>> itself is responsible for not accepting reads until the rebuild is >>>>>>> complete? E.g. by marking it as "Joining", similar to a node that is >>>>>>> being >>>>>>> bootstrapped? >>>>>>> >>>>>> >>>>>> Yes, and Cassandra 2.0.7 and above contain this long desired >>>>>> functionality. >>>>>> >>>>>> https://issues.apache.org/jira/browse/CASSANDRA-6961 >>>>>> >>>>>> I presume that one can also run a rebuild in this state, though I >>>>>> haven't tried. Driftx gives it an 80% chance... try it and see and let us >>>>>> know? :D >>>>>> >>>>>> =Rob >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
