@Thibaut wrong email? Or how's "Avoid dropping messages off the client request path" (CASSANDRA-1676) related to the bootstrap questions I had?
On Wed, Jan 5, 2011 at 5:23 PM, Thibaut Britz <thibaut.br...@trendiction.com > wrote: > https://issues.apache.org/jira/browse/CASSANDRA-1676 > > you have to use at least 0.6.7 > > > > On Wed, Jan 5, 2011 at 4:19 PM, Edward Capriolo <edlinuxg...@gmail.com>wrote: > >> On Wed, Jan 5, 2011 at 10:05 AM, Ran Tavory <ran...@gmail.com> wrote: >> > In storage-conf I see this comment [1] from which I understand that the >> > recommended way to bootstrap a new node is to set AutoBootstrap=true and >> > remove itself from the seeds list. >> > Moreover, I did try to set AutoBootstrap=true and have the node in its >> own >> > seeds list, but it would not bootstrap. I don't recall the exact message >> but >> > it was something like "I found myself in the seeds list therefore I'm >> not >> > going to bootstrap even though AutoBootstrap is true". >> > >> > [1] >> > <!-- >> > ~ Turn on to make new [non-seed] nodes automatically migrate the >> right >> > data >> > ~ to themselves. (If no InitialToken is specified, they will pick >> one >> > ~ such that they will get half the range of the most-loaded node.) >> > ~ If a node starts up without bootstrapping, it will mark itself >> > bootstrapped >> > ~ so that you can't subsequently accidently bootstrap a node with >> > ~ data on it. (You can reset this by wiping your data and commitlog >> > ~ directories.) >> > ~ >> > ~ Off by default so that new clusters and upgraders from 0.4 don't >> > ~ bootstrap immediately. You should turn this on when you start >> adding >> > ~ new nodes to a cluster that already has data on it. (If you are >> > upgrading >> > ~ from 0.4, start your cluster with it off once before changing it to >> > true. >> > ~ Otherwise, no data will be lost but you will incur a lot of >> unnecessary >> > ~ I/O before your cluster starts up.) >> > --> >> > <AutoBootstrap>false</AutoBootstrap> >> > On Wed, Jan 5, 2011 at 4:58 PM, David Boxenhorn <da...@lookin2.com> >> wrote: >> >> >> >> If "seed list should be the same across the cluster" that means that >> nodes >> >> *should* have themselves as a seed. If that doesn't work for Ran, then >> that >> >> is the first problem, no? >> >> >> >> >> >> On Wed, Jan 5, 2011 at 3:56 PM, Jake Luciani <jak...@gmail.com> wrote: >> >>> >> >>> Well your ring issues don't make sense to me, seed list should be the >> >>> same across the cluster. >> >>> I'm just thinking of other things to try, non-boostrapped nodes should >> >>> join the ring instantly but reads will fail if you aren't using >> quorum. >> >>> >> >>> On Wed, Jan 5, 2011 at 8:51 AM, Ran Tavory <ran...@gmail.com> wrote: >> >>>> >> >>>> I haven't tried repair. Should I? >> >>>> >> >>>> On Jan 5, 2011 3:48 PM, "Jake Luciani" <jak...@gmail.com> wrote: >> >>>> > Have you tried not bootstrapping but setting the token and manually >> >>>> > calling >> >>>> > repair? >> >>>> > >> >>>> > On Wed, Jan 5, 2011 at 7:07 AM, Ran Tavory <ran...@gmail.com> >> wrote: >> >>>> > >> >>>> >> My conclusion is lame: I tried this on several hosts and saw the >> same >> >>>> >> behavior, the only way I was able to join new nodes was to first >> >>>> >> start them >> >>>> >> when they are *not in* their own seeds list and after they >> >>>> >> finish transferring the data, then restart them with themselves >> *in* >> >>>> >> their >> >>>> >> own seeds list. After doing that the node would join the ring. >> >>>> >> This is either my misunderstanding or a bug, but the only place I >> >>>> >> found it >> >>>> >> documented stated that the new node should not be in its own seeds >> >>>> >> list. >> >>>> >> Version 0.6.6. >> >>>> >> >> >>>> >> On Wed, Jan 5, 2011 at 10:35 AM, David Boxenhorn >> >>>> >> <da...@lookin2.com>wrote: >> >>>> >> >> >>>> >>> My nodes all have themselves in their list of seeds - always did >> - >> >>>> >>> and >> >>>> >>> everything works. (You may ask why I did this. I don't know, I >> must >> >>>> >>> have >> >>>> >>> copied it from an example somewhere.) >> >>>> >>> >> >>>> >>> On Wed, Jan 5, 2011 at 9:42 AM, Ran Tavory <ran...@gmail.com> >> wrote: >> >>>> >>> >> >>>> >>>> I was able to make the node join the ring but I'm confused. >> >>>> >>>> What I did is, first when adding the node, this node was not in >> the >> >>>> >>>> seeds >> >>>> >>>> list of itself. AFAIK this is how it's supposed to be. So it was >> >>>> >>>> able to >> >>>> >>>> transfer all data to itself from other nodes but then it stayed >> in >> >>>> >>>> the >> >>>> >>>> bootstrapping state. >> >>>> >>>> So what I did (and I don't know why it works), is add this node >> to >> >>>> >>>> the >> >>>> >>>> seeds list in its own storage-conf.xml file. Then restart the >> >>>> >>>> server and >> >>>> >>>> then I finally see it in the ring... >> >>>> >>>> If I had added the node to the seeds list of itself when first >> >>>> >>>> joining >> >>>> >>>> it, it would not join the ring but if I do it in two phases it >> did >> >>>> >>>> work. >> >>>> >>>> So it's either my misunderstanding or a bug... >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> On Wed, Jan 5, 2011 at 7:14 AM, Ran Tavory <ran...@gmail.com> >> >>>> >>>> wrote: >> >>>> >>>> >> >>>> >>>>> The new node does not see itself as part of the ring, it sees >> all >> >>>> >>>>> others >> >>>> >>>>> but itself, so from that perspective the view is consistent. >> >>>> >>>>> The only problem is that the node never finishes to bootstrap. >> It >> >>>> >>>>> stays >> >>>> >>>>> in this state for hours (It's been 20 hours now...) >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> $ bin/nodetool -p 9004 -h localhost streams >> >>>> >>>>>> Mode: Bootstrapping >> >>>> >>>>>> Not sending any streams. >> >>>> >>>>>> Not receiving any streams. >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> On Wed, Jan 5, 2011 at 1:20 AM, Nate McCall <n...@riptano.com> >> >>>> >>>>> wrote: >> >>>> >>>>> >> >>>> >>>>>> Does the new node have itself in the list of seeds per chance? >> >>>> >>>>>> This >> >>>> >>>>>> could cause some issues if so. >> >>>> >>>>>> >> >>>> >>>>>> On Tue, Jan 4, 2011 at 4:10 PM, Ran Tavory <ran...@gmail.com> >> >>>> >>>>>> wrote: >> >>>> >>>>>> > I'm still at lost. I haven't been able to resolve this. I >> tried >> >>>> >>>>>> > adding another node at a different location on the ring but >> >>>> >>>>>> > this node >> >>>> >>>>>> > too remains stuck in the bootstrapping state for many hours >> >>>> >>>>>> > without >> >>>> >>>>>> > any of the other nodes being busy with anti compaction or >> >>>> >>>>>> > anything >> >>>> >>>>>> > else. I don't know what's keeping it from finishing the >> >>>> >>>>>> > bootstrap,no >> >>>> >>>>>> > CPU, no io, files were already streamed so what is it >> waiting >> >>>> >>>>>> > for? >> >>>> >>>>>> > I read the release notes of 0.6.7 and 0.6.8 and there didn't >> >>>> >>>>>> > seem to >> >>>> >>>>>> > be anything addressing a similar issue so I figured there >> was >> >>>> >>>>>> > no >> >>>> >>>>>> point >> >>>> >>>>>> > in upgrading. But let me know if you think there is. >> >>>> >>>>>> > Or any other advice... >> >>>> >>>>>> > >> >>>> >>>>>> > On Tuesday, January 4, 2011, Ran Tavory <ran...@gmail.com> >> >>>> >>>>>> > wrote: >> >>>> >>>>>> >> Thanks Jake, but unfortunately the streams directory is >> empty >> >>>> >>>>>> >> so I >> >>>> >>>>>> don't think that any of the nodes is anti-compacting data >> right >> >>>> >>>>>> now or had >> >>>> >>>>>> been in the past 5 hours. It seems that all the data was >> already >> >>>> >>>>>> transferred >> >>>> >>>>>> to the joining host but the joining node, after having >> received >> >>>> >>>>>> the data >> >>>> >>>>>> would still remain in bootstrapping mode and not join the >> >>>> >>>>>> cluster. I'm not >> >>>> >>>>>> sure that *all* data was transferred (perhaps other nodes need >> to >> >>>> >>>>>> transfer >> >>>> >>>>>> more data) but nothing is actually happening so I assume all >> has >> >>>> >>>>>> been moved. >> >>>> >>>>>> >> Perhaps it's a configuration error from my part. Should I >> use >> >>>> >>>>>> >> I use >> >>>> >>>>>> AutoBootstrap=true ? Anything else I should look out for in >> the >> >>>> >>>>>> configuration file or something else? >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> On Tue, Jan 4, 2011 at 4:08 PM, Jake Luciani >> >>>> >>>>>> >> <jak...@gmail.com> >> >>>> >>>>>> wrote: >> >>>> >>>>>> >> >> >>>> >>>>>> >> In 0.6, locate the node doing anti-compaction and look in >> the >> >>>> >>>>>> "streams" subdirectory in the keyspace data dir to monitor the >> >>>> >>>>>> anti-compaction progress (it puts new SSTables for >> bootstrapping >> >>>> >>>>>> node in >> >>>> >>>>>> there) >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> On Tue, Jan 4, 2011 at 8:01 AM, Ran Tavory < >> ran...@gmail.com> >> >>>> >>>>>> wrote: >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> Running nodetool decommission didn't help. Actually the >> node >> >>>> >>>>>> >> refused >> >>>> >>>>>> to decommission itself (b/c it wasn't part of the ring). So I >> >>>> >>>>>> simply stopped >> >>>> >>>>>> the process, deleted all the data directories and started it >> >>>> >>>>>> again. It >> >>>> >>>>>> worked in the sense of the node bootstrapped again but as >> before, >> >>>> >>>>>> after it >> >>>> >>>>>> had finished moving the data nothing happened for a long time >> >>>> >>>>>> (I'm still >> >>>> >>>>>> waiting, but nothing seems to be happening). >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> Any hints how to analyze a "stuck" bootstrapping >> node??thanks >> >>>> >>>>>> >> On Tue, Jan 4, 2011 at 1:51 PM, Ran Tavory < >> ran...@gmail.com> >> >>>> >>>>>> wrote: >> >>>> >>>>>> >> Thanks Shimi, so indeed anticompaction was run on one of >> the >> >>>> >>>>>> >> other >> >>>> >>>>>> nodes from the same DC but to my understanding it has already >> >>>> >>>>>> ended. A few >> >>>> >>>>>> hour ago... >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> I plenty of log messages such as [1] which ended a couple >> of >> >>>> >>>>>> >> hours >> >>>> >>>>>> ago, and I've seen the new node streaming and accepting the >> data >> >>>> >>>>>> from the >> >>>> >>>>>> node which performed the anticompaction and so far it was >> normal >> >>>> >>>>>> so it >> >>>> >>>>>> seemed that data is at its right place. But now the new node >> >>>> >>>>>> seems sort of >> >>>> >>>>>> stuck. None of the other nodes is anticompacting right now or >> had >> >>>> >>>>>> been >> >>>> >>>>>> anticompacting since then. >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> The new node's CPU is close to zero, it's iostats are >> almost >> >>>> >>>>>> >> zero so >> >>>> >>>>>> I can't find another bottleneck that would keep it hanging. >> >>>> >>>>>> >> On the IRC someone suggested I'd maybe retry to join this >> >>>> >>>>>> >> node, >> >>>> >>>>>> e.g. decommission and rejoin it again. I'll try it now... >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> [1] INFO [COMPACTION-POOL:1] 2011-01-04 04:04:09,721 >> >>>> >>>>>> CompactionManager.java (line 338) AntiCompacting >> >>>> >>>>>> >> >>>> >>>>>> >> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')] >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:18,683 >> >>>> >>>>>> CompactionManager.java (line 338) AntiCompacting >> >>>> >>>>>> >> >>>> >>>>>> >> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3874-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3873-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3876-Data.db')] >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:19,132 >> >>>> >>>>>> CompactionManager.java (line 338) AntiCompacting >> >>>> >>>>>> >> >>>> >>>>>> >> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-951-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-976-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-978-Data.db')] >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:26,486 >> >>>> >>>>>> CompactionManager.java (line 338) AntiCompacting >> >>>> >>>>>> >> >>>> >>>>>> >> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')] >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> On Tue, Jan 4, 2011 at 12:45 PM, shimi <shim...@gmail.com> >> >>>> >>>>>> >> wrote: >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> In my experience most of the time it takes for a node to >> join >> >>>> >>>>>> >> the >> >>>> >>>>>> cluster is the anticompaction on the other nodes. The >> streaming >> >>>> >>>>>> part is very >> >>>> >>>>>> fast. >> >>>> >>>>>> >> Check the other nodes logs to see if there is any node >> doing >> >>>> >>>>>> anticompaction.I don't remember how much data I had in the >> >>>> >>>>>> cluster when I >> >>>> >>>>>> needed to add/remove nodes. I do remember that it took a few >> >>>> >>>>>> hours. >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> The node will join the ring only when it will finish the >> >>>> >>>>>> >> bootstrap. >> >>>> >>>>>> >> -- >> >>>> >>>>>> >> /Ran >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> > >> >>>> >>>>>> > -- >> >>>> >>>>>> > /Ran >> >>>> >>>>>> > >> >>>> >>>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> -- >> >>>> >>>>> /Ran >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> -- >> >>>> >>>> /Ran >> >>>> >>>> >> >>>> >>>> >> >>>> >>> >> >>>> >> >> >>>> >> >> >>>> >> -- >> >>>> >> /Ran >> >>>> >> >> >>>> >> >> >>> >> >> >> > >> > >> > >> > -- >> > /Ran >> > >> >> If non-auto-bootstrap nodes to not join they check to make sure good >> old iptables is not on. >> >> Edward >> > > -- /Ran
