"we will always see unassigned tablets on the monitor page, when tserver goes down." Not necessarily. From your previous email, it sounded as if you were expecting unassigned tablets to become suspended. I was trying to make it clear that once a tablet becomes unassigned, it can't be suspended. Only tablets that are currently hosted on a tserver can be suspended.
On Thu, Dec 2, 2021 at 10:49 AM Ligade, Shailesh [USA] < ligade_shail...@bah.com> wrote: > Thanks Mike, > > > > So let me see if I understood this, > > > > Doesn’t matter what this suspend.duration setting is, we will always see > unassigned tablets on the monitor page, when tserver goes down. > > > > If the setting is high enough then master is basically assigning the same > old tablets to that tserver, when it is back online, and thus will not move > any tablets around. > > > > If the duration is default (0s) or short then, master will start > reassigning tablets to other tserver. And when the original tablet comes > back up, master will try to rebalance tablets (may not get the old tablets > back). > > > > And thus having that setting high enough will make things faster to > recover. > > > > Thanks > > > > -S > > > > *From:* Mike Miller <mmil...@apache.org> > *Sent:* Thursday, December 2, 2021 10:39 AM > *To:* user@accumulo.apache.org > *Subject:* Re: [External] Re: [EXTERNAL EMAIL] - Re: accumulo tserver > rolling restart > > > > Some things to keep in mind... The Master will wait the > table.suspend.duration before reassigning the SUSPENDED tablets to new > tservers. With the table.suspend.duration set > 0, a tablet will go from > HOSTED to SUSPENDED if it's tserver is shutdown. It will then stay > SUSPENDED until it's old tserver is available or table.suspend.duration has > passed. If table.suspend.duration has passed before it's tserver has > returned, it will then be UNASSIGNED. Once a tablet is UNASSIGNED it won't > enter the SUSPENDED state. > > > > On Thu, Dec 2, 2021 at 9:43 AM Ligade, Shailesh [USA] < > ligade_shail...@bah.com> wrote: > > Thanks Mike, > > > > If I set the value to 0s (default) or if set it to 5m, when I restart > tserver system (it is pretty quick in the order of second), I still get > unassigned tablets on monitor page. My understand is that with that setting > of 5m (or 200s etc), master will wait for that mush time before start > moving unassigned tablets. In my situation, unassigned tablet counts goes > back to zero after long time, and hence rolling restarts take lot longer > (hours in most cases – depends on how many tablets/tserver) > > > > This setting appears to be working on accumulo 2.0.1, but since that is > not my prod version I have not tested it completely. > > > > Thanks > > > > -S > > *From:* Mike Miller <mmil...@apache.org> > *Sent:* Thursday, December 2, 2021 9:38 AM > *To:* user@accumulo.apache.org > *Subject:* [External] Re: [EXTERNAL EMAIL] - Re: accumulo tserver rolling > restart > > > > When you say "since that setting (table.suspend.duration) is not working > for me in accumulo 1.10.0" do you mean that the feature is not helping to > solve your problem? Or that the feature is not working and there could be a > bug? > > > > On Thu, Dec 2, 2021 at 8:00 AM Ligade, Shailesh [USA] < > ligade_shail...@bah.com> wrote: > > Thanks for detail steps! Really appreciated. > > > > Just curious, since that setting (table.suspend.duration) is not working > for me in accumulo 1.10.0, can I just stop both the masters and then > restart tserver one at a time (or all at once)? Will that speed up the > restart without getting into this offline tablet situation and or data loss > type situation? I can stop the ingest, flush the tables and then bring down > the master… > > > > We can take short downtime and my understanding is that the master is the > one keeping track of tservers and offline tablets situation. So just > curious… > > > > Thanks again > > > > -S > > > > *From:* dev1 <d...@etcoleman.com> > *Sent:* Monday, November 29, 2021 2:56 PM > *To:* 'user@accumulo.apache.org' <user@accumulo.apache.org> > *Subject:* [External] RE: [EXTERNAL EMAIL] - Re: accumulo tserver rolling > restart > > > > I believe the property is table.suspend.duration (not > tablet.suspended.duration > as you have in this email) – but the shell should have thrown an error > saying the property cannot be set in zookeeper if you had it wrong. > > > > What do you mean by: > > > > *but when i issued restart tserver (one at a time without waiting for > first to come up)* > > > > I’m assuming the requirement is to keep the cluster up and serving users > without major disruption – not to rip through the restart as fast as > possible. With 6 – 8 nodes you should still be able to do this in under an > hour. If you had a much larger cluster then the concept is the same but > you would want to use some number of tservers that is a fraction of the > total available that would be cycled at any given point in time. > > > > In general the way that I would do a conservative, rolling restart: > > > > 1. [optional] pause ingest – or be prepared for recovering any failed > ingests if they occur. > 2. [optional] Flush tables that have continuous ingest using the wait > option – this should help minimize recovery. > 3. Set the table.suspend.duration > 4. For each tserver – one (or a small group for large cluster) at a > time > > > 1. Stop the tserver > 2. Pause long enough that ZooKeeper recognizes the lost connection > 3. Restart the tserver > 4. Pause to allow for any recovery > > > 1. Reset the table.suspend.duration back to 0s (the default) > > > > If you tail the master / manager debug log you should get a good idea of > what is going on – there should be messages showing the tserver leaving and > then rejoining and any other activity related to recovery. With a rolling > restart the idea is to keep the cluster up and serving tables – only one > (or a few) tservers go offline and for a short duration (general less than > a minute) and between each tserver restart, time is allowed for things to > stabilize. > > > > > > *From:* Shailesh Ligade <slig...@fbi.gov> > *Sent:* Monday, November 29, 2021 11:17 AM > *To:* user@accumulo.apache.org > *Subject:* Re: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart > > > > > > Uhmm updated the setting tablet.suspended.duration to 5m > > > > config -s tablet.suspended.duration=5m > > > > but when i issued restart tserver (one at a time without waiting for first > to come up), i still get all tablets unassigned 🙁 may be, I need to > bring masters down first? > > > > btw this is for accumulo 1.10.0 > > > > am I missing anything? > > > > -S > ------------------------------ > > *From:* Shailesh Ligade <slig...@fbi.gov> > *Sent:* Monday, November 29, 2021 10:35 AM > *To:* user@accumulo.apache.org <user@accumulo.apache.org> > *Subject:* Re: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart > > > > Thanks Michael, > > > > stop cluster using admin stop? The issue is that, since we are using > systemd with restart=always, it interferes with any of those stop > (stop-all, stop-here etc) commands/scripts. So either we have to modify > systemd settings or may be just shutdown vm type of operation (i think that > is little brutal) > > > > -S > ------------------------------ > > *From:* Michael Wall <mjw...@gmail.com> > *Sent:* Monday, November 29, 2021 9:54 AM > *To:* user@accumulo.apache.org <user@accumulo.apache.org> > *Subject:* [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart > > > > Is there a reason to not just stop the cluster, reset the heap and restart > the cluster? That is simpler. > > > > On Mon, Nov 29, 2021 at 9:37 AM dev1 <d...@etcoleman.com> wrote: > > Yes – and don’t forget to reset it back when you are done. > > > > *From:* Ligade, Shailesh [USA] <ligade_shail...@bah.com> > *Sent:* Monday, November 29, 2021 9:36 AM > *To:* user@accumulo.apache.org > *Subject:* RE: accumulo tserver rolling restart > > > > Thanks, > > > > I am assuming I can set that property using shell and it will take effect > immediately? > > > > Thanks > > > > -S > > > > *From:* dev1 <d...@etcoleman.com> > *Sent:* Monday, November 29, 2021 9:25 AM > *To:* 'user@accumulo.apache.org' <user@accumulo.apache.org> > *Subject:* [External] RE: accumulo tserver rolling restart > > > > See > https://accumulo.apache.org/1.10/accumulo_user_manual.html#_restarting_process_on_a_node > <https://urldefense.com/v3/__https:/usg02.safelinks.protection.office365.us/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2Faccumulo.apache.org*2F1.10*2Faccumulo_user_manual.html*_restarting_process_on_a_node__*3BIw!!May37g!evyseDphy3PM_d8-tSlk89Sw1fFlSXHtH7vhiQedtcADc_P7OLEHw2kVZjlQ4Q8G_Q*24&data=04*7C01*7CSLIGADE*40FBI.GOV*7C363899b757914815738508d9b34de39b*7C022914a9b95f4b7bbace551ce1a04071*7C0*7C0*7C637737969389540183*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000&sdata=p*2FOeqj*2BgzX5PV4H*2Bd3TluGSvACs2CERSRhwEnifXX1c*3D&reserved=0__;JSUlJSUlJSUlKiUlJSUlJSUlJSUlJSUlJSUlJQ!!May37g!e_nAdxcZ_YbW8DCkWUX6TA7ZQTyaCUgOoHwNBzElKw28V3WJEuUD93wefizCiH0Epg$> > – A note on rolling restarts. > > > > There is property that can be set (table.suspend.duration) that will delay > the reassignment while a tserver is restarting – there is a trade-off on > the data not being available so try to minimize the time the tserver is > off-line. > > > > *From:* Ligade, Shailesh [USA] <ligade_shail...@bah.com> > *Sent:* Monday, November 29, 2021 9:19 AM > *To:* user@accumulo.apache.org > *Subject:* accumulo tserver rolling restart > > > > Hello, > > > > I want to restart al the tservers, say I updated the tserver heap size. > Since we ar eusing system, I can issue restart command on a tserver. This > causes all sorts of tablet movements even though accumulo is down for may > be a second. If I wait for all unassigned tables to become 0, then to > restart next tserver, then to completely restart a small cluster (6-8 > nodes) take hours (roughly 4k+ tablets per tserver) > > > > What may be right way to perform such routine maintenance operation? Is > there a delay setting we can change so that it will not move tablets > around? What may be a safe delay value? > > > > -S > >
