Thanks Michael, stop cluster using admin stop? The issue is that, since we are using systemd with restart=always, it interferes with any of those stop (stop-all, stop-here etc) commands/scripts. So either we have to modify systemd settings or may be just shutdown vm type of operation (i think that is little brutal)
-S ________________________________ From: Michael Wall <[email protected]> Sent: Monday, November 29, 2021 9:54 AM To: [email protected] <[email protected]> Subject: [EXTERNAL EMAIL] - Re: accumulo tserver rolling restart Is there a reason to not just stop the cluster, reset the heap and restart the cluster? That is simpler. On Mon, Nov 29, 2021 at 9:37 AM dev1 <[email protected]<mailto:[email protected]>> wrote: Yes – and don’t forget to reset it back when you are done. From: Ligade, Shailesh [USA] <[email protected]<mailto:[email protected]>> Sent: Monday, November 29, 2021 9:36 AM To: [email protected]<mailto:[email protected]> Subject: RE: accumulo tserver rolling restart Thanks, I am assuming I can set that property using shell and it will take effect immediately? Thanks -S From: dev1 <[email protected]<mailto:[email protected]>> Sent: Monday, November 29, 2021 9:25 AM To: '[email protected]<mailto:[email protected]>' <[email protected]<mailto:[email protected]>> Subject: [External] RE: accumulo tserver rolling restart See https://accumulo.apache.org/1.10/accumulo_user_manual.html#_restarting_process_on_a_node<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Faccumulo.apache.org%2F1.10%2Faccumulo_user_manual.html*_restarting_process_on_a_node__%3BIw!!May37g!evyseDphy3PM_d8-tSlk89Sw1fFlSXHtH7vhiQedtcADc_P7OLEHw2kVZjlQ4Q8G_Q%24&data=04%7C01%7CSLIGADE%40FBI.GOV%7Cf2b58075b8d7483d123108d9b3483d35%7C022914a9b95f4b7bbace551ce1a04071%7C0%7C0%7C637737946823965404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=KIpAbI4rSXKj4S1wBwGndZvZ1UYdeJvlO1m%2Fu7Hhc3o%3D&reserved=0> – A note on rolling restarts. There is property that can be set (table.suspend.duration) that will delay the reassignment while a tserver is restarting – there is a trade-off on the data not being available so try to minimize the time the tserver is off-line. From: Ligade, Shailesh [USA] <[email protected]<mailto:[email protected]>> Sent: Monday, November 29, 2021 9:19 AM To: [email protected]<mailto:[email protected]> Subject: accumulo tserver rolling restart Hello, I want to restart al the tservers, say I updated the tserver heap size. Since we ar eusing system, I can issue restart command on a tserver. This causes all sorts of tablet movements even though accumulo is down for may be a second. If I wait for all unassigned tables to become 0, then to restart next tserver, then to completely restart a small cluster (6-8 nodes) take hours (roughly 4k+ tablets per tserver) What may be right way to perform such routine maintenance operation? Is there a delay setting we can change so that it will not move tablets around? What may be a safe delay value? -S
