Re: Best Practice to check Drillbit status(Cluster mode)
I think logs may be the only way to figure it out, at the present. You could have a watch on your logs to be informed of such events. For notifications, I would say file an enhancement JIRA - if it gathers enough attention, perhaps someone would volunteer to work or comment on it. On Mon, Jul 16, 2018 at 2:08 AM Divya Gehlot wrote: > Hi , > Thanks Abhishek ! > I would like to have a notification of that orphan drillbit process when it > gets disconnected from other running drillbits for some reason , definitely > not because of the unclean shut down as those drill bits are running for > months . > I know I can check the logs and kill that orphaned , which what I did in my > case, but I would like to have notification for down drillbit. > > > Thanks, > Divya > > On Fri, 13 Jul 2018 at 04:15, Abhishek Girish wrote: > > > Hey Divya, > > > > It would depend on the situation, afaik. The sys.drillbits table > contains a > > list of all running drillibits. If one of the Drillbit has issues and > > cannot stay connected to the cluster, I would assume it would be > > unregistered and may not show up in the output of sys.drillbits. If it's > an > > intermittent issue and Drillbit process maintains it's heartbeat > > connection, it may show up in the output. > > > > If you take a look at the logs, you might be able to figure out what is > > causing the issue. There may be orphan Drillbit processes which may be > have > > left behind due to a previous unclean shutdown. Can you clean up all > > Drillbit processes (using 'ps -ef | grep -i drillbit' and then a kill -9) > > on nodes where you suspect issues and restart Drillbits? > > > > -Abhishek > > > > On Tue, Jul 10, 2018 at 7:16 PM Divya Gehlot > > wrote: > > > > > Hi , > > > select * from sys.drillbits; > > > What does above query shows if drillbits process hangs ? > > > > > > > > > Thanks > > > > > > On Tue, 10 Jul 2018 at 15:36, Khurram Faraaz wrote: > > > > > > > You can run the below query, and look for the *state *column in the > > > result > > > > of the query. Online drillbits will be marked as ONLINE. > > > > > > > > select * from sys.drillbits; > > > > > > > > - Khurram > > > > > > > > On Tue, Jul 10, 2018 at 12:24 AM, Divya Gehlot < > > divya.htco...@gmail.com> > > > > wrote: > > > > > > > > > Hi, > > > > > I would like to know the best practice to check the Drillbits > status > > in > > > > > cluster mode. > > > > > I have encountered the scenario when check Drillbits process > running > > > fine > > > > > and When check in Drll WebUI , some of the Drillbits are down. > > > > > When do RCA(root cause analysis) , got to know due to some reason > > > > drillbits > > > > > process hanged . > > > > > For now the alert system which I have implemented now is checking > the > > > > > > > > > > > > > > > > drill/bin/drillbit.sh status > > > > > > > > > > > > > > > Is there any other best way to catch the hung Drillbit process? > > > > > Appreciate the advise from Drill community users. > > > > > > > > > > Thanks, > > > > > Divya > > > > > > > > > > > > > > >
Re: Best Practice to check Drillbit status(Cluster mode)
Hi , Thanks Abhishek ! I would like to have a notification of that orphan drillbit process when it gets disconnected from other running drillbits for some reason , definitely not because of the unclean shut down as those drill bits are running for months . I know I can check the logs and kill that orphaned , which what I did in my case, but I would like to have notification for down drillbit. Thanks, Divya On Fri, 13 Jul 2018 at 04:15, Abhishek Girish wrote: > Hey Divya, > > It would depend on the situation, afaik. The sys.drillbits table contains a > list of all running drillibits. If one of the Drillbit has issues and > cannot stay connected to the cluster, I would assume it would be > unregistered and may not show up in the output of sys.drillbits. If it's an > intermittent issue and Drillbit process maintains it's heartbeat > connection, it may show up in the output. > > If you take a look at the logs, you might be able to figure out what is > causing the issue. There may be orphan Drillbit processes which may be have > left behind due to a previous unclean shutdown. Can you clean up all > Drillbit processes (using 'ps -ef | grep -i drillbit' and then a kill -9) > on nodes where you suspect issues and restart Drillbits? > > -Abhishek > > On Tue, Jul 10, 2018 at 7:16 PM Divya Gehlot > wrote: > > > Hi , > > select * from sys.drillbits; > > What does above query shows if drillbits process hangs ? > > > > > > Thanks > > > > On Tue, 10 Jul 2018 at 15:36, Khurram Faraaz wrote: > > > > > You can run the below query, and look for the *state *column in the > > result > > > of the query. Online drillbits will be marked as ONLINE. > > > > > > select * from sys.drillbits; > > > > > > - Khurram > > > > > > On Tue, Jul 10, 2018 at 12:24 AM, Divya Gehlot < > divya.htco...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > I would like to know the best practice to check the Drillbits status > in > > > > cluster mode. > > > > I have encountered the scenario when check Drillbits process running > > fine > > > > and When check in Drll WebUI , some of the Drillbits are down. > > > > When do RCA(root cause analysis) , got to know due to some reason > > > drillbits > > > > process hanged . > > > > For now the alert system which I have implemented now is checking the > > > > > > > > > > > > > drill/bin/drillbit.sh status > > > > > > > > > > > > Is there any other best way to catch the hung Drillbit process? > > > > Appreciate the advise from Drill community users. > > > > > > > > Thanks, > > > > Divya > > > > > > > > > >
Re: Best Practice to check Drillbit status(Cluster mode)
Hey Divya, It would depend on the situation, afaik. The sys.drillbits table contains a list of all running drillibits. If one of the Drillbit has issues and cannot stay connected to the cluster, I would assume it would be unregistered and may not show up in the output of sys.drillbits. If it's an intermittent issue and Drillbit process maintains it's heartbeat connection, it may show up in the output. If you take a look at the logs, you might be able to figure out what is causing the issue. There may be orphan Drillbit processes which may be have left behind due to a previous unclean shutdown. Can you clean up all Drillbit processes (using 'ps -ef | grep -i drillbit' and then a kill -9) on nodes where you suspect issues and restart Drillbits? -Abhishek On Tue, Jul 10, 2018 at 7:16 PM Divya Gehlot wrote: > Hi , > select * from sys.drillbits; > What does above query shows if drillbits process hangs ? > > > Thanks > > On Tue, 10 Jul 2018 at 15:36, Khurram Faraaz wrote: > > > You can run the below query, and look for the *state *column in the > result > > of the query. Online drillbits will be marked as ONLINE. > > > > select * from sys.drillbits; > > > > - Khurram > > > > On Tue, Jul 10, 2018 at 12:24 AM, Divya Gehlot > > wrote: > > > > > Hi, > > > I would like to know the best practice to check the Drillbits status in > > > cluster mode. > > > I have encountered the scenario when check Drillbits process running > fine > > > and When check in Drll WebUI , some of the Drillbits are down. > > > When do RCA(root cause analysis) , got to know due to some reason > > drillbits > > > process hanged . > > > For now the alert system which I have implemented now is checking the > > > > > > > > > > drill/bin/drillbit.sh status > > > > > > > > > Is there any other best way to catch the hung Drillbit process? > > > Appreciate the advise from Drill community users. > > > > > > Thanks, > > > Divya > > > > > >
Re: Best Practice to check Drillbit status(Cluster mode)
Hi , select * from sys.drillbits; What does above query shows if drillbits process hangs ? Thanks On Tue, 10 Jul 2018 at 15:36, Khurram Faraaz wrote: > You can run the below query, and look for the *state *column in the result > of the query. Online drillbits will be marked as ONLINE. > > select * from sys.drillbits; > > - Khurram > > On Tue, Jul 10, 2018 at 12:24 AM, Divya Gehlot > wrote: > > > Hi, > > I would like to know the best practice to check the Drillbits status in > > cluster mode. > > I have encountered the scenario when check Drillbits process running fine > > and When check in Drll WebUI , some of the Drillbits are down. > > When do RCA(root cause analysis) , got to know due to some reason > drillbits > > process hanged . > > For now the alert system which I have implemented now is checking the > > > > > > > drill/bin/drillbit.sh status > > > > > > Is there any other best way to catch the hung Drillbit process? > > Appreciate the advise from Drill community users. > > > > Thanks, > > Divya > > >
Re: Best Practice to check Drillbit status(Cluster mode)
You can run the below query, and look for the *state *column in the result of the query. Online drillbits will be marked as ONLINE. select * from sys.drillbits; - Khurram On Tue, Jul 10, 2018 at 12:24 AM, Divya Gehlot wrote: > Hi, > I would like to know the best practice to check the Drillbits status in > cluster mode. > I have encountered the scenario when check Drillbits process running fine > and When check in Drll WebUI , some of the Drillbits are down. > When do RCA(root cause analysis) , got to know due to some reason drillbits > process hanged . > For now the alert system which I have implemented now is checking the > > > > drill/bin/drillbit.sh status > > > Is there any other best way to catch the hung Drillbit process? > Appreciate the advise from Drill community users. > > Thanks, > Divya >
Best Practice to check Drillbit status(Cluster mode)
Hi, I would like to know the best practice to check the Drillbits status in cluster mode. I have encountered the scenario when check Drillbits process running fine and When check in Drll WebUI , some of the Drillbits are down. When do RCA(root cause analysis) , got to know due to some reason drillbits process hanged . For now the alert system which I have implemented now is checking the > drill/bin/drillbit.sh status Is there any other best way to catch the hung Drillbit process? Appreciate the advise from Drill community users. Thanks, Divya
