Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Thanks.. will upgrade to 1.13.0 and let you know. On Tue, Mar 20, 2018 11:08 AM, Parth Chandra par...@apache.org wrote: Hi Anup, I don't have full context for the proposed hack, and it might have worked, but looks like Vlad has addressed the issue in the right place. Perhaps you

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi Anup, I don't have full context for the proposed hack, and it might have worked, but looks like Vlad has addressed the issue in the right place. Perhaps you can try out 1.13.0 and let us all know. Thanks Parth On Sat, Mar 17, 2018 at 11:43 AM, Anup Tiwari wrote:

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Thanks Parth for Info. I am really looking forward to it. But can you tell me if the second part(about hack) was right or not? Because i really want to test it as we got this issue several time in last 2-3 days post upgrading to 1.12.0. Also i have seen sometimes after lost connection , drillbit

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

On Fri, Mar 16, 2018 at 8:10 PM, Anup Tiwari wrote: > Hi All, > I was just going through this post and found very good suggestions. > But this issue is still there in Drill 1.12.0 and i can see > https://issues.apache.org/jira/browse/DRILL-4708 is now marked as >

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi All, I was just going through this post and found very good suggestions. But this issue is still there in Drill 1.12.0 and i can see https://issues.apache.org/jira/browse/DRILL-4708 is now marked as resolved in "1.13.0" so i am hoping that this will be fixed in drill 1.13.0. Few things i want

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Very interesting findings, Francois. Thanks for sharing them with the community. The change of max_per_node and affinity_factor seems to reduce the possibility of one drillbit was hitting overload issue because of either CPU or Network contention. In our in-house testing, we also noticed that

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Very interesting findings, Francois. Thanks for sharing them with the community. The change of max_per_node and affinity_factor seems to reduce the possibility of one drillbit was hitting overload issue because of either CPU or Network contention. In our in-house testing, we also noticed that

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi, We have been having client-foreman connection and ZkConnection issue few months ago. It went from annoying to a show stopper when we moved from a 12 nodes cluster to a 220 nodes cluster. Nodes specs - 8 cores total (2 x E5620) - 72 GB RAM Total - Other applications share the same hardware.

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi John, First of all sorry for delayed response and thanks for your suggestion, reducing value of "planner.width.max_per_node" helped me a lot, above issue which was coming 8 out of 10 times earlier now it is coming only 2 out of 10 times. As mentioned above occurrences of connection error came

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Have you tried disabling hash joins or hash agg on the query or changing the planning width? Here are some docs to check out: https://drill.apache.org/docs/configuring-resources-for-a-shared-drillbit/ https://drill.apache.org/docs/guidelines-for-optimizing-aggregation/

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi John, I have tried above config as well but still getting this issue. And please note that we were using similar configuration params for Drill 1.6 where this issue was not coming. Anything else which i can try? Regards, *Anup Tiwari* On Fri, Mar 3, 2017 at 11:01 PM, Abhishek Girish

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

+1 on John's suggestion. On Fri, Mar 3, 2017 at 6:24 AM, John Omernik wrote: > So your node has 32G of ram yet you are allowing Drill to use 36G. I would > change your settings to be 8GB of Heap, and 22GB of Direct Memory. See if > this helps with your issues. Also, are you

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

So your node has 32G of ram yet you are allowing Drill to use 36G. I would change your settings to be 8GB of Heap, and 22GB of Direct Memory. See if this helps with your issues. Also, are you using a distributed filesystem? If so you may want to allow even more free ram...i.e. 8GB of Heap and

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi, Please find our configuration details :- Number of Nodes : 4 RAM/Node : 32GB Core/Node : 8 DRILL_MAX_DIRECT_MEMORY="20G" DRILL_HEAP="16G" And all other variables are set to default. Since we have tried some of the settings suggested above but still facing this issue more frequently, kindly

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Another thing to consider is ensure you have a Spill Location setup, and then disable hashagg/hashjoin for the query... On Wed, Mar 1, 2017 at 1:25 PM, Abhishek Girish wrote: > Hey Anup, > > This is indeed an issue, and I can understand that having an unstable > environment

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hey Anup, This is indeed an issue, and I can understand that having an unstable environment is not something anyone wants. DRILL-4708 is still unresolved - hopefully someone will get to it soon. I've bumped up the priority. Unfortunately we do not publish any sizing guidelines, so you'd have to

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi, Can someone look into it? As we are now getting this more frequently in Adhoc queries as well. And for automation jobs, we are moving to Hive as in drill we are getting this more frequently. Regards, *Anup Tiwari* On Sat, Dec 31, 2016 at 12:11 PM, Anup Tiwari

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi, We are getting this issue bit more frequently. can someone please look into it and tell us that why it is happening since as mention in earlier mail when this query gets executed no other query is running at that time. Thanks in advance. Regards, *Anup Tiwari* On Sat, Dec 24, 2016 at 10:20

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi Sudheesh, Please find below ans :- 1. Total 4,(3 Datanodes, 1 namenode) 2. Only one query, as this query is part of daily dump and runs in early morning. And as @chun mentioned , it seems similar to DRILL-4708 , so any update on progress of this ticket? On 22-Dec-2016 12:13 AM, "Sudheesh

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Two more questions.. (1) How many nodes in your cluster? (2) How many queries are running when the failure is seen? If you have multiple large queries running at the same time, the load on the system could cause those failures (which are heartbeat related). The two options I suggested decrease

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

@sudheesh, yes drill bit is running on datanodeN/10.*.*.5:31010). Can you tell me how this will impact to query and do i have to set this at session level OR system level? Regards, *Anup Tiwari* On Tue, Dec 20, 2016 at 11:59 PM, Chun Chang wrote: > I am pretty sure this

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

I am pretty sure this is the same as DRILL-4708. On Tue, Dec 20, 2016 at 10:27 AM, Sudheesh Katkam wrote: > Is the drillbit service (running on datanodeN/10.*.*.5:31010) actually > down when the error is seen? > > If not, try lowering parallelism using these two session

Re: [Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Is the drillbit service (running on datanodeN/10.*.*.5:31010) actually down when the error is seen? If not, try lowering parallelism using these two session options, before running the queries: planner.width.max_per_node (decrease this) planner.slice_target (increase this) Thank you, Sudheesh

[Drill 1.9.0] : [CONNECTION ERROR] :- (user client) closed unexpectedly. Drillbit down?

Hi Team, We are running some drill automation script on a daily basis and we often see that some query gets failed frequently by giving below error , Also i came across DRILL-4708 which seems similar, Can anyone give me update on that OR