Re: Drill Hangout 2/7/2017

2017-02-07 Thread Gautam Parai 

Hi Jasbir,


Which version of Drill are you using? This issue was fixed in 1.8.0. Please see 
DRILL-3710<https://issues.apache.org/jira/browse/DRILL-3710>


As Jinfeng mentioned, the default value of `planner.in_subquery_threshold` is 
20. So you should see the Join plan for ~300 in clause size.

Gautam


From: jasbir.s...@accenture.com <jasbir.s...@accenture.com>
Sent: Tuesday, February 7, 2017 1:04:06 AM
To: u...@drill.apache.org
Cc: dev@drill.apache.org
Subject: RE: Drill Hangout 2/7/2017

Thanks for the reply.

Attached in a query in which we have a member SUBDMA which has around 300 
members in IN CLAUSE. If you run this type of query fetching data from parquet 
file, drill gives no result. And if we reduce IN CLAUSE to less than 19 it 
works.

Can this be solved by changing any parameter?


Regards,
Jasbir Singh

-Original Message-
From: Jinfeng Ni [mailto:j...@apache.org]
Sent: Tuesday, February 07, 2017 1:01 PM
To: user <u...@drill.apache.org>
Cc: dev <dev@drill.apache.org>; Sareen, Nitin A. 
<nitin.a.sar...@accenture.com>; Kothari, Maneesh 
<maneesh.koth...@accenture.com>; Kumar, H. P. <h.p.ku...@accenture.com>
Subject: Re: Drill Hangout 2/7/2017

Regarding item 2 & 3,  IIKC, there is no upper limit of IN clause. An IN-list 
predicate  could be converted to a JOIN between T1 and VALUES operator, when 
the number of values in IN-LIST is beyond certain threshold. An Aggregate 
operator is applied to remove possible duplicates values in the list. That's 
why you may see HashAgg in the query plan.

If the number of values in IN-list is under the threshold, IN-list is evaluated 
as a OR-ed predicates.

The default threshold is 20 [1]. But you can change it, by running following:

alter session set `planner.in_subquery_threshold` = some_number;


>From T1
WHERE  T1.expression in (value1, value2, ..., value_n)?

==>
   Join
  /\
   T1 Agg
  \
  Values (values1, values2, ..., values_n)


[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_drill_blob_master_exec_java-2Dexec_src_main_java_org_apache_drill_exec_planner_physical_PlannerSettings.java-23L99-2DL100=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=7scIIjM0jY9x3fjvY6a_yERLxMA2NwA8l0DnuyrL6yA=pJ2Zg_dw2Uqydx8WOtKp7fRJ2_93ggT4MlawkE_77d4=XfXo4Gh524K0A_hwKSgiQA2Kzt7L2C-x0RawQFmlpDE=

On Mon, Feb 6, 2017 at 9:16 PM,  <jasbir.s...@accenture.com> wrote:
> Hi,
>
> I am interested in joining this session.
>
> My area of interest would be -
>
> 1. Future roadmap of Apache Drill
> 2. How apache drill creates HashAgg when there are lot of IN members
> in a where clause 3. What is the upper limit of IN CLAUSE.
>
> Regards,
> Jasbir Singh
>
> -Original Message-
> From: Jinfeng Ni [mailto:j...@apache.org]
> Sent: Tuesday, February 07, 2017 1:19 AM
> To: dev <dev@drill.apache.org>; user <u...@drill.apache.org>
> Subject: Drill Hangout 2/7/2017
>
> Hi drillers,
>
> We are going to have Drill Hangout tomorrow (02/07/2017, 10 AM PT). If you 
> have any suggestions for hangout topics, you can add them to this thread. We 
> will also ask around at the beginning of the hangout for topics.
>
> Thank you,
>
> Jinfeng
>
> 
>
> This message is for the designated recipient only and may contain privileged, 
> proprietary, or otherwise confidential information. If you have received it 
> in error, please notify the sender immediately and delete the original. Any 
> other use of the e-mail by you is prohibited. Where allowed by local law, 
> electronic communications with Accenture and its affiliates, including e-mail 
> and instant messaging (including content), may be scanned by our systems for 
> the purposes of information security and assessment of internal compliance 
> with Accenture policy.
> __
> 
>
> www.accenture.com<http://www.accenture.com>

Re: Drill Hangout 2/7/2017

2017-02-07 Thread Jinfeng Ni 
Drill hangout will start shortly. Join us at:

https://hangouts.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

On Tue, Feb 7, 2017 at 8:58 AM, Gautam Parai <gpa...@mapr.com> wrote:
>
> Hi Jasbir,
>
>
> Which version of Drill are you using? This issue was fixed in 1.8.0. Please 
> see DRILL-3710<https://issues.apache.org/jira/browse/DRILL-3710>
>
>
> As Jinfeng mentioned, the default value of `planner.in_subquery_threshold` is 
> 20. So you should see the Join plan for ~300 in clause size.
>
> Gautam
>
>
> From: jasbir.s...@accenture.com <jasbir.s...@accenture.com>
> Sent: Tuesday, February 7, 2017 1:04:06 AM
> To: u...@drill.apache.org
> Cc: dev@drill.apache.org
> Subject: RE: Drill Hangout 2/7/2017
>
> Thanks for the reply.
>
> Attached in a query in which we have a member SUBDMA which has around 300 
> members in IN CLAUSE. If you run this type of query fetching data from 
> parquet file, drill gives no result. And if we reduce IN CLAUSE to less than 
> 19 it works.
>
> Can this be solved by changing any parameter?
>
>
> Regards,
> Jasbir Singh
>
> -Original Message-
> From: Jinfeng Ni [mailto:j...@apache.org]
> Sent: Tuesday, February 07, 2017 1:01 PM
> To: user <u...@drill.apache.org>
> Cc: dev <dev@drill.apache.org>; Sareen, Nitin A. 
> <nitin.a.sar...@accenture.com>; Kothari, Maneesh 
> <maneesh.koth...@accenture.com>; Kumar, H. P. <h.p.ku...@accenture.com>
> Subject: Re: Drill Hangout 2/7/2017
>
> Regarding item 2 & 3,  IIKC, there is no upper limit of IN clause. An IN-list 
> predicate  could be converted to a JOIN between T1 and VALUES operator, when 
> the number of values in IN-LIST is beyond certain threshold. An Aggregate 
> operator is applied to remove possible duplicates values in the list. That's 
> why you may see HashAgg in the query plan.
>
> If the number of values in IN-list is under the threshold, IN-list is 
> evaluated as a OR-ed predicates.
>
> The default threshold is 20 [1]. But you can change it, by running following:
>
> alter session set `planner.in_subquery_threshold` = some_number;
>
>
> From T1
> WHERE  T1.expression in (value1, value2, ..., value_n)?
>
> ==>
>Join
>   /\
>T1 Agg
>   \
>   Values (values1, values2, ..., values_n)
>
>
> [1] 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_drill_blob_master_exec_java-2Dexec_src_main_java_org_apache_drill_exec_planner_physical_PlannerSettings.java-23L99-2DL100=DwIFaQ=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU=7scIIjM0jY9x3fjvY6a_yERLxMA2NwA8l0DnuyrL6yA=pJ2Zg_dw2Uqydx8WOtKp7fRJ2_93ggT4MlawkE_77d4=XfXo4Gh524K0A_hwKSgiQA2Kzt7L2C-x0RawQFmlpDE=
>
> On Mon, Feb 6, 2017 at 9:16 PM,  <jasbir.s...@accenture.com> wrote:
>> Hi,
>>
>> I am interested in joining this session.
>>
>> My area of interest would be -
>>
>> 1. Future roadmap of Apache Drill
>> 2. How apache drill creates HashAgg when there are lot of IN members
>> in a where clause 3. What is the upper limit of IN CLAUSE.
>>
>> Regards,
>> Jasbir Singh
>>
>> -Original Message-
>> From: Jinfeng Ni [mailto:j...@apache.org]
>> Sent: Tuesday, February 07, 2017 1:19 AM
>> To: dev <dev@drill.apache.org>; user <u...@drill.apache.org>
>> Subject: Drill Hangout 2/7/2017
>>
>> Hi drillers,
>>
>> We are going to have Drill Hangout tomorrow (02/07/2017, 10 AM PT). If you 
>> have any suggestions for hangout topics, you can add them to this thread. We 
>> will also ask around at the beginning of the hangout for topics.
>>
>> Thank you,
>>
>> Jinfeng
>>
>> 
>>
>> This message is for the designated recipient only and may contain 
>> privileged, proprietary, or otherwise confidential information. If you have 
>> received it in error, please notify the sender immediately and delete the 
>> original. Any other use of the e-mail by you is prohibited. Where allowed by 
>> local law, electronic communications with Accenture and its affiliates, 
>> including e-mail and instant messaging (including content), may be scanned 
>> by our systems for the purposes of information security and assessment of 
>> internal compliance with Accenture policy.
>> __
>> 
>>
>> www.accenture.com<http://www.accenture.com>

Re: Drill Hangout 2/7/2017

2017-02-06 Thread Jinfeng Ni 
Regarding item 2 & 3,  IIKC, there is no upper limit of IN clause. An
IN-list predicate  could be converted to a JOIN between T1 and VALUES
operator, when the number of values in IN-LIST is beyond certain
threshold. An Aggregate operator is applied to remove possible
duplicates values in the list. That's why you may see HashAgg in the
query plan.

If the number of values in IN-list is under the threshold, IN-list is
evaluated as a OR-ed predicates.

The default threshold is 20 [1]. But you can change it, by running following:

alter session set `planner.in_subquery_threshold` = some_number;


>From T1
WHERE  T1.expression in (value1, value2, ..., value_n)?

==>
   Join
  /\
   T1 Agg
  \
  Values (values1, values2, ..., values_n)


[1] 
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java#L99-L100

On Mon, Feb 6, 2017 at 9:16 PM,   wrote:
> Hi,
>
> I am interested in joining this session.
>
> My area of interest would be -
>
> 1. Future roadmap of Apache Drill
> 2. How apache drill creates HashAgg when there are lot of IN members in a 
> where clause
> 3. What is the upper limit of IN CLAUSE.
>
> Regards,
> Jasbir Singh
>
> -Original Message-
> From: Jinfeng Ni [mailto:j...@apache.org]
> Sent: Tuesday, February 07, 2017 1:19 AM
> To: dev ; user 
> Subject: Drill Hangout 2/7/2017
>
> Hi drillers,
>
> We are going to have Drill Hangout tomorrow (02/07/2017, 10 AM PT). If you 
> have any suggestions for hangout topics, you can add them to this thread. We 
> will also ask around at the beginning of the hangout for topics.
>
> Thank you,
>
> Jinfeng
>
> 
>
> This message is for the designated recipient only and may contain privileged, 
> proprietary, or otherwise confidential information. If you have received it 
> in error, please notify the sender immediately and delete the original. Any 
> other use of the e-mail by you is prohibited. Where allowed by local law, 
> electronic communications with Accenture and its affiliates, including e-mail 
> and instant messaging (including content), may be scanned by our systems for 
> the purposes of information security and assessment of internal compliance 
> with Accenture policy.
> __
>
> www.accenture.com

RE: Drill Hangout 2/7/2017

2017-02-06 Thread jasbir.sing 
Hi,

I am interested in joining this session.

My area of interest would be -

1. Future roadmap of Apache Drill
2. How apache drill creates HashAgg when there are lot of IN members in a where 
clause
3. What is the upper limit of IN CLAUSE.

Regards,
Jasbir Singh

-Original Message-
From: Jinfeng Ni [mailto:j...@apache.org]
Sent: Tuesday, February 07, 2017 1:19 AM
To: dev ; user 
Subject: Drill Hangout 2/7/2017

Hi drillers,

We are going to have Drill Hangout tomorrow (02/07/2017, 10 AM PT). If you have 
any suggestions for hangout topics, you can add them to this thread. We will 
also ask around at the beginning of the hangout for topics.

Thank you,

Jinfeng



This message is for the designated recipient only and may contain privileged, 
proprietary, or otherwise confidential information. If you have received it in 
error, please notify the sender immediately and delete the original. Any other 
use of the e-mail by you is prohibited. Where allowed by local law, electronic 
communications with Accenture and its affiliates, including e-mail and instant 
messaging (including content), may be scanned by our systems for the purposes 
of information security and assessment of internal compliance with Accenture 
policy.
__

www.accenture.com