[jira] [Resolved] (DRILL-4704) select statement behavior is inconsistent for decimal values in parquet

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni resolved DRILL-4704.
---
   Resolution: Fixed
Fix Version/s: (was: Future)
   1.8.0

Fixed in commit: aaf220ffdddaa329e38237d69b438f4e09d16a87

> select statement behavior is inconsistent for decimal values in parquet
> ---
>
> Key: DRILL-4704
> URL: https://issues.apache.org/jira/browse/DRILL-4704
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.6.0
> Environment: Windows 7 Pro, Java 1.8.0_91
>Reporter: Dave Oshinsky
> Fix For: 1.8.0
>
>
> A select statement that searches a parquet file for a decimal value matching 
> a specific value behaves inconsistently.  The query expressed most simply 
> finds nothing:
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> employee_id = 100;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  | 
> HIRE_DATE |
> +--+-+++---+---+
> +--+-+++---+---+
> No rows selected (0.348 seconds)
> The query can be modified to find the matching row in a few ways, such as the 
> following (using between instead of '=', changing 100 to 100.0, or casting as 
> decimal:
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> employee_id between 100 and 100;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  |   
> HIR |
> +--+-+++---+---+
> | 100  | Steven  | King   | SKING  | 515.123.4567  | 
> 2003-06-1 |
> +--+-+++---+---+
> 1 row selected (0.226 seconds)
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> employee_id = 100.0;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  |   
> HIR |
> +--+-+++---+---+
> | 100  | Steven  | King   | SKING  | 515.123.4567  | 
> 2003-06-1 |
> +--+-+++---+---+
> 1 row selected (0.259 seconds)
> 0: jdbc:drill:zk=local> select * from dfs.`c:/archiveHR/HR.EMPLOYEES` where 
> cast(employee_id AS DECIMAL) = 100;
> +--+-+++---+---+
> | EMPLOYEE_ID  | FIRST_NAME  | LAST_NAME  | EMAIL  | PHONE_NUMBER  |   
> HIR |
> +--+-+++---+---+
> | 100  | Steven  | King   | SKING  | 515.123.4567  | 
> 2003-06-1 |
> +--+-+++---+---+
> 1 row selected (0.232 seconds)
> 0: jdbc:drill:zk=local>
> The schema of the parquet data that is being searched is as follows:
> $ java -jar parquet-tools*1.jar meta c:/archiveHR/HR.EMPLOYEES/1.parquet
> file:   file:/c:/archiveHR/HR.EMPLOYEES/1.parquet
> creator:parquet-mr version 1.8.1 (build 
> 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)
> .
> file schema:HR.EMPLOYEES
> 
> EMPLOYEE_ID:REQUIRED FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:0
> FIRST_NAME: OPTIONAL BINARY O:UTF8 R:0 D:1
> LAST_NAME:  REQUIRED BINARY O:UTF8 R:0 D:0
> EMAIL:  REQUIRED BINARY O:UTF8 R:0 D:0
> PHONE_NUMBER:   OPTIONAL BINARY O:UTF8 R:0 D:1
> HIRE_DATE:  REQUIRED BINARY O:UTF8 R:0 D:0
> JOB_ID: REQUIRED BINARY O:UTF8 R:0 D:0
> SALARY: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> COMMISSION_PCT: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> MANAGER_ID: OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> DEPARTMENT_ID:  OPTIONAL FIXED_LEN_BYTE_ARRAY O:DECIMAL R:0 D:1
> row group 1:RC:107 TS:9943 OFFSET:4
> 
> EMPLOYEE_ID: FIXED_LEN_BYTE_ARRAY SNAPPY DO:0 FPO:4 SZ:360/355/0.99 
> VC:107 ENC:PLAIN,BIT_PACKED
> FIRST_NAME:  BINARY SNAPPY DO:0 FPO:364 SZ:902/1058/1.17 VC:107 
> ENC:PLAIN_DICTIONARY,RLE,BIT_PACKED
> LAST_NAME:   BINARY SNAPPY DO:0 FPO:1266 SZ:913//1.22 VC:107 
> ENC:PLAIN,BIT_PACKED
> EMAIL:   BINARY SNAPPY DO:0 FPO:2179 SZ:977/1184/1.21 VC:107 
> ENC:PLAIN,BIT_PACKED
> PHONE_NUMBER:BINARY SNAPPY 

[GitHub] drill pull request #517: DRILL-4704 fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/517


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for a 1.8 release

[Jinfeng Ni]

Here is the latest status of 1.8.0 release.

(venki) DRILL-4732, DRILL-4729, DRILL-4728 : merged
(sudheesh) DRILL-4822, DRILL-4819, DRILL-4623 : merged
DRILL-4792 : not merged.

DRILL-4836 was opened just now, which seems to be a regression from
several release back. I think it would make sense to try to get
DRILL-4836 fixed in this release.

I'll also try to merge Dave's DRILL-4704 tonight.


On Fri, Aug 5, 2016 at 1:35 PM, Venki Korukanti
 wrote:
> I will try to get these sometime today or eod tomorrow.
>
> 1) DRILL-4728: Add support for new metadata fetch APIs,
> https://github.com/apache/drill/pull/527 (Ready to commit after few minor
> changes)
>
> 2) DRILL-4729: Add support for prepared statement API on server side,
> https://github.com/apache/drill/pull/530 (need to address a review comment)
>
> 3) DRILL-4732: Update JDBC driver to use new prepared statement API in
> DrillClient, https://github.com/apache/drill/pull/532 (need to address a
> comment).
>
> Thanks
> Venki
>
> On Fri, Aug 5, 2016 at 1:31 PM, Sudheesh Katkam 
> wrote:
>
>> Yes, that’s a typo. DRILL-3623 was for LIMIT 0 changes. That number will
>> forever remain in my subconscious ;)
>>
>> Thank you,
>> Sudheesh
>>
>> > On Aug 5, 2016, at 1:27 PM, Zelaine Fong  wrote:
>> >
>> > I assume #3 has a typo.  It should be DRILL-4623.
>> >
>> > -- Zelaine
>> >
>> > On Fri, Aug 5, 2016 at 1:15 PM, Sudheesh Katkam 
>> wrote:
>> >
>> >> Here’s my list that I will try to get in before Monday.
>> >>
>> >> 1) DRILL-4822: Pending commit.
>> >> PR: https://github.com/apache/drill/pull/558
>> >>
>> >> 2) DRILL-4819: Pending review from Aditya.
>> >> PR: https://github.com/apache/drill/pull/556
>> >>
>> >> 3) DRILL-3623: Pending tests (or test results).
>> >> PR: https://github.com/apache/drill/pull/486
>> >>
>> >> 4) DRILL-4792: Pending changes from Arina.
>> >> PR: https://github.com/apache/drill/pull/551
>> >>
>> >> The first three will make it; the last one may not.
>> >>
>> >> Thank you,
>> >> Sudheesh
>> >>
>> >>
>> >> On Wed, Aug 3, 2016 at 11:49 AM, Jinfeng Ni 
>> wrote:
>> >>
>> >>> I would like to propose we set Monday 8/8 as the tentative cut-off
>> >>> date for 1.8 release.
>> >>>
>> >>> If people have any working-in-progress patches, and would like to
>> >>> include in 1.8 release, please submit PR, or ping people to review
>> >>> your PR if there has been a PR under review.
>> >>>
>> >>> Thanks,
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Aug 2, 2016 at 3:00 PM, Jinfeng Ni 
>> >> wrote:
>>  Hello everyone,
>> 
>>   I would like to start the train for 1.8 release.
>> 
>>   If you have been working on some open issues, and want to include
>>  them in 1.8, please rely this email and let us know the JIRAs number
>>  and corresponding pull requests.
>> 
>>   Based on the response, we may go through the list of opens JIRAs for
>>  1.8, and come up with a tentatively cut-off date for 1.8 release.
>> 
>>   Thanks,
>> 
>> 
>> 
>>  On Tue, Aug 2, 2016 at 2:14 PM, Aditya  wrote:
>> > +1 for Jinfeng as the RM for 1.8 release.
>> >
>> > On Tue, Aug 2, 2016 at 11:59 AM, Jinfeng Ni 
>> >>> wrote:
>> >
>> >> I'll volunteer to be the release manager for 1.8, if no one else
>> >> would
>> >> like to do so.
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Aug 1, 2016 at 9:40 PM, Parth Chandra 
>> >>> wrote:
>> >>> Hi Everyone,
>> >>>
>> >>>   I think its time to roll out 1.8.  Would any PMC
>> >> member/committer
>> >> like
>> >>> to volunteer to be the release manager?
>> >>>
>> >>> Parth
>> >>
>> >>>
>> >>
>>
>>

[GitHub] drill issue #517: DRILL-4704 fix

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/517
  
I'll make minor change to the commit based on Aman's comment, and re-run 
regression and merge if everything is fine. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #563: [DRILL-4795] Nested aggregate windowed query fails-...

[gparai]

GitHub user gparai opened a pull request:

https://github.com/apache/drill/pull/563

[DRILL-4795] Nested aggregate windowed query fails-IllegalStateException



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gparai/drill Drill-4795-ADM

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/563.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #563


commit a04c9397460f978a00915f0591255a474e9e1e19
Author: Gautam Parai 
Date:   2016-08-06T01:06:34Z

[DRILL-4795] Nested aggregate windowed query fails - IllegalStateException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-4836:
--

 Summary: ZK Issue during Drillbit startup, possibly due to race 
condition
 Key: DRILL-4836
 URL: https://issues.apache.org/jira/browse/DRILL-4836
 Project: Apache Drill
  Issue Type: Bug
  Components:  Server
Reporter: Abhishek Girish
Assignee: Paul Rogers
 Fix For: 1.8.0


During a parallel launch of Drillbits on a 4 node cluster, I hit this issue 
during startup:

{code}
Exception in thread "main" 
org.apache.drill.exec.exception.DrillbitStartupException: Failure during 
initial startup of Drillbit.
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257)
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable to 
put
at 
org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196)
at 
org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94)
...
at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281)
... 2 more
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs
at 
org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191)
... 7 more
{code}

And similarly,
{code}
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #517: DRILL-4704 fix

[daveoshinsky]

Github user daveoshinsky commented on a diff in the pull request:

https://github.com/apache/drill/pull/517#discussion_r73981580
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
 ---
@@ -267,6 +267,17 @@ public LogicalExpression 
visitBooleanOperator(BooleanOperator op, FunctionLookup
   return new BooleanOperator(op.getName(), args, op.getPosition());
 }
 
+static public int computePrecision(LogicalExpression currentArg) {
+int precision = currentArg.getMajorType().getPrecision();
--- End diff --

Well, if the type has no precision, what do you suggest?  Move the fix 
back to CastIntDecimal.java (calculate the precision based on the value), as I 
originally had it, but Jinfeng insisted was incorrect?
You two fellows, Jinfeng and Aman, can decide among yourselves.  Let me 
know when you're finished duking it out.  If there are any survivors, we can 
discuss on the hangout tomorrow. 

On Monday, August 8, 2016 8:04 PM, Aman Sinha 
 wrote:
 

 In 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java:>
 @@ -267,6 +267,17 @@ public LogicalExpression 
visitBooleanOperator(BooleanOperator op, FunctionLookup
>return new BooleanOperator(op.getName(), args, op.getPosition());
>  }
>  
> +static public int computePrecision(LogicalExpression currentArg) {
> +int precision = currentArg.getMajorType().getPrecision();
Did you consider checking whether the type has a precision or not using 
getMajorType().hasPrecision() ? That way, you would only call getPrecision() if 
it returned True and otherwise set the precision for INT, BIGINT. —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.  

  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[amansinha100]

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/517
  
couple of minor comments; otherwise LGTM.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #517: DRILL-4704 fix

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/517#discussion_r73978291
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
 ---
@@ -267,6 +267,17 @@ public LogicalExpression 
visitBooleanOperator(BooleanOperator op, FunctionLookup
   return new BooleanOperator(op.getName(), args, op.getPosition());
 }
 
+static public int computePrecision(LogicalExpression currentArg) {
+int precision = currentArg.getMajorType().getPrecision();
--- End diff --

Did you consider checking whether the type has a precision or not using 
getMajorType().hasPrecision()  ?  That way, you would only call getPrecision() 
if it returned True and otherwise set the precision for INT, BIGINT. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #517: DRILL-4704 fix

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/517#discussion_r73977447
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java
 ---
@@ -267,6 +267,17 @@ public LogicalExpression 
visitBooleanOperator(BooleanOperator op, FunctionLookup
   return new BooleanOperator(op.getName(), args, op.getPosition());
 }
 
+static public int computePrecision(LogicalExpression currentArg) {
+int precision = currentArg.getMajorType().getPrecision();
+if (currentArg.getMajorType().getMinorType() == MinorType.INT) {
+precision = 10;
--- End diff --

These values should preferably be defined as static final constants. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #562: DRILL-4833: Handle union-all physical operator during the ...

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/562
  
+1 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #562: DRILL-4833: Handle union-all physical operator duri...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/562#discussion_r73969588
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
 ---
@@ -72,6 +73,47 @@ public Prel visitScan(ScanPrel prel, MajorFragmentStat 
s) throws RuntimeExceptio
 return prel;
   }
 
+  /**
+   * A union-all should be treated differently compared to a join operator 
because joins impose
+   * a co-location requirement and therefore insert an exchange on both 
sides of the
+   * join (e.g HashToRandomExchange or BroadcastExchange), thus the major 
fragment of the join itself
+   * is different from the major fragment of its children.  Union-All does 
not impose the co-location
+   * requirement on its children, hence the major fragment of the 
union-all may be the same as that of
+   * its children. Thus, we should take an 'aggregate' view of all its 
children to decide the parallelism.
+   */
+  @Override
+  public Prel visitUnionAll(UnionAllPrel prel, MajorFragmentStat s) throws 
RuntimeException {
+List children = Lists.newArrayList();
+s.add(prel);
+
+List statList = Lists.newArrayList();
+for (Prel p : prel) {
+  // for each input of union-all, create a temporary MajorFragmentStat 
instance
+  MajorFragmentStat childStat = new MajorFragmentStat(s /* use 
existing stat to initialize */);
+  statList.add(childStat);
+  childStat.add(p);
+}
+
+int i = 0;
+for(Prel p : prel) {
+  children.add(p.accept(this, statList.get(i++)));
+}
+
+MajorFragmentStat maxStat = statList.get(0);
+// get the max width of all child stats
+for (int j=1; j < statList.size(); j++) {
+  if (statList.get(j).getMaxWidth() > maxStat.getMaxWidth()) {
+maxStat = statList.get(j);
+  }
+}
+
+// width of the major fragment that contains union-all should be the 
maximum
+// width of all its inputs
+s.setMaxWidth(maxStat.getMaxWidth());
--- End diff --

After discussion with @amansinha100 , I realized that maxRows has been 
taken care of for Union-All operator, since for union-all, maxRows will be sum 
of both children's maxRows.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/517
  
Since we will have a discussion for long term plan of decimal type support 
(tomorrow hangout), I think it makes sense to apply a small-scoped change to 
fix this incorrect query result issue. For the overflow problem or decimal type 
design in general, we could  continue our discussion either in hangout or dev 
list.

Based on that, I'm inclined to merge this patch. @amansinha100 , could you 
please take a look at the new change, and see if it addressed your prior 
concern?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4732) Update JDBC driver to use the new prepared statement APIs on DrillClient

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4732.

Resolution: Fixed

Fixed in 
[e02aa59|https://github.com/apache/drill/commit/e02aa596fa64f38bce773fd108f15032f8086601]

> Update JDBC driver to use the new prepared statement APIs on DrillClient
> 
>
> Key: DRILL-4732
> URL: https://issues.apache.org/jira/browse/DRILL-4732
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Metadata
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
> Fix For: 1.8.0
>
>
> DRILL-4729 is adding new prepared statement implementation on server side and 
> it provides APIs on DrillClient to create new prepared statement which 
> returns metadata along with a opaque handle and submit prepared statement for 
> execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #562: DRILL-4833: Handle union-all physical operator duri...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/562#discussion_r73959459
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
 ---
@@ -72,6 +73,47 @@ public Prel visitScan(ScanPrel prel, MajorFragmentStat 
s) throws RuntimeExceptio
 return prel;
   }
 
+  /**
+   * A union-all should be treated differently compared to a join operator 
because joins impose
+   * a co-location requirement and therefore insert an exchange on both 
sides of the
+   * join (e.g HashToRandomExchange or BroadcastExchange), thus the major 
fragment of the join itself
+   * is different from the major fragment of its children.  Union-All does 
not impose the co-location
+   * requirement on its children, hence the major fragment of the 
union-all may be the same as that of
+   * its children. Thus, we should take an 'aggregate' view of all its 
children to decide the parallelism.
+   */
+  @Override
+  public Prel visitUnionAll(UnionAllPrel prel, MajorFragmentStat s) throws 
RuntimeException {
+List children = Lists.newArrayList();
+s.add(prel);
+
+List statList = Lists.newArrayList();
+for (Prel p : prel) {
+  // for each input of union-all, create a temporary MajorFragmentStat 
instance
+  MajorFragmentStat childStat = new MajorFragmentStat(s /* use 
existing stat to initialize */);
+  statList.add(childStat);
+  childStat.add(p);
+}
+
+int i = 0;
+for(Prel p : prel) {
+  children.add(p.accept(this, statList.get(i++)));
+}
+
+MajorFragmentStat maxStat = statList.get(0);
+// get the max width of all child stats
+for (int j=1; j < statList.size(); j++) {
+  if (statList.get(j).getMaxWidth() > maxStat.getMaxWidth()) {
+maxStat = statList.get(j);
+  }
+}
+
+// width of the major fragment that contains union-all should be the 
maximum
+// width of all its inputs
+s.setMaxWidth(maxStat.getMaxWidth());
--- End diff --

In addition to maxWidth, should we maintain maxRows as well for the stat 
object of Union-all operator?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #532: DRILL-4732: Update JDBC driver to use the new prepared sta...

[parthchandra]

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/532
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4729) Add support for prepared statement implementation on server side

[Venki Korukanti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venki Korukanti resolved DRILL-4729.

Resolution: Fixed

Fixed in 
[14f6ec7|https://github.com/apache/drill/commit/14f6ec7dd9b010de6c884431e443eb788ce54339].

> Add support for prepared statement implementation on server side
> 
>
> Key: DRILL-4729
> URL: https://issues.apache.org/jira/browse/DRILL-4729
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Metadata
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
> Fix For: 1.8.0
>
>
> Currently Drill JDBC/ODBC driver implements its own prepared statement 
> implementation, which basically issues limit 0 query to get the metadata and 
> then executes the actual query. So the query is planned twice (for metadata 
> fetch and actual execution). Proposal is to move that logic to server where 
> we can make optimizations without disrupting/updating the JDBC/ODBC drivers.
> *  {{PreparedStatement createPreparedStatement(String query)}}. 
> {{PreparedStatement}} object contains the following:
> ** {{ResultSetMetadata getResultSetMetadata()}}
> *** {{ResultsSetMetadata}} contains methods to fetch info about output 
> columns of the query. What info these methods provide is given in this 
> [spreadsheet|https://docs.google.com/spreadsheets/d/1A6nqUQo5xJaZDQlDTittpVrK7t4Kylycs3P32Yn_O5k/edit?usp=sharing].
>  It lists the ODBC/JDBC requirements and what Drill will provided through 
> object {{ResultsSetMetadata}}.
> *** Server can put more info here which is opaque to client and use it in 
> server when the client sends execute prepared statement query request. 
> Overload the current submit query API to take the {{PreparedStatement}} 
> returned above. 
> In the initial implementation, server side implementation of 
> {{createPreparedStatement}} API is implemented as follows:
> * Runs the query with {{LIMIT 0}}, gets the schema
> * Convert the query into a binary blob and set it as opaque object in 
> {{PreparedStatement}}.
> When the {{PreparedStatement}} is submitted for execution, reconstruct the 
> query from binary blob in opaque component of {{PreparedStatement}} and 
> execute it from scratch. 
> Opaque component of the {{PreparedStatement}} is where we can save more 
> information which we can use for optimizations/speedups.
> NOTE: We are not going to worry about parameters in prepared query in initial 
> implementation. We can provide the functionality later if there is sufficient 
> demand from Drill community.
> Changes in this patch are going to include protobuf messages, server side 
> messages and Java client APIs. Native client changes are going to be tracked 
> in a separate JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/530


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #530: DRILL-4729: Add support for prepared statement implementat...

[vkorukanti]

Github user vkorukanti commented on the issue:

https://github.com/apache/drill/pull/530
  
pushed the squashed commit that contains the review changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73945771
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,427 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import static 
org.apache.drill.exec.ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.FAILED;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.OK;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.TIMEOUT;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  /**
+   * Static list of mappings from {@link MinorType} to JDBC ResultSet 
class name (to be returned through
+   * {@link ResultSetMetaData#getColumnClassName(int)}.
+   */
+  private static final Map DRILL_TYPE_TO_JDBC_CLASSNAME 
= ImmutableMap.builder()
+  .put(MinorType.INT, Integer.class.getName())
+  .put(MinorType.BIGINT, 

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73945212
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,427 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import static 
org.apache.drill.exec.ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.FAILED;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.OK;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.TIMEOUT;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  /**
+   * Static list of mappings from {@link MinorType} to JDBC ResultSet 
class name (to be returned through
+   * {@link ResultSetMetaData#getColumnClassName(int)}.
+   */
+  private static final Map DRILL_TYPE_TO_JDBC_CLASSNAME 
= ImmutableMap.builder()
+  .put(MinorType.INT, Integer.class.getName())
+  .put(MinorType.BIGINT, 

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73944827
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,427 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import static 
org.apache.drill.exec.ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.FAILED;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.OK;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.TIMEOUT;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  /**
+   * Static list of mappings from {@link MinorType} to JDBC ResultSet 
class name (to be returned through
+   * {@link ResultSetMetaData#getColumnClassName(int)}.
+   */
+  private static final Map DRILL_TYPE_TO_JDBC_CLASSNAME 
= ImmutableMap.builder()
+  .put(MinorType.INT, Integer.class.getName())
+  .put(MinorType.BIGINT, 

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73944757
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -512,6 +526,61 @@ public void runQuery(QueryType type, 
List planFragments, UserResul
   }
 
   /**
+   * Create a prepared statement for given query.
+   *
+   * @param query
+   * @return
+   */
+  public DrillRpcFuture 
createPreparedStatement(final String query) {
+final CreatePreparedStatementReq req =
+CreatePreparedStatementReq.newBuilder()
+.setSqlQuery(query)
+.build();
+
+return client.send(RpcType.CREATE_PREPARED_STATEMENT, req, 
CreatePreparedStatementResp.class);
+  }
+
+  /**
+   * Execute the given prepared statement.
+   *
+   * @param preparedStatementHandle Prepared statement handle returned in 
response to
+   *{@link 
#createPreparedStatement(String)}.
+   * @param resultsListener {@link UserResultsListener} instance for 
listening for query results.
+   */
+  public void executePreparedStatement(final PreparedStatementHandle 
preparedStatementHandle,
+  final UserResultsListener resultsListener) {
+final RunQuery runQuery = newBuilder()
+.setResultsMode(STREAM_FULL)
+.setType(QueryType.PREPARED_STATEMENT)
+.setPreparedStatementHandle(preparedStatementHandle)
+.build();
+client.submitQuery(resultsListener, runQuery);
+  }
+
+  /**
+   * Execute the given prepared statement and return the results.
+   *
+   * @param preparedStatementHandle Prepared statement handle returned in 
response to
+   *{@link 
#createPreparedStatement(String)}.
+   * @return List of {@link QueryDataBatch}s. It is responsibility of the 
caller to release query data batches.
+   * @throws RpcException
+   */
+  public List executePreparedStatement(final 
PreparedStatementHandle preparedStatementHandle)
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73944776
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,427 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import static 
org.apache.drill.exec.ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.FAILED;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.OK;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.TIMEOUT;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  /**
+   * Static list of mappings from {@link MinorType} to JDBC ResultSet 
class name (to be returned through
+   * {@link ResultSetMetaData#getColumnClassName(int)}.
+   */
+  private static final Map DRILL_TYPE_TO_JDBC_CLASSNAME 
= ImmutableMap.builder()
+  .put(MinorType.INT, Integer.class.getName())
+  .put(MinorType.BIGINT, 

[GitHub] drill issue #561: DRILL-4766: FragmentExecutor should use EventProcessor and...

[sudheeshkatkam]

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/561
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/517
  
This is the prototype patch I tried [1]. I replied with the link in our 
prior direct communication. 

[1] https://github.com/jinfengni/incubator-drill/commits/review/DRILL-4704



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73942398
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,427 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import static 
org.apache.drill.exec.ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.FAILED;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.OK;
+import static org.apache.drill.exec.proto.UserProtos.RequestStatus.TIMEOUT;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  /**
+   * Static list of mappings from {@link MinorType} to JDBC ResultSet 
class name (to be returned through
+   * {@link ResultSetMetaData#getColumnClassName(int)}.
+   */
+  private static final Map DRILL_TYPE_TO_JDBC_CLASSNAME 
= ImmutableMap.builder()
+  .put(MinorType.INT, Integer.class.getName())
+  .put(MinorType.BIGINT, 

Re: [GitHub] drill issue #518: DRILL-4653.json - Malformed JSON should not stop the entir...

[Jason Altekruse]

Hey Parth and Subbu,

Sorry for missing the last message on this thread, I will be able to attend
the hangout tomorrow to discuss my concern. As I had said previously, I am
mostly trying to make sure that there is agreement about the impact of this
change on user behavior and expectations before we merge.

Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Thu, Aug 4, 2016 at 5:11 PM, Parth Chandra  wrote:

> Hi Subbu,
>
>   Yes we can discuss this on the next hangout. If Jason is able to attend
> we can discuss some way to address his concern.
>
> Parth
>
> On Wed, Aug 3, 2016 at 10:24 AM, Subbu Srinivasan  >
> wrote:
>
> > Hi Folks,
> > When can we discuss this feature? Would next hangout be appropriate?
> >
> > Thanks
> > Subbu
> >
> > On Mon, Jul 25, 2016 at 10:20 AM, Subbu Srinivasan <
> > ssriniva...@zscaler.com>
> > wrote:
> >
> > > This mechanism falls in line with other JSON processing similar to
> > serde's
> > > with Hive, UDF's enabled at global level will apply to all users and is
> > > outlined using documentation.
> > >
> > >
> > > What is your stance if we move to the JSONFormatPlugin?
> > >
> > > On Fri, Jul 15, 2016 at 2:08 PM, jaltekruse 
> wrote:
> > >
> > >> Github user jaltekruse commented on the issue:
> > >>
> > >> https://github.com/apache/drill/pull/518
> > >>
> > >> I don't think we should merge this without a mechanism to return a
> > >> warning to the user to tell them at least that some data was ignored,
> > and
> > >> ideally some indication of how much data was discarded. While I do
> > >> understand this is not the default behavior, I think there is still
> too
> > >> high of a risk that an admin could set this at a global level and
> users
> > >> would be unaware of some of their data being discarded.
> > >>
> > >> I am willing to discuss the benefits of merging this before such a
> > >> system exists, but until this issue has been thoroughly evaluated I am
> > -1
> > >> on the change.
> > >>
> > >> One improvement you could make to the current implementation is
> > >> moving the option to the format plugin instead of the system/session
> > list.
> > >> This enables users to include setting the option in there query with
> the
> > >> "table with options" syntax that was added last fall. We already have
> a
> > >> JIRA open for moving the all_text_mode and read_numbers_as_double
> > options
> > >> to this location, because it doesn't really make sense to change query
> > >> results based on session state. Unfortunately this change does not
> > >> completely remove my initial concern, because not all users can modify
> > or
> > >> see the storage plugins in the case when web UI security is enabled.
> > >> Non-admin users in these cases could be surprised by this behavior.
> > >>
> > >> For examples of how this is done, you can look at the text plugin
> > >> config, you would just need to add these options as properties to the
> > json
> > >> config which is currently mostly empty.
> > >>
> > >> https://github.com/apache/drill/blob/master/exec/java-
> > exec/src/main/java/org/apache/drill/exec/store/easy/json/
> > JSONFormatPlugin.java#L93
> > >>
> > >>
> > >> https://github.com/apache/drill/blob/master/exec/java-
> > exec/src/main/java/org/apache/drill/exec/store/easy/text/
> > TextFormatPlugin.java#L135
> > >>
> > >> Select with options: https://issues.apache.org/
> > jira/browse/DRILL-4047
> > >> Jira for moving the existing options:
> > >> https://issues.apache.org/jira/browse/DRILL-4206
> > >>
> > >>
> > >> ---
> > >> If your project is set up for it, you can reply to this email and have
> > >> your
> > >> reply appear on GitHub as well. If your project does not have this
> > feature
> > >> enabled and wishes so, or if the feature is enabled but not working,
> > >> please
> > >> contact infrastructure at infrastruct...@apache.org or file a JIRA
> > ticket
> > >> with INFRA.
> > >> ---
> > >>
> > >
> > >
> >
>

Re: Suggestions for hangout topics for 08/09

[Jason Altekruse]

Yeah, I can join the hangout tomorrow to talk about the PR, thanks for the
heads up.

Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Mon, Aug 8, 2016 at 12:09 PM, Zelaine Fong  wrote:

> Jason -- will you be able to join tomorrow's hangout, since you had raised
> questions about Subbu's pull request?
>
> -- Zelaine
>
>
> On Mon, Aug 8, 2016 at 11:33 AM, Gautam Parai  wrote:
>
>> Tomorrow's hangout is scheduled for 10AM - 11AM PST
>>
>> On Mon, Aug 8, 2016 at 11:30 AM, Subbu Srinivasan <
>> ssriniva...@zscaler.com>
>> wrote:
>>
>> > What time is tomorrow's mtg scheduled for?
>> >
>> >
>> > On Mon, Aug 8, 2016 at 10:48 AM, Gautam Parai 
>> wrote:
>> >
>> > > If you have any suggestions for Drill hangout topics for tomorrow,
>> you
>> > can
>> > > add it to this thread.  We will also ask around at the beginning of
>> the
>> > > hangout for any topics.  We will try to cover whatever possible during
>> > the
>> > > 1 hr.
>> > >
>> > > Topics:
>> > >   1.  DRILL-4653:  Malformed JSON should not stop the entire query
>> from
>> > > progressing.
>> > >Discussion about the PR.
>> > >
>> >
>>
>
>

[GitHub] drill pull request #532: DRILL-4732: Update JDBC driver to use the new prepa...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/532#discussion_r73940855
  
--- Diff: 
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillConnectionImpl.java ---
@@ -362,12 +364,33 @@ public PreparedStatement prepareStatement(String sql, 
int resultSetType,
 int resultSetHoldability) 
throws SQLException {
 throwIfClosed();
 try {
-  DrillPrepareResult prepareResult = new DrillPrepareResult(sql);
+  DrillRpcFuture respFuture = 
client.createPreparedStatement(sql);
+
+  CreatePreparedStatementResp resp;
+  try {
+resp = respFuture.get();
+  } catch (InterruptedException e) {
+// Preserve evidence that the interruption occurred so that code 
higher up
+// on the call stack can learn of the interruption and respond to 
it if it
+// wants to.
+Thread.currentThread().interrupt();
+
+throw new SQLException( "Interrupted", e );
--- End diff --

From [1], we remove the RpcFuture when the response is received, and set 
the status in Future. Nobody is going to process it, as the client is not 
looking for a response anymore. I am trying to understand who you mean by 
processing?

[1] 
https://github.com/apache/drill/blob/master/exec/rpc/src/main/java/org/apache/drill/exec/rpc/RpcBus.java#L412


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[daveoshinsky]

Github user daveoshinsky commented on the issue:

https://github.com/apache/drill/pull/517
  
The overflow problem will require separate changes to fix, from any changes 
we make in the short-term to fix this issue (see DRILL-4834).  However, if I 
understand some of Paul's findings earlier, the two issues are related as 
follows.  If we choose a large precision based only on the kind of integer 
that is input to the cast (either an int, or a long, but not minimized to 
represent the integer value to be casted), then we are more likely to encounter 
the overflow issue, because this precision is more likely to exceed the 
capacity of the destination decimal of the cast.  Is that right, Paul?  Given 
the risk of overflow, it is safer to choose the absolute minimum precision 
possible, which is what the changes I made to CastIntDecimal.java are doing.  
We can't avoid overflow in every situation, but we can try to minimize the 
chances of it.  My understanding is that the actual integer value to be casted 
isn't available in ExpressionTreeMaterializer.java; at least, I did not see
  how to get it.  And that integer value is required to compute the minimum 
precision to represent that value.  I don't see how to fix this problem 
reliably in ExpressionTreeMaterializer.java, as I said earlier.  If you 
(Jinfeng) know how to do that, why don't you just take the changes in PR-517, 
and then check changes in on top of those to fix the problem your way?  At 
least, you get my unit test, which was for a condition (DRILL-4704) that was 
never tested by any earlier unit test. 

On Monday, August 8, 2016 3:25 PM, Jinfeng Ni 
 wrote:
 

 I'm not fully convinced that we should check the precision for each input 
value for this castIntDecimal function. The argument of proposed patch is 
parameter "precision=0" is not valid, and has to be calculated on-the-fly or 
"dynamically" for each input value. To me, 1) if "precision=0" is a wrong input 
to this function, then we should fix the place where precision=0 is passed in. 
2) why would you treat "precision=0" only? what if I pass in "precision = 1", 
and a integer value of 123456? in such case, do we think "precison=1" is valid 
or not? Regarding the overflow problem, that seems to be a separate issue, and 
probably is true for most of the existing decimal functions. —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.  

  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/517
  
I'm not fully convinced that we should check the precision for each input 
value for this castIntDecimal function. 

The argument of proposed patch is parameter "precision=0" is not valid, and 
has to be calculated on-the-fly or "dynamically" for each input value.  To me, 
1) if "precision=0" is a wrong input to this function, then we should fix the 
place where precision=0 is passed in.  2) why would you treat "precision=0" 
only?  what if I pass in "precision = 1", and a integer value of 123456? in 
such case, do we think "precison=1" is valid or not? 

Regarding the overflow problem, that seems to be a separate issue, and 
probably is true for most of the existing decimal functions. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Fwd: Suggestions for hangout topics for 08/09

[Zelaine Fong]

Jason -- will you be able to join tomorrow's hangout, since you had raised
questions about Subbu's pull request?

-- Zelaine

On Mon, Aug 8, 2016 at 11:33 AM, Gautam Parai  wrote:

> Tomorrow's hangout is scheduled for 10AM - 11AM PST
>
> On Mon, Aug 8, 2016 at 11:30 AM, Subbu Srinivasan  >
> wrote:
>
> > What time is tomorrow's mtg scheduled for?
> >
> >
> > On Mon, Aug 8, 2016 at 10:48 AM, Gautam Parai 
> wrote:
> >
> > > If you have any suggestions for Drill hangout topics for tomorrow,  you
> > can
> > > add it to this thread.  We will also ask around at the beginning of the
> > > hangout for any topics.  We will try to cover whatever possible during
> > the
> > > 1 hr.
> > >
> > > Topics:
> > >   1.  DRILL-4653:  Malformed JSON should not stop the entire query from
> > > progressing.
> > >Discussion about the PR.
> > >
> >
>

Re: JDBC jar file size issues

[Jinfeng Ni]

Are you building Apache release or something else?

The error message means your jdbc-all jar file is too large
(93086002), probably because you brought more dependency and did not
exclude them in jdbc-all packaging.

[1] https://github.com/apache/drill/blob/master/exec/jdbc-all/pom.xml#L434-L446


On Mon, Aug 8, 2016 at 11:28 AM, rahul challapalli
 wrote:
> Hi Team,
>
> When I tried to build the latest master of Drill, the JDBC all module
> failed with the below error. Any idea about this?
>
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireFilesSize failed
> with message:
> The file drill-jdbc-all-1.8.0-SNAPSHOT.jar is outside the expected size
> range.
>
>   This is likely due to you adding new dependencies to a
> java-exec and not updating the excludes in this module. This is important
> as it minimizes the size of the dependency of Drill application users.
> /root/drillAutomation/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.8.0-SNAPSHOT.jar
> size (93086002) too large. Max. is
> 2100/root/drillAutomation/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.8.0-SNAPSHOT.jar
>
> - Rahul

[GitHub] drill pull request #532: DRILL-4732: Update JDBC driver to use the new prepa...

[parthchandra]

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/532#discussion_r73930026
  
--- Diff: 
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillConnectionImpl.java ---
@@ -362,12 +364,33 @@ public PreparedStatement prepareStatement(String sql, 
int resultSetType,
 int resultSetHoldability) 
throws SQLException {
 throwIfClosed();
 try {
-  DrillPrepareResult prepareResult = new DrillPrepareResult(sql);
+  DrillRpcFuture respFuture = 
client.createPreparedStatement(sql);
+
+  CreatePreparedStatementResp resp;
+  try {
+resp = respFuture.get();
+  } catch (InterruptedException e) {
+// Preserve evidence that the interruption occurred so that code 
higher up
+// on the call stack can learn of the interruption and respond to 
it if it
+// wants to.
+Thread.currentThread().interrupt();
+
+throw new SQLException( "Interrupted", e );
--- End diff --

What I was trying to understand was what happens if there is a message from 
the server on the wire and the respFuture gets interrupted. Now there is no 
listener to process the protobuf message. So who processes it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[vkorukanti]

Github user vkorukanti commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73929975
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,419 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  private static final int LIMIT_0_QUERY_TIMEOUT_MINS = 10;
--- End diff --

I added a session/system option in latest commit. In my opinion if a `LIMIT 
0` query is taking longer than 10mins, we should try to fix it, otherwise it is 
not an interactive experience when using Drill with BI tools.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73929114
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,419 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  private static final int LIMIT_0_QUERY_TIMEOUT_MINS = 10;
--- End diff --

Or make this a system property?

We have seen cases where `LIMIT 0` queries take longer than 10 minutes. 
Although there is [an option (disabled by 
default)](https://issues.apache.org/jira/browse/DRILL-3623) to avoid 
full-fledged execution for queries on _tables with known schema_, on other 
tables, `LIMIT 0` queries may take 10 minutes. They should not, but they may 
(due to single fragment execution).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact 

Re: Suggestions for hangout topics for 08/09

[Gautam Parai]

Tomorrow's hangout is scheduled for 10AM - 11AM PST

On Mon, Aug 8, 2016 at 11:30 AM, Subbu Srinivasan 
wrote:

> What time is tomorrow's mtg scheduled for?
>
>
> On Mon, Aug 8, 2016 at 10:48 AM, Gautam Parai  wrote:
>
> > If you have any suggestions for Drill hangout topics for tomorrow,  you
> can
> > add it to this thread.  We will also ask around at the beginning of the
> > hangout for any topics.  We will try to cover whatever possible during
> the
> > 1 hr.
> >
> > Topics:
> >   1.  DRILL-4653:  Malformed JSON should not stop the entire query from
> > progressing.
> >Discussion about the PR.
> >
>

[GitHub] drill issue #530: DRILL-4729: Add support for prepared statement implementat...

[vkorukanti]

Github user vkorukanti commented on the issue:

https://github.com/apache/drill/pull/530
  
Added option for changing the timeout. Also added a new RequestStatus type 
`TIMEOUT`. @parthchandra New changes are in a separate commit. I will create a 
JIRA for adding cancel.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Suggestions for hangout topics for 08/09

[Subbu Srinivasan]

What time is tomorrow's mtg scheduled for?


On Mon, Aug 8, 2016 at 10:48 AM, Gautam Parai  wrote:

> If you have any suggestions for Drill hangout topics for tomorrow,  you can
> add it to this thread.  We will also ask around at the beginning of the
> hangout for any topics.  We will try to cover whatever possible during the
> 1 hr.
>
> Topics:
>   1.  DRILL-4653:  Malformed JSON should not stop the entire query from
> progressing.
>Discussion about the PR.
>

JDBC jar file size issues

[rahul challapalli]

Hi Team,

When I tried to build the latest master of Drill, the JDBC all module
failed with the below error. Any idea about this?

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireFilesSize failed
with message:
The file drill-jdbc-all-1.8.0-SNAPSHOT.jar is outside the expected size
range.

  This is likely due to you adding new dependencies to a
java-exec and not updating the excludes in this module. This is important
as it minimizes the size of the dependency of Drill application users.
/root/drillAutomation/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.8.0-SNAPSHOT.jar
size (93086002) too large. Max. is
2100/root/drillAutomation/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.8.0-SNAPSHOT.jar

- Rahul

[GitHub] drill issue #517: DRILL-4704 fix

[daveoshinsky]

Github user daveoshinsky commented on the issue:

https://github.com/apache/drill/pull/517
  
Paul, do you mean the long piece of code with conditions on various 
destination precision types, that sets inPrec and maxPrec, and then selects the 
smaller of the two for the resulting precision?  I think it's just way simpler 
to use the calculated precision, as I suggested earlier.  That also works, and 
it's less code to look at, and less code to potentially be broken.  I do have 
a bias against the heavy, heavy use of FreeMarker everywhere.  It's just a 
recipe for bugs piled on top of bugs, and all of the numerous FreeMarker 
conditions are not likely to be tested for quite some time. 

On Monday, August 8, 2016 1:48 PM, Paul Rogers 
 wrote:
 

 What is the resolution here? Dave, do you want to review & adopt the 
suggested code several posts back? That code passed the unit tests that you 
provided, plus those aready in Drill (using the unit tests you suggested in an 
earlier post.) Then, update this PR with the results. Sounds like Jinfeng can 
then review the code & accept the PR. As we've said, the fix is not perfect, 
but it does work for every int/decimal type if the user is careful to provide 
int values that fit within the target decimal type.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.  

  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Suggestions for hangout topics for 08/09

[Gautam Parai]

If you have any suggestions for Drill hangout topics for tomorrow,  you can
add it to this thread.  We will also ask around at the beginning of the
hangout for any topics.  We will try to cover whatever possible during the
1 hr.

Topics:
  1.  DRILL-4653:  Malformed JSON should not stop the entire query from
progressing.
   Discussion about the PR.

[GitHub] drill issue #517: DRILL-4704 fix

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/517
  
What is the resolution here? Dave, do you want to review & adopt the 
suggested code several posts back? That code passed the unit tests that you 
provided, plus those aready in Drill (using the unit tests you suggested in an 
earlier post.) Then, update this PR with the results. Sounds like Jinfeng can 
then review the code & accept the PR. As we've said, the fix is not perfect, 
but it does work for every int/decimal type if the user is careful to provide 
int values that fit within the target decimal type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/517
  
"Dynamic" meant "for each data value." "Static" meant "at compile time." We 
make our type decisions at compile time. That is too early to adjust scale 
based on actual input values. It is a real nuisance that scale is associated 
with the type, not the value.

Dave filed a bug to track the additional issues. See DRILL-4834.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #530: DRILL-4729: Add support for prepared statement impl...

[parthchandra]

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/530#discussion_r73917579
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/prepare/PreparedStatementProvider.java
 ---
@@ -0,0 +1,419 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.prepare;
+
+import org.apache.drill.common.exceptions.ErrorHelper;
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.impl.materialize.QueryWritableBatch;
+import org.apache.drill.exec.proto.ExecProtos.ServerPreparedStatementState;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError;
+import org.apache.drill.exec.proto.UserBitShared.DrillPBError.ErrorType;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.UserBitShared.QueryType;
+import org.apache.drill.exec.proto.UserBitShared.SerializedField;
+import org.apache.drill.exec.proto.UserProtos.ColumnSearchability;
+import org.apache.drill.exec.proto.UserProtos.ColumnUpdatability;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementReq;
+import org.apache.drill.exec.proto.UserProtos.CreatePreparedStatementResp;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatement;
+import org.apache.drill.exec.proto.UserProtos.PreparedStatementHandle;
+import org.apache.drill.exec.proto.UserProtos.RequestStatus;
+import org.apache.drill.exec.proto.UserProtos.ResultColumnMetadata;
+import org.apache.drill.exec.proto.UserProtos.RpcType;
+import org.apache.drill.exec.proto.UserProtos.RunQuery;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.ResponseSender;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.rpc.user.UserServer.UserClientConnection;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.store.ischema.InfoSchemaConstants;
+import org.apache.drill.exec.work.user.UserWorker;
+import org.joda.time.Period;
+
+import com.google.common.collect.ImmutableMap;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.ChannelFuture;
+import java.math.BigDecimal;
+import java.net.SocketAddress;
+import java.sql.Date;
+import java.sql.ResultSetMetaData;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Contains worker {@link Runnable} for creating a prepared statement and 
helper methods.
+ */
+public class PreparedStatementProvider {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PreparedStatementProvider.class);
+
+  private static final int LIMIT_0_QUERY_TIMEOUT_MINS = 10;
--- End diff --

I agree, 10 min is a long time. Your call on whether to make it 
configurable, but given our current implementation, it is not impossible for us 
to hit it.
For the Prepare call, a cancel is probably a good idea in the wire 
protocol; that's just standard behaviour for any async protocol. But that may 
be a future change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---