Re: [DISCUSSION] current project state

2018-08-13 Thread Charles Givre
I’d like to weigh in here as well. As a long time user of Drill, I really would like to see more people using it and I think there are a few key aspects that could really help on that front. The first of which is the Arrow integration. I’m not enough of a software engineer to understand all t

[GitHub] sohami commented on issue #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-412738302 @weijietong - Thanks for making the changes. But I am seeing issues in implementation related to how internals work, following protocol of iterative model

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209793080 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/RuntimeFilterRecordBatch.java ##

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209803392 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/RuntimeFilterRecordBatch.java ##

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209791450 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/RuntimeFilterPrelVisitor.java ###

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209723766 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java ## @@ -59

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209718874 ## File path: protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ## @@ -585,6 +585,10 @@ private Fr

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209815818 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/filter/RuntimeFilterRecordBatch.java ##

[GitHub] sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#discussion_r209791940 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/RuntimeFilterPrel.java ## @@ -0,0 +1

[GitHub] ilooner commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
ilooner commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#issuecomment-412734987 Thanks for the help @ppadma. Ran the tests. There are a few failures but those are known issues unrelated to your

Re: [DISCUSSION] current project state

2018-08-13 Thread Paul Rogers
Hi Arina, Another topic would be whether/how to round out Drill's data model. Drill's scalar and nullable types are pretty solid. Great work was done recently for Decimal (though the old types still remain.) Good support is now available for nested types to do implicit joins to produce SQL-frie

[GitHub] paul-rogers commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
paul-rogers commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#discussion_r209803870 ## File path: exec/vector/src/main/java/org/apache/drill/exec/vector/complex/ListVe

[GitHub] paul-rogers commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
paul-rogers commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#discussion_r209803508 ## File path: exec/vector/src/main/java/org/apache/drill/exec/vector/complex/ListVe

[GitHub] paul-rogers commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
paul-rogers commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#issuecomment-412715369 @ilooner, thanks much for taking a look at this PR. Agree it would be great for @ppadma to review this as she

[GitHub] ppadma commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
ppadma commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#issuecomment-412711621 @paul-rogers @ilooner I can review this PR. It will take some time. I will post comments as I make progress. mean

[GitHub] ilooner commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
ilooner commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#issuecomment-412703438 @paul-rogers I added some more checkstyle checks recently, and they fail when the PR is rebased onto the latest m

[GitHub] ilooner commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
ilooner commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#discussion_r209791763 ## File path: exec/vector/src/main/java/org/apache/drill/exec/vector/complex/ListVector

[GitHub] ilooner commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
ilooner commented on a change in pull request #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#discussion_r209791651 ## File path: exec/vector/src/main/java/org/apache/drill/exec/vector/complex/ListVector

Re: [DISCUSS] sqlline upgrade

2018-08-13 Thread Parth Chandra
#3 was considered a bit of a requirement when it was done. Having SQLLine show a different version from that of Drill caused confusion among users. #4 seems another usability issue. I would feel less confident in a product whose command line tool shows me commands that are not supported. Also you w

Re: [DISCUSS] sqlline upgrade

2018-08-13 Thread Boaz Ben-Zvi
supports scrolling through multi-line SQL statements as single snippets in history   I opened an issue (#73) for that back in April ( https://github.com/julianhyde/sqlline/issues/73 ), but no progress so far On 8/13/18 10:15 AM, Abhishek Girish wrote: +1. Need to try out (2) to unders

[jira] [Created] (DRILL-6685) Error in parquet record reader

2018-08-13 Thread Robert Hou (JIRA)
Robert Hou created DRILL-6685: - Summary: Error in parquet record reader Key: DRILL-6685 URL: https://issues.apache.org/jira/browse/DRILL-6685 Project: Apache Drill Issue Type: Bug Compo

[GitHub] ilooner commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader

2018-08-13 Thread GitBox
ilooner commented on issue #1429: DRILL-6676: Add Union, List and Repeated List types to Result Set Loader URL: https://github.com/apache/drill/pull/1429#issuecomment-412693176 Thanks @paul-rogers . I'll run the functional tests and let you know if there are any issues. Since your code is

[jira] [Created] (DRILL-6684) Swap sys.options and sys.options_val tables

2018-08-13 Thread Kunal Khatua (JIRA)
Kunal Khatua created DRILL-6684: --- Summary: Swap sys.options and sys.options_val tables Key: DRILL-6684 URL: https://issues.apache.org/jira/browse/DRILL-6684 Project: Apache Drill Issue Type: Im

[jira] [Created] (DRILL-6683) move getSelectionVector2 and getSelectionVector4 from VectorAccessible interface to RecordBatch interface

2018-08-13 Thread Timothy Farkas (JIRA)
Timothy Farkas created DRILL-6683: - Summary: move getSelectionVector2 and getSelectionVector4 from VectorAccessible interface to RecordBatch interface Key: DRILL-6683 URL: https://issues.apache.org/jira/browse/DRI

[GitHub] ilooner commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework.

2018-08-13 Thread GitBox
ilooner commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework. URL: https://github.com/apache/drill/pull/1344#discussion_r209779765 ## File path: exec/java-exec/src/main/java/org/apach

[GitHub] ilooner commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework.

2018-08-13 Thread GitBox
ilooner commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework. URL: https://github.com/apache/drill/pull/1344#discussion_r209779483 ## File path: exec/java-exec/src/test/java/org/apach

[GitHub] ilooner commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework.

2018-08-13 Thread GitBox
ilooner commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework. URL: https://github.com/apache/drill/pull/1344#discussion_r209779109 ## File path: exec/java-exec/src/main/java/org/apach

Re: Slowness in ctas json to parquet in 1.14

2018-08-13 Thread Kunal Khatua
That certainly sounds unusual.  Are you seeing this when the JSON source files are identical?  For the 1.14 build, the operator overview of that CTAS query's profile, you should be able to what operator is consuming the most time.  There isn't any direct change in 1.14 that would explain the sl

Slowness in ctas json to parquet in 1.14

2018-08-13 Thread Mehran Dashti [ BRCO - PD ]
Congratulations for drill 1.14 But I see a very annoying slowness in ctas json to parquet in recent releases. Specifically in 1.14 I see that it seems to be a blocker bug. 5 minute takes to insert a json file to parquet, where in 1.13 it takes 10 seconds. Is it sth that I should do to get bett

Re: [DISCUSSION] current project state

2018-08-13 Thread Paul Rogers
Hi Arina, Thanks for launching this discussion. A few minor suggestions. The developers have done a fantastic job stabilizing and improving Drill's core functionality. Now the opportunity is to expand the use cases for Drill so that it gets wider adoption within the community. Drill competes fo

Re: [DISCUSS] sqlline upgrade

2018-08-13 Thread John Omernik
Since we are "shipping" sqlline, can we include a sqline shell script that puts the command line arguments we need, and then include another that people can use if they want to use without our arguments? This way anyone who is used to just "sqlline" doesn't have to change, it's the "new" functional

Re: [DISCUSS] sqlline upgrade

2018-08-13 Thread Abhishek Girish
+1. Need to try out (2) to understand it's impact on usability. Okay with (3). Not sure of consequences of (4) - what will happen to options Drill doesn't support? Also, do we know if the latest sqlline supports scrolling through multi-line SQL statements as single snippets in history? On Mon, Au

[DISCUSS] sqlline upgrade

2018-08-13 Thread Arina Yelchiyeva
Hi all, currently Apache Drill is using custom version of Sqlline [1]. It has some custom fixes, the importance of which I propose to discuss. *1. Isolation.* Drill supports TRANSACTION_NONE only, default in Sqlline is TRANSACTION_REPEATABLE_READ. Apparently the was not possibility to override th

[GitHub] sohami commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework.

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework. URL: https://github.com/apache/drill/pull/1344#discussion_r209677503 ## File path: exec/java-exec/src/test/java/org/apache

[GitHub] sohami commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework.

2018-08-13 Thread GitBox
sohami commented on a change in pull request #1344: DRILL-6461: Added basic data correctness tests for hash agg, and improved operator unit testing framework. URL: https://github.com/apache/drill/pull/1344#discussion_r209672777 ## File path: exec/java-exec/src/main/java/org/apache

Drillbit client connect authorization

2018-08-13 Thread Oleksandr Kalinin
Hello Drill community, In multi-tenant YARN clusters, running multiple Drill-on-YARN clusters seems as attractive feature as it enables leveraging on YARN mechanisms of resource management and isolation. However, there seems to be simple access restriction issue. Assume : - Cluster A launched by

Requesting ETA on drill/apache-drill-centos:1.14.0 docker image

2018-08-13 Thread Vedant Naik
Hi, We are using Apache Drill (love it so far!!) We use the drill/apache-drill-centos docker image to deploy on kubernetes. We tried out the latest drill version 1.14.0 locally, and are keen to try it out on the k8s cluster. However, the drill/apache-drill-centos:1.14.0-SNAPSHOT image ends in "Cr

[GitHub] arina-ielchiieva commented on issue #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA

2018-08-13 Thread GitBox
arina-ielchiieva commented on issue #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA URL: https://github.com/apache/drill/pull/1430#issuecomment-412522612 @vvysotskyi thanks for the code review, addressed CR comments.

[DISCUSSION] current project state

2018-08-13 Thread Arina Ielchiieva
Hi all, as a new PMC Chair I would like to thank users for choosing and using Apache Drill and contributors / committers for making improvements and fixes. Recently Apache Drill 1.14 was released bundled up with many improvements and new features. Please feel free to try it out and share your exp

[GitHub] okalinin commented on issue #1428: DRILL-6670: align Parquet TIMESTAMP_MICROS logical type handling with earlier versions + minor fixes

2018-08-13 Thread GitBox
okalinin commented on issue #1428: DRILL-6670: align Parquet TIMESTAMP_MICROS logical type handling with earlier versions + minor fixes URL: https://github.com/apache/drill/pull/1428#issuecomment-412493398 Comment addressed, conflicts resolved. PR should be OK for further review. -

[GitHub] asfgit closed pull request #1408: DRILL-6453: Resolve deadlock when reading from build and probe sides simultaneously in HashJoin

2018-08-13 Thread GitBox
asfgit closed pull request #1408: DRILL-6453: Resolve deadlock when reading from build and probe sides simultaneously in HashJoin URL: https://github.com/apache/drill/pull/1408 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below

[GitHub] vvysotskyi commented on a change in pull request #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA

2018-08-13 Thread GitBox
vvysotskyi commented on a change in pull request #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA URL: https://github.com/apache/drill/pull/1430#discussion_r209544436 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ShowFiles

[GitHub] vvysotskyi commented on a change in pull request #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA

2018-08-13 Thread GitBox
vvysotskyi commented on a change in pull request #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA URL: https://github.com/apache/drill/pull/1430#discussion_r209556134 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/Records.java ##

[GitHub] vvysotskyi commented on a change in pull request #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA

2018-08-13 Thread GitBox
vvysotskyi commented on a change in pull request #1430: DRILL-6680: Expose show files command into INFORMATION_SCHEMA URL: https://github.com/apache/drill/pull/1430#discussion_r209540538 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ShowFiles