Re: About integration of drill and arrow

2020-01-13 Thread Paul Rogers
Thanks Andy! Very helpful. You have hit on one of the questions that we've been wrestling with: which tools would consume Drill data as Arrow? More generally, what are the use cases for Arrow data interchange? Flight makes sense for transferring large data sets, such as in exchanges within a d

Re: About integration of drill and arrow

2020-01-13 Thread Andy Grove
Hi Paul, There is a test flight server in the Arrow Java project [1] that might be a good starting point, although I haven't used it myself. I was looking at Arrow Flight for my Ballista Poc [2] although I don't really have time to spend on that right now. I'm less sure of the value of having an

Re: About integration of drill and arrow

2020-01-13 Thread Paul Rogers
Hi Andy & Charles, We've discussed two ways for Drill to interface to Arrow: either as an input or an output: Arrow Producer --> Drill --> Arrow Consumer Given how Drill works, the easiest of the two is to create a storage plugin to read from an Arrow Producer, perhaps using Arrow Flight (than

Re: About integration of drill and arrow

2020-01-13 Thread Paul Rogers
Hi Igor, Thanks much for volunteering to create some POCs for our various options! It is not entirely obvious what we want to test, so let's think about it a bit. We want to identify those areas that are either the biggest risk or benefit to performance. We want to do that without the cost of a

[GitHub] [drill] ihuzenko opened a new pull request #1955: DRILL-7491: Incorrect count() returned for complex types in parquet

2020-01-13 Thread GitBox
ihuzenko opened a new pull request #1955: DRILL-7491: Incorrect count() returned for complex types in parquet URL: https://github.com/apache/drill/pull/1955 # [DRILL-7491](https://issues.apache.org/jira/browse/DRILL-7491): Incorrect count() returned for complex types in parquet ## D

[GitHub] [drill] KazydubB commented on a change in pull request #1948: DRILL-7506: Simplify code gen error handling

2020-01-13 Thread GitBox
KazydubB commented on a change in pull request #1948: DRILL-7506: Simplify code gen error handling URL: https://github.com/apache/drill/pull/1948#discussion_r366000321 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/LateralJoinBatch.java ###

[GitHub] [drill] KazydubB commented on a change in pull request #1948: DRILL-7506: Simplify code gen error handling

2020-01-13 Thread GitBox
KazydubB commented on a change in pull request #1948: DRILL-7506: Simplify code gen error handling URL: https://github.com/apache/drill/pull/1948#discussion_r366002749 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/LateralJoinBatch.java ###

[GitHub] [drill] KazydubB commented on a change in pull request #1948: DRILL-7506: Simplify code gen error handling

2020-01-13 Thread GitBox
KazydubB commented on a change in pull request #1948: DRILL-7506: Simplify code gen error handling URL: https://github.com/apache/drill/pull/1948#discussion_r366001563 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperatorUtil.java

[GitHub] [drill] KazydubB opened a new pull request #1954: DRILL-7509: Incorrect TupleSchema is created for DICT column when querying Parquet files

2020-01-13 Thread GitBox
KazydubB opened a new pull request #1954: DRILL-7509: Incorrect TupleSchema is created for DICT column when querying Parquet files URL: https://github.com/apache/drill/pull/1954 # [DRILL-7509](https://issues.apache.org/jira/browse/DRILL-7509): Incorrect TupleSchema is created for DICT colu

Re: About integration of drill and arrow

2020-01-13 Thread Andy Grove
I just started working with Drill and I am a PMC member of Apache Arrow. I am in the process of writing my first storage plugin for Drill, and I think it would be interesting to build a storage plugin for the Apache Arrow Flight protocol as a way for Drill to query Arrow data, although I'm not sure

Apache Drill documentation updates

2020-01-13 Thread Bevens, Bridget
Hi, I wanted to let everyone know that I won’t be working regularly on the Apache Drill documentation, but I’ll be available to update the website for the releases, which includes: * Generating the release notes, blog, What’s New pages * Updating files with the new release version/poi

Re: About integration of drill and arrow

2020-01-13 Thread Igor Guzenko
Hi Paul and Volodymyr, Thank you very much Volodymyr and Paul for defining the good migration strategy. It really should work for a smooth migration. What also I really like in the discussion is that excellent questions appeared: - Aren't we just suffering from premature optimizations? - Were

[jira] [Created] (DRILL-7527) DROP METADATA doesn't work with table name starting with '/' inside workspace

2020-01-13 Thread Denys Ordynskiy (Jira)
Denys Ordynskiy created DRILL-7527: -- Summary: DROP METADATA doesn't work with table name starting with '/' inside workspace Key: DRILL-7527 URL: https://issues.apache.org/jira/browse/DRILL-7527 Proje

[GitHub] [drill] arina-ielchiieva edited a comment on issue #1953: Add docs for Drill Metastore

2020-01-13 Thread GitBox
arina-ielchiieva edited a comment on issue #1953: Add docs for Drill Metastore URL: https://github.com/apache/drill/pull/1953#issuecomment-573703899 @vvysotskyi overall good regarding analyze command but I think we need to add more info about Drill Metastore notion in general. I suggest we

[GitHub] [drill] arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore

2020-01-13 Thread GitBox
arina-ielchiieva commented on issue #1953: Add docs for Drill Metastore URL: https://github.com/apache/drill/pull/1953#issuecomment-573703899 @vvysotskyi overall good regarding analyze command but I think we need to add more info about Drill Metastore notion in general. I suggest we create

Re: Drill Hangout

2020-01-13 Thread Igor Guzenko
Hello Drillers, I'm fine with recovery of hangouts with one good suggestion. I think it will be fine to have a bi-weekly call like it was earlier, but also I'd like to suggest skip calls if one day before no one introduced good topic for discussion. >From previous experience, we had a few times wh

[GitHub] [drill] vvysotskyi opened a new pull request #1953: Add docs for Drill Metastore

2020-01-13 Thread GitBox
vvysotskyi opened a new pull request #1953: Add docs for Drill Metastore URL: https://github.com/apache/drill/pull/1953 This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: Drill Hangout

2020-01-13 Thread Charles Givre
This week is a little busy for me as well. I was thinking next week. I'd like to get as many of us as possible. @Paul are you an early riser? -- C > On Jan 13, 2020, at 9:37 AM, Vova Vysotskyi wrote: > > Hi Charles, > > I wouldn't be able to join Drill Hangouts during this week. > > Ki

Re: Drill Hangout

2020-01-13 Thread Vova Vysotskyi
Hi Charles, I wouldn't be able to join Drill Hangouts during this week. Kind regards, Volodymyr Vysotskyi On Mon, Jan 13, 2020 at 4:26 PM Charles Givre wrote: > Hello Drill Community, > I'd like to propose a reinstitution of Drill Hangouts, perhaps every other > week. Given that the bulk of

Drill Hangout

2020-01-13 Thread Charles Givre
Hello Drill Community, I'd like to propose a reinstitution of Drill Hangouts, perhaps every other week. Given that the bulk of the development at the moment has shifted from the US to Europe, I'd like to propose 0930 ET / 1630EET / 0630 PT for the first one. Depending on interest, we can rota

[GitHub] [drill] cgivre commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5

2020-01-13 Thread GitBox
cgivre commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5 URL: https://github.com/apache/drill/pull/1778#discussion_r365817001 ## File path: contrib/format-hdf5/src/test/java/org/apache/drill/exec/store/hdf5/TestHDF5Format.java ## @@ -863,14 +

[GitHub] [drill] cgivre commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5

2020-01-13 Thread GitBox
cgivre commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5 URL: https://github.com/apache/drill/pull/1778#discussion_r365816254 ## File path: contrib/format-hdf5/src/test/java/org/apache/drill/exec/store/hdf5/TestHDF5Format.java ## @@ -887,6 +8

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5

2020-01-13 Thread GitBox
arina-ielchiieva commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5 URL: https://github.com/apache/drill/pull/1778#discussion_r364640539 ## File path: contrib/format-hdf5/src/test/java/org/apache/drill/exec/store/hdf5/TestHDF5Format.java ## @@

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5

2020-01-13 Thread GitBox
arina-ielchiieva commented on a change in pull request #1778: DRILL-7233: Format Plugin for HDF5 URL: https://github.com/apache/drill/pull/1778#discussion_r364636869 ## File path: contrib/format-hdf5/src/test/java/org/apache/drill/exec/store/hdf5/TestHDF5Format.java ## @@

[jira] [Created] (DRILL-7526) Assertion Error when only type with used with schema in table function

2020-01-13 Thread Arina Ielchiieva (Jira)
Arina Ielchiieva created DRILL-7526: --- Summary: Assertion Error when only type with used with schema in table function Key: DRILL-7526 URL: https://issues.apache.org/jira/browse/DRILL-7526 Project: A

[GitHub] [drill] cgivre commented on issue #1951: DRILL-7454: Convert Avro to EVF

2020-01-13 Thread GitBox
cgivre commented on issue #1951: DRILL-7454: Convert Avro to EVF URL: https://github.com/apache/drill/pull/1951#issuecomment-573638073 Nice work everyone! This is an automated message from the Apache Git Service. To respond to

[GitHub] [drill] arina-ielchiieva commented on issue #1951: DRILL-7454: Convert Avro to EVF

2020-01-13 Thread GitBox
arina-ielchiieva commented on issue #1951: DRILL-7454: Convert Avro to EVF URL: https://github.com/apache/drill/pull/1951#issuecomment-573637810 @paul-rogers / @vvysotskyi thank you for the review! This is an automated message

[jira] [Created] (DRILL-7525) Convert SequqenceFiles to EVF

2020-01-13 Thread Arina Ielchiieva (Jira)
Arina Ielchiieva created DRILL-7525: --- Summary: Convert SequqenceFiles to EVF Key: DRILL-7525 URL: https://issues.apache.org/jira/browse/DRILL-7525 Project: Apache Drill Issue Type: Improvem

[jira] [Created] (DRILL-7524) Distinct on array with any_value

2020-01-13 Thread benj (Jira)
benj created DRILL-7524: --- Summary: Distinct on array with any_value Key: DRILL-7524 URL: https://issues.apache.org/jira/browse/DRILL-7524 Project: Apache Drill Issue Type: Bug Components: Fun

[jira] [Created] (DRILL-7523) Update Calcite to 1.22.0

2020-01-13 Thread Vova Vysotskyi (Jira)
Vova Vysotskyi created DRILL-7523: - Summary: Update Calcite to 1.22.0 Key: DRILL-7523 URL: https://issues.apache.org/jira/browse/DRILL-7523 Project: Apache Drill Issue Type: Task Affects