Re: Naming the new ValueVector Initiative

[Jacques Nadeau]

Hey guys, one other note on PR. To make arrow known and have a good launch,
it is best to not announce anything publicly until the press release comes
out. Will send out a press release draft shortly and then we can work with
Apache PR to set an announcement date.
On Jan 20, 2016 7:15 PM, "Jacques Nadeau"  wrote:

> Yep, straight to TLP.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Jan 20, 2016 at 5:21 PM, Jake Luciani  wrote:
>
>> That's great! So it's going straight to TLP?
>> Hey Everyone,
>>
>> Good news! The Apache board has approved the Apache Arrow as a new TLP.
>> I've asked the Apache INFRA team to set up required resources so we can
>> start moving forward (ML, Git, Website, etc).
>>
>> I've started working on a press release to announce the Apache Arrow
>> project and will circulate a draft shortly. Once the project mailing lists
>> are established, we can move this thread over there to continue
>> discussions. They had us do one of change to the proposal during the board
>> call which was to remove the initial committers (separate from initial
>> pmc). Once we establish the PMC list, we can immediately add the additional
>> committers as our first PMC action.
>>
>> thanks to everyone!
>> Jacques
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem 
>> wrote:
>>
>>> +1 on a repo for the spec.
>>> I do have questions as well.
>>> In particular for the metadata.
>>>
>>> On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:
>>>
 On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra 
 wrote:
 >
 >
 > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney 
 wrote:
 >>
 >>
 >> >
 >> > As far as the existing work is concerned, I'm not sure everyone is
 aware
 >> > of
 >> > the C++ code inside of Drill that can represent at least the scalar
 >> > types in
 >> > Drill's existing Value Vectors [1]. This is currently used by the
 native
 >> > client written to hook up an ODBC driver.
 >> >
 >>
 >> I have read this code. From my perspective, it would be less work to
 >> collaborate on a self-contained implementation that closely models
 the
 >> Arrow / VV spec that includes builder classes and its own memory
 >> management without coupling to Drill details. I started prototyping
 >> something here (warning: only a few actual days of coding here):
 >>
 >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
 >>
 >> For example, you can see an example constructing an Array or
 >> String (== Array) column in the tests here
 >>
 >>
 >>
 https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
 >>
 >> I've been planning to use this as the basis of a C++ Parquet
 >> reader-writer and the associated Python pandas-like layer which
 >> includes in-memory analytics on Arrow data structures.
 >>
 >> > Parth who is included here has been the primary owner of this C++
 code
 >> > throughout it's life in Drill. Parth, what do you think is the best
 >> > strategy
 >> > for managing the C++ code right now? As the C++ build is not tied
 into
 >> > the
 >> > Java one, as I understand it we just run it manually when updates
 are
 >> > made
 >> > there and we need to update ODBC. Would it be disruptive to move
 the
 >> > code to
 >> > the arrow repo? If so, we could include Drill as a submodule in
 the new
 >> > repo, or put Wes's work so far in the Drill repo.
 >>
 >> If we can enumerate the non-Drill-client related parts (i.e. the
 array
 >> accessors and data structures-oriented code) that would make sense in
 >> a standalone Arrow library it would be great to start a side
 >> discussion about the design of the C++ reference implementation
 >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
 >> this is a quite urgent for me (intending to deliver a minimally
 viable
 >> pandas-like Arrow + Parquet in Python stack in the next ~3 months) it
 >> would be great to do this sooner rather than later.
 >>
 >
 > Most of the code for  Drill C++ Value Vectors is independent of Drill
 -
 > mostly the code upto line 787 in this file -
 >
 https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
 >
 > My thought was to leave the Drill implementation alone and borrow
 copiously
 > from it when convenient for Arrow. Seems like we can still do that
 building
 > on Wes' work.
 >

 Makes sense. Speaking of code, would you all like me to set up a
 temporary repo for the specification itself? I already have a few
 questions like how and where to track array null counts.

 > Wes, let me know if you want to have a quick hangout on this.
 >

Re: Naming the new ValueVector Initiative

[Jacques Nadeau]

Yep, straight to TLP.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Jan 20, 2016 at 5:21 PM, Jake Luciani  wrote:

> That's great! So it's going straight to TLP?
> Hey Everyone,
>
> Good news! The Apache board has approved the Apache Arrow as a new TLP.
> I've asked the Apache INFRA team to set up required resources so we can
> start moving forward (ML, Git, Website, etc).
>
> I've started working on a press release to announce the Apache Arrow
> project and will circulate a draft shortly. Once the project mailing lists
> are established, we can move this thread over there to continue
> discussions. They had us do one of change to the proposal during the board
> call which was to remove the initial committers (separate from initial
> pmc). Once we establish the PMC list, we can immediately add the additional
> committers as our first PMC action.
>
> thanks to everyone!
> Jacques
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem  wrote:
>
>> +1 on a repo for the spec.
>> I do have questions as well.
>> In particular for the metadata.
>>
>> On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:
>>
>>> On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra 
>>> wrote:
>>> >
>>> >
>>> > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney 
>>> wrote:
>>> >>
>>> >>
>>> >> >
>>> >> > As far as the existing work is concerned, I'm not sure everyone is
>>> aware
>>> >> > of
>>> >> > the C++ code inside of Drill that can represent at least the scalar
>>> >> > types in
>>> >> > Drill's existing Value Vectors [1]. This is currently used by the
>>> native
>>> >> > client written to hook up an ODBC driver.
>>> >> >
>>> >>
>>> >> I have read this code. From my perspective, it would be less work to
>>> >> collaborate on a self-contained implementation that closely models the
>>> >> Arrow / VV spec that includes builder classes and its own memory
>>> >> management without coupling to Drill details. I started prototyping
>>> >> something here (warning: only a few actual days of coding here):
>>> >>
>>> >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
>>> >>
>>> >> For example, you can see an example constructing an Array or
>>> >> String (== Array) column in the tests here
>>> >>
>>> >>
>>> >>
>>> https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
>>> >>
>>> >> I've been planning to use this as the basis of a C++ Parquet
>>> >> reader-writer and the associated Python pandas-like layer which
>>> >> includes in-memory analytics on Arrow data structures.
>>> >>
>>> >> > Parth who is included here has been the primary owner of this C++
>>> code
>>> >> > throughout it's life in Drill. Parth, what do you think is the best
>>> >> > strategy
>>> >> > for managing the C++ code right now? As the C++ build is not tied
>>> into
>>> >> > the
>>> >> > Java one, as I understand it we just run it manually when updates
>>> are
>>> >> > made
>>> >> > there and we need to update ODBC. Would it be disruptive to move the
>>> >> > code to
>>> >> > the arrow repo? If so, we could include Drill as a submodule in the
>>> new
>>> >> > repo, or put Wes's work so far in the Drill repo.
>>> >>
>>> >> If we can enumerate the non-Drill-client related parts (i.e. the array
>>> >> accessors and data structures-oriented code) that would make sense in
>>> >> a standalone Arrow library it would be great to start a side
>>> >> discussion about the design of the C++ reference implementation
>>> >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
>>> >> this is a quite urgent for me (intending to deliver a minimally viable
>>> >> pandas-like Arrow + Parquet in Python stack in the next ~3 months) it
>>> >> would be great to do this sooner rather than later.
>>> >>
>>> >
>>> > Most of the code for  Drill C++ Value Vectors is independent of Drill -
>>> > mostly the code upto line 787 in this file -
>>> >
>>> https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
>>> >
>>> > My thought was to leave the Drill implementation alone and borrow
>>> copiously
>>> > from it when convenient for Arrow. Seems like we can still do that
>>> building
>>> > on Wes' work.
>>> >
>>>
>>> Makes sense. Speaking of code, would you all like me to set up a
>>> temporary repo for the specification itself? I already have a few
>>> questions like how and where to track array null counts.
>>>
>>> > Wes, let me know if you want to have a quick hangout on this.
>>> >
>>>
>>> Sure, I'll follow up separately to get something on the calendar.
>>> Looking forward to connecting!
>>>
>>> > Parth
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Julien
>>
>
>

[jira] [Created] (DRILL-4293) [Drill JDBC] java.lang.NoClassDefFoundError: oadd/org/apache/commons/logging/LogFactory

[hide kojima (JIRA)]

hide kojima created DRILL-4293:
--

 Summary: [Drill JDBC] java.lang.NoClassDefFoundError: 
oadd/org/apache/commons/logging/LogFactory
 Key: DRILL-4293
 URL: https://issues.apache.org/jira/browse/DRILL-4293
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 1.4.0
 Environment: Platform: x86_64-apple-darwin13.4.0 (64-bit)
OS X 10.11.2 (El Capitan)

java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

Drill version: 1.4.0
Reporter: hide kojima


When I connect embedded drill from RJDBC it raises

java.lang.NoClassDefFoundError:oadd/org/apache/commons/logging/LogFactory

And it seems drill-jdbc-all-1.4.0.jar does not contain  the class

Here is the MANIFEST.MF

Manifest-Version: 1.0
Implementation-Vendor: The Apache Software Foundation
Implementation-Title: exec/JDBC Driver using dependencies
Implementation-Version: 1.4.0
Implementation-Vendor-Id: org.apache.drill.exec
Build-Jdk: 1.7.0_45
Specification-Vendor: The Apache Software Foundation
Built-By: 
Specification-Title: exec/JDBC Driver using dependencies
Created-By: Apache Maven 3.0.5
Specification-Version: 1.4.0
Extension-Name: org.apache.drill
url: http://drill.apache.org/
Archiver-Version: Plexus Archiver



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Naming the new ValueVector Initiative

[Jake Luciani]

That's great! So it's going straight to TLP?
Hey Everyone,

Good news! The Apache board has approved the Apache Arrow as a new TLP.
I've asked the Apache INFRA team to set up required resources so we can
start moving forward (ML, Git, Website, etc).

I've started working on a press release to announce the Apache Arrow
project and will circulate a draft shortly. Once the project mailing lists
are established, we can move this thread over there to continue
discussions. They had us do one of change to the proposal during the board
call which was to remove the initial committers (separate from initial
pmc). Once we establish the PMC list, we can immediately add the additional
committers as our first PMC action.

thanks to everyone!
Jacques


--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem  wrote:

> +1 on a repo for the spec.
> I do have questions as well.
> In particular for the metadata.
>
> On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:
>
>> On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra  wrote:
>> >
>> >
>> > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney  wrote:
>> >>
>> >>
>> >> >
>> >> > As far as the existing work is concerned, I'm not sure everyone is
>> aware
>> >> > of
>> >> > the C++ code inside of Drill that can represent at least the scalar
>> >> > types in
>> >> > Drill's existing Value Vectors [1]. This is currently used by the
>> native
>> >> > client written to hook up an ODBC driver.
>> >> >
>> >>
>> >> I have read this code. From my perspective, it would be less work to
>> >> collaborate on a self-contained implementation that closely models the
>> >> Arrow / VV spec that includes builder classes and its own memory
>> >> management without coupling to Drill details. I started prototyping
>> >> something here (warning: only a few actual days of coding here):
>> >>
>> >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
>> >>
>> >> For example, you can see an example constructing an Array or
>> >> String (== Array) column in the tests here
>> >>
>> >>
>> >>
>> https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
>> >>
>> >> I've been planning to use this as the basis of a C++ Parquet
>> >> reader-writer and the associated Python pandas-like layer which
>> >> includes in-memory analytics on Arrow data structures.
>> >>
>> >> > Parth who is included here has been the primary owner of this C++
>> code
>> >> > throughout it's life in Drill. Parth, what do you think is the best
>> >> > strategy
>> >> > for managing the C++ code right now? As the C++ build is not tied
>> into
>> >> > the
>> >> > Java one, as I understand it we just run it manually when updates are
>> >> > made
>> >> > there and we need to update ODBC. Would it be disruptive to move the
>> >> > code to
>> >> > the arrow repo? If so, we could include Drill as a submodule in the
>> new
>> >> > repo, or put Wes's work so far in the Drill repo.
>> >>
>> >> If we can enumerate the non-Drill-client related parts (i.e. the array
>> >> accessors and data structures-oriented code) that would make sense in
>> >> a standalone Arrow library it would be great to start a side
>> >> discussion about the design of the C++ reference implementation
>> >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
>> >> this is a quite urgent for me (intending to deliver a minimally viable
>> >> pandas-like Arrow + Parquet in Python stack in the next ~3 months) it
>> >> would be great to do this sooner rather than later.
>> >>
>> >
>> > Most of the code for  Drill C++ Value Vectors is independent of Drill -
>> > mostly the code upto line 787 in this file -
>> >
>> https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
>> >
>> > My thought was to leave the Drill implementation alone and borrow
>> copiously
>> > from it when convenient for Arrow. Seems like we can still do that
>> building
>> > on Wes' work.
>> >
>>
>> Makes sense. Speaking of code, would you all like me to set up a
>> temporary repo for the specification itself? I already have a few
>> questions like how and where to track array null counts.
>>
>> > Wes, let me know if you want to have a quick hangout on this.
>> >
>>
>> Sure, I'll follow up separately to get something on the calendar.
>> Looking forward to connecting!
>>
>> > Parth
>> >
>> >
>>
>
>
>
> --
> Julien
>

[jira] [Created] (DRILL-4292) Avoid multiple new Configuration/FileSystem instantiation

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4292:
-

 Summary: Avoid multiple new Configuration/FileSystem instantiation 
 Key: DRILL-4292
 URL: https://issues.apache.org/jira/browse/DRILL-4292
 Project: Apache Drill
  Issue Type: Task
Reporter: Laurent Goujon


There are lots of places where Drill code has the following pattern:
{noformat}
conf = new Configuration();
{noformat}

or
{noformat}
Configuration conf = new Configuration();
[...]
fs = FileSystem.get(conf);
{noformat}

Creating Configuration instances is a pretty expensive operation as it triggers 
a classpath scan to find resources (this lazily happens when accessing a key). 
Also, these extra instances use more memory than expected (because the default 
resources are not shared between Configuration instances.

FileSystem instances should not be expensive to create, because by default, 
instances are cached (by scheme/authority/ugi), but it also means that the 
Configuration instance has little value when getting the FileSystem instnce 
(except when getting the default filesystem (can be replaced by 
FileSystem#get(URI, Configuration)) or creating the FileSystem for the given 
scheme, IF not already in the cache).

If possible, all these examples should be refactored to inject the right 
FileSystem instance, instead of letting each class trying to create its own.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Naming the new ValueVector Initiative

[P. Taylor Goetz]

> On Jan 20, 2016, at 6:28 PM, Jacques Nadeau  wrote:
> 
> Good news! The Apache board has approved the Apache Arrow as a new TLP. I've 
> asked the Apache INFRA team to set up required resources so we can start 
> moving forward (ML, Git, Website, etc). 

Mostly due to Jacques' editing and svn mojo during the board meeting to address 
comments. :) Otherwise we would have waited a month. Nice work Jacques!

The feedback from the board was mainly around the "additional contributors" 
clause in the resolution. That had to be removed.

All that means is that one of the first actions as a PMC will be to invite 
those individuals.

-Taylor

Re: Naming the new ValueVector Initiative

[Hanifi GUNES]

Awesome! Thanks for heavy lifting Jacques.

2016-01-20 15:30 GMT-08:00 Wes McKinney :

> Fantastic!
>
> Really looking forward to working more with everyone.
>
> Thanks to Jacques for stewarding the process. This is a really important
> step for the ecosystem.
>
> On Wed, Jan 20, 2016 at 3:29 PM, Reynold Xin  wrote:
>
>> Thanks and great job driving this, Jacques!
>>
>>
>> On Wed, Jan 20, 2016 at 3:28 PM, Jacques Nadeau 
>> wrote:
>>
>>> Hey Everyone,
>>>
>>> Good news! The Apache board has approved the Apache Arrow as a new TLP.
>>> I've asked the Apache INFRA team to set up required resources so we can
>>> start moving forward (ML, Git, Website, etc).
>>>
>>> I've started working on a press release to announce the Apache Arrow
>>> project and will circulate a draft shortly. Once the project mailing lists
>>> are established, we can move this thread over there to continue
>>> discussions. They had us do one of change to the proposal during the board
>>> call which was to remove the initial committers (separate from initial
>>> pmc). Once we establish the PMC list, we can immediately add the additional
>>> committers as our first PMC action.
>>>
>>> thanks to everyone!
>>> Jacques
>>>
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>> On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem 
>>> wrote:
>>>
 +1 on a repo for the spec.
 I do have questions as well.
 In particular for the metadata.

 On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:

> On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra 
> wrote:
> >
> >
> > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney 
> wrote:
> >>
> >>
> >> >
> >> > As far as the existing work is concerned, I'm not sure everyone
> is aware
> >> > of
> >> > the C++ code inside of Drill that can represent at least the
> scalar
> >> > types in
> >> > Drill's existing Value Vectors [1]. This is currently used by the
> native
> >> > client written to hook up an ODBC driver.
> >> >
> >>
> >> I have read this code. From my perspective, it would be less work to
> >> collaborate on a self-contained implementation that closely models
> the
> >> Arrow / VV spec that includes builder classes and its own memory
> >> management without coupling to Drill details. I started prototyping
> >> something here (warning: only a few actual days of coding here):
> >>
> >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
> >>
> >> For example, you can see an example constructing an Array or
> >> String (== Array) column in the tests here
> >>
> >>
> >>
> https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
> >>
> >> I've been planning to use this as the basis of a C++ Parquet
> >> reader-writer and the associated Python pandas-like layer which
> >> includes in-memory analytics on Arrow data structures.
> >>
> >> > Parth who is included here has been the primary owner of this C++
> code
> >> > throughout it's life in Drill. Parth, what do you think is the
> best
> >> > strategy
> >> > for managing the C++ code right now? As the C++ build is not tied
> into
> >> > the
> >> > Java one, as I understand it we just run it manually when updates
> are
> >> > made
> >> > there and we need to update ODBC. Would it be disruptive to move
> the
> >> > code to
> >> > the arrow repo? If so, we could include Drill as a submodule in
> the new
> >> > repo, or put Wes's work so far in the Drill repo.
> >>
> >> If we can enumerate the non-Drill-client related parts (i.e. the
> array
> >> accessors and data structures-oriented code) that would make sense
> in
> >> a standalone Arrow library it would be great to start a side
> >> discussion about the design of the C++ reference implementation
> >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
> >> this is a quite urgent for me (intending to deliver a minimally
> viable
> >> pandas-like Arrow + Parquet in Python stack in the next ~3 months)
> it
> >> would be great to do this sooner rather than later.
> >>
> >
> > Most of the code for  Drill C++ Value Vectors is independent of
> Drill -
> > mostly the code upto line 787 in this file -
> >
> https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
> >
> > My thought was to leave the Drill implementation alone and borrow
> copiously
> > from it when convenient for Arrow. Seems like we can still do that
> building
> > on Wes' work.
> >
>
> Makes sense. Speaking of code, would you all like me to set up a
> temporary repo for the specification itself? I already have a few
> questions like how and where to track array null counts.
>

Re: Naming the new ValueVector Initiative

[Wes McKinney]

Fantastic!

Really looking forward to working more with everyone.

Thanks to Jacques for stewarding the process. This is a really important
step for the ecosystem.

On Wed, Jan 20, 2016 at 3:29 PM, Reynold Xin  wrote:

> Thanks and great job driving this, Jacques!
>
>
> On Wed, Jan 20, 2016 at 3:28 PM, Jacques Nadeau 
> wrote:
>
>> Hey Everyone,
>>
>> Good news! The Apache board has approved the Apache Arrow as a new TLP.
>> I've asked the Apache INFRA team to set up required resources so we can
>> start moving forward (ML, Git, Website, etc).
>>
>> I've started working on a press release to announce the Apache Arrow
>> project and will circulate a draft shortly. Once the project mailing lists
>> are established, we can move this thread over there to continue
>> discussions. They had us do one of change to the proposal during the board
>> call which was to remove the initial committers (separate from initial
>> pmc). Once we establish the PMC list, we can immediately add the additional
>> committers as our first PMC action.
>>
>> thanks to everyone!
>> Jacques
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem 
>> wrote:
>>
>>> +1 on a repo for the spec.
>>> I do have questions as well.
>>> In particular for the metadata.
>>>
>>> On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:
>>>
 On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra 
 wrote:
 >
 >
 > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney 
 wrote:
 >>
 >>
 >> >
 >> > As far as the existing work is concerned, I'm not sure everyone is
 aware
 >> > of
 >> > the C++ code inside of Drill that can represent at least the scalar
 >> > types in
 >> > Drill's existing Value Vectors [1]. This is currently used by the
 native
 >> > client written to hook up an ODBC driver.
 >> >
 >>
 >> I have read this code. From my perspective, it would be less work to
 >> collaborate on a self-contained implementation that closely models
 the
 >> Arrow / VV spec that includes builder classes and its own memory
 >> management without coupling to Drill details. I started prototyping
 >> something here (warning: only a few actual days of coding here):
 >>
 >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
 >>
 >> For example, you can see an example constructing an Array or
 >> String (== Array) column in the tests here
 >>
 >>
 >>
 https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
 >>
 >> I've been planning to use this as the basis of a C++ Parquet
 >> reader-writer and the associated Python pandas-like layer which
 >> includes in-memory analytics on Arrow data structures.
 >>
 >> > Parth who is included here has been the primary owner of this C++
 code
 >> > throughout it's life in Drill. Parth, what do you think is the best
 >> > strategy
 >> > for managing the C++ code right now? As the C++ build is not tied
 into
 >> > the
 >> > Java one, as I understand it we just run it manually when updates
 are
 >> > made
 >> > there and we need to update ODBC. Would it be disruptive to move
 the
 >> > code to
 >> > the arrow repo? If so, we could include Drill as a submodule in
 the new
 >> > repo, or put Wes's work so far in the Drill repo.
 >>
 >> If we can enumerate the non-Drill-client related parts (i.e. the
 array
 >> accessors and data structures-oriented code) that would make sense in
 >> a standalone Arrow library it would be great to start a side
 >> discussion about the design of the C++ reference implementation
 >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
 >> this is a quite urgent for me (intending to deliver a minimally
 viable
 >> pandas-like Arrow + Parquet in Python stack in the next ~3 months) it
 >> would be great to do this sooner rather than later.
 >>
 >
 > Most of the code for  Drill C++ Value Vectors is independent of Drill
 -
 > mostly the code upto line 787 in this file -
 >
 https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
 >
 > My thought was to leave the Drill implementation alone and borrow
 copiously
 > from it when convenient for Arrow. Seems like we can still do that
 building
 > on Wes' work.
 >

 Makes sense. Speaking of code, would you all like me to set up a
 temporary repo for the specification itself? I already have a few
 questions like how and where to track array null counts.

 > Wes, let me know if you want to have a quick hangout on this.
 >

 Sure, I'll follow up separately to get something on the calendar.
 Looking forward to connecting!

 > Parth
 >
 >

>>>
>>>
>>>
>>> --
>>> Jul

Re: Naming the new ValueVector Initiative

[Reynold Xin]

Thanks and great job driving this, Jacques!


On Wed, Jan 20, 2016 at 3:28 PM, Jacques Nadeau  wrote:

> Hey Everyone,
>
> Good news! The Apache board has approved the Apache Arrow as a new TLP.
> I've asked the Apache INFRA team to set up required resources so we can
> start moving forward (ML, Git, Website, etc).
>
> I've started working on a press release to announce the Apache Arrow
> project and will circulate a draft shortly. Once the project mailing lists
> are established, we can move this thread over there to continue
> discussions. They had us do one of change to the proposal during the board
> call which was to remove the initial committers (separate from initial
> pmc). Once we establish the PMC list, we can immediately add the additional
> committers as our first PMC action.
>
> thanks to everyone!
> Jacques
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem  wrote:
>
>> +1 on a repo for the spec.
>> I do have questions as well.
>> In particular for the metadata.
>>
>> On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:
>>
>>> On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra 
>>> wrote:
>>> >
>>> >
>>> > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney 
>>> wrote:
>>> >>
>>> >>
>>> >> >
>>> >> > As far as the existing work is concerned, I'm not sure everyone is
>>> aware
>>> >> > of
>>> >> > the C++ code inside of Drill that can represent at least the scalar
>>> >> > types in
>>> >> > Drill's existing Value Vectors [1]. This is currently used by the
>>> native
>>> >> > client written to hook up an ODBC driver.
>>> >> >
>>> >>
>>> >> I have read this code. From my perspective, it would be less work to
>>> >> collaborate on a self-contained implementation that closely models the
>>> >> Arrow / VV spec that includes builder classes and its own memory
>>> >> management without coupling to Drill details. I started prototyping
>>> >> something here (warning: only a few actual days of coding here):
>>> >>
>>> >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
>>> >>
>>> >> For example, you can see an example constructing an Array or
>>> >> String (== Array) column in the tests here
>>> >>
>>> >>
>>> >>
>>> https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
>>> >>
>>> >> I've been planning to use this as the basis of a C++ Parquet
>>> >> reader-writer and the associated Python pandas-like layer which
>>> >> includes in-memory analytics on Arrow data structures.
>>> >>
>>> >> > Parth who is included here has been the primary owner of this C++
>>> code
>>> >> > throughout it's life in Drill. Parth, what do you think is the best
>>> >> > strategy
>>> >> > for managing the C++ code right now? As the C++ build is not tied
>>> into
>>> >> > the
>>> >> > Java one, as I understand it we just run it manually when updates
>>> are
>>> >> > made
>>> >> > there and we need to update ODBC. Would it be disruptive to move the
>>> >> > code to
>>> >> > the arrow repo? If so, we could include Drill as a submodule in the
>>> new
>>> >> > repo, or put Wes's work so far in the Drill repo.
>>> >>
>>> >> If we can enumerate the non-Drill-client related parts (i.e. the array
>>> >> accessors and data structures-oriented code) that would make sense in
>>> >> a standalone Arrow library it would be great to start a side
>>> >> discussion about the design of the C++ reference implementation
>>> >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
>>> >> this is a quite urgent for me (intending to deliver a minimally viable
>>> >> pandas-like Arrow + Parquet in Python stack in the next ~3 months) it
>>> >> would be great to do this sooner rather than later.
>>> >>
>>> >
>>> > Most of the code for  Drill C++ Value Vectors is independent of Drill -
>>> > mostly the code upto line 787 in this file -
>>> >
>>> https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
>>> >
>>> > My thought was to leave the Drill implementation alone and borrow
>>> copiously
>>> > from it when convenient for Arrow. Seems like we can still do that
>>> building
>>> > on Wes' work.
>>> >
>>>
>>> Makes sense. Speaking of code, would you all like me to set up a
>>> temporary repo for the specification itself? I already have a few
>>> questions like how and where to track array null counts.
>>>
>>> > Wes, let me know if you want to have a quick hangout on this.
>>> >
>>>
>>> Sure, I'll follow up separately to get something on the calendar.
>>> Looking forward to connecting!
>>>
>>> > Parth
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Julien
>>
>
>

Re: Naming the new ValueVector Initiative

[Jacques Nadeau]

Hey Everyone,

Good news! The Apache board has approved the Apache Arrow as a new TLP.
I've asked the Apache INFRA team to set up required resources so we can
start moving forward (ML, Git, Website, etc).

I've started working on a press release to announce the Apache Arrow
project and will circulate a draft shortly. Once the project mailing lists
are established, we can move this thread over there to continue
discussions. They had us do one of change to the proposal during the board
call which was to remove the initial committers (separate from initial
pmc). Once we establish the PMC list, we can immediately add the additional
committers as our first PMC action.

thanks to everyone!
Jacques


--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Jan 12, 2016 at 11:03 PM, Julien Le Dem  wrote:

> +1 on a repo for the spec.
> I do have questions as well.
> In particular for the metadata.
>
> On Tue, Jan 12, 2016 at 6:59 PM, Wes McKinney  wrote:
>
>> On Tue, Jan 12, 2016 at 6:21 PM, Parth Chandra  wrote:
>> >
>> >
>> > On Tue, Jan 12, 2016 at 9:57 AM, Wes McKinney  wrote:
>> >>
>> >>
>> >> >
>> >> > As far as the existing work is concerned, I'm not sure everyone is
>> aware
>> >> > of
>> >> > the C++ code inside of Drill that can represent at least the scalar
>> >> > types in
>> >> > Drill's existing Value Vectors [1]. This is currently used by the
>> native
>> >> > client written to hook up an ODBC driver.
>> >> >
>> >>
>> >> I have read this code. From my perspective, it would be less work to
>> >> collaborate on a self-contained implementation that closely models the
>> >> Arrow / VV spec that includes builder classes and its own memory
>> >> management without coupling to Drill details. I started prototyping
>> >> something here (warning: only a few actual days of coding here):
>> >>
>> >> https://github.com/arrow-data/arrow-cpp/tree/master/src/arrow
>> >>
>> >> For example, you can see an example constructing an Array or
>> >> String (== Array) column in the tests here
>> >>
>> >>
>> >>
>> https://github.com/arrow-data/arrow-cpp/blob/master/src/arrow/builder-test.cc#L328
>> >>
>> >> I've been planning to use this as the basis of a C++ Parquet
>> >> reader-writer and the associated Python pandas-like layer which
>> >> includes in-memory analytics on Arrow data structures.
>> >>
>> >> > Parth who is included here has been the primary owner of this C++
>> code
>> >> > throughout it's life in Drill. Parth, what do you think is the best
>> >> > strategy
>> >> > for managing the C++ code right now? As the C++ build is not tied
>> into
>> >> > the
>> >> > Java one, as I understand it we just run it manually when updates are
>> >> > made
>> >> > there and we need to update ODBC. Would it be disruptive to move the
>> >> > code to
>> >> > the arrow repo? If so, we could include Drill as a submodule in the
>> new
>> >> > repo, or put Wes's work so far in the Drill repo.
>> >>
>> >> If we can enumerate the non-Drill-client related parts (i.e. the array
>> >> accessors and data structures-oriented code) that would make sense in
>> >> a standalone Arrow library it would be great to start a side
>> >> discussion about the design of the C++ reference implementation
>> >> (metadata / schemas, IPC, array builders and accessors, etc.). Since
>> >> this is a quite urgent for me (intending to deliver a minimally viable
>> >> pandas-like Arrow + Parquet in Python stack in the next ~3 months) it
>> >> would be great to do this sooner rather than later.
>> >>
>> >
>> > Most of the code for  Drill C++ Value Vectors is independent of Drill -
>> > mostly the code upto line 787 in this file -
>> >
>> https://github.com/apache/drill/blob/master/contrib/native/client/src/include/drill/recordBatch.hpp
>> >
>> > My thought was to leave the Drill implementation alone and borrow
>> copiously
>> > from it when convenient for Arrow. Seems like we can still do that
>> building
>> > on Wes' work.
>> >
>>
>> Makes sense. Speaking of code, would you all like me to set up a
>> temporary repo for the specification itself? I already have a few
>> questions like how and where to track array null counts.
>>
>> > Wes, let me know if you want to have a quick hangout on this.
>> >
>>
>> Sure, I'll follow up separately to get something on the calendar.
>> Looking forward to connecting!
>>
>> > Parth
>> >
>> >
>>
>
>
>
> --
> Julien
>

Re: query hanging in CANCELLATION_REQUEST

[Abdel Hakim Deneche]

I found the issue, I think. Sort was spilling to disk and I run out of disk
space, for some reason this caused Zookeeper to behave incorrectly, I could
still connect to Drill after the first query hang, but once I run a second
query ZK died.

Could this explain the query hang ?

On Tue, Jan 19, 2016 at 1:57 PM, Jacques Nadeau  wrote:

> It sounds like the connection break is not correctly marking an ack fail.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Jan 19, 2016 at 11:47 AM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Ok, I will create a JIRA
> >
> > On Tue, Jan 19, 2016 at 11:45 AM, Hanifi Gunes 
> > wrote:
> >
> > > I had reported this problem sometime last year verbally. I don't
> remember
> > > creating a JIRA though. In general, I dislike this sort of blocking
> calls
> > > anywhere in the execution even though one could argue it simplifies the
> > > code flow.
> > >
> > > A JIRA would be appreciated.
> > >
> > > On Tue, Jan 19, 2016 at 11:10 AM, Abdel Hakim Deneche <
> > > adene...@maprtech.com
> > > > wrote:
> > >
> > > > I was running a query with a hash join that was generating lot's of
> > > > results. I cancelled the query from sqlline then closed it. Now the
> > query
> > > > is stuck in CANCELLATION_REQUEST state.
> > > >
> > > > Looking at jstack it looks like screenRoot is blocked waiting for
> data
> > > sent
> > > > to the client to be acknowledged.
> > > >
> > > > Do we have a JIRA similar to this ?
> > > >
> > > > --
> > > >
> > > > Abdelhakim Deneche
> > > >
> > > > Software Engineer
> > > >
> > > >   
> > > >
> > > >
> > > > Now Available - Free Hadoop On-Demand Training
> > > > <
> > > >
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Re: resource manager proposal -- initial write-ups

[Hanifi Gunes]

Sure. Just did it.

-H+

On Wed, Jan 20, 2016 at 12:14 PM, Neeraja Rentachintala <
nrentachint...@maprtech.com> wrote:

> Hanifi
> thanks for putting this together.
> Seems like the doc has only view rights.
> Can you share it such that we can add comments.
>
> -thanks
>
> On Wed, Jan 20, 2016 at 10:59 AM, Hanifi GUNES 
> wrote:
>
> > Folks,
> >
> > I have been working on designing the new resource manager. I have a
> > *moving*
> > design document living at [1]. Note that this is purely a
> work-in-progress,
> > has a lot of incomplete pieces. In the meantime, however, you can get the
> > broad idea of what we are targeting there as well as dropping your
> feedback
> > on the side for further clarification, suggestions etc.
> >
> >
> > Cheers.
> > -Hanifi
> >
> > 1: https://goo.gl/rpcjVR
> >
>

Re: resource manager proposal -- initial write-ups

[Neeraja Rentachintala]

Hanifi
thanks for putting this together.
Seems like the doc has only view rights.
Can you share it such that we can add comments.

-thanks

On Wed, Jan 20, 2016 at 10:59 AM, Hanifi GUNES 
wrote:

> Folks,
>
> I have been working on designing the new resource manager. I have a
> *moving*
> design document living at [1]. Note that this is purely a work-in-progress,
> has a lot of incomplete pieces. In the meantime, however, you can get the
> broad idea of what we are targeting there as well as dropping your feedback
> on the side for further clarification, suggestions etc.
>
>
> Cheers.
> -Hanifi
>
> 1: https://goo.gl/rpcjVR
>

[GitHub] drill pull request: Drill 2517: Apply directory-based partition pr...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/328#discussion_r50302540
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
 ---
@@ -78,71 +94,68 @@ public PruneScanRule(RelOptRuleOperand operand, String 
id, OptimizerRulesContext
 this.optimizerContext = optimizerContext;
   }
 
-  public static final RelOptRule getFilterOnProject(OptimizerRulesContext 
optimizerRulesContext) {
-return new PruneScanRule(
-RelOptHelper.some(DrillFilterRel.class, 
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
-"PruneScanRule:Filter_On_Project",
-optimizerRulesContext) {
+  private static class DirPruneScanFilterOnProjectRule extends 
PruneScanRule {
+private final boolean calciteLogical;
--- End diff --

You are right. The flag is not needed now. I will refactor the rules. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Drill 2517: Apply directory-based partition pr...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/328#discussion_r50302251
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java
 ---
@@ -84,9 +99,17 @@ public int getMaxHierarchyLevel() {
 
   @Override
   public GroupScan createNewGroupScan(List newFiles) throws 
IOException {
-final FileSelection newSelection = FileSelection.create(null, 
newFiles, getBaseTableLocation());
-final FileGroupScan newScan = 
((FileGroupScan)scanRel.getGroupScan()).clone(newSelection);
-return newScan;
+if (scanRel instanceof DrillScanRel) {
+  final FileSelection newFileSelection = new FileSelection(null, 
newFiles, getBaseTableLocation());
+  final FileGroupScan newScan = 
((FileGroupScan)((DrillScanRel)scanRel).getGroupScan()).clone(newFileSelection);
+  return newScan;
+} else {
+  throw new UnsupportedOperationException("Does not allow to get 
groupScan for EnumerableTableScan");
--- End diff --

The else condition could be anything other than DrillScanRel, so you need 
not mention EnumerableTableScan. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

resource manager proposal -- initial write-ups

[Hanifi GUNES]

Folks,

I have been working on designing the new resource manager. I have a *moving*
design document living at [1]. Note that this is purely a work-in-progress,
has a lot of incomplete pieces. In the meantime, however, you can get the
broad idea of what we are targeting there as well as dropping your feedback
on the side for further clarification, suggestions etc.


Cheers.
-Hanifi

1: https://goo.gl/rpcjVR

[GitHub] drill pull request: Drill 2517: Apply directory-based partition pr...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/328#discussion_r50300305
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/partition/PruneScanRule.java
 ---
@@ -78,71 +94,68 @@ public PruneScanRule(RelOptRuleOperand operand, String 
id, OptimizerRulesContext
 this.optimizerContext = optimizerContext;
   }
 
-  public static final RelOptRule getFilterOnProject(OptimizerRulesContext 
optimizerRulesContext) {
-return new PruneScanRule(
-RelOptHelper.some(DrillFilterRel.class, 
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
-"PruneScanRule:Filter_On_Project",
-optimizerRulesContext) {
+  private static class DirPruneScanFilterOnProjectRule extends 
PruneScanRule {
+private final boolean calciteLogical;
--- End diff --

It's not clear to me why a flag is needed to keep track of Calcite 
logical...ideally this should be handled by the type of the RelNode. We may 
need to do some refactoring of this rule to separate out the ones that apply 
during Calcite logical planning vs. Drill logical planning.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4256: Create HiveConf per HiveStoragePlu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/329


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for a 1.5 release?

[Jinfeng Ni]

I still saw mvn full build failed on linux due to the following unit
test case sometime:

Tests in error:
  TestTopNSchemaChanges.testMissingColumn:206 »  at position 0 column
'`kl1`' mi...

Either we comment out that unit test, or we should fix the test case.
Otherwise, people may see maven build failure from the 1.5.0 release.



On Tue, Jan 19, 2016 at 4:18 PM, Jacques Nadeau  wrote:
> Bumping this thread...
>
> Here are the issues that were mentioned in this thread along with a
> proposed categorization:
>
> Release Blockers
> In-progress Amit https://issues.apache.org/jira/browse/DRILL-4190
> In-progress Amit https://issues.apache.org/jira/browse/DRILL-4196
> Ready to merge Jacques https://issues.apache.org/jira/browse/DRILL-4246
> In-review Jinfeng https://issues.apache.org/jira/browse/DRILL-4256
> In-progress Jacques https://issues.apache.org/jira/browse/DRILL-4278
> Ready to merge Laurent https://issues.apache.org/jira/browse/DRILL-4285
> Nice to Have
> Open Jason/Hakim https://issues.apache.org/jira/browse/DRILL-4247
> In-progress Jason https://issues.apache.org/jira/browse/DRILL-4203
> Open Jacques https://issues.apache.org/jira/browse/DRILL-4266
> Ready to merge Jacques https://issues.apache.org/jira/browse/DRILL-4131
>
> What do others think? Let's try to get the blockers wrapped up in the next
> day or two and start a release vote...
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jan 4, 2016 at 1:48 PM, Jason Altekruse 
> wrote:
>
>> Hello All,
>>
>> With the allocator changes merged and about a month since the last release
>> I think it would be good to start a vote soon. I would like to volunteer to
>> be release manager.
>>
>> I know that there were some issues that were identified after the transfer
>> patch was merged. I think that these issues should be fixed before we cut a
>> release candidate.
>>
>> From looking at the associated JIRAs it looked like there was a possible
>> short term fix just adjusting the max_query_memory_per_node option, and
>> some more involved work to change how we determine the correct time to
>> spill during external sort. I believe it makes sense to make external sort
>> work well with the newly improved memory accounting before cutting a
>> release, but I'm not sure how much work is left to be done there. [1]
>>
>> Please respond with your thoughts on a release soon and any JIRAs you would
>> like to include in the release.
>>
>> [1] - https://issues.apache.org/jira/browse/DRILL-4243
>>
>> Thanks,
>> Jason
>>

[jira] [Created] (DRILL-4291) JDBC Driver regression (DRILL-2482 creeping back in)

[JIRA]

Stefán Baxter created DRILL-4291:


 Summary: JDBC Driver regression (DRILL-2482 creeping back in)
 Key: DRILL-4291
 URL: https://issues.apache.org/jira/browse/DRILL-4291
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 0.5.0
 Environment: Linux / 1.5-SNAPSHOT
Reporter: Stefán Baxter


Hi,

We are using the 1.5-SNAPSHOT version of the JDBC drilver (all) and we seem to 
be getting this old thing:

https://issues.apache.org/jira/browse/DRILL-2482

We are either doing something wrong or this or this is a regression. Has anyone 
else experienced not being able to get nested structures via the latest JDBC 
driver?

(I'm going to pull the lastest from master to be sure this has not been solved)

The error we get when accessing a field containing a sub-structure is :
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Text

at 
oadd.org.apache.drill.exec.util.JsonStringArrayList.(JsonStringArrayList.java:35)
at 
oadd.org.apache.drill.exec.vector.RepeatedVarCharVector$Accessor.getObject(RepeatedVarCharVector.java:293)
at 
oadd.org.apache.drill.exec.vector.RepeatedVarCharVector$Accessor.getObject(RepeatedVarCharVector.java:290)
at 
oadd.org.apache.drill.exec.vector.accessor.GenericAccessor.getObject(GenericAccessor.java:44)
at 
oadd.org.apache.drill.exec.vector.accessor.BoundCheckingAccessor.getObject(BoundCheckingAccessor.java:148)
at 
org.apache.drill.jdbc.impl.TypeConvertingSqlAccessor.getObject(TypeConvertingSqlAccessor.java:795)
at 
org.apache.drill.jdbc.impl.AvaticaDrillSqlAccessor.getObject(AvaticaDrillSqlAccessor.java:179)
at 
oadd.net.hydromatic.avatica.AvaticaResultSet.getObject(AvaticaResultSet.java:351)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.getObject(DrillResultSetImpl.java:420)

Regards,
 -Stefan



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4277) Jdbc Storage Plugin doesn't correctly add metadata for physical serialization

[Jacques Nadeau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau resolved DRILL-4277.
---
Resolution: Fixed

Fixed in 664d34e3c64c3b52e1ed6347cab9ea27a4722a28

> Jdbc Storage Plugin doesn't correctly add metadata for physical serialization
> -
>
> Key: DRILL-4277
> URL: https://issues.apache.org/jira/browse/DRILL-4277
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>
> Example Trace:
> {code}
> Error: SYSTEM ERROR: IllegalStateException: Already had POJO for id
> (java.lang.Integer)
> [com.fasterxml.jackson.annotation.ObjectIdGenerator$IdKey@3372bbe8]*
> *Fragment 2:0*
> *[Error Id: 57494209-04e8-4580-860d-461cf50b41f8 on
> ip-x-x-x-x.ec2.internal:31010]*
> *  (com.fasterxml.jackson.databind.JsonMappingException) Already had POJO
> for id (java.lang.Integer)
> [com.fasterxml.jackson.annotation.ObjectIdGenerator$IdKey@3372bbe8]
> (through reference chain:
> org.apache.drill.exec.physical.config.BroadcastSender["destinations"])*
> *com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath():210*
> *com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath():177*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow():1420*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():351*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1056*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject():264*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1028*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():154*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():126*
> *
> com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():113*
> *
> com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():84*
> *
> com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():132*
> *
> com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize():41*
> *com.fasterxml.jackson.databind.ObjectReader._bindAndClose():1269*
> *com.fasterxml.jackson.databind.ObjectReader.readValue():896*
> *
> org.apache.drill.exec.planner.PhysicalPlanReader.readFragmentOperator():94*
> *org.apache.drill.exec.work.fragment.FragmentExecutor.run():227*
> *org.apache.drill.common.SelfCleaningRunnable.run():38*
> *java.util.concurrent.ThreadPoolExecutor.runWorker():1145*
> *java.util.concurrent.ThreadPoolExecutor$Worker.run():615*
> *java.lang.Thread.run():745*
> *  Caused By (java.lang.IllegalStateException) Already had POJO for id
> (java.lang.Integer)
> [com.fasterxml.jackson.annotation.ObjectIdGenerator$IdKey@3372bbe8]*
> *com.fasterxml.jackson.annotation.SimpleObjectIdResolver.bindItem():20*
> *
> com.fasterxml.jackson.databind.deser.impl.ReadableObjectId.bindItem():66*
> *
> com.fasterxml.jackson.databind.deser.impl.PropertyValueBuffer.handleIdValue():117*
> *
> com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():169*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():349*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1056*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject():264*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1028*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():154*
> *
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():126*
> *
> com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():113*
> *
> com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():84*
> *
> com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():132*
> *
> com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize():41*
> *com.fasterxml.jackson.databind.ObjectReader._bindAndClose():1269*
> *com.fasterxml.jackson.databind.ObjectReader.readValue():896*
> *
> org.apache.drill.exec.planner.PhysicalPlanReader.readFragmentOperator():94*
> *org.apache.drill.exec.work.fragment.FragmentExecutor.run():227*
> *org.apache.drill.common.SelfCleaningRunnable.run():38*
> *java.util.concurrent.ThreadPoolExecutor.runWorker():1145*
> *java.util.concurrent.ThreadPoolExecutor$Worker.run():615*
> *java.lang.Thread.run():745 (state=,code=0)*
> {cod

[GitHub] drill pull request: DRILL-4246: Fix Allocator concurrency bug and ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/323


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4277: Fix issue with JdbcPrel serializat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/326


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4285: Bumps FMPP version to 0.9.15

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/330


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4288) Obsolete generated files under protocol/

[Jacques Nadeau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau resolved DRILL-4288.
---
   Resolution: Fixed
Fix Version/s: 1.5.0

Fixed in 9cc7e116fe6e315eeb8c7bc4ca2cfcb2b8ad7556

> Obsolete generated files under protocol/
> 
>
> Key: DRILL-4288
> URL: https://issues.apache.org/jira/browse/DRILL-4288
> Project: Apache Drill
>  Issue Type: Task
>  Components: Tools, Build & Test
>Reporter: Laurent Goujon
>Priority: Trivial
> Fix For: 1.5.0
>
>
> The protocol module contains the following files directly at the root of the 
> module:
> {noformat}
> protocol/org/apache/drill/exec/proto/BitControlHandshake.java
> protocol/org/apache/drill/exec/proto/BitStatus.java
> protocol/org/apache/drill/exec/proto/FragmentStatus.java
> protocol/org/apache/drill/exec/proto/PlanFragment.java
> protocol/org/apache/drill/exec/proto/RpcType.java
> protocol/org/apache/drill/exec/proto/WorkQueueStatus.java
> {noformat}
> These files are not generated anymore when running {{mvn process-sources -P 
> proto-compile}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4285) maven cannot find Could not find artifact org.freemarker:freemarker:jar:2.3.24-SNAPSHOT in apache.snapshots

[Jacques Nadeau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau resolved DRILL-4285.
---
   Resolution: Fixed
Fix Version/s: 1.5.0

Fixed in 06e93e61d05e0499d980d2dd514e105ba53fa790

> maven cannot find Could not find artifact 
> org.freemarker:freemarker:jar:2.3.24-SNAPSHOT in apache.snapshots
> ---
>
> Key: DRILL-4285
> URL: https://issues.apache.org/jira/browse/DRILL-4285
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
> Environment: MacOS 10.11, Java 7 (1.7.0_80), maven 3.2.x and 3.3.x
>Reporter: Laurent Goujon
>Priority: Minor
> Fix For: 1.5.0
>
>
> When building from master, build fails in component drill-fmpp-maven-plugin, 
> with the following error message:
> {noformat}
> [ERROR] Failed to execute goal on project drill-fmpp-maven-plugin: Could not 
> resolve dependencies for project 
> org.apache.drill.tools:drill-fmpp-maven-plugin:maven-plugin:1.5.0-SNAPSHOT: 
> Could not find artifact org.freemarker:freemarker:jar:2.3.24-SNAPSHOT in 
> apache.snapshots (http://repository.apache.org/snapshots) -> [Help 1]
> {noformat} 
> The project itself doesn't depend directly on the SNAPSHOT version, but FMPP 
> dependency specifies a version range for FreeMarker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4131) Update RPC layer to use child allocators of the RootAllocator rather than using the PooledByteBufAllocatorL directly

[Jacques Nadeau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau resolved DRILL-4131.
---
Resolution: Fixed

Fixed in 412d08f9ffd28b877eab2130f2b3f907c5cd43ca

> Update RPC layer to use child allocators of the RootAllocator rather than 
> using the PooledByteBufAllocatorL directly
> 
>
> Key: DRILL-4131
> URL: https://issues.apache.org/jira/browse/DRILL-4131
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Flow
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4278) Memory leak when using LIMIT

[Jacques Nadeau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau resolved DRILL-4278.
---
   Resolution: Fixed
 Assignee: Jacques Nadeau
Fix Version/s: 1.5.0

Resolved in 43414e1e2731c85a52febaaedebfa2f392dd6d2f

> Memory leak when using LIMIT
> 
>
> Key: DRILL-4278
> URL: https://issues.apache.org/jira/browse/DRILL-4278
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.4.0, 1.5.0
> Environment: OS X
> 0: jdbc:drill:zk=local> select * from sys.version;
> +--+---+-++++
> | version  | commit_id |   
> commit_message|commit_time |
> build_email | build_time |
> +--+---+-++++
> | 1.4.0| 32b871b24c7b69f59a1d2e70f444eed6e599e825  | 
> [maven-release-plugin] prepare release drill-1.4.0  | 08.12.2015 @ 00:24:59 
> PST  | venki.koruka...@gmail.com  | 08.12.2015 @ 01:14:39 PST  |
> +--+---+-++++
> 0: jdbc:drill:zk=local> select * from sys.options where status <> 'DEFAULT';
> +-+---+-+--+--+-+---++
> |name | kind  |  type   |  status  | num_val  | 
> string_val  | bool_val  | float_val  |
> +-+---+-+--+--+-+---++
> | planner.slice_target| LONG  | SYSTEM  | CHANGED  | 10   | null  
>   | null  | null   |
> | planner.width.max_per_node  | LONG  | SYSTEM  | CHANGED  | 5| null  
>   | null  | null   |
> +-+---+-+--+--+-+---++
> 2 rows selected (0.16 seconds)
>Reporter: jean-claude
>Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>
> copy the parquet files in the samples directory so that you have a 12 or so
> $ ls -lha /apache-drill-1.4.0/sample-data/nationsMF/
> nationsMF1.parquet
> nationsMF2.parquet
> nationsMF3.parquet
> create a file with a few thousand lines like these
> select * from dfs.`/Users/jccote/apache-drill-1.4.0/sample-data/nationsMF` 
> limit 500;
> start drill
> $ /apache-drill-1.4.0/bin/drill-embeded
> reduce the slice target size to force drill to use multiple fragment/threads
> jdbc:drill:zk=local> system set planner.slice_target=10;
> now run the list of queries from the file your created above
> jdbc:drill:zk=local> !run /Users/jccote/test-memory-leak-using-limit.sql
> the java heap space keeps going up until the old space is at 100% and 
> eventually you get an OutOfMemoryException in drill
> $ jstat -gccause 86850 5s
>   S0 S1 E  O  M CCSYGC YGCTFGCFGCT 
> GCTLGCC GCC 
>   0.00   0.00 100.00 100.00  98.56  96.71   2279   26.682   240  458.139  
> 484.821 GCLocker Initiated GC Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   242  461.347  
> 488.028 Allocation Failure   Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   245  466.630  
> 493.311 Allocation Failure   Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   247  470.020  
> 496.702 Allocation Failure   Ergonomics  
> If you do the same test but do not use the LIMIT then the memory usage does 
> not go up.
> If you add a where clause so that no results are returned, then the memory 
> usage does not go up.
> Something with the RPC layer?
> Also it seems sensitive to the number of fragments/threads. If you limit it 
> to one fragment/thread the memory usage goes up much slower.
> I have used parquet files and CSV files. In either case the behaviour is the 
> same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4131: Move RPC allocators under Drill's ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/327


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4288: Removes obsolete protobuf-generate...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/332


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4278: Fix issue where WorkspaceConfig wa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/331


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4256: Create HiveConf per HiveStoragePlu...

[jinfengni]

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/329#issuecomment-173229548
  
LGTM. +1



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Drill - question

[Viktor Kvachan]

Good afternoon!
Your Drill is perfect!

But i have some questions, 
how can i select all of columns by FamilyColum, for example

select 
CONVERT_FROM(row_key, 'UTF8') as id, 
CONVERT_FROM(h.basicproperties.link, 'UTF8') as link,
CONVERT_FROM(h.goodproperties.vendorname, 'UTF8') as vendor
h.alternativegoods.*
from hbase.newheureka h
where CONVERT_FROM(h.goodproperties.vendorname, 'UTF8') like '%Lenovo%';

OR

select 
CONVERT_FROM(row_key, 'UTF8') as id, 
CONVERT_FROM(h.basicproperties.link, 'UTF8') as link,
CONVERT_FROM(h.goodproperties.vendorname, 'UTF8') as vendor
h.alternativegoods.link-%%%
from hbase.newheureka h
where CONVERT_FROM(h.goodproperties.vendorname, 'UTF8') like '%Lenovo%';


Thank you
Maybe, if it possible, give me contact or forum where i can ask about queries 
by Drill!


Yours sincerely, 
Viktor Kvachan 

[jira] [Resolved] (DRILL-4290) Run select count(1) from mysql.user,it throws error.Error: PARSE ERROR: Encountered ". user" at line 1

[Jacques Nadeau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau resolved DRILL-4290.
---
Resolution: Not A Problem

User is a reserved word so it needs to be a quoted identifier. Try `user`

> Run select count(1) from mysql.user,it throws error.Error: PARSE ERROR: 
> Encountered ". user" at line 1
> --
>
> Key: DRILL-4290
> URL: https://issues.apache.org/jira/browse/DRILL-4290
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.4.0
> Environment: drill1.4
>Reporter: david_hudavy
>
> I have performing some select commands tests of drill.
> Test case 1:  user is mysql table, it is error. 
> 0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from 
> mysql.user;
> Error: PARSE ERROR: Encountered ". user" at line 1, column 32.
> Was expecting one of:
> 
> "ORDER" ...
> while parsing SQL query:
> select count(1) from mysql.user
>^
> [Error Id: 484485a4-6e0a-45cd-892a-099236319109 on SC-1:31010] (state=,code=0)
> Test case 2:  user is mysql table, i add ` on user table. it is right. 
> 0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from 
> mysql.`user`;
> +-+
> | EXPR$0  |
> +-+
> | 2   |
> +-+
> 1 row selected (0.179 seconds)
> Test case 3: other mysql table,i don't add ' on the table,but it is right.
> 0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from 
> mysql.event;
> +-+
> | EXPR$0  |
> +-+
> | 0   |
> +-+
> 1 row selected (0.128 seconds)
> 0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select * from 
> performance_schema.users;
> +---+--++
> | USER  | CURRENT_CONNECTIONS  | TOTAL_CONNECTIONS  |
> +---+--++
> | null  | 18   | 24 |
> | root  | 5| 130|
> +---+--++
> 2 rows selected (0.117 seconds)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4290) Run select count(1) from mysql.user,it throws error.Error: PARSE ERROR: Encountered ". user" at line 1

[david_hudavy (JIRA)]

david_hudavy created DRILL-4290:
---

 Summary: Run select count(1) from mysql.user,it throws 
error.Error: PARSE ERROR: Encountered ". user" at line 1
 Key: DRILL-4290
 URL: https://issues.apache.org/jira/browse/DRILL-4290
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.4.0
 Environment: drill1.4
Reporter: david_hudavy



I have performing some select commands tests of drill.

Test case 1:  user is mysql table, it is error. 
0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from mysql.user;
Error: PARSE ERROR: Encountered ". user" at line 1, column 32.
Was expecting one of:

"ORDER" ...
while parsing SQL query:
select count(1) from mysql.user
   ^
[Error Id: 484485a4-6e0a-45cd-892a-099236319109 on SC-1:31010] (state=,code=0)

Test case 2:  user is mysql table, i add ` on user table. it is right. 
0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from 
mysql.`user`;
+-+
| EXPR$0  |
+-+
| 2   |
+-+
1 row selected (0.179 seconds)

Test case 3: other mysql table,i don't add ' on the table,but it is right.
0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from mysql.event;
+-+
| EXPR$0  |
+-+
| 0   |
+-+
1 row selected (0.128 seconds)
0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select * from 
performance_schema.users;
+---+--++
| USER  | CURRENT_CONNECTIONS  | TOTAL_CONNECTIONS  |
+---+--++
| null  | 18   | 24 |
| root  | 5| 130|
+---+--++
2 rows selected (0.117 seconds)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)