Re: Create 1.4 Release branch soon?

[Steven Phillips]

I just ran the tests on a linux machine, and did not see this failure. Do
you see it consistently?

On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni  wrote:

> I run mvn full build on linux box against the latest master branch.
> There was one unit test failure. However, on mac, it's successful. Has
> anyone experienced the same?
>
> Failed tests:
>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196 Result
> mismatch.
> Expected:
> Year|Model|Category
> 1999|Venture "Extended Edition"|
> 1999|Venture "Extended Edition, Very Large"|
> Year|Model|Category
> 1999||Venture "Extended Edition"
> 1999||Venture "Extended Edition, Very Large"
>
> Received:
> Year|Model|Category
> 1999||Venture "Extended Edition"
> 1999||Venture "Extended Edition, Very Large"
> Year|Model|Category
> 1999|Venture "Extended Edition"|
> 1999|Venture "Extended Edition, Very Large"|
>  expected:<...Model|Category
> 1999|[Venture "Extended Edition"|
> 1999|Venture "Extended Edition, Very Large"|
> Year|Model|Category
> 1999||Venture "Extended Edition"
> 1999||Venture "Extended Edition, Very Large"]
> > but was:<...Model|Category
> 1999|[|Venture "Extended Edition"
> 1999||Venture "Extended Edition, Very Large"
> Year|Model|Category
> 1999|Venture "Extended Edition"|
> 1999|Venture "Extended Edition, Very Large"|]
> >
>
> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
>
> git log
> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
>
>
> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau  wrote:
> > I think we should roll forward to 1.5-S..
> > On Dec 2, 2015 6:44 PM, "Venki Korukanti" 
> wrote:
> >
> >> 1.4.0 branch is cut and available here:
> >> https://github.com/vkorukanti/drill/tree/1.4.0.
> >>
> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC is
> >> passed?
> >>
> >> Thanks
> >> Venki
> >>
> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
> venki.koruka...@gmail.com
> >> >
> >> wrote:
> >>
> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
> verify. If
> >> > the changes are reviewed lets merge them today. Once the branch is cut
> >> > today, MapR will do the release sanity for next couple of days before
> RC0
> >> > voting goes out. If any issues are found, we still have time to fix
> >> before
> >> > the RC0 voting.
> >> >
> >> > Thanks
> >> > Venki
> >> >
> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau 
> >> wrote:
> >> >
> >> >> Sounds good.
> >> >>
> >> >> --
> >> >> Jacques Nadeau
> >> >> CTO and Co-Founder, Dremio
> >> >>
> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips 
> >> >> wrote:
> >> >>
> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
> >> >> >
> >> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
> >> >> venki.koruka...@gmail.com
> >> >> > >
> >> >> > wrote:
> >> >> >
> >> >> > > For DRILL-4145: ran the regression suite which includes customer
> and
> >> >> > > extended tests. No regressions found.
> >> >> > >
> >> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
> >> >> > venki.koruka...@gmail.com
> >> >> > > >
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Sure, I will trigger a regression run with DRILL-4145.
> >> >> > > >
> >> >> > > > Thanks
> >> >> > > > Venki
> >> >> > > >
> >> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
> >> jacq...@dremio.com>
> >> >> > > wrote:
> >> >> > > >
> >> >> > > >> I believe 4109 is ready but if I understand correctly, it
> can't
> >> go
> >> >> in
> >> >> > > >> without 4125. Amit said he needs another hour for that.
> >> >> > > >>
> >> >> > > >> On 4145: Venki, can someone on your side running any
> >> >> extended/customer
> >> >> > > >> tests you have against this to make sure that it doesn't cause
> >> any
> >> >> > > >> regressions. We've run on our side without issue but I'd like
> to
> >> >> get
> >> >> > as
> >> >> > > >> broad of coverage as possible to ensure no issues.
> >> >> > > >>
> >> >> > > >> --
> >> >> > > >> Jacques Nadeau
> >> >> > > >> CTO and Co-Founder, Dremio
> >> >> > > >>
> >> >> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
> >> >> > > >> venki.koruka...@gmail.com>
> >> >> > > >> wrote:
> >> >> > > >>
> >> >> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any
> idea
> >> >> when
> >> >> > > the
> >> >> > > >> > following JIRAs are going to be ready to commit?
> >> >> > > >> >
> >> >> > > >> > DRILL-4109 (in review)
> >> >> > > >> > DRILL-4125
> >> >> > > >> > DRILL-4145 (in review)
> >> >> > > >> >
> >> >> > > >> > Jinfeng and I discussed about including metastore caching
> and
> >> >> > decided
> >> >> > > to
> >> >> > > >> > move it to 1.5.
> >> >> > > >> >
> >> >> > > >> > Lets make 5pm as the cutoff time for 1.4 branch.
> >> >> > > >> >
> >> >> > > >> > Thanks
> >> >> > > >> > Venki
> >> >> > > >> >
> >> >> > > >> > On Wed, Dec 2, 2015 at 11:42 AM, Julien Le Dem <
> >> >> jul...@dremio.com>
> >> >> > > >> wrote:
> >> >> > > >> >
> >> >> > > >> > > DRILL-4124 is ready to merge:
> >> >> > > >> https://github.com/apache/drill/

Re: Naming the new ValueVector Initiative

[Marcel Kornacker]

Just added my vote.

On Thu, Dec 3, 2015 at 12:51 PM, Wes McKinney  wrote:
> Shall we call the voting closed? Any last stragglers?
>
> On Tue, Dec 1, 2015 at 5:39 PM, Ted Dunning  wrote:
>>
>> Apache can handle this if we set the groundwork in place.
>>
>> Also, Twitter's lawyers work for Twitter, not for Apache. As such, their
>> opinions can't be taken by Apache as legal advice.  There are issues of
>> privilege, conflict of interest and so on.
>>
>>
>>
>> On Wed, Dec 2, 2015 at 7:51 AM, Alex Levenson 
>> wrote:
>>>
>>> I can ask about whether Twitter's lawyers can help out -- is that
>>> something we need? Or is that something apache helps out with in the next
>>> step?
>>>
>>> On Mon, Nov 30, 2015 at 9:32 PM, Julian Hyde  wrote:

 +1 to have a vote tomorrow.

 Assuming that Vector is out of play, I just did a quick search for the
 top 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
 sourceforge, open hub, trademarkia, and on google. There are no trademarks
 for these in similar subject areas. There is a moderately active project
 called “joist” [1].

 I will point out that “Apache Arrow” has native-american connotations
 that we may or may not want to live with (just ask the Washington Redskins
 how they feel about their name).

 If someone would like to vet other names, use the links on
 https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill out
 column C in the spreadsheet.

 Julian

 [1] https://github.com/stephenh/joist


 On Nov 30, 2015, at 7:01 PM, Jacques Nadeau  wrote:

 +1

 --
 Jacques Nadeau
 CTO and Co-Founder, Dremio

 On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney  wrote:

 Should we have a last call for votes, closing EOD tomorrow (Tuesday)? I
 missed this for a few days last week with holiday travel.

 On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
 wrote:

 Consulting a lawyer is part of the Apache branding process but the first
 stage is to gather a list of potential conflicts -
 https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an example.

 The other part, frankly, is to pick your battles.

 A year or so ago Actian re-branded Vectorwise as Vector.

 http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
 Given that it is an analytic database in the Hadoop space I think that is
 as close to a “direct hit” as it gets. I don’t think we need a lawyer to
 tell us that. Certainly it makes sense to look for conflicts for the
 other
 alternatives before consulting lawyers.

 Julian




 On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
 wrote:



 On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
 wrote:

 Ok guys,

 I don't think anyone is doing a thorough analysis of viaability. I did a
 quick glance and the top one (Vector) seems like it would have an issue
 with conflict of an Actian product. The may be fine. Let's do a second
 phase vote.


 I'm assuming you mean Vectorwise?

 Before we do anything else, could we have a lawyer look into this? Last
 time around that I remember (Parquet), Twitter's lawyers did a good job
 of
 weeding out the potential trademark violations.

 Alex, could Twitter get involved this time around as well?



 Pick your top 3 (1,2,3 with 3 being top preference)

 Let's get this done by Friday and then we can do a podling name search
 starting with the top one.

 Link again:



 https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1

 thanks


 --
 Jacques Nadeau
 CTO and Co-Founder, Dremio

 On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
 wrote:

 Ok, it looks like we have a candidate list (we actually got 11 since
 there was a three-way tie for ninth place):

 VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
 Next we need to do trademark searches on each of these to see whether
 we're likely to have success. I've moved candidates to a second tab:



 https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532

 Anybody want to give a hand in analyzing potential conflicts?



 --
 Jacques Nadeau
 CTO and Co-Founder, Dremio

 On Mon, Nov 16, 2015 at 12:10 PM, Jacques Nadeau 
 wrote:

 Everybody should pick their ten favorites using the numbers 1 to 10.

 10 is most preferred



 --
 Jacques Nadeau
 CTO and Co-Founder, Dremio

 On Mon, Nov 16, 2015 at 10:17 AM, Ted Dunning 
 wrote:


 Single vote for most preferred?

 Sin

[jira] [Created] (DRILL-4156) Parquet group converter cannot cope with ENUM original type values

[Chris Jansen (JIRA)]

Chris Jansen created DRILL-4156:
---

 Summary: Parquet group converter cannot cope with ENUM original 
type values 
 Key: DRILL-4156
 URL: https://issues.apache.org/jira/browse/DRILL-4156
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.3.0
Reporter: Chris Jansen


When importing parquet data written using a Thrift schema, Drill cannot cope 
with ENUMs. This was fixed elsewhere in DRILL-1775 but not in the 
DrillParquetGroupConverter class ([see mailing list 
thread|https://mail-archives.apache.org/mod_mbox/drill-user/201508.mbox/%3CCAK55V=+VjPAehMiJ8U=crh182k4zvhgfxzfwznaa9edwzwg...@mail.gmail.com%3E]).

It appears that the ParquetToDrillTypeConverter class was updated to use a 
varbinary as it's default behaviour, but the DrillParquetGroupConverter class 
was not updated to do the same. 

{code}
Caused by: java.lang.UnsupportedOperationException: Unsupported type ENUM
at 
org.apache.drill.exec.store.parquet2.DrillParquetGroupConverter.getConverterForType(DrillParquetGroupConverter.java:249)
at 
org.apache.drill.exec.store.parquet2.DrillParquetGroupConverter.(DrillParquetGroupConverter.java:154)
at 
org.apache.drill.exec.store.parquet2.DrillParquetGroupConverter.(DrillParquetGroupConverter.java:147)
at 
org.apache.drill.exec.store.parquet2.DrillParquetGroupConverter.(DrillParquetGroupConverter.java:147)
at 
org.apache.drill.exec.store.parquet2.DrillParquetGroupConverter.(DrillParquetGroupConverter.java:147)
at 
org.apache.drill.exec.store.parquet2.DrillParquetRecordMaterializer.(DrillParquetRecordMaterializer.java:40)
at 
org.apache.drill.exec.store.parquet2.DrillParquetReader.setup(DrillParquetReader.java:267)
... 14 more
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4157) Add LIMIT push down rule for relational database sources

[daniel (JIRA)]

daniel created DRILL-4157:
-

 Summary: Add LIMIT push down rule for relational database sources
 Key: DRILL-4157
 URL: https://issues.apache.org/jira/browse/DRILL-4157
 Project: Apache Drill
  Issue Type: New Feature
  Components: Storage - Other
Reporter: daniel


Hi,

It will be very useful if queries with LIMIT or OFFSET clauses could be 
translated to native SQL when using relational databases as input.

Supose that I have a mysql connection configured. Right now, if I execute:

select * from mysql.schema.table LIMIT 1

This will be translated into:

select * from schema.table

And executed in the relational database server. As you can see, all the rows 
will be retrieved from the table. Then, apache drill will filter the result and 
show only one row.

The problem is that this is a huge performance penalty that can be avoided 
translating the query having this into account.


I'm not quite sure if this is a mysql problem or all the relational database 
sources share the same limitation. So, excuse me if I open this issue as a 
feature request.

Regards



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Create 1.4 Release branch soon?

[Jinfeng Ni]

I run twice and hit the same error.


On Thu, Dec 3, 2015 at 12:10 AM, Steven Phillips  wrote:
> I just ran the tests on a linux machine, and did not see this failure. Do
> you see it consistently?
>
> On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni  wrote:
>
>> I run mvn full build on linux box against the latest master branch.
>> There was one unit test failure. However, on mac, it's successful. Has
>> anyone experienced the same?
>>
>> Failed tests:
>>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196 Result
>> mismatch.
>> Expected:
>> Year|Model|Category
>> 1999|Venture "Extended Edition"|
>> 1999|Venture "Extended Edition, Very Large"|
>> Year|Model|Category
>> 1999||Venture "Extended Edition"
>> 1999||Venture "Extended Edition, Very Large"
>>
>> Received:
>> Year|Model|Category
>> 1999||Venture "Extended Edition"
>> 1999||Venture "Extended Edition, Very Large"
>> Year|Model|Category
>> 1999|Venture "Extended Edition"|
>> 1999|Venture "Extended Edition, Very Large"|
>>  expected:<...Model|Category
>> 1999|[Venture "Extended Edition"|
>> 1999|Venture "Extended Edition, Very Large"|
>> Year|Model|Category
>> 1999||Venture "Extended Edition"
>> 1999||Venture "Extended Edition, Very Large"]
>> > but was:<...Model|Category
>> 1999|[|Venture "Extended Edition"
>> 1999||Venture "Extended Edition, Very Large"
>> Year|Model|Category
>> 1999|Venture "Extended Edition"|
>> 1999|Venture "Extended Edition, Very Large"|]
>> >
>>
>> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
>>
>> git log
>> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
>>
>>
>> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau  wrote:
>> > I think we should roll forward to 1.5-S..
>> > On Dec 2, 2015 6:44 PM, "Venki Korukanti" 
>> wrote:
>> >
>> >> 1.4.0 branch is cut and available here:
>> >> https://github.com/vkorukanti/drill/tree/1.4.0.
>> >>
>> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC is
>> >> passed?
>> >>
>> >> Thanks
>> >> Venki
>> >>
>> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
>> venki.koruka...@gmail.com
>> >> >
>> >> wrote:
>> >>
>> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
>> verify. If
>> >> > the changes are reviewed lets merge them today. Once the branch is cut
>> >> > today, MapR will do the release sanity for next couple of days before
>> RC0
>> >> > voting goes out. If any issues are found, we still have time to fix
>> >> before
>> >> > the RC0 voting.
>> >> >
>> >> > Thanks
>> >> > Venki
>> >> >
>> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau 
>> >> wrote:
>> >> >
>> >> >> Sounds good.
>> >> >>
>> >> >> --
>> >> >> Jacques Nadeau
>> >> >> CTO and Co-Founder, Dremio
>> >> >>
>> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips 
>> >> >> wrote:
>> >> >>
>> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
>> >> >> >
>> >> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
>> >> >> venki.koruka...@gmail.com
>> >> >> > >
>> >> >> > wrote:
>> >> >> >
>> >> >> > > For DRILL-4145: ran the regression suite which includes customer
>> and
>> >> >> > > extended tests. No regressions found.
>> >> >> > >
>> >> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
>> >> >> > venki.koruka...@gmail.com
>> >> >> > > >
>> >> >> > > wrote:
>> >> >> > >
>> >> >> > > > Sure, I will trigger a regression run with DRILL-4145.
>> >> >> > > >
>> >> >> > > > Thanks
>> >> >> > > > Venki
>> >> >> > > >
>> >> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
>> >> jacq...@dremio.com>
>> >> >> > > wrote:
>> >> >> > > >
>> >> >> > > >> I believe 4109 is ready but if I understand correctly, it
>> can't
>> >> go
>> >> >> in
>> >> >> > > >> without 4125. Amit said he needs another hour for that.
>> >> >> > > >>
>> >> >> > > >> On 4145: Venki, can someone on your side running any
>> >> >> extended/customer
>> >> >> > > >> tests you have against this to make sure that it doesn't cause
>> >> any
>> >> >> > > >> regressions. We've run on our side without issue but I'd like
>> to
>> >> >> get
>> >> >> > as
>> >> >> > > >> broad of coverage as possible to ensure no issues.
>> >> >> > > >>
>> >> >> > > >> --
>> >> >> > > >> Jacques Nadeau
>> >> >> > > >> CTO and Co-Founder, Dremio
>> >> >> > > >>
>> >> >> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
>> >> >> > > >> venki.koruka...@gmail.com>
>> >> >> > > >> wrote:
>> >> >> > > >>
>> >> >> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any
>> idea
>> >> >> when
>> >> >> > > the
>> >> >> > > >> > following JIRAs are going to be ready to commit?
>> >> >> > > >> >
>> >> >> > > >> > DRILL-4109 (in review)
>> >> >> > > >> > DRILL-4125
>> >> >> > > >> > DRILL-4145 (in review)
>> >> >> > > >> >
>> >> >> > > >> > Jinfeng and I discussed about including metastore caching
>> and
>> >> >> > decided
>> >> >> > > to
>> >> >> > > >> > move it to 1.5.
>> >> >> > > >> >
>> >> >> > > >> > Lets make 5pm as the cutoff time for 1.4 branch.
>> >> >> > > >> >
>> >> >> > > >> > Thanks
>> >> >> > >

Re: Can we pass the #skipped records with RecordBatch?

[Zelaine Fong]

Yes, it would be great to get your thoughts so we can assess the scope of
what's involved.

Thanks.

-- Zelaine

On Wed, Dec 2, 2015 at 7:29 PM, Jacques Nadeau  wrote:

> Definitely agree that we shouldn't boil the ocean.  That said, I don't
> think we should make RecordBatch interface changes without deliberate
> design. Same for RPC protocol changes. Part of my internal struggle with
> the warning patch is exactly this lack of broader design. I think this is
> especially true given the drive to supports backwards compatibility.
>
> I don't think we're talking about a massive undertaking. I'll try to write
> up some thoughts later this week to get the ball rolling. Sound good?
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> +1 on having a framework.
> OTOH, as with the warnings implementation, we might want to go ahead with a
> simpler implementation while we get a more generic framework design in
> place.
>
> Jacques, do you have any preliminary thoughts on the framework?
>
> On Tue, Dec 1, 2015 at 2:08 PM, Julian Hyde  wrote:
>
> > +1 for a sideband mechanism.
> >
> > Sideband can also allow correlated restart of sub-queries.
> >
> > In sideband use cases you described, the messages ran in the opposite
> > direction to the data. Would the sideband also run in the same direction
> as
> > the data? If so it could carry warnings, rejected rows, progress
> > indications, and (for online aggregation[1]) notifications that a better
> > approximate query result is available.
> >
> > Julian
> >
> > [1] https://en.wikipedia.org/wiki/Online_aggregation
> >
> >
> >
> > > On Dec 1, 2015, at 1:51 PM, Jacques Nadeau  wrote:
> > >
> > > This seems like a form of sideband communication. I think we should
> have
> > a
> > > framework for this type of thing in general rather than a one-off for
> > this
> > > particular need. Other forms of sideband might be small table
> bloomfilter
> > > generation and pushdown into hbase, separate file
> assignment/partitioning
> > > providers balancing/generating scanner workloads, statistics generation
> > for
> > > adaptive execution, etc.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Tue, Dec 1, 2015 at 11:35 AM, Hsuan Yi Chu 
> > wrote:
> > >
> > >> I am trying to deal with the following scenario:
> > >>
> > >> A bunch of minor fragments are doing things in parallel. Each of them
> > could
> > >> skip some records. Since the downstream minor fragment needs to know
> the
> > >> sum of skipped-record-counts (in order to just display or see if the
> > number
> > >> exceeds the threshold) in the upstreams, each upstream minor fragment
> > needs
> > >> to pass this scalar with RecordBatch.
> > >>
> > >> Since this seems impacting the protocol of RecordBatch, I am looking
> for
> > >> some advice here.
> > >>
> > >> Thanks.
> > >>
> >
> >
>

Re: Create 1.4 Release branch soon?

[Jinfeng Ni]

I switch to a different linux box, and hit the same error. So, seems
it's consistent from what I tried.


On Thu, Dec 3, 2015 at 7:58 AM, Jinfeng Ni  wrote:
> I run twice and hit the same error.
>
>
> On Thu, Dec 3, 2015 at 12:10 AM, Steven Phillips  wrote:
>> I just ran the tests on a linux machine, and did not see this failure. Do
>> you see it consistently?
>>
>> On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni  wrote:
>>
>>> I run mvn full build on linux box against the latest master branch.
>>> There was one unit test failure. However, on mac, it's successful. Has
>>> anyone experienced the same?
>>>
>>> Failed tests:
>>>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196 Result
>>> mismatch.
>>> Expected:
>>> Year|Model|Category
>>> 1999|Venture "Extended Edition"|
>>> 1999|Venture "Extended Edition, Very Large"|
>>> Year|Model|Category
>>> 1999||Venture "Extended Edition"
>>> 1999||Venture "Extended Edition, Very Large"
>>>
>>> Received:
>>> Year|Model|Category
>>> 1999||Venture "Extended Edition"
>>> 1999||Venture "Extended Edition, Very Large"
>>> Year|Model|Category
>>> 1999|Venture "Extended Edition"|
>>> 1999|Venture "Extended Edition, Very Large"|
>>>  expected:<...Model|Category
>>> 1999|[Venture "Extended Edition"|
>>> 1999|Venture "Extended Edition, Very Large"|
>>> Year|Model|Category
>>> 1999||Venture "Extended Edition"
>>> 1999||Venture "Extended Edition, Very Large"]
>>> > but was:<...Model|Category
>>> 1999|[|Venture "Extended Edition"
>>> 1999||Venture "Extended Edition, Very Large"
>>> Year|Model|Category
>>> 1999|Venture "Extended Edition"|
>>> 1999|Venture "Extended Edition, Very Large"|]
>>> >
>>>
>>> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
>>>
>>> git log
>>> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
>>>
>>>
>>> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau  wrote:
>>> > I think we should roll forward to 1.5-S..
>>> > On Dec 2, 2015 6:44 PM, "Venki Korukanti" 
>>> wrote:
>>> >
>>> >> 1.4.0 branch is cut and available here:
>>> >> https://github.com/vkorukanti/drill/tree/1.4.0.
>>> >>
>>> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC is
>>> >> passed?
>>> >>
>>> >> Thanks
>>> >> Venki
>>> >>
>>> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
>>> venki.koruka...@gmail.com
>>> >> >
>>> >> wrote:
>>> >>
>>> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
>>> verify. If
>>> >> > the changes are reviewed lets merge them today. Once the branch is cut
>>> >> > today, MapR will do the release sanity for next couple of days before
>>> RC0
>>> >> > voting goes out. If any issues are found, we still have time to fix
>>> >> before
>>> >> > the RC0 voting.
>>> >> >
>>> >> > Thanks
>>> >> > Venki
>>> >> >
>>> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau 
>>> >> wrote:
>>> >> >
>>> >> >> Sounds good.
>>> >> >>
>>> >> >> --
>>> >> >> Jacques Nadeau
>>> >> >> CTO and Co-Founder, Dremio
>>> >> >>
>>> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips 
>>> >> >> wrote:
>>> >> >>
>>> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
>>> >> >> >
>>> >> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
>>> >> >> venki.koruka...@gmail.com
>>> >> >> > >
>>> >> >> > wrote:
>>> >> >> >
>>> >> >> > > For DRILL-4145: ran the regression suite which includes customer
>>> and
>>> >> >> > > extended tests. No regressions found.
>>> >> >> > >
>>> >> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
>>> >> >> > venki.koruka...@gmail.com
>>> >> >> > > >
>>> >> >> > > wrote:
>>> >> >> > >
>>> >> >> > > > Sure, I will trigger a regression run with DRILL-4145.
>>> >> >> > > >
>>> >> >> > > > Thanks
>>> >> >> > > > Venki
>>> >> >> > > >
>>> >> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
>>> >> jacq...@dremio.com>
>>> >> >> > > wrote:
>>> >> >> > > >
>>> >> >> > > >> I believe 4109 is ready but if I understand correctly, it
>>> can't
>>> >> go
>>> >> >> in
>>> >> >> > > >> without 4125. Amit said he needs another hour for that.
>>> >> >> > > >>
>>> >> >> > > >> On 4145: Venki, can someone on your side running any
>>> >> >> extended/customer
>>> >> >> > > >> tests you have against this to make sure that it doesn't cause
>>> >> any
>>> >> >> > > >> regressions. We've run on our side without issue but I'd like
>>> to
>>> >> >> get
>>> >> >> > as
>>> >> >> > > >> broad of coverage as possible to ensure no issues.
>>> >> >> > > >>
>>> >> >> > > >> --
>>> >> >> > > >> Jacques Nadeau
>>> >> >> > > >> CTO and Co-Founder, Dremio
>>> >> >> > > >>
>>> >> >> > > >> On Wed, Dec 2, 2015 at 11:52 AM, Venki Korukanti <
>>> >> >> > > >> venki.koruka...@gmail.com>
>>> >> >> > > >> wrote:
>>> >> >> > > >>
>>> >> >> > > >> > Sure, we can include DRILL-4124. I will merge it soon. Any
>>> idea
>>> >> >> when
>>> >> >> > > the
>>> >> >> > > >> > following JIRAs are going to be ready to commit?
>>> >> >> > > >> >
>>> >> >> > > >> > DRILL-4109 (in review)
>>> >> >> > > >> > DRILL-4125
>>> >> >> > > >> > DRILL-4145 (in review)

Re: Naming the new ValueVector Initiative

[Jacques Nadeau]

I think we can call the voting closed. Top vote getters:

Apache Arrow (17)
Apache Herringbone (9)
Apache Joist (8)
Apache Colbuf (8)

I'll up a PODLINGNAMESEARCH-* shortly for Arrow.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Thu, Dec 3, 2015 at 1:23 AM, Marcel Kornacker 
wrote:

> Just added my vote.
>
> On Thu, Dec 3, 2015 at 12:51 PM, Wes McKinney  wrote:
> > Shall we call the voting closed? Any last stragglers?
> >
> > On Tue, Dec 1, 2015 at 5:39 PM, Ted Dunning 
> wrote:
> >>
> >> Apache can handle this if we set the groundwork in place.
> >>
> >> Also, Twitter's lawyers work for Twitter, not for Apache. As such, their
> >> opinions can't be taken by Apache as legal advice.  There are issues of
> >> privilege, conflict of interest and so on.
> >>
> >>
> >>
> >> On Wed, Dec 2, 2015 at 7:51 AM, Alex Levenson  >
> >> wrote:
> >>>
> >>> I can ask about whether Twitter's lawyers can help out -- is that
> >>> something we need? Or is that something apache helps out with in the
> next
> >>> step?
> >>>
> >>> On Mon, Nov 30, 2015 at 9:32 PM, Julian Hyde  wrote:
> 
>  +1 to have a vote tomorrow.
> 
>  Assuming that Vector is out of play, I just did a quick search for the
>  top 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
>  sourceforge, open hub, trademarkia, and on google. There are no
> trademarks
>  for these in similar subject areas. There is a moderately active
> project
>  called “joist” [1].
> 
>  I will point out that “Apache Arrow” has native-american connotations
>  that we may or may not want to live with (just ask the Washington
> Redskins
>  how they feel about their name).
> 
>  If someone would like to vet other names, use the links on
>  https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill
> out
>  column C in the spreadsheet.
> 
>  Julian
> 
>  [1] https://github.com/stephenh/joist
> 
> 
>  On Nov 30, 2015, at 7:01 PM, Jacques Nadeau 
> wrote:
> 
>  +1
> 
>  --
>  Jacques Nadeau
>  CTO and Co-Founder, Dremio
> 
>  On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney 
> wrote:
> 
>  Should we have a last call for votes, closing EOD tomorrow (Tuesday)?
> I
>  missed this for a few days last week with holiday travel.
> 
>  On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
>  wrote:
> 
>  Consulting a lawyer is part of the Apache branding process but the
> first
>  stage is to gather a list of potential conflicts -
>  https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an
> example.
> 
>  The other part, frankly, is to pick your battles.
> 
>  A year or so ago Actian re-branded Vectorwise as Vector.
> 
> 
> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
>  Given that it is an analytic database in the Hadoop space I think
> that is
>  as close to a “direct hit” as it gets. I don’t think we need a lawyer
> to
>  tell us that. Certainly it makes sense to look for conflicts for the
>  other
>  alternatives before consulting lawyers.
> 
>  Julian
> 
> 
> 
> 
>  On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
>  wrote:
> 
> 
> 
>  On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
>  wrote:
> 
>  Ok guys,
> 
>  I don't think anyone is doing a thorough analysis of viaability. I
> did a
>  quick glance and the top one (Vector) seems like it would have an
> issue
>  with conflict of an Actian product. The may be fine. Let's do a second
>  phase vote.
> 
> 
>  I'm assuming you mean Vectorwise?
> 
>  Before we do anything else, could we have a lawyer look into this?
> Last
>  time around that I remember (Parquet), Twitter's lawyers did a good
> job
>  of
>  weeding out the potential trademark violations.
> 
>  Alex, could Twitter get involved this time around as well?
> 
> 
> 
>  Pick your top 3 (1,2,3 with 3 being top preference)
> 
>  Let's get this done by Friday and then we can do a podling name search
>  starting with the top one.
> 
>  Link again:
> 
> 
> 
> 
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
> 
>  thanks
> 
> 
>  --
>  Jacques Nadeau
>  CTO and Co-Founder, Dremio
> 
>  On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
>  wrote:
> 
>  Ok, it looks like we have a candidate list (we actually got 11 since
>  there was a three-way tie for ninth place):
> 
>  VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpulsevictor
>  Next we need to do trademark searches on each of these to see whether
>  we're likely to have success. I've moved candidates to a second tab:
> 
> 
> 
> 
> https://docs.google.com/spre

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46585304
  
--- Diff: common/src/main/java/org/apache/drill/common/HistoricalLog.java 
---
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common;
+
+import java.util.Arrays;
+import java.util.LinkedList;
+
+import org.slf4j.Logger;
+
+/**
+ * Utility class that can be used to log activity within a class
+ * for later logging and debugging. Supports recording events and
+ * recording the stack at the time they occur.
+ */
+public class HistoricalLog {
+  private static class Event {
+private final String note; // the event text
+private final StackTrace stackTrace; // where the event occurred
+
+public Event(final String note) {
+  this.note = note;
+  stackTrace = new StackTrace();
+}
+  }
+
+  private final LinkedList history = new LinkedList<>();
+  private final String idString; // the formatted id string
+  private Event firstEvent; // the first stack trace recorded
+  private final int limit; // the limit on the number of events kept
+
+  /**
+   * Constructor. The format string will be formatted and have its 
arguments
+   * substituted at the time this is called.
+   *
+   * @param idStringFormat {@link String#format} format string that can be 
used
+   * to identify this object in a log. Including some kind of unique 
identifier
+   * that can be associated with the object instance is best.
+   * @param args for the format string, or nothing if none are required
+   */
+  public HistoricalLog(final String idStringFormat, Object... args) {
+this(Integer.MAX_VALUE, idStringFormat, args);
+  }
+
+  /**
+   * Constructor. The format string will be formatted and have its 
arguments
+   * substituted at the time this is called.
+   *
+   * This form supports the specification of a limit that will limit the
+   * number of historical entries kept (which keeps down the amount of 
memory
+   * used). With the limit, the first entry made is always kept (under the
+   * assumption that this is the creation site of the object, which is 
usually
+   * interesting), and then up to the limit number of entries are kept 
after that.
+   * Each time a new entry is made, the oldest that is not the first is 
dropped.
+   * 
+   *
+   * @param limit the maximum number of historical entries that will be 
kept,
+   *   not including the first entry made
+   * @param idStringFormat {@link String#format} format string that can be 
used
+   * to identify this object in a log. Including some kind of unique 
identifier
+   * that can be associated with the object instance is best.
+   * @param args for the format string, or nothing if none are required
+   */
+  public HistoricalLog(final int limit, final String idStringFormat, 
Object... args) {
+this.limit = limit;
+this.idString = String.format(idStringFormat, args);
+  }
+
+  /**
+   * Record an event. Automatically captures the stack trace at the time 
this is
+   * called. The format string will be formatted and have its arguments 
substituted
+   * at the time this is called.
+   *
+   * @param noteFormat {@link String#format} format string that describes 
the event
+   * @param args for the format string, or nothing if none are required
+   */
+  public synchronized void recordEvent(final String noteFormat, Object... 
args) {
--- End diff --

would it make sense to also store a timestamp for each event ? This would 
help debugging sliced buffers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.

Re: Create 1.4 Release branch soon?

[Steven Phillips]

Okay, after looking at it more closely, it looks like its an ordering
problem. We should rewrite the tests using the test framework, and be sure
to set the validation as unordered.

I can take care of this, but won't get to it until after lunch. If someone
else wants to do it now in order to get an RC out sooner, feel free.

On Thu, Dec 3, 2015 at 8:39 AM, Jinfeng Ni  wrote:

> I switch to a different linux box, and hit the same error. So, seems
> it's consistent from what I tried.
>
>
> On Thu, Dec 3, 2015 at 7:58 AM, Jinfeng Ni  wrote:
> > I run twice and hit the same error.
> >
> >
> > On Thu, Dec 3, 2015 at 12:10 AM, Steven Phillips 
> wrote:
> >> I just ran the tests on a linux machine, and did not see this failure.
> Do
> >> you see it consistently?
> >>
> >> On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni 
> wrote:
> >>
> >>> I run mvn full build on linux box against the latest master branch.
> >>> There was one unit test failure. However, on mac, it's successful. Has
> >>> anyone experienced the same?
> >>>
> >>> Failed tests:
> >>>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196 Result
> >>> mismatch.
> >>> Expected:
> >>> Year|Model|Category
> >>> 1999|Venture "Extended Edition"|
> >>> 1999|Venture "Extended Edition, Very Large"|
> >>> Year|Model|Category
> >>> 1999||Venture "Extended Edition"
> >>> 1999||Venture "Extended Edition, Very Large"
> >>>
> >>> Received:
> >>> Year|Model|Category
> >>> 1999||Venture "Extended Edition"
> >>> 1999||Venture "Extended Edition, Very Large"
> >>> Year|Model|Category
> >>> 1999|Venture "Extended Edition"|
> >>> 1999|Venture "Extended Edition, Very Large"|
> >>>  expected:<...Model|Category
> >>> 1999|[Venture "Extended Edition"|
> >>> 1999|Venture "Extended Edition, Very Large"|
> >>> Year|Model|Category
> >>> 1999||Venture "Extended Edition"
> >>> 1999||Venture "Extended Edition, Very Large"]
> >>> > but was:<...Model|Category
> >>> 1999|[|Venture "Extended Edition"
> >>> 1999||Venture "Extended Edition, Very Large"
> >>> Year|Model|Category
> >>> 1999|Venture "Extended Edition"|
> >>> 1999|Venture "Extended Edition, Very Large"|]
> >>> >
> >>>
> >>> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
> >>>
> >>> git log
> >>> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
> >>>
> >>>
> >>> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau 
> wrote:
> >>> > I think we should roll forward to 1.5-S..
> >>> > On Dec 2, 2015 6:44 PM, "Venki Korukanti"  >
> >>> wrote:
> >>> >
> >>> >> 1.4.0 branch is cut and available here:
> >>> >> https://github.com/vkorukanti/drill/tree/1.4.0.
> >>> >>
> >>> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC
> is
> >>> >> passed?
> >>> >>
> >>> >> Thanks
> >>> >> Venki
> >>> >>
> >>> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
> >>> venki.koruka...@gmail.com
> >>> >> >
> >>> >> wrote:
> >>> >>
> >>> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
> >>> verify. If
> >>> >> > the changes are reviewed lets merge them today. Once the branch
> is cut
> >>> >> > today, MapR will do the release sanity for next couple of days
> before
> >>> RC0
> >>> >> > voting goes out. If any issues are found, we still have time to
> fix
> >>> >> before
> >>> >> > the RC0 voting.
> >>> >> >
> >>> >> > Thanks
> >>> >> > Venki
> >>> >> >
> >>> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau <
> jacq...@dremio.com>
> >>> >> wrote:
> >>> >> >
> >>> >> >> Sounds good.
> >>> >> >>
> >>> >> >> --
> >>> >> >> Jacques Nadeau
> >>> >> >> CTO and Co-Founder, Dremio
> >>> >> >>
> >>> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips <
> ste...@dremio.com>
> >>> >> >> wrote:
> >>> >> >>
> >>> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
> >>> >> >> >
> >>> >> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
> >>> >> >> venki.koruka...@gmail.com
> >>> >> >> > >
> >>> >> >> > wrote:
> >>> >> >> >
> >>> >> >> > > For DRILL-4145: ran the regression suite which includes
> customer
> >>> and
> >>> >> >> > > extended tests. No regressions found.
> >>> >> >> > >
> >>> >> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
> >>> >> >> > venki.koruka...@gmail.com
> >>> >> >> > > >
> >>> >> >> > > wrote:
> >>> >> >> > >
> >>> >> >> > > > Sure, I will trigger a regression run with DRILL-4145.
> >>> >> >> > > >
> >>> >> >> > > > Thanks
> >>> >> >> > > > Venki
> >>> >> >> > > >
> >>> >> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
> >>> >> jacq...@dremio.com>
> >>> >> >> > > wrote:
> >>> >> >> > > >
> >>> >> >> > > >> I believe 4109 is ready but if I understand correctly, it
> >>> can't
> >>> >> go
> >>> >> >> in
> >>> >> >> > > >> without 4125. Amit said he needs another hour for that.
> >>> >> >> > > >>
> >>> >> >> > > >> On 4145: Venki, can someone on your side running any
> >>> >> >> extended/customer
> >>> >> >> > > >> tests you have against this to make sure that it doesn't
> cause
> >>> >> any
> >>> >> >> > > >> regressions. We've run on our side without issue but I'

Re: Parquet pushdown filtering

[Julien Le Dem]

Hey Adam,
If you have questions about the Parquet side of things, I'm happy to chat.
Julien

On Tue, Dec 1, 2015 at 10:20 PM, Parth Chandra  wrote:

> Parquet metadata has the rowCount for every rowGroup which is also the
> value count for every column in the rowGroup. Isn't that what you need?
>
> On Tue, Dec 1, 2015 at 10:10 PM, Adam Gilmore 
> wrote:
>
> > Hi guys,
> >
> > I'm trying to (re)implement pushdown filtering for Parquet with the new
> > Parquet metadata caching implementation.
> >
> > I've run into a couple of challenges:
> >
> >1. Scan batches don't allow empty batches.  This means if a particular
> >filter filters out *all* rows, we get an exception.  I haven't read
> the
> >full comments on the relevant JIRA items, but it seems odd that we
> can't
> >query an empty JSON file, for example.  This is a bit of a blocker to
> >implement the pushdown filtering properly.
> >2. The Parquet metadata doesn't include all the relevant metadata.
> >Specifically, count of values is not included, therefore the default
> >Parquet statistics filter has issues because it compares the count of
> >values with count of nulls to work out if it can drop it.  This isn't
> >necessarily a blocker, but it feels ugly simulating there's "1" row
> in a
> >block (just to get around the null comparison).
> >
> > Also, it feels a bit ugly rehydrating the standard Parquet metadata
> objects
> > manually.  I'm not sure I understand why we created our own objects for
> the
> > Parquet metadata as opposed to simply writing a custom serializer for
> those
> > objects which we store.
> >
> > Thoughts would be great - I'd love to get a patch out for this.
> >
>



-- 
Julien

Re: Parquet pushdown filtering

[Hanifi GUNES]

Regarding your point  #1. I guess Daniel struggled with this limitation as
well. I merged few of his patches which addressed empty batch(no data)
handling in various places during execution. That said, however, we still
could not have time to develop a solid way to handle empty batches with no
schema.

*- Scan batches don't allow empty batches.  This means if a
particular filter filters out *all* rows, we get an exception.*
Looks to me, you are referring to no data rather than no schema here. I
would expect graceful execution in this case. Do you mind sharing a simple
reproduction?


-Hanifi

2015-12-03 10:56 GMT-08:00 Julien Le Dem :

> Hey Adam,
> If you have questions about the Parquet side of things, I'm happy to chat.
> Julien
>
> On Tue, Dec 1, 2015 at 10:20 PM, Parth Chandra  wrote:
>
> > Parquet metadata has the rowCount for every rowGroup which is also the
> > value count for every column in the rowGroup. Isn't that what you need?
> >
> > On Tue, Dec 1, 2015 at 10:10 PM, Adam Gilmore 
> > wrote:
> >
> > > Hi guys,
> > >
> > > I'm trying to (re)implement pushdown filtering for Parquet with the new
> > > Parquet metadata caching implementation.
> > >
> > > I've run into a couple of challenges:
> > >
> > >1. Scan batches don't allow empty batches.  This means if a
> particular
> > >filter filters out *all* rows, we get an exception.  I haven't read
> > the
> > >full comments on the relevant JIRA items, but it seems odd that we
> > can't
> > >query an empty JSON file, for example.  This is a bit of a blocker
> to
> > >implement the pushdown filtering properly.
> > >2. The Parquet metadata doesn't include all the relevant metadata.
> > >Specifically, count of values is not included, therefore the default
> > >Parquet statistics filter has issues because it compares the count
> of
> > >values with count of nulls to work out if it can drop it.  This
> isn't
> > >necessarily a blocker, but it feels ugly simulating there's "1" row
> > in a
> > >block (just to get around the null comparison).
> > >
> > > Also, it feels a bit ugly rehydrating the standard Parquet metadata
> > objects
> > > manually.  I'm not sure I understand why we created our own objects for
> > the
> > > Parquet metadata as opposed to simply writing a custom serializer for
> > those
> > > objects which we store.
> > >
> > > Thoughts would be great - I'd love to get a patch out for this.
> > >
> >
>
>
>
> --
> Julien
>

Re: Create 1.4 Release branch soon?

[Venki Korukanti]

As the changes are only in test, it should be ok if we get the fix after
lunch. Currently the branch is going through regression testing.

On Thu, Dec 3, 2015 at 9:47 AM, Steven Phillips  wrote:

> Okay, after looking at it more closely, it looks like its an ordering
> problem. We should rewrite the tests using the test framework, and be sure
> to set the validation as unordered.
>
> I can take care of this, but won't get to it until after lunch. If someone
> else wants to do it now in order to get an RC out sooner, feel free.
>
> On Thu, Dec 3, 2015 at 8:39 AM, Jinfeng Ni  wrote:
>
> > I switch to a different linux box, and hit the same error. So, seems
> > it's consistent from what I tried.
> >
> >
> > On Thu, Dec 3, 2015 at 7:58 AM, Jinfeng Ni 
> wrote:
> > > I run twice and hit the same error.
> > >
> > >
> > > On Thu, Dec 3, 2015 at 12:10 AM, Steven Phillips 
> > wrote:
> > >> I just ran the tests on a linux machine, and did not see this failure.
> > Do
> > >> you see it consistently?
> > >>
> > >> On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni 
> > wrote:
> > >>
> > >>> I run mvn full build on linux box against the latest master branch.
> > >>> There was one unit test failure. However, on mac, it's successful.
> Has
> > >>> anyone experienced the same?
> > >>>
> > >>> Failed tests:
> > >>>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196 Result
> > >>> mismatch.
> > >>> Expected:
> > >>> Year|Model|Category
> > >>> 1999|Venture "Extended Edition"|
> > >>> 1999|Venture "Extended Edition, Very Large"|
> > >>> Year|Model|Category
> > >>> 1999||Venture "Extended Edition"
> > >>> 1999||Venture "Extended Edition, Very Large"
> > >>>
> > >>> Received:
> > >>> Year|Model|Category
> > >>> 1999||Venture "Extended Edition"
> > >>> 1999||Venture "Extended Edition, Very Large"
> > >>> Year|Model|Category
> > >>> 1999|Venture "Extended Edition"|
> > >>> 1999|Venture "Extended Edition, Very Large"|
> > >>>  expected:<...Model|Category
> > >>> 1999|[Venture "Extended Edition"|
> > >>> 1999|Venture "Extended Edition, Very Large"|
> > >>> Year|Model|Category
> > >>> 1999||Venture "Extended Edition"
> > >>> 1999||Venture "Extended Edition, Very Large"]
> > >>> > but was:<...Model|Category
> > >>> 1999|[|Venture "Extended Edition"
> > >>> 1999||Venture "Extended Edition, Very Large"
> > >>> Year|Model|Category
> > >>> 1999|Venture "Extended Edition"|
> > >>> 1999|Venture "Extended Edition, Very Large"|]
> > >>> >
> > >>>
> > >>> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
> > >>>
> > >>> git log
> > >>> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
> > >>>
> > >>>
> > >>> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau 
> > wrote:
> > >>> > I think we should roll forward to 1.5-S..
> > >>> > On Dec 2, 2015 6:44 PM, "Venki Korukanti" <
> venki.koruka...@gmail.com
> > >
> > >>> wrote:
> > >>> >
> > >>> >> 1.4.0 branch is cut and available here:
> > >>> >> https://github.com/vkorukanti/drill/tree/1.4.0.
> > >>> >>
> > >>> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an RC
> > is
> > >>> >> passed?
> > >>> >>
> > >>> >> Thanks
> > >>> >> Venki
> > >>> >>
> > >>> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
> > >>> venki.koruka...@gmail.com
> > >>> >> >
> > >>> >> wrote:
> > >>> >>
> > >>> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
> > >>> verify. If
> > >>> >> > the changes are reviewed lets merge them today. Once the branch
> > is cut
> > >>> >> > today, MapR will do the release sanity for next couple of days
> > before
> > >>> RC0
> > >>> >> > voting goes out. If any issues are found, we still have time to
> > fix
> > >>> >> before
> > >>> >> > the RC0 voting.
> > >>> >> >
> > >>> >> > Thanks
> > >>> >> > Venki
> > >>> >> >
> > >>> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau <
> > jacq...@dremio.com>
> > >>> >> wrote:
> > >>> >> >
> > >>> >> >> Sounds good.
> > >>> >> >>
> > >>> >> >> --
> > >>> >> >> Jacques Nadeau
> > >>> >> >> CTO and Co-Founder, Dremio
> > >>> >> >>
> > >>> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips <
> > ste...@dremio.com>
> > >>> >> >> wrote:
> > >>> >> >>
> > >>> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
> > >>> >> >> >
> > >>> >> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
> > >>> >> >> venki.koruka...@gmail.com
> > >>> >> >> > >
> > >>> >> >> > wrote:
> > >>> >> >> >
> > >>> >> >> > > For DRILL-4145: ran the regression suite which includes
> > customer
> > >>> and
> > >>> >> >> > > extended tests. No regressions found.
> > >>> >> >> > >
> > >>> >> >> > > On Wed, Dec 2, 2015 at 1:41 PM, Venki Korukanti <
> > >>> >> >> > venki.koruka...@gmail.com
> > >>> >> >> > > >
> > >>> >> >> > > wrote:
> > >>> >> >> > >
> > >>> >> >> > > > Sure, I will trigger a regression run with DRILL-4145.
> > >>> >> >> > > >
> > >>> >> >> > > > Thanks
> > >>> >> >> > > > Venki
> > >>> >> >> > > >
> > >>> >> >> > > > On Wed, Dec 2, 2015 at 1:36 PM, Jacques Nadeau <
> > >>> >> jacq...@dremio.com>
> > >>> >>

[jira] [Created] (DRILL-4158) Using CASE statement with heterogeneous schemas, NullPointers or returns Nonrelated error

[john schneider (JIRA)]

john schneider created DRILL-4158:
-

 Summary: Using CASE statement with heterogeneous schemas, 
NullPointers or returns Nonrelated error
 Key: DRILL-4158
 URL: https://issues.apache.org/jira/browse/DRILL-4158
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.3.0
 Environment: running locally on MAC OS X and remotely on Centos 7
Reporter: john schneider


I'm trying to use case statements to manage a heterogeneous stream of json 
objects as 
shown in the example from  
https://drill.apache.org/blog/2015/11/23/drill-1.3-released/
trying to get this to work I have encounted a number of failure modes.

For this report there are two files casetest-1.json and casetest-2.json

casetest-1.json has the following two lines - there are two schemas represented 
by
the two lines
{"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":"ndagdagan_a...@apixio.com"}}
{"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":"ndagdagan_a...@apixio.com","roles":null,"isNotadmins":true,"iscoders":true}}}

casetest-2.json has the following two lines - there is just one schema 
represented
{"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":"ndagdagan_a...@apixio.com","roles":null,"isNotadmins":true,"iscoders":true}}}
{"level":"EVENT","time":1448844983160,"user_info":{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":"ndagdagan_a...@apixio.com","roles":null,"isNotadmins":true,"iscoders":true}}}


I'll outline all of the things I've tried and include stacktraces where 
appropriate

Prior to running tests, I "enable_union_type"
0: jdbc:drill:zk=local> ALTER SESSION SET `exec.enable_union_type` = true;
+---+--+
|  ok   | summary  |
+---+--+
| true  | exec.enable_union_type updated.  |
+---+--+
1 row selected (1.344 seconds)


## FIRST TEST, two lines two schemas, one with a field that's a string and 
second field is a map
## first lets just select all records, I expect this to barf since there are 
two schemas
## WORKS AS EXPECTED for this version of software

: jdbc:drill:zk=local> select *  from 
dfs.`/Users/jos/work/drill/casetest-1.json` t ;
Error: DATA_READ ERROR: Error parsing JSON - You tried to start when you are 
using a ValueWriter of type NullableVarCharWriterImpl.

File  /Users/jos/work/drill/casetest-1.json
Record  2
Fragment 0:0

[Error Id: 1385aea5-68cb-4775-ae17-fad6b4901ea6 on 10.0.1.9:31010] 
(state=,code=0)

## SECOND TEST, now lets use a case statement to sort out the schemas, I don't
## expect this to barf but barf it does and hard w/ a NULLPOINTER, THIS SHOULD 
HAVE WORKED
## StackTrace at bottom

0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then 'map' 
else 'string' end from dfs.`/Users/jos/work/drill/casetest-1.json` t ;
Error: SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 2568df13-d11a-402d-9cc9-82bd0ddb3c50 on 10.19.220.63:31010] 
(state=,code=0)
0: jdbc:drill:zk=local>


## THRID TEST now lets see if any case will work on any structure - for this we
## will use casetest-2.json which has just one schema
## select * WORKS AS EXPECTED

0: jdbc:drill:zk=local> select * from 
dfs.`/Users/jos/work/drill/casetest-2.json` t ;
+---+--+---+
| level | time | user_info |
+---+--+---+
| EVENT | 1448844983160 | 
{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":"ndagdagan_a...@apixio.com","isNotadmins":true,"iscoders":true}}
 |
| EVENT | 1448844983160 | 
{"session":"9OOLJ8HEGEQ0sTCVSXsK9ddJWVpFM5wM","user":{"id":"ndagdagan_a...@apixio.com","isNotadmins":true,"iscoders":true}}
 |
+---+--+---+
2 rows selected (1.701 seconds)

## FOURTH TEST - now lets try to use a  case statement w/ same schema
## THIS SHOULD HAVE WORKED, it doesn't work, and we get different more puzzling 
errors

0: jdbc:drill:zk=local> select case when is_map(t.user_info.`user`) then 'map' 
else 'string' end  from dfs.`/Users/jos/work/drill/casetest-2.json` t ;
Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize 
incoming schema.  Errors:

Error in expression at index -1.  Error: Missing function implementation: 
[is_map(MAP-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
Error in expression at index -1.  Error: Failure composing If Expression.  All 
conditions must return a boolean type.  Condition was of Type NULL..  Full 
expression: --UNKNOWN EXPRESSION--..

Fragment 0:0

[Error Id: c3a7f989-4d93-48c0-9a16-a38dd195314c on 10.19.220.63:31010] 
(state=,code=0)
0: jdbc:drill:zk=local>


 Stack trace from TEST 2 ===

Error: SYSTEM ERROR: NullPointerException

F

[jira] [Created] (DRILL-4159) TestCsvHeader sometimes fails due to ordering issue

[Steven Phillips (JIRA)]

Steven Phillips created DRILL-4159:
--

 Summary: TestCsvHeader sometimes fails due to ordering issue
 Key: DRILL-4159
 URL: https://issues.apache.org/jira/browse/DRILL-4159
 Project: Apache Drill
  Issue Type: Bug
Reporter: Steven Phillips
Assignee: Steven Phillips


This test should be rewritten to use the query test framework, rather than 
doing a string comparison of the entire result set. And it should be specified 
as unordered, so that results aren't affected by the random order in which 
files are read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[StevenMPhillips]

GitHub user StevenMPhillips opened a pull request:

https://github.com/apache/drill/pull/289

DRILL-4159: Use test framework in TestCsvHeader



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StevenMPhillips/drill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #289


commit f4e34ccac6add649df469f85e38f064962eae90a
Author: Steven Phillips 
Date:   2015-12-03T21:39:03Z

DRILL-4159: Use test framework in TestCsvHeader




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4134: Allocator updates

[adeneche]

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/283#discussion_r46617075
  
--- Diff: 
exec/memory/base/src/main/java/org/apache/drill/exec/memory/AllocatorManager.java
 ---
@@ -0,0 +1,386 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import static org.apache.drill.exec.memory.BaseAllocator.indent;
+import io.netty.buffer.DrillBuf;
+import io.netty.buffer.PooledByteBufAllocatorL;
+import io.netty.buffer.UnsafeDirectLittleEndian;
+
+import java.util.IdentityHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import org.apache.drill.common.HistoricalLog;
+import org.apache.drill.exec.memory.BaseAllocator.Verbosity;
+import org.apache.drill.exec.metrics.DrillMetrics;
+import org.apache.drill.exec.ops.BufferManager;
+
+import com.carrotsearch.hppc.LongObjectOpenHashMap;
+import com.google.common.base.Preconditions;
+
+/**
+ * Manages the relationship between one or more allocators and a 
particular UDLE. Ensures that one allocator owns the
+ * memory that multiple allocators may be referencing. Manages a 
BufferLedger between each of its associated allocators.
+ * This class is also responsible for managing when memory is allocated 
and returned to the Netty-based
+ * PooledByteBufAllocatorL.
+ *
+ * The only reason that this isn't package private is we're forced to put 
DrillBuf in Netty's package which need access
+ * to these objects or methods.
+ *
+ * Threading: AllocatorManager manages thread-safety internally. 
Operations within the context of a single BufferLedger
+ * are lockless in nature and can be leveraged by multiple threads. 
Operations that cross the context of two ledgers
+ * will acquire a lock on the AllocatorManager instance. Important note, 
there is one AllocatorManager per
+ * UnsafeDirectLittleEndian buffer allocation. As such, there will be 
thousands of these in a typical query. The
+ * contention of acquiring a lock on AllocatorManager should be very low.
+ *
+ */
+public class AllocatorManager {
+  // private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(AllocatorManager.class);
+
+  private static final AtomicLong LEDGER_ID_GENERATOR = new AtomicLong(0);
+  static final PooledByteBufAllocatorL INNER_ALLOCATOR = new 
PooledByteBufAllocatorL(DrillMetrics.getInstance());
+
+  private final RootAllocator root;
+  private volatile BufferLedger owningLedger;
+  private final int size;
+  private final UnsafeDirectLittleEndian underlying;
+  private final ReadWriteLock lock = new ReentrantReadWriteLock();
+  private final LongObjectOpenHashMap map = new 
LongObjectOpenHashMap<>();
+  private final AutoCloseableLock readLock = new 
AutoCloseableLock(lock.readLock());
+  private final AutoCloseableLock writeLock = new 
AutoCloseableLock(lock.writeLock());
+  private final IdentityHashMap buffers =
+  BaseAllocator.DEBUG ? new IdentityHashMap() : null;
+
+  AllocatorManager(BaseAllocator accountingAllocator, int size) {
+Preconditions.checkNotNull(accountingAllocator);
+this.root = accountingAllocator.root;
+this.underlying = INNER_ALLOCATOR.allocate(size);
+this.owningLedger = associate(accountingAllocator);
+this.size = underlying.capacity();
+  }
+
+  /**
+   * Associate the existing underlying buffer with a new allocator.
+   *
+   * @param allocator
+   *  The target allocator to associate this buffer with.
+   * @return The Ledger (new or existing) that associates the underlying 
buffer to this new ledger.
+   */
+  public BufferLedger associate(BaseAllocator allocator) {
+if (root != allocator.root) {

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[vkorukanti]

Github user vkorukanti commented on the pull request:

https://github.com/apache/drill/pull/289#issuecomment-161794179
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/289


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Steven Phillips]

I just pushed a fix to the test that I am pretty confident resolves the
problem that Jin Feng was hitting. Jin Feng, can you confirm this fixes
your problem?

On Thu, Dec 3, 2015 at 11:52 AM, Venki Korukanti 
wrote:

> As the changes are only in test, it should be ok if we get the fix after
> lunch. Currently the branch is going through regression testing.
>
> On Thu, Dec 3, 2015 at 9:47 AM, Steven Phillips  wrote:
>
> > Okay, after looking at it more closely, it looks like its an ordering
> > problem. We should rewrite the tests using the test framework, and be
> sure
> > to set the validation as unordered.
> >
> > I can take care of this, but won't get to it until after lunch. If
> someone
> > else wants to do it now in order to get an RC out sooner, feel free.
> >
> > On Thu, Dec 3, 2015 at 8:39 AM, Jinfeng Ni 
> wrote:
> >
> > > I switch to a different linux box, and hit the same error. So, seems
> > > it's consistent from what I tried.
> > >
> > >
> > > On Thu, Dec 3, 2015 at 7:58 AM, Jinfeng Ni 
> > wrote:
> > > > I run twice and hit the same error.
> > > >
> > > >
> > > > On Thu, Dec 3, 2015 at 12:10 AM, Steven Phillips 
> > > wrote:
> > > >> I just ran the tests on a linux machine, and did not see this
> failure.
> > > Do
> > > >> you see it consistently?
> > > >>
> > > >> On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni 
> > > wrote:
> > > >>
> > > >>> I run mvn full build on linux box against the latest master branch.
> > > >>> There was one unit test failure. However, on mac, it's successful.
> > Has
> > > >>> anyone experienced the same?
> > > >>>
> > > >>> Failed tests:
> > > >>>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196
> Result
> > > >>> mismatch.
> > > >>> Expected:
> > > >>> Year|Model|Category
> > > >>> 1999|Venture "Extended Edition"|
> > > >>> 1999|Venture "Extended Edition, Very Large"|
> > > >>> Year|Model|Category
> > > >>> 1999||Venture "Extended Edition"
> > > >>> 1999||Venture "Extended Edition, Very Large"
> > > >>>
> > > >>> Received:
> > > >>> Year|Model|Category
> > > >>> 1999||Venture "Extended Edition"
> > > >>> 1999||Venture "Extended Edition, Very Large"
> > > >>> Year|Model|Category
> > > >>> 1999|Venture "Extended Edition"|
> > > >>> 1999|Venture "Extended Edition, Very Large"|
> > > >>>  expected:<...Model|Category
> > > >>> 1999|[Venture "Extended Edition"|
> > > >>> 1999|Venture "Extended Edition, Very Large"|
> > > >>> Year|Model|Category
> > > >>> 1999||Venture "Extended Edition"
> > > >>> 1999||Venture "Extended Edition, Very Large"]
> > > >>> > but was:<...Model|Category
> > > >>> 1999|[|Venture "Extended Edition"
> > > >>> 1999||Venture "Extended Edition, Very Large"
> > > >>> Year|Model|Category
> > > >>> 1999|Venture "Extended Edition"|
> > > >>> 1999|Venture "Extended Edition, Very Large"|]
> > > >>> >
> > > >>>
> > > >>> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
> > > >>>
> > > >>> git log
> > > >>> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
> > > >>>
> > > >>>
> > > >>> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau  >
> > > wrote:
> > > >>> > I think we should roll forward to 1.5-S..
> > > >>> > On Dec 2, 2015 6:44 PM, "Venki Korukanti" <
> > venki.koruka...@gmail.com
> > > >
> > > >>> wrote:
> > > >>> >
> > > >>> >> 1.4.0 branch is cut and available here:
> > > >>> >> https://github.com/vkorukanti/drill/tree/1.4.0.
> > > >>> >>
> > > >>> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an
> RC
> > > is
> > > >>> >> passed?
> > > >>> >>
> > > >>> >> Thanks
> > > >>> >> Venki
> > > >>> >>
> > > >>> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
> > > >>> venki.koruka...@gmail.com
> > > >>> >> >
> > > >>> >> wrote:
> > > >>> >>
> > > >>> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
> > > >>> verify. If
> > > >>> >> > the changes are reviewed lets merge them today. Once the
> branch
> > > is cut
> > > >>> >> > today, MapR will do the release sanity for next couple of days
> > > before
> > > >>> RC0
> > > >>> >> > voting goes out. If any issues are found, we still have time
> to
> > > fix
> > > >>> >> before
> > > >>> >> > the RC0 voting.
> > > >>> >> >
> > > >>> >> > Thanks
> > > >>> >> > Venki
> > > >>> >> >
> > > >>> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau <
> > > jacq...@dremio.com>
> > > >>> >> wrote:
> > > >>> >> >
> > > >>> >> >> Sounds good.
> > > >>> >> >>
> > > >>> >> >> --
> > > >>> >> >> Jacques Nadeau
> > > >>> >> >> CTO and Co-Founder, Dremio
> > > >>> >> >>
> > > >>> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips <
> > > ste...@dremio.com>
> > > >>> >> >> wrote:
> > > >>> >> >>
> > > >>> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
> > > >>> >> >> >
> > > >>> >> >> > On Wed, Dec 2, 2015 at 2:45 PM, Venki Korukanti <
> > > >>> >> >> venki.koruka...@gmail.com
> > > >>> >> >> > >
> > > >>> >> >> > wrote:
> > > >>> >> >> >
> > > >>> >> >> > > For DRILL-4145: ran the regression suite which includes
> > > customer
> > > >>> and
> > > >>> >> >> > 

Build failed in Jenkins: drill-scm #612

[Apache Jenkins Server]

See 

Changes:

[smp] DRILL-4159: Use test framework in TestCsvHeader

--
[...truncated 4930 lines...]
[info] --match = 
[info] Tag refs [ 
[Ref[refs/tags/0.6.0-incubating=3daef7f553431c5aa76ff3074a8e00943de6dedd], 
Ref[refs/tags/0.9.0=78fd658d20767f8c627e29f58a6ede05a055bfdf], 
Ref[refs/tags/drill-1.0.0-m1=04020a8fca8b287874528d86dc7b8be0269ad788], 
Ref[refs/tags/drill-root-0.4.0-incubating=caa15e2629329cb56903189ff294bbd490a3fca8],
 Ref[refs/tags/drill-root-1.0.0-m1=ad638d9e41aa9efdb1e877cfe7e0a4b910f539fc], 
Ref[refs/tags/oscon_workshop=eaf95ed3c30d7bb147afe337e0e0477be6518d90], 
Ref[refs/tags/pre_exec_merge=a97a22b0a9547f8639e92258c0a3475b01742f15]] ]
[info] Resolved tag [ 0.6.0-incubating ] [ PersonIdent[Steven Phillips, 
sphill...@maprtech.com, Thu Oct 2 02:27:36 2014 -0700] ], points at [ commit 
5fa257b38df77dedf5609952e364ce0cfc7383d2 0 -- ] 
[info] Resolved tag [ 0.9.0 ] [ PersonIdent[Jacques Nadeau, jacq...@apache.org, 
Wed Apr 29 00:33:38 2015 -0700] ], points at [ commit 
c7cb36c8b45613bab299dfe6cafb36d4d2e00add 0 -- ] 
[info] Resolved tag [ drill-1.0.0-m1 ] [ PersonIdent[Jacques Nadeau, 
jacq...@apache.org, Fri Sep 6 13:05:42 2013 -0700] ], points at [ commit 
a0d3c6977820516983142c96d7f9374681529968 0 -- ] 
[info] Resolved tag [ drill-root-0.4.0-incubating ] [ PersonIdent[Jacques 
Nadeau, jacq...@apache.org, Wed Jul 30 22:21:41 2014 -0700] ], points at [ 
commit 98b208e262dd9b54988f5eeffacea7d0e83c958c 0 -- ] 
[info] Resolved tag [ drill-root-1.0.0-m1 ] [ PersonIdent[Jacques Nadeau, 
jacq...@apache.org, Wed Sep 4 04:23:47 2013 -0700] ], points at [ commit 
41c18197e3b8ae3c42d55089d641e9a0b68c6f29 0 -- ] 
[info] Resolved tag [ pre_exec_merge ] [ PersonIdent[Jacques Nadeau, 
jacq...@apache.org, Fri Jul 19 18:33:56 2013 -0700] ], points at [ commit 
5052b64d9953857575f8f40995b8da05160e5457 0 -- ] 
[info] key [ commit 41c18197e3b8ae3c42d55089d641e9a0b68c6f29 0 -- ], tags 
=> [ [DatedRevTag{id=ad638d9e41aa9efdb1e877cfe7e0a4b910f539fc, 
tagName='drill-root-1.0.0-m1', date=September 4, 2013 11:23:47 AM UTC}] ] 
[info] key [ commit 5fa257b38df77dedf5609952e364ce0cfc7383d2 0 -- ], tags 
=> [ [DatedRevTag{id=3daef7f553431c5aa76ff3074a8e00943de6dedd, 
tagName='0.6.0-incubating', date=October 2, 2014 9:27:36 AM UTC}] ] 
[info] key [ commit 5052b64d9953857575f8f40995b8da05160e5457 0 -- ], tags 
=> [ [DatedRevTag{id=a97a22b0a9547f8639e92258c0a3475b01742f15, 
tagName='pre_exec_merge', date=July 20, 2013 1:33:56 AM UTC}] ] 
[info] key [ commit a0d3c6977820516983142c96d7f9374681529968 0 -- ], tags 
=> [ [DatedRevTag{id=04020a8fca8b287874528d86dc7b8be0269ad788, 
tagName='drill-1.0.0-m1', date=September 6, 2013 8:05:42 PM UTC}] ] 
[info] key [ commit 98b208e262dd9b54988f5eeffacea7d0e83c958c 0 -- ], tags 
=> [ [DatedRevTag{id=caa15e2629329cb56903189ff294bbd490a3fca8, 
tagName='drill-root-0.4.0-incubating', date=July 31, 2014 5:21:41 AM UTC}] ] 
[info] key [ commit c7cb36c8b45613bab299dfe6cafb36d4d2e00add 0 -- ], tags 
=> [ [DatedRevTag{id=78fd658d20767f8c627e29f58a6ede05a055bfdf, tagName='0.9.0', 
date=April 29, 2015 7:33:38 AM UTC}] ] 
[info] Created map: [ {commit 41c18197e3b8ae3c42d55089d641e9a0b68c6f29 0 
--=[drill-root-1.0.0-m1], commit 5fa257b38df77dedf5609952e364ce0cfc7383d2 0 
--=[0.6.0-incubating], commit 5052b64d9953857575f8f40995b8da05160e5457 0 
--=[pre_exec_merge], commit a0d3c6977820516983142c96d7f9374681529968 0 
--=[drill-1.0.0-m1], commit 98b208e262dd9b54988f5eeffacea7d0e83c958c 0 
--=[drill-root-0.4.0-incubating], commit 
c7cb36c8b45613bab299dfe6cafb36d4d2e00add 0 --=[0.9.0]} ] 
[info] HEAD is [ f4e34ccac6add649df469f85e38f064962eae90a ] 
[info] Repo is in dirty state [ false ]
[info] git.commit.id.describe 0.9.0-514-gf4e34cc
[info] git.commit.id f4e34ccac6add649df469f85e38f064962eae90a
[info] git.commit.id.abbrev f4e34cc
[info] git.commit.user.name Steven Phillips
[info] git.commit.user.email s...@apache.org
[info] git.commit.message.full DRILL-4159: Use test framework in TestCsvHeader

[info] git.commit.message.short DRILL-4159: Use test framework in TestCsvHeader
[info] git.commit.time 03.12.2015 @ 21:39:03 UTC
[info] git.remote.origin.url https://git-wip-us.apache.org/repos/asf/drill.git
[info] git.build.time 03.12.2015 @ 22:00:51 UTC
[info] found property git.commit.id.abbrev
[info] found property git.commit.user.email
[info] found property git.commit.message.full
[info] found property git.commit.id
[info] found property git.commit.message.short
[info] found property git.commit.user.name
[info] found property git.build.user.name
[info] found property git.commit.id.describe
[info] found property git.build.user.email
[info] found property git.branch
[info] found property git.commit.time
[info] found property git.build.time
[info] found property git.remote.origin.url
[info] Writing properties file to [ 

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/289#issuecomment-161800836
  
I didn't do a full build with tests before merging, and it looks like there 
is a checkstyle problem.
I fixed it, and am now doing a full build to verify before merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[jaltekruse]

Github user jaltekruse commented on the pull request:

https://github.com/apache/drill/pull/289#issuecomment-161813807
  
As long as you are changing it do you want to remove the two test classes 
still using this method and then just delete it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : drill-scm #613

[Apache Jenkins Server]

See 

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/289#issuecomment-161816994
  
Which method?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Update classpath scanning doc

[julienledem]

Github user julienledem closed the pull request at:

https://github.com/apache/drill/pull/277


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: Update classpath scanning doc

[julienledem]

Github user julienledem commented on the pull request:

https://github.com/apache/drill/pull/277#issuecomment-161817032
  
merged in 

https://github.com/apache/drill/commit/b45e966f0a3067e1f8dda5ad106d2c3ac41d6019

https://github.com/apache/drill/commit/e138bb984e1778f6ed3d3335523a323e483bab8f


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: remove deprecated constructor and related test...

[julienledem]

Github user julienledem closed the pull request at:

https://github.com/apache/drill/pull/125


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4159: Use test framework in TestCsvHeade...

[jaltekruse]

Github user jaltekruse commented on the pull request:

https://github.com/apache/drill/pull/289#issuecomment-161818595
  
The one that is generating a string representation of the result set, which 
caused the bug. getResultString()


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Create 1.4 Release branch soon?

[Jinfeng Ni]

I had a successful run on linux machine against the latest master branch.


On Thu, Dec 3, 2015 at 1:51 PM, Steven Phillips  wrote:
> I just pushed a fix to the test that I am pretty confident resolves the
> problem that Jin Feng was hitting. Jin Feng, can you confirm this fixes
> your problem?
>
> On Thu, Dec 3, 2015 at 11:52 AM, Venki Korukanti 
> wrote:
>
>> As the changes are only in test, it should be ok if we get the fix after
>> lunch. Currently the branch is going through regression testing.
>>
>> On Thu, Dec 3, 2015 at 9:47 AM, Steven Phillips  wrote:
>>
>> > Okay, after looking at it more closely, it looks like its an ordering
>> > problem. We should rewrite the tests using the test framework, and be
>> sure
>> > to set the validation as unordered.
>> >
>> > I can take care of this, but won't get to it until after lunch. If
>> someone
>> > else wants to do it now in order to get an RC out sooner, feel free.
>> >
>> > On Thu, Dec 3, 2015 at 8:39 AM, Jinfeng Ni 
>> wrote:
>> >
>> > > I switch to a different linux box, and hit the same error. So, seems
>> > > it's consistent from what I tried.
>> > >
>> > >
>> > > On Thu, Dec 3, 2015 at 7:58 AM, Jinfeng Ni 
>> > wrote:
>> > > > I run twice and hit the same error.
>> > > >
>> > > >
>> > > > On Thu, Dec 3, 2015 at 12:10 AM, Steven Phillips 
>> > > wrote:
>> > > >> I just ran the tests on a linux machine, and did not see this
>> failure.
>> > > Do
>> > > >> you see it consistently?
>> > > >>
>> > > >> On Wed, Dec 2, 2015 at 10:20 PM, Jinfeng Ni 
>> > > wrote:
>> > > >>
>> > > >>> I run mvn full build on linux box against the latest master branch.
>> > > >>> There was one unit test failure. However, on mac, it's successful.
>> > Has
>> > > >>> anyone experienced the same?
>> > > >>>
>> > > >>> Failed tests:
>> > > >>>   TestCsvHeader.testCsvHeaderMismatch:151->validateResults:196
>> Result
>> > > >>> mismatch.
>> > > >>> Expected:
>> > > >>> Year|Model|Category
>> > > >>> 1999|Venture "Extended Edition"|
>> > > >>> 1999|Venture "Extended Edition, Very Large"|
>> > > >>> Year|Model|Category
>> > > >>> 1999||Venture "Extended Edition"
>> > > >>> 1999||Venture "Extended Edition, Very Large"
>> > > >>>
>> > > >>> Received:
>> > > >>> Year|Model|Category
>> > > >>> 1999||Venture "Extended Edition"
>> > > >>> 1999||Venture "Extended Edition, Very Large"
>> > > >>> Year|Model|Category
>> > > >>> 1999|Venture "Extended Edition"|
>> > > >>> 1999|Venture "Extended Edition, Very Large"|
>> > > >>>  expected:<...Model|Category
>> > > >>> 1999|[Venture "Extended Edition"|
>> > > >>> 1999|Venture "Extended Edition, Very Large"|
>> > > >>> Year|Model|Category
>> > > >>> 1999||Venture "Extended Edition"
>> > > >>> 1999||Venture "Extended Edition, Very Large"]
>> > > >>> > but was:<...Model|Category
>> > > >>> 1999|[|Venture "Extended Edition"
>> > > >>> 1999||Venture "Extended Edition, Very Large"
>> > > >>> Year|Model|Category
>> > > >>> 1999|Venture "Extended Edition"|
>> > > >>> 1999|Venture "Extended Edition, Very Large"|]
>> > > >>> >
>> > > >>>
>> > > >>> Tests run: 1505, Failures: 1, Errors: 0, Skipped: 121
>> > > >>>
>> > > >>> git log
>> > > >>> commit 3ae3bf5e127b4384c7b91d797d36ea4d51a058ae
>> > > >>>
>> > > >>>
>> > > >>> On Wed, Dec 2, 2015 at 7:21 PM, Jacques Nadeau > >
>> > > wrote:
>> > > >>> > I think we should roll forward to 1.5-S..
>> > > >>> > On Dec 2, 2015 6:44 PM, "Venki Korukanti" <
>> > venki.koruka...@gmail.com
>> > > >
>> > > >>> wrote:
>> > > >>> >
>> > > >>> >> 1.4.0 branch is cut and available here:
>> > > >>> >> https://github.com/vkorukanti/drill/tree/1.4.0.
>> > > >>> >>
>> > > >>> >> Should I move the master to 1.5.0-SNAPSHOT now or wait until an
>> RC
>> > > is
>> > > >>> >> passed?
>> > > >>> >>
>> > > >>> >> Thanks
>> > > >>> >> Venki
>> > > >>> >>
>> > > >>> >> On Wed, Dec 2, 2015 at 4:25 PM, Venki Korukanti <
>> > > >>> venki.koruka...@gmail.com
>> > > >>> >> >
>> > > >>> >> wrote:
>> > > >>> >>
>> > > >>> >> > For DRILL-4109 and DRILL-4125, Vicky is not available today to
>> > > >>> verify. If
>> > > >>> >> > the changes are reviewed lets merge them today. Once the
>> branch
>> > > is cut
>> > > >>> >> > today, MapR will do the release sanity for next couple of days
>> > > before
>> > > >>> RC0
>> > > >>> >> > voting goes out. If any issues are found, we still have time
>> to
>> > > fix
>> > > >>> >> before
>> > > >>> >> > the RC0 voting.
>> > > >>> >> >
>> > > >>> >> > Thanks
>> > > >>> >> > Venki
>> > > >>> >> >
>> > > >>> >> > On Wed, Dec 2, 2015 at 4:02 PM, Jacques Nadeau <
>> > > jacq...@dremio.com>
>> > > >>> >> wrote:
>> > > >>> >> >
>> > > >>> >> >> Sounds good.
>> > > >>> >> >>
>> > > >>> >> >> --
>> > > >>> >> >> Jacques Nadeau
>> > > >>> >> >> CTO and Co-Founder, Dremio
>> > > >>> >> >>
>> > > >>> >> >> On Wed, Dec 2, 2015 at 2:47 PM, Steven Phillips <
>> > > ste...@dremio.com>
>> > > >>> >> >> wrote:
>> > > >>> >> >>
>> > > >>> >> >> > Okay, I'm going to go ahead and merge DRILL-4145
>> > > >>> >> >> >
>> > > >>> >> 

[GitHub] drill pull request: Add doc for Select with options

[julienledem]

GitHub user julienledem opened a pull request:

https://github.com/apache/drill/pull/290

Add doc for Select with options



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/julienledem/drill patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #290


commit b99825f1dae5167d7bb86fcac3af0bd40002f2ef
Author: Julien Le Dem 
Date:   2015-12-03T23:53:05Z

Add doc for Select with options




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Naming the new ValueVector Initiative

[Jacques Nadeau]

I've opened a name search for our top vote getter.

https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-92


I also just realized that my previously email dropped other recipients.
Here it is below.


I think we can call the voting closed. Top vote getters:

Apache Arrow (17)
Apache Herringbone (9)
Apache Joist (8)
Apache Colbuf (8)

I'll up a PODLINGNAMESEARCH-* shortly for Arrow.

---






--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Thu, Dec 3, 2015 at 1:23 AM, Marcel Kornacker 
wrote:

> Just added my vote.
>
> On Thu, Dec 3, 2015 at 12:51 PM, Wes McKinney  wrote:
> > Shall we call the voting closed? Any last stragglers?
> >
> > On Tue, Dec 1, 2015 at 5:39 PM, Ted Dunning 
> wrote:
> >>
> >> Apache can handle this if we set the groundwork in place.
> >>
> >> Also, Twitter's lawyers work for Twitter, not for Apache. As such, their
> >> opinions can't be taken by Apache as legal advice.  There are issues of
> >> privilege, conflict of interest and so on.
> >>
> >>
> >>
> >> On Wed, Dec 2, 2015 at 7:51 AM, Alex Levenson  >
> >> wrote:
> >>>
> >>> I can ask about whether Twitter's lawyers can help out -- is that
> >>> something we need? Or is that something apache helps out with in the
> next
> >>> step?
> >>>
> >>> On Mon, Nov 30, 2015 at 9:32 PM, Julian Hyde  wrote:
> 
>  +1 to have a vote tomorrow.
> 
>  Assuming that Vector is out of play, I just did a quick search for the
>  top 4 remaining, (“arrow”, “honeycomb”, “herringbone”, “joist"), at
>  sourceforge, open hub, trademarkia, and on google. There are no
> trademarks
>  for these in similar subject areas. There is a moderately active
> project
>  called “joist” [1].
> 
>  I will point out that “Apache Arrow” has native-american connotations
>  that we may or may not want to live with (just ask the Washington
> Redskins
>  how they feel about their name).
> 
>  If someone would like to vet other names, use the links on
>  https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90, and fill
> out
>  column C in the spreadsheet.
> 
>  Julian
> 
>  [1] https://github.com/stephenh/joist
> 
> 
>  On Nov 30, 2015, at 7:01 PM, Jacques Nadeau 
> wrote:
> 
>  +1
> 
>  --
>  Jacques Nadeau
>  CTO and Co-Founder, Dremio
> 
>  On Mon, Nov 30, 2015 at 6:34 PM, Wes McKinney 
> wrote:
> 
>  Should we have a last call for votes, closing EOD tomorrow (Tuesday)?
> I
>  missed this for a few days last week with holiday travel.
> 
>  On Thu, Nov 26, 2015 at 3:04 PM, Julian Hyde 
>  wrote:
> 
>  Consulting a lawyer is part of the Apache branding process but the
> first
>  stage is to gather a list of potential conflicts -
>  https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-90 is an
> example.
> 
>  The other part, frankly, is to pick your battles.
> 
>  A year or so ago Actian re-branded Vectorwise as Vector.
> 
> 
> http://www.zdnet.com/article/actian-consolidates-its-analytics-portfolio/.
>  Given that it is an analytic database in the Hadoop space I think
> that is
>  as close to a “direct hit” as it gets. I don’t think we need a lawyer
> to
>  tell us that. Certainly it makes sense to look for conflicts for the
>  other
>  alternatives before consulting lawyers.
> 
>  Julian
> 
> 
> 
> 
>  On Nov 25, 2015, at 9:42 PM, Marcel Kornacker 
>  wrote:
> 
> 
> 
>  On Tue, Nov 24, 2015 at 3:25 PM, Jacques Nadeau 
>  wrote:
> 
>  Ok guys,
> 
>  I don't think anyone is doing a thorough analysis of viaability. I
> did a
>  quick glance and the top one (Vector) seems like it would have an
> issue
>  with conflict of an Actian product. The may be fine. Let's do a second
>  phase vote.
> 
> 
>  I'm assuming you mean Vectorwise?
> 
>  Before we do anything else, could we have a lawyer look into this?
> Last
>  time around that I remember (Parquet), Twitter's lawyers did a good
> job
>  of
>  weeding out the potential trademark violations.
> 
>  Alex, could Twitter get involved this time around as well?
> 
> 
> 
>  Pick your top 3 (1,2,3 with 3 being top preference)
> 
>  Let's get this done by Friday and then we can do a podling name search
>  starting with the top one.
> 
>  Link again:
> 
> 
> 
> 
> https://docs.google.com/spreadsheets/d/1q6UqluW6SLuMKRwW2TBGBzHfYLlXYm37eKJlIxWQGQM/edit#gid=304381532&vpid=A1
> 
>  thanks
> 
> 
>  --
>  Jacques Nadeau
>  CTO and Co-Founder, Dremio
> 
>  On Fri, Nov 20, 2015 at 9:24 AM, Jacques Nadeau 
>  wrote:
> 
>  Ok, it looks like we have a candidate list (we actually got 11 since
>  there was a three-way tie for ninth place):
> 
>  VectorArrowhoneycombHerringbonejoistV2Pietcolbufbatonimpuls

[jira] [Created] (DRILL-4160) Order By on a flattened column throws SchemaChangeException - Missing function implementation

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-4160:
--

 Summary: Order By on a flattened column throws 
SchemaChangeException - Missing function implementation
 Key: DRILL-4160
 URL: https://issues.apache.org/jira/browse/DRILL-4160
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser, Storage - JSON
Reporter: Abhishek Girish


Query with an ORDER BY clause on a flattened column fails:
{code}
> select `name`, `type`, flatten(kvgen(`compliments`)) as `compliments` from 
> `user` order by `name`, `type`, `compliments` limit 2;
Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize 
incoming schema.  Errors:
Error in expression at index 2.  Error: Missing function implementation: 
[hash64asdouble(MAP-REQUIRED, BIGINT-REQUIRED)].  Full expression: null..
Fragment 3:0
[Error Id: 3b3d3224-953a-46a2-8caa-fa6949e58ffd on abhi1:31010] (state=,code=0)
{code}
Query without order by on the flatten column executes fine. 
{code}
> select `name`, `type`, flatten(kvgen(`compliments`)) as `compliments` from 
> `user` order by `name`, `type` limit 2;
++---++
|name| type  |  compliments   |
++---++
|  Kurt  | user  | {"key":"cute","value":1.0} |
|  Kurt  | user  | {"key":"writer","value":1.0}   |
++---++
2 rows selected (4.239 seconds)
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)