Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

[qiang li]

Hi all,

Just to report my test progress.

As the SQL always throw schema changes exception. I reformatted my SQL, now
it will not throw exception.

I changed SQL
*from *
select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
convert_from(`ref0`.`v`.`v`,'UTF8') as v
from hbase.`offers_nation_idx` as `nation` join hbase.offers_ref0 as `ref0`
on BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v`
where `nation`.row_key  > '0br' and `nation`.row_key  < '0bs' limit 10

*to *
select t1.uid, t1.v from
  (select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as
uid, convert_from(`ref0`.`v`.`v`,'UTF8') as v
   from hbase.offers_ref0 as `ref0` where `ref0`.row_key > '0' and
`ref0`.row_key < '1' ) t1
join
  (select CONVERT_FROM(nation.`v`.`v`, 'BIGINT_BE') uid
   from hbase.`offers_nation_idx` as `nation` where `nation`.row_key  >
'0br' and `nation`.row_key  < '0bs' ) t2
on t1.uid = t2.uid limit 10

Now, my work can going on. But I think the first SQL should working too,
hope someone can take a look at it.






2016-06-23 10:59 GMT+08:00 qiang li :

> Dear all,
>
> Today ,I retest it again step by step. This time I get exception in both
> drillbit.log and drillbit.out, also I tested with released 1.6.0, its throw
> Hash join does not support schema changes exception too.
>
>
> I have reattached the error msg to 4734, Could some one help me solve this
> issue, we are very need this function working.
>
> Thanks
>
> 2016-06-22 9:39 GMT+08:00 qiang li :
>
>> Thanks Aman.
>>
>> I think the second is not related to hbase too. Its
>> happened occasionally. I wil give you more detail when its happen again.
>>
>> Hi Aditya, there is no error in drillbit.out, but have error in
>> drillbit.log, I have attached to the DRILL-4734
>>  , please check it .
>>
>>
>>
>> 2016-06-22 4:31 GMT+08:00 Aman Sinha :
>>
>>> Qiang,
>>> The last issue mentioned about failure to start drillbit seems unrelated
>>> to HBase isn't it ?  For now, I am going with the assumption that it is not
>>> a regression, so does not impact the 1.7 release.  Of course, if you find
>>> out otherwise, we would need to address it.
>>> Regarding the SchemaChangeException you reported, I have created a JIRA:
>>> DRILL-4734.
>>>
>>>
>>> -Aman
>>>
>>> On Tue, Jun 21, 2016 at 10:04 AM, Aditya 
>>> wrote:
>>>
 There should be another stack trace in drillbit.out when this happens,
 could you please check that file.

 On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:

 > Another issue is some time when I restart the node, the node can not
 be
 > startup.
 >
 > Here is the exception.
 > ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
 >
 jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
 > took 2800ms
 > 2016-06-20 19:10:18,313 [main] INFO
 o.a.d.e.s.s.PersistentStoreRegistry -
 > Using the configured PStoreProvider class:
 >
 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
 > 2016-06-20 19:10:19,221 [main] INFO
 o.apache.drill.exec.server.Drillbit -
 > Construction completed (1529 ms).
 > 2016-06-20 19:10:31,136 [main] WARN
 o.apache.drill.exec.server.Drillbit -
 > Failure on close()
 > java.lang.NullPointerException: null
 > at
 > org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
 > ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
 > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
 > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > 2016-06-20 19:10:31,137 [main] INFO
 o.apache.drill.exec.server.Drillbit -
 > Shutdown completed (1914 ms).
 >
 > I did nothing and start it at next day, then it can startup.
 >
 > 2016-06-21 9:48 GMT+08:00 qiang li :
 >
 >> Hi Aman,
 >>
 >> I did not fully test with the old version.
 >>
 >> Cloud you please help me create the JIRA issue,  I think my account
 have
 >> not the privilege, my account is griffinli and can not find the
 place to
 >> create new issue. Below is the explain detail for the same SQL in
 diffe

Re: Getting back on Calcite master: only a few steps left

[Jinfeng Ni]

Hi All,

It has been a while since we had discussion about getting Drill back
to Calcite master. We have made good progress by merging patches to
Calcite master. Thanks everyone who helped on this effort!

I took a look at the spreadsheet [1], and marked the patches based on
what I understand. So far, there are 4 patches marked as "TODO" (in
red), 2 patches marked as "PR pending review/merge" (in yellow).

1.  Can people of those 4 patches (Julian Le Dam, Jacques, Venki,
Aman) see if the patches have been merged to Calcite master (I may
miss something)?
2.  Once the necessary patches are in Calcite master, we probably need
have a discussion about the plan to do the REAL rebasing work.

Thoughts?

[1]  
https://docs.google.com/spreadsheets/d/1Z0J5aMCBfqq_9SEl-LXsi_B8Ahaosygqke4cH6pPwmo/edit#gid=0

On Wed, Apr 20, 2016 at 1:04 PM, Julian Hyde  wrote:
> Regarding https://issues.apache.org/jira/browse/CALCITE-1150, "Create the a 
> new DynamicRecordType, avoiding star expansion when working with this type”. 
> This feature will be useful, and as I have said to Jacques would fit well 
> within Calcite, but it’s a bit shapeless to me right now. I would like to see 
> some validator tests (any maybe one or two sql-to-rel converter tests) so I 
> get a feel for how it would work. When you have some tests can you post them 
> in the JIRA case? I don’t think you should charge ahead with the 
> implementation because I might not agree with the specification (i.e. the 
> test cases).
>
> I’l make the same comment in the JIRA case, and let’s continue the discussion 
> there.
>
> Julian
>
>
>
>> On Apr 19, 2016, at 9:51 AM, Jinfeng Ni  wrote:
>>
>> @Jacques,
>>
>> Sorry for the delay. Let me spend couple of days this week to get
>> CALCITE-1150 back on track. Initially, I encountered one conflicting
>> change in Calcite master, and broke couple of unit tests. If I can not
>> get them solved, I'll ping you or Minji for a discussion.
>>
>>
>>
>> On Mon, Apr 18, 2016 at 5:31 PM, Jacques Nadeau  wrote:
>>> Hey All,
>>>
>>> Following up to get a status update. We made some good initial progress but
>>> it seems like people may have hit some challenges (or distractions). Can
>>> everyone report on how they are doing?
>>>
>>> Jinfeng, how are tests for CALCITE-1150 going? Can Minji help get together
>>> test cases for CALCITE-1150? Maybe you could provide guidance on the set of
>>> queries to test?
>>>
>>> thanks,
>>> Jacques
>>>
>>>
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>>
>>> On Thu, Mar 31, 2016 at 4:19 PM, Julian Hyde  wrote:
>>>
 I’ve closed 1149, if we don’t need the feature.

 Yes, we need a unit test for 1151. I offered a suggestion how.

> On Mar 31, 2016, at 11:59 AM, Sudheesh Katkam 
 wrote:
>
> I submitted a patch for CALCITE-1151 <
 https://issues.apache.org/jira/browse/CALCITE-1151> (with changes to
 resolve a checkstyle error). I am waiting for comments regarding the unit
 test.
>
> I added a comment to CALCITE-1149 <
 https://issues.apache.org/jira/browse/CALCITE-1149> with the workaround
 being used.
>
> Thank you,
> Sudheesh
>
>> On Mar 16, 2016, at 5:19 PM, Jacques Nadeau  wrote:
>>
>> Yes, I'm trying to work through the failing unit tests.
>>
>> I merged your change.
>>
>> In the future you can pick compare & create pull request on your branch
 and
>> then change the target repo from apache to mine.
>>
>> thanks,
>> Jacques
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Wed, Mar 16, 2016 at 4:39 PM, Aman Sinha 
 wrote:
>>
>>> Jacques, I wasn't sure how to create a pull request against your
 branch;
>>> for  CALCITE-1108 you can cherry-pick from here:
>>>
>>>
 https://github.com/amansinha100/incubator-calcite/commits/calcite-drill-2
>>>
>>> BTW,  there are unit test failures on your branch which I assume is
>>> expected for now ?
>>>
>>> On Tue, Mar 15, 2016 at 6:56 PM, Jacques Nadeau 
>>> wrote:
>>>
 Why don't you guys propose patches for my branch and I'll incorporate
>>> until
 we get to a good state. Once we feel good about it, I'll clean up the
 revision history.

 --
 Jacques Nadeau
 CTO and Co-Founder, Dremio

 On Tue, Mar 15, 2016 at 11:01 AM, Jinfeng Ni 
 wrote:

> I'll add test for CALCITE-1150.
>
>
>
> On Tue, Mar 15, 2016 at 9:45 AM, Sudheesh Katkam <
 skat...@maprtech.com

> wrote:
>> CALCITE-1149 [Extend CALCITE-845] <
>

>>>
 https://github.com/mapr/incubator-calcite/commit/bd73728a8297e15331ae956096eab0e15b3f
>
> does not need to be committed into Calcite. DRILL-4372 <
> https://issues.apache.org/jira/browse/DRILL-4372> supersedes that

[GitHub] drill issue #518: DRILL-4653.json - Malformed JSON should not stop the entir...

[ssriniva123]

Github user ssriniva123 commented on the issue:

https://github.com/apache/drill/pull/518
  
What is the recommendation with this bug? Should we just remove the line 
nos for now and just name the file name. I was not able to work on this last 
couple of days, not sure if there is sufficient time until the next candidate 
release.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

[qiang li]

Dear all,

Today ,I retest it again step by step. This time I get exception in both
drillbit.log and drillbit.out, also I tested with released 1.6.0, its throw
Hash join does not support schema changes exception too.


I have reattached the error msg to 4734, Could some one help me solve this
issue, we are very need this function working.

Thanks

2016-06-22 9:39 GMT+08:00 qiang li :

> Thanks Aman.
>
> I think the second is not related to hbase too. Its happened occasionally.
> I wil give you more detail when its happen again.
>
> Hi Aditya, there is no error in drillbit.out, but have error in
> drillbit.log, I have attached to the DRILL-4734
>  , please check it .
>
>
>
> 2016-06-22 4:31 GMT+08:00 Aman Sinha :
>
>> Qiang,
>> The last issue mentioned about failure to start drillbit seems unrelated
>> to HBase isn't it ?  For now, I am going with the assumption that it is not
>> a regression, so does not impact the 1.7 release.  Of course, if you find
>> out otherwise, we would need to address it.
>> Regarding the SchemaChangeException you reported, I have created a JIRA:
>> DRILL-4734.
>>
>>
>> -Aman
>>
>> On Tue, Jun 21, 2016 at 10:04 AM, Aditya  wrote:
>>
>>> There should be another stack trace in drillbit.out when this happens,
>>> could you please check that file.
>>>
>>> On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:
>>>
>>> > Another issue is some time when I restart the node, the node can not be
>>> > startup.
>>> >
>>> > Here is the exception.
>>> > ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
>>> >
>>> jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
>>> > took 2800ms
>>> > 2016-06-20 19:10:18,313 [main] INFO
>>> o.a.d.e.s.s.PersistentStoreRegistry -
>>> > Using the configured PStoreProvider class:
>>> >
>>> 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
>>> > 2016-06-20 19:10:19,221 [main] INFO
>>> o.apache.drill.exec.server.Drillbit -
>>> > Construction completed (1529 ms).
>>> > 2016-06-20 19:10:31,136 [main] WARN
>>> o.apache.drill.exec.server.Drillbit -
>>> > Failure on close()
>>> > java.lang.NullPointerException: null
>>> > at
>>> > org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
>>> > ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
>>> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
>>> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > 2016-06-20 19:10:31,137 [main] INFO
>>> o.apache.drill.exec.server.Drillbit -
>>> > Shutdown completed (1914 ms).
>>> >
>>> > I did nothing and start it at next day, then it can startup.
>>> >
>>> > 2016-06-21 9:48 GMT+08:00 qiang li :
>>> >
>>> >> Hi Aman,
>>> >>
>>> >> I did not fully test with the old version.
>>> >>
>>> >> Cloud you please help me create the JIRA issue,  I think my account
>>> have
>>> >> not the privilege, my account is griffinli and can not find the place
>>> to
>>> >> create new issue. Below is the explain detail for the same SQL in
>>> different
>>> >> nodes of cluster.
>>> >>
>>> >>
>>> >> This is the correct plan which only have two nodes:
>>> >> 0: jdbc:drill:zk=xxx:> explain plan for select
>>> >> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>>> >> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from
>>> hbase.`offers_nation_idx` as
>>> >> `nation` join hbase.offers_ref0 as `ref0` on
>>> >>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where
>>> `nation`.row_key
>>> >>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>>> >> +--+--+
>>> >> | text | json |
>>> >> +--+--+
>>> >> | 00-00Screen
>>> >> 00-01  Project(uid=[$0], v=[$1])
>>> >> 00-02SelectionVectorRemover
>>> >> 00-03  Limit(fetch=[10])
>>> >> 00-04UnionExchange
>>> >> 01-01  SelectionVectorRemover
>>> >> 01-02Limit(fetch=[10])
>>> >> 01-03
>>> Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
>>> >> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
>>> >> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
>>> >> row_key0=[$0], v0=[$1], $f2=[$2])
>>> >> 01-05  HashJoin(condition=[=($2, $5)],
>>> >> joinType=[inner])
>>> >> 01-07   

Re: Time for a 1.7 release

[Aman Sinha]

DRILL-4733 patch is reviewed and tested, so I will include it in the
release (pending a minor update to the unit test).

A regression was reported for a few decimal tests, see
https://issues.apache.org/jira/browse/DRILL-4746,  which was caused by
DRILL-3149, so I will be excluding this commit from the release.

I hope to send out a release candidate tomorrow morning.

-Aman

On Mon, Jun 20, 2016 at 6:03 PM, Aman Sinha  wrote:

> Quick update:  DRILL-4733 (
> https://issues.apache.org/jira/browse/DRILL-4733) is a regression that
> Drill QA team found today, so I will have to wait to have it resolved for
> 1.7.0 before creating a release candidate.
>
>
> On Mon, Jun 20, 2016 at 1:52 PM, Johannes Schulte <
> johannes.schu...@gmail.com> wrote:
>
>> Speaking for DRILL-4574:
>>
>> I can get a simple mvn test to run, even on the master. I always get
>>
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-dependency-plugin:2.8:unpack
>> (unpack-vector-types) on project drill-java-exec: Artifact has not been
>> packaged yet. When used on reactor artifact, unpack should be executed
>> after packaging: see MDEP-98. -> [Help 1]
>>
>> I tried some things but nothing so far worked. It's not really unit tests
>> failing, it's the building. If somebody could check it out and run the
>> tests i'd be really happy
>>
>> On Mon, Jun 20, 2016 at 5:53 AM, Aman Sinha  wrote:
>>
>> > For the fixes that were committed in the last few days, could the
>> > committers close the pull requests and update the JIRAs with 'Fixed'
>> status
>> > for 1.7.
>> >
>> > For the remaining JIRAs mentioned in this thread, here are the status:
>> >
>> > 1. DRILL-4525 (BETWEEN clause on Date and Timestamp):  the right place
>> to
>> > fix this would be an enhancement in Calcite.  In the meantime, a
>> workaround
>> > is to do explicit CASTing as suggested in the JIRA.
>> > 2. DRILL-4653 (Skip malformed JSON):  mostly reviewed but needs some
>> more
>> > review/testing.
>> > 3. DRILL-4704 (Decimal type):  unit test being added.  needs review.
>> > 4. DRILL-4574 (Avro):  Rebased but unit tests failing for some other
>> > reason.
>> >
>> > I would like to finalize the content by EOD tomorrow.  Clearly, 1 has
>> been
>> > pushed out of 1.7 and I think it  would be quite a stretch to get the
>> rest
>> > in, so I would be in favor of pushing 2, 3, 4 into the next release.
>> > However, since all 3 have good momentum going right now, let's try to
>> get
>> > the pending issues resolved soon.
>> >
>> > Thanks !
>> > Aman
>> >
>> >
>> > On Thu, Jun 16, 2016 at 1:57 PM, Aman Sinha 
>> wrote:
>> >
>> > > It does look like DRILL-4574 was previously reviewed and ready to be
>> > > merged.  Right now it will need to be rebased on master branch.  Since
>> > this
>> > > is in the Avro plugin,  I am unsure about the types of tests that
>> need to
>> > > be run.. I would prefer if Jason Altekruse could take a quick look and
>> > > merge into master if everything looks ok.
>> > >
>> > > On Thu, Jun 16, 2016 at 1:22 PM, Johannes Schulte <
>> > > johannes.schu...@gmail.com> wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> https://github.com/apache/drill/pull/459 (
>> > >> https://issues.apache.org/jira/browse/DRILL-4574) is still not
>> merged
>> > >> but i
>> > >> think it is ready for a merge. Are there any other actions necessary?
>> > >>
>> > >> Johannes
>> > >>
>> > >> On Thu, Jun 16, 2016 at 8:07 PM, Jinfeng Ni 
>> > >> wrote:
>> > >>
>> > >> > I will review the Sean's PR for DRILL-4525, since it's a regression
>> > from
>> > >> > 1.6.
>> > >> >
>> > >> >
>> > >> > On Thu, Jun 16, 2016 at 9:39 AM, rahul challapalli
>> > >> >  wrote:
>> > >> > > I would like to have DRILL-4525 as this is a regression (most
>> likely
>> > >> from
>> > >> > > 1.6). Any takers for this?
>> > >> > >
>> > >> > > - Rahul
>> > >> > >
>> > >> > > On Wed, Jun 15, 2016 at 4:03 PM, Aman Sinha <
>> amansi...@apache.org>
>> > >> > wrote:
>> > >> > >
>> > >> > >> I can take a look at DRILL-4653.
>> > >> > >>
>> > >> > >> Could someone familiar with the Decimal type take a look at
>> > >> DRILL-4704 ?
>> > >> > >> Agree with Dave that it is a simple case that should be fixed
>> > (note,
>> > >> > >> however, that decimal is disabled by default currently).
>> > >> > >>
>> > >> > >>
>> > >> > >> On Wed, Jun 15, 2016 at 3:12 PM, Subbu Srinivasan <
>> > >> > ssriniva...@zscaler.com
>> > >> > >> >
>> > >> > >> wrote:
>> > >> > >>
>> > >> > >> > Who can  review
>> https://issues.apache.org/jira/browse/DRILL-4653
>> > ?
>> > >> > >> >
>> > >> > >> > On Wed, Jun 15, 2016 at 1:37 PM, Parth Chandra <
>> > >> pchan...@maprtech.com
>> > >> > >
>> > >> > >> > wrote:
>> > >> > >> >
>> > >> > >> > > +1 on the 1.7 release
>> > >> > >> > >
>> > >> > >> > > I'm reviewing the following and hope to get them in the
>> release
>> > >> > before
>> > >> > >> > > cutoff:
>> > >> > >> > > https://issues.apache.org/jira/browse/DRILL-2593
>> > >> > >> > > https://issues.apache.org/jira/browse/DRILL-4309
>> > >> > >>

[GitHub] drill pull request #531: DRILL-4733: max(dir0) reading more columns than nec...

[amansinha100]

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/531#discussion_r68161349
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/TestImplicitFileColumns.java
 ---
@@ -110,4 +111,20 @@ public void testImplicitColumnsForParquet() throws 
Exception {
 .go();
   }
 
+  @Test // DRILL-4733
+  public void testMultilevelParquetWithSchemaChange() throws Exception {
+try {
+  test("alter session set `planner.enable_decimal_data_type` = true");
+  testBuilder()
+  .sqlQuery(String.format("select max(dir0) as max_dir from 
dfs_test.`%s/src/test/resources/multilevel/parquetWithSchemaChange`",
+  TestTools.getWorkingPath()))
+  .unOrdered()
+  .baselineColumns("max_dir")
+  .baselineValues("voter50.parquet")
--- End diff --

It does seem confusing to have the .parquet extension to the folder name.  
Could you change it and update the pull request ? 

Perf tests look good so far.  

+1 on the patch. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68148042
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +186,9 @@ public boolean isFunctionComplexOutput(String name) {
 }
 return false;
   }
+
+  public OptionManager getOptionManager() {
--- End diff --

Removed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68148064
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestCastFunctions.java
 ---
@@ -17,18 +17,16 @@
  */
 package org.apache.drill.exec.physical.impl;
 
+import static org.apache.drill.exec.RunRootExec.c;
--- End diff --

removed 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68148055
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 
---
@@ -17,17 +17,54 @@
  */
 package org.apache.drill.exec;
 
+import com.codahale.metrics.MetricRegistry;
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.ClassPathScanner;
+import org.apache.drill.exec.compile.CodeCompilerTestFactory;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.memory.RootAllocatorFactory;
 import org.apache.drill.exec.metrics.DrillMetrics;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReaderTestFactory;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import 
org.apache.drill.exec.store.sys.store.provider.LocalPersistentStoreProvider;
 import org.apache.drill.test.DrillTest;
 import org.junit.After;
+import org.junit.BeforeClass;
+
+import static org.apache.drill.exec.RunRootExec.c;
 
 public class ExecTest extends DrillTest {
 
+  protected static SystemOptionManager optionManager;
+
   @After
   public void clear(){
 // TODO:  (Re DRILL-1735) Check whether still needed now that
 // BootstrapContext.close() resets the metrics.
 DrillMetrics.resetMetrics();
   }
 
+
+  @BeforeClass
+  public static void setupOptionManager() throws Exception{
+final LocalPersistentStoreProvider provider = new 
LocalPersistentStoreProvider(c);
+provider.start();
+optionManager = new 
SystemOptionManager(PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(c),
 provider);
+optionManager.init();
+  }
+
+  protected void initDrillbitContext(final DrillbitContext bitContext) 
throws Exception {
--- End diff --

changed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68148021
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -247,11 +275,13 @@ void flushCode() {
   outer._throws(SchemaChangeException.class);
 
   int methodIndex = 0;
-  int blocksInMethod = 0;
+  int exprsInMethod = 0;
   boolean isVoidMethod = method.getReturnType() == void.class;
-  for(JBlock b : blocks[i++]) {
+  for(SizedJBlock sb : blocks[i++]) {
+JBlock b = sb.getBlock();
 if(!b.isEmpty()) {
-  if (blocksInMethod > MAX_BLOCKS_IN_FUNCTION) {
+  if (optionManager != null &&
+  exprsInMethod > 
optionManager.getOption(ExecConstants.CODE_GEN_EXP_IN_METHOD_SIZE).num_val) {
--- End diff --

Done.  thx. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68147991
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -215,22 +218,47 @@ public JVar 
declareVectorValueSetupAndMember(DirectExpression batchName, TypedFi
 return vv;
   }
 
+  public enum BlkCreateMode {
+TRUE,  // Create new block
+FALSE, // Do not create block; put into existing block.
+TRUE_IF_BOUND // Create new block only if # of expressions added hit 
upper-bound (ExecConstants.CODE_GEN_EXP_IN_METHOD_SIZE)
+  }
+
   public HoldingContainer addExpr(LogicalExpression ex) {
-return addExpr(ex, true);
+// default behavior is always to put expression into new block.
+return addExpr(ex, BlkCreateMode.TRUE);
   }
 
-  public HoldingContainer addExpr(LogicalExpression ex, boolean rotate) {
-//logger.debug("Adding next write {}", ex);
-if (rotate) {
-  rotateBlock();
+  public HoldingContainer addExpr(LogicalExpression ex, BlkCreateMode 
mode) {
+if (mode == BlkCreateMode.TRUE || mode == BlkCreateMode.TRUE_IF_BOUND) 
{
+  rotateBlock(mode);
 }
+
+for (LinkedList b : blocks) {
+  b.getLast().incCounter();
+}
+
 return evaluationVisitor.addExpr(ex, this);
   }
 
   public void rotateBlock() {
-evaluationVisitor.previousExpressions.clear();
-for (LinkedList b : blocks) {
-  b.add(new JBlock(true, true));
+// default behavior is always to create new block.
+rotateBlock(BlkCreateMode.TRUE);
+  }
+
+  private void rotateBlock(BlkCreateMode mode) {
+boolean blockRotated = false;
+for (LinkedList b : blocks) {
+  if (mode == BlkCreateMode.TRUE ||
+  (mode == BlkCreateMode.TRUE_IF_BOUND &&
+optionManager != null &&
+b.getLast().getCount() > 
optionManager.getOption(ExecConstants.CODE_GEN_EXP_IN_METHOD_SIZE).num_val)) {
--- End diff --

Done. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68133101
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -247,11 +275,13 @@ void flushCode() {
   outer._throws(SchemaChangeException.class);
 
   int methodIndex = 0;
-  int blocksInMethod = 0;
+  int exprsInMethod = 0;
   boolean isVoidMethod = method.getReturnType() == void.class;
-  for(JBlock b : blocks[i++]) {
+  for(SizedJBlock sb : blocks[i++]) {
+JBlock b = sb.getBlock();
 if(!b.isEmpty()) {
-  if (blocksInMethod > MAX_BLOCKS_IN_FUNCTION) {
+  if (optionManager != null &&
+  exprsInMethod > 
optionManager.getOption(ExecConstants.CODE_GEN_EXP_IN_METHOD_SIZE).num_val) {
--- End diff --

here as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68133124
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 
---
@@ -17,17 +17,54 @@
  */
 package org.apache.drill.exec;
 
+import com.codahale.metrics.MetricRegistry;
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.ClassPathScanner;
+import org.apache.drill.exec.compile.CodeCompilerTestFactory;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.memory.RootAllocatorFactory;
 import org.apache.drill.exec.metrics.DrillMetrics;
+import org.apache.drill.exec.physical.impl.OperatorCreatorRegistry;
+import org.apache.drill.exec.planner.PhysicalPlanReaderTestFactory;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import 
org.apache.drill.exec.store.sys.store.provider.LocalPersistentStoreProvider;
 import org.apache.drill.test.DrillTest;
 import org.junit.After;
+import org.junit.BeforeClass;
+
+import static org.apache.drill.exec.RunRootExec.c;
 
 public class ExecTest extends DrillTest {
 
+  protected static SystemOptionManager optionManager;
+
   @After
   public void clear(){
 // TODO:  (Re DRILL-1735) Check whether still needed now that
 // BootstrapContext.close() resets the metrics.
 DrillMetrics.resetMetrics();
   }
 
+
+  @BeforeClass
+  public static void setupOptionManager() throws Exception{
+final LocalPersistentStoreProvider provider = new 
LocalPersistentStoreProvider(c);
+provider.start();
+optionManager = new 
SystemOptionManager(PhysicalPlanReaderTestFactory.defaultLogicalPlanPersistence(c),
 provider);
+optionManager.init();
+  }
+
+  protected void initDrillbitContext(final DrillbitContext bitContext) 
throws Exception {
--- End diff --

How about `mockDrillbitContext(...)`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68133110
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +186,9 @@ public boolean isFunctionComplexOutput(String name) {
 }
 return false;
   }
+
+  public OptionManager getOptionManager() {
--- End diff --

Unused?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68133140
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestCastFunctions.java
 ---
@@ -17,18 +17,16 @@
  */
 package org.apache.drill.exec.physical.impl;
 
+import static org.apache.drill.exec.RunRootExec.c;
--- End diff --

unused?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[sudheeshkatkam]

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68133092
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -215,22 +218,47 @@ public JVar 
declareVectorValueSetupAndMember(DirectExpression batchName, TypedFi
 return vv;
   }
 
+  public enum BlkCreateMode {
+TRUE,  // Create new block
+FALSE, // Do not create block; put into existing block.
+TRUE_IF_BOUND // Create new block only if # of expressions added hit 
upper-bound (ExecConstants.CODE_GEN_EXP_IN_METHOD_SIZE)
+  }
+
   public HoldingContainer addExpr(LogicalExpression ex) {
-return addExpr(ex, true);
+// default behavior is always to put expression into new block.
+return addExpr(ex, BlkCreateMode.TRUE);
   }
 
-  public HoldingContainer addExpr(LogicalExpression ex, boolean rotate) {
-//logger.debug("Adding next write {}", ex);
-if (rotate) {
-  rotateBlock();
+  public HoldingContainer addExpr(LogicalExpression ex, BlkCreateMode 
mode) {
+if (mode == BlkCreateMode.TRUE || mode == BlkCreateMode.TRUE_IF_BOUND) 
{
+  rotateBlock(mode);
 }
+
+for (LinkedList b : blocks) {
+  b.getLast().incCounter();
+}
+
 return evaluationVisitor.addExpr(ex, this);
   }
 
   public void rotateBlock() {
-evaluationVisitor.previousExpressions.clear();
-for (LinkedList b : blocks) {
-  b.add(new JBlock(true, true));
+// default behavior is always to create new block.
+rotateBlock(BlkCreateMode.TRUE);
+  }
+
+  private void rotateBlock(BlkCreateMode mode) {
+boolean blockRotated = false;
+for (LinkedList b : blocks) {
+  if (mode == BlkCreateMode.TRUE ||
+  (mode == BlkCreateMode.TRUE_IF_BOUND &&
+optionManager != null &&
+b.getLast().getCount() > 
optionManager.getOption(ExecConstants.CODE_GEN_EXP_IN_METHOD_SIZE).num_val)) {
--- End diff --

Drop `.num_val`. The OptionValidator is typed (LongValidator).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4750) Document supported/unsupported hive functions

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4750:


 Summary: Document supported/unsupported hive functions
 Key: DRILL-4750
 URL: https://issues.apache.org/jira/browse/DRILL-4750
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Reporter: Rahul Challapalli


>From our documentation it is not clear that Drill supports using hive 
>functions natively. We should document all the supported hive functions. Few 
>things to consider :

 1. Drill does not support any hive functions which return unsupported data 
types (List, Map etc)
 2. DATE and TIMESTAMP representations are incompatible between hive and 
drill. Some hive functions involving date and timestamp will work on data in a 
hive table, but will not work on non-hive data. Similarly some drill functions 
involving date and timestamp functions will work on non-hive data but will not 
work on hive data (could throw an error or report wrong data)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4749) Verification Failures (Decimal values) in drill's regression tests

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4749:


 Summary: Verification Failures (Decimal values) in drill's 
regression tests
 Key: DRILL-4749
 URL: https://issues.apache.org/jira/browse/DRILL-4749
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types, Storage - Text & CSV
Affects Versions: 1.7.0
Reporter: Rahul Challapalli
Priority: Critical


We started seeing the below 4 functional test failures in drill's extended 
tests [1]. The data for the below tests can be downloaded from [2]

{code}
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate28.q
framework/resources/Functional/tpcds/impala/text/q43.q
framework/resources/Functional/tpcds/variants/text/q6_1.sql
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate29.q
{code}

The failures started showing up from the commit [3]

[1] https://github.com/mapr/drill-test-framework
[2] http://apache-drill.s3.amazonaws.com/files/tpcds-sf1-text.tgz
[3] 
https://github.com/apache/drill/commit/223507b76ff6c2227e667ae4a53f743c92edd295

Let me know if more information is needed to reproduce this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4748) Verification Failures (Decimal values) in drill's regression tests

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4748:


 Summary: Verification Failures (Decimal values) in drill's 
regression tests
 Key: DRILL-4748
 URL: https://issues.apache.org/jira/browse/DRILL-4748
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types, Storage - Text & CSV
Affects Versions: 1.7.0
Reporter: Rahul Challapalli
Priority: Critical


We started seeing the below 4 functional test failures in drill's extended 
tests [1]. The data for the below tests can be downloaded from [2]

{code}
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate28.q
framework/resources/Functional/tpcds/impala/text/q43.q
framework/resources/Functional/tpcds/variants/text/q6_1.sql
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate29.q
{code}

The failures started showing up from the commit [3]

[1] https://github.com/mapr/drill-test-framework
[2] http://apache-drill.s3.amazonaws.com/files/tpcds-sf1-text.tgz
[3] 
https://github.com/apache/drill/commit/223507b76ff6c2227e667ae4a53f743c92edd295

Let me know if more information is needed to reproduce this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4747) Verification Failures (Decimal values) in drill's regression tests

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4747:


 Summary: Verification Failures (Decimal values) in drill's 
regression tests
 Key: DRILL-4747
 URL: https://issues.apache.org/jira/browse/DRILL-4747
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types, Storage - Text & CSV
Affects Versions: 1.7.0
Reporter: Rahul Challapalli
Priority: Critical


We started seeing the below 4 functional test failures in drill's extended 
tests [1]. The data for the below tests can be downloaded from [2]

{code}
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate28.q
framework/resources/Functional/tpcds/impala/text/q43.q
framework/resources/Functional/tpcds/variants/text/q6_1.sql
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate29.q
{code}

The failures started showing up from the commit [3]

[1] https://github.com/mapr/drill-test-framework
[2] http://apache-drill.s3.amazonaws.com/files/tpcds-sf1-text.tgz
[3] 
https://github.com/apache/drill/commit/223507b76ff6c2227e667ae4a53f743c92edd295

Let me know if more information is needed to reproduce this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4746) Verification Failures (Decimal values) in drill's regression tests

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4746:


 Summary: Verification Failures (Decimal values) in drill's 
regression tests
 Key: DRILL-4746
 URL: https://issues.apache.org/jira/browse/DRILL-4746
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types, Storage - Text & CSV
Affects Versions: 1.7.0
Reporter: Rahul Challapalli
Priority: Critical


We started seeing the below 4 functional test failures in drill's extended 
tests [1]. The data for the below tests can be downloaded from [2]

{code}
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate28.q
framework/resources/Functional/tpcds/impala/text/q43.q
framework/resources/Functional/tpcds/variants/text/q6_1.sql
framework/resources/Functional/aggregates/tpcds_variants/text/aggregate29.q
{code}

The failures started showing up from the commit [3]

[1] https://github.com/mapr/drill-test-framework
[2] http://apache-drill.s3.amazonaws.com/files/tpcds-sf1-text.tgz
[3] 
https://github.com/apache/drill/commit/223507b76ff6c2227e667ae4a53f743c92edd295

Let me know if more information is needed to reproduce this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68102519
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/SizedJBlock.java ---
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr;
+
+import com.sun.codemodel.JBlock;
+
+/**
+ * Uses this class to keep track # of Drill Logical Expressions that are
+ * put to JBlock.
+ */
+public class SizedJBlock {
+  private final JBlock block;
--- End diff --

Added comment. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68102435
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -215,22 +219,47 @@ public JVar 
declareVectorValueSetupAndMember(DirectExpression batchName, TypedFi
 return vv;
   }
 
+  public enum RotateMode {
--- End diff --

Rename it to BlcCreationMode. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #521: DRILL-4715: Fix java compilation error in run-time ...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/521#discussion_r68102305
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java ---
@@ -86,14 +88,16 @@ public static MappingSet getDefaultMapping() {
   }
 
   @SuppressWarnings("unchecked")
-  ClassGenerator(CodeGenerator codeGenerator, MappingSet mappingSet, 
SignatureHolder signature, EvaluationVisitor eval, JDefinedClass clazz, 
JCodeModel model) throws JClassAlreadyExistsException {
+  ClassGenerator(CodeGenerator codeGenerator, MappingSet mappingSet, 
SignatureHolder signature, EvaluationVisitor eval, JDefinedClass clazz, 
JCodeModel model, OptionManager optionManager) throws 
JClassAlreadyExistsException {
 this.codeGenerator = codeGenerator;
 this.clazz = clazz;
 this.mappings = mappingSet;
 this.sig = signature;
 this.evaluationVisitor = eval;
 this.model = model;
-blocks = (LinkedList[]) new LinkedList[sig.size()];
+this.optionManager = optionManager;
--- End diff --

Revise the code to pass in session option manager. 

Also refactor some existing unit tests to pass in option manager to unit 
tests. 

@sudheeshkatkam , could you please take a quick look to see if it addresses 
your comment? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[amansinha100]

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/517
  
Thanks for adding the tests.  I will take a look likely by tomorrow.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4745) Large compilation unit tests due to large code size or timing out

[Jinfeng Ni (JIRA)]

Jinfeng Ni created DRILL-4745:
-

 Summary: Large compilation unit tests due to large code size or 
timing out
 Key: DRILL-4745
 URL: https://issues.apache.org/jira/browse/DRILL-4745
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Codegen
Reporter: Jinfeng Ni


There unit tests in TestLargeFileCompilation fails on master branch. Part of 
the reason that the failures were not exposed is that TestLargeFileCompilation 
was excluded from normal "mvn install" build. 

1. testPARQUET_WRITER() : code too large in Copier operator
2. testTEXT_WRITER(): code too large in Copier operator
3. testFILTER() : hit error in Scalar Replacement code. Then time out.  The SR 
error might be related to DRILL-3995. 

DRILL-4715 is to address the issues of large code in project operator, and 
would not solve the above 3 failures.  Open this JIRA to keep track of those  
failures. 

[1]  
https://github.com/apache/drill/blob/master/exec/java-exec/pom.xml#L672-L674 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #531: DRILL-4733: max(dir0) reading more columns than nec...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/531#discussion_r68070931
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/TestImplicitFileColumns.java
 ---
@@ -110,4 +111,20 @@ public void testImplicitColumnsForParquet() throws 
Exception {
 .go();
   }
 
+  @Test // DRILL-4733
+  public void testMultilevelParquetWithSchemaChange() throws Exception {
+try {
+  test("alter session set `planner.enable_decimal_data_type` = true");
+  testBuilder()
+  .sqlQuery(String.format("select max(dir0) as max_dir from 
dfs_test.`%s/src/test/resources/multilevel/parquetWithSchemaChange`",
+  TestTools.getWorkingPath()))
+  .unOrdered()
+  .baselineColumns("max_dir")
+  .baselineValues("voter50.parquet")
--- End diff --

Got it. "voter50.parquet" is actually a folder name. I thought it's a 
parquet file. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #517: DRILL-4704 fix

[daveoshinsky]

Github user daveoshinsky commented on the issue:

https://github.com/apache/drill/pull/517
  
I saw no emails yesterday associated with my adding a unit test here for 
DRILL-4704 fix.  Just in case those got lost somewhere, please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #516: DRILL-4664: ScanBatch.isNewSchema() returns wrong result f...

[vdiravka]

Github user vdiravka commented on the issue:

https://github.com/apache/drill/pull/516
  
@hnfgns Could you please review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4744) Fully Qualified JDBC Plugin Tables return Table not Found via Rest API

[John Omernik (JIRA)]

John Omernik created DRILL-4744:
---

 Summary: Fully Qualified JDBC Plugin Tables return Table not Found 
via Rest API
 Key: DRILL-4744
 URL: https://issues.apache.org/jira/browse/DRILL-4744
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - JDBC
Affects Versions: 1.6.0
Reporter: John Omernik
Priority: Minor


When trying to query a JDBC table via authenticated Rest API, using a fully 
qualified table name returns table not found.  This does not occur in sqlline, 
and a workaround is to "use pluginname.mysqldatabase" prior to the query. (Then 
the fully qualified table name will work)

Plugin Name: mysql
Mysql Database: events
Mysql Table: curevents

Via Rest:

select * from mysql.events.curevents limit 10;

Fail with "VALIDATION ERROR "Table 'mysql.events.curevents' not found

Via Rest:

use mysql.events;
select * from mysql.events.curevents limit 10;

- Success. 

Via SQL line, authenticating with the same username, you can connect, and run 
select * from mysql.events.curevents limit 10;

without issue. (and without the use mysql.events)






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)