[GitHub] drill pull request: DRILL-4479: Use varchar for default column whe...

[amansinha100]

GitHub user amansinha100 opened a pull request:

https://github.com/apache/drill/pull/420

DRILL-4479: Use varchar for default column when all_text_mode is enab…

…led.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amansinha100/incubator-drill DRILL-4479

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #420


commit c5b4aef5b35547561ea71ce880391429643a6ee0
Author: Aman Sinha 
Date:   2016-03-08T17:27:32Z

DRILL-4479: Use varchar for default column when all_text_mode is enabled.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4492) TestMergeJoinWithSchemaChanges depends on order files in a directory are read to pass, should be refactored

[Jason Altekruse (JIRA)]

Jason Altekruse created DRILL-4492:
--

 Summary: TestMergeJoinWithSchemaChanges depends on order files in 
a directory are read to pass, should be refactored
 Key: DRILL-4492
 URL: https://issues.apache.org/jira/browse/DRILL-4492
 Project: Apache Drill
  Issue Type: Bug
Reporter: Jason Altekruse
Assignee: amit hadke


I was running unit tests and saw a failure that seemed unrelated to the changes 
I was making. The test runs fine in isolation both from IntelliJ and the maven 
command line (with -Dtest=TestMergeJoinWithSchemaChanges in the java-exec 
module).

Not sure what about the particular test run made it change the order the files 
were read, but we cannot rely on any particular system to read the files in a 
given order. The test should be updated to remove this assumption.

This is the error I received on one run of the full unit tests:
{code}
testMissingAndNewColumns(TestMergeJoinWithSchemaChanges.java:265)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: 
UNSUPPORTED_OPERATION ERROR: Sort doesn't currently supportsorts with 
changing schemas

Fragment 0:0

[Error Id: bf84bffb-f643-493b-9ed5-720eb18d55f2 on 10.1.10.225:31010]

  (org.apache.drill.exec.exception.SchemaChangeException) Sort currently only 
supports a single schema.
org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.build():146
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():442
org.apache.drill.exec.record.AbstractRecordBatch.next():162

org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():215
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51

org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
org.apache.drill.exec.record.AbstractRecordBatch.next():162

org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():215
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.RecordIterator.nextBatch():97
org.apache.drill.exec.record.RecordIterator.next():183
org.apache.drill.exec.record.RecordIterator.prepare():167
org.apache.drill.exec.physical.impl.join.JoinStatus.prepare():87
org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():162
org.apache.drill.exec.record.AbstractRecordBatch.next():162

org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():215
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51

org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
org.apache.drill.exec.record.AbstractRecordBatch.next():162

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4491: FormatPluginOptionsDescriptor requ...

[adityakishore]

Github user adityakishore commented on the pull request:

https://github.com/apache/drill/pull/418#issuecomment-194097775
  
Until I looked at the code, I was under assumption that we are using 
Jackson to extract the serializable properties. We can, and should, definitely 
go that route.

The way code currently works is that it iterate through allthe table 
options and see if there is a Java field present in the corresponding 
FormatPluginConfig class. If it does find one, and this is why I say it is a 
bug in the current implementation, it makes is accessible 
(`setAccesible(true)`) implying that it is expected to work with non-public 
fields and sets the value to the one passed as parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4482: Avro subselection broken by 4382

[StevenMPhillips]

Github user StevenMPhillips commented on the pull request:

https://github.com/apache/drill/pull/419#issuecomment-194060258
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4482: Avro subselection broken by 4382

[jaltekruse]

GitHub user jaltekruse opened a pull request:

https://github.com/apache/drill/pull/419

DRILL-4482: Avro subselection broken by 4382

This fix includes a number of test updates to ensure Avro files are being 
read correctly.

The branch includes 4441, which is on a different PR, but touched some of 
the same code, so I just based this fix on that branch.

The actual regression fix is in the AvroRecordReader, in the case of a 
Union, we should not be created a child of the fieldSelection, which was 
properly done in the case with maps and records.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaltekruse/incubator-drill 4441-4482-avro-bugs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #419


commit 56048dae231a6c11c2650384da6893a8c1011fee
Author: Jason Altekruse 
Date:   2016-02-26T17:55:05Z

DRILL-4441: Fix varchar data read out of Avro filtering incorrectly due to 
metadata bug

The precision of the Varchar datatype was not being set causing inconsistent
truncation of values to the default length of 1. Fixed the same issue with 
varbinary.

The test framework was previously taking a string as the baseline for a 
binary value,
which cannot express all possible values. Fixed the test to intstead use a 
byte array.
Thie required updating the hive tests that were using the old method of 
specifying
baselines with a String.

Fix cast to varbinary when reading from a data source with schema needed 
for writing
a test.

commit 15209ea07a41b0a7bdccb382950b5738bd229b18
Author: Jason Altekruse 
Date:   2016-03-08T22:16:03Z

DRILL-4482: Fix Avro nested field selection regression

Update some of the Avro tests to properly verify their results,
others still need to be fixed. These will be addressed in DRILL-4110.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/416


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for the 1.6 Release

[Parth Chandra]

Okay we are down to the final one -

DRILL-4482 - Avro no longer selects data correctly from a
sub-structure.(Jason)

Note that MapR QA team is going to start testing 1.6 snapshot now before I
roll out the release candidate. DRILL-4482 can be merged in later as it is
not likely to affect the  Hopefully there will be no show stoppers (.

The plan is to roll out the release candidate by Thursday.

Thanks

Parth


On Tue, Mar 8, 2016 at 9:31 AM, Parth Chandra  wrote:

> OK, let's leave it out then.
>
> On Tue, Mar 8, 2016 at 9:25 AM, Jason Altekruse 
> wrote:
>
>> To be honest I was expecting a longer review cycle so I hadn't run the
>> unit
>> tests before posting it for review. There were only very minor functional
>> changes, so I wasn't thinking it would be an issue, and I was anticipating
>> having to update the patch before merging it. I could update the test that
>> is failing but I don't see much sense in trying to get it into the release
>> because it only introduces new tests and some small core refactoring.
>>
>> I'm all for getting it merged so everyone can start using it, I just think
>> it doesn't really matter if it happens on the release branch or back on
>> master once we cut a release branch.
>>
>> I would rather try to focus on getting the Avro issues resolved, which is
>> what I'm working on right now.
>>
>> - Jason
>>
>> On Tue, Mar 8, 2016 at 8:58 AM, Parth Chandra  wrote:
>>
>> > Sounds good Jason. Let's finalize this in the hangout.
>> > Do you have the expected plans for the failing tests? If so can you
>> update
>> > those and put in a pull request and we'll merge and run the tests.
>> > Any reason for the operator test framework to be punted? You have a +1
>> to
>> > merge it.
>> >
>> >
>> >
>> > On Mon, Mar 7, 2016 at 9:33 PM, Khurram Faraaz 
>> > wrote:
>> >
>> > > We should update the expected results (i.e. the expected query plan in
>> > this
>> > > case) and not mark them as Failing. We do not have a Failing test
>> > directory
>> > > today.
>> > >
>> > > If we are sure the expected query plans for these tests are incorrect
>> > > (today), we should go ahead and update the expected results files with
>> > the
>> > > correct query plans.
>> > >
>> > > Functional/ctas/ctas_auto_partition/existing_partition_
>> > > pruning/hierarchical/plan/count2_MD-185.q
>> > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_2.q
>> > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_4.q
>> > >
>> > > Thanks,
>> > > Khurram
>> > >
>> > > On Tue, Mar 8, 2016 at 8:58 AM, Jason Altekruse <
>> > altekruseja...@gmail.com>
>> > > wrote:
>> > >
>> > > > The cutoff time sounds good to me. I don't think I'm too
>> overburdened,
>> > I
>> > > > mostly tried putting together a merge branch today and had to kick a
>> > few
>> > > > things out as I found issues.
>> > > >
>> > > > As we are not running the full regression cluster at Dremio, it
>> would
>> > be
>> > > > helpful if someone could merge Jacques patch (DRILL-4474) after
>> running
>> > > the
>> > > > complete set of tests. Could someone at MapR pick up the test
>> updates
>> > and
>> > > > merging this patch?
>> > > >
>> > > > These are the failures I was seeing so far, my fix was just going
>> to be
>> > > to
>> > > > mark the tests as failing:
>> > > >
>> > > >
>> > >
>> >
>> Functional/ctas/ctas_auto_partition/existing_partition_pruning/hierarchical/plan/count2_MD-185.q
>> > > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_2.q
>> > > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_4.q
>> > > >
>> > > > I will just go ahead and merge the other two fixes as I had a clean
>> > unit
>> > > > test run and neither of the patches will have any impact on the
>> > > regression
>> > > > tests. One is a build fix, that only changes behavior when running
>> the
>> > > > release profile and the other is a unit test fix.
>> > > >
>> > > > I had punted the operator unit tests out of the release, so those
>> can
>> > be
>> > > > removed from your list.
>> > > >
>> > > > I will start looking back at both of the Avro issues right after
>> > merging.
>> > > >
>> > > > On Mon, Mar 7, 2016 at 6:32 PM, Parth Chandra <
>> pchan...@maprtech.com>
>> > > > wrote:
>> > > >
>> > > > > The train is about to leave :). I'd like to propose a cutoff for
>> > > > tomorrow 5
>> > > > > pm.
>> > > > > (This will be a full week after the initial mail so plenty of time
>> > for
>> > > > > everyone to get their issues aired.)
>> > > > >
>> > > > > Here are the open items (that I am aware of) -
>> > > > >
>> > > > > Waiting to be merged
>> > > > > DRILL-4483/pr 411 (Fix text plan regression in query profiles)
>> > (Venki)
>> > > > > DRILL-4437 (and others)/pr 394 (Operator unit test framework).
>> > (Jason)
>> > > > > DRILL-4332/pr 389 (Make vector comparison order stable in test
>> > > framework)
>> > > > > (?)
>> > > > > DRILL-4474/pr 406 (Ensure 

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/406


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4491: FormatPluginOptionsDescriptor requ...

[jaltekruse]

Github user jaltekruse commented on the pull request:

https://github.com/apache/drill/pull/418#issuecomment-194049783
  
Is there a way to ask Jackson to look at a POJO and give you what it 
considers to be the serializable fields? This isn't quite the right fix because 
it is valid to have private members in a class and only have some of the 
serialized and deserialized by Jackson based on gettters/setters. This would 
expose the non-serializable fields as options.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/416#discussion_r55457004
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -117,6 +117,24 @@ public void onMatch(RelOptRuleCall call) {
   } else if (aggCall.getArgList().size() == 1) {
   // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
 int index = aggCall.getArgList().get(0);
+
+if (proj != null) {
+  // project in the middle of Agg and Scan : Only when input of 
AggCall is a RexInputRef in Project, we find the index of Scan's field.
+  // For instance,
+  // Agg - count($0)
+  //  \
+  //  Proj - Exp={$1}
+  //\
+  //   Scan (col1, col2).
+  // return count of "col2" in Scan's metadata, if found.
+
+  if (proj.getProjects().get(index) instanceof RexInputRef) {
+index = ((RexInputRef) 
proj.getProjects().get(index)).getIndex();
--- End diff --

I added two more case: count(100) and count(1+2) to check the count 
pushdown happen. 

Currently, count( 1 + non-nulllable-col) will not pushdown count. That's 
because Drill return nullable-ANY for "+" function.  With Sean's type exposure 
patch, it should get the right type for "+" in planning time, and it will get 
count pushddown as well. 
 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4491: FormatPluginOptionsDescriptor requ...

[adityakishore]

GitHub user adityakishore opened a pull request:

https://github.com/apache/drill/pull/418

DRILL-4491: FormatPluginOptionsDescriptor requires FormatPluginConfig…

… fields to be public

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adityakishore/drill DRILL-4491

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/418.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #418


commit cce8467c2476da871891bad7db6cab3236537f7c
Author: Aditya Kishore 
Date:   2016-03-09T00:49:55Z

DRILL-4491: FormatPluginOptionsDescriptor requires FormatPluginConfig 
fields to be public




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[amansinha100]

Github user amansinha100 commented on the pull request:

https://github.com/apache/drill/pull/416#issuecomment-194042039
  
Overall, LGTM.  +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4491) FormatPluginOptionsDescriptor requires FormatPluginConfig fields to be public

[Aditya Kishore (JIRA)]

Aditya Kishore created DRILL-4491:
-

 Summary: FormatPluginOptionsDescriptor requires FormatPluginConfig 
fields to be public
 Key: DRILL-4491
 URL: https://issues.apache.org/jira/browse/DRILL-4491
 Project: Apache Drill
  Issue Type: Bug
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Minor


The code uses {{getField()}} instead of {{getDeclaredField()}}, which returns 
only the public fields.

{code:title=FormatPluginOptionsDescriptor.java:165|borderStyle=solid}
Field field = pluginConfigClass.getField(paramDef.name);
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/416#discussion_r55455931
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -117,6 +117,24 @@ public void onMatch(RelOptRuleCall call) {
   } else if (aggCall.getArgList().size() == 1) {
   // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
 int index = aggCall.getArgList().get(0);
+
+if (proj != null) {
+  // project in the middle of Agg and Scan : Only when input of 
AggCall is a RexInputRef in Project, we find the index of Scan's field.
+  // For instance,
+  // Agg - count($0)
+  //  \
+  //  Proj - Exp={$1}
+  //\
+  //   Scan (col1, col2).
+  // return count of "col2" in Scan's metadata, if found.
+
+  if (proj.getProjects().get(index) instanceof RexInputRef) {
+index = ((RexInputRef) 
proj.getProjects().get(index)).getIndex();
--- End diff --

Make sense. Let me add more unit test in the patch. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/416#discussion_r55455592
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -117,6 +117,24 @@ public void onMatch(RelOptRuleCall call) {
   } else if (aggCall.getArgList().size() == 1) {
   // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
 int index = aggCall.getArgList().get(0);
+
+if (proj != null) {
+  // project in the middle of Agg and Scan : Only when input of 
AggCall is a RexInputRef in Project, we find the index of Scan's field.
+  // For instance,
+  // Agg - count($0)
+  //  \
+  //  Proj - Exp={$1}
+  //\
+  //   Scan (col1, col2).
+  // return count of "col2" in Scan's metadata, if found.
+
+  if (proj.getProjects().get(index) instanceof RexInputRef) {
+index = ((RexInputRef) 
proj.getProjects().get(index)).getIndex();
--- End diff --

might be good to add a test case for that just in case calcite changes this 
behavior in future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4485 - MapR profile - switch to MapR 5.1...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/416#discussion_r55454105
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -117,6 +117,24 @@ public void onMatch(RelOptRuleCall call) {
   } else if (aggCall.getArgList().size() == 1) {
   // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
 int index = aggCall.getArgList().get(0);
+
+if (proj != null) {
+  // project in the middle of Agg and Scan : Only when input of 
AggCall is a RexInputRef in Project, we find the index of Scan's field.
+  // For instance,
+  // Agg - count($0)
+  //  \
+  //  Proj - Exp={$1}
+  //\
+  //   Scan (col1, col2).
+  // return count of "col2" in Scan's metadata, if found.
+
+  if (proj.getProjects().get(index) instanceof RexInputRef) {
+index = ((RexInputRef) 
proj.getProjects().get(index)).getIndex();
--- End diff --

Calcite rewrote count(100) or count(1) into count() ==> 
aggCall.getArgList.isEmpty() is true. So Line 113 will take care of those 
cases. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jacques-n]

Github user jacques-n commented on a diff in the pull request:

https://github.com/apache/drill/pull/416#discussion_r55453543
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -117,6 +117,24 @@ public void onMatch(RelOptRuleCall call) {
   } else if (aggCall.getArgList().size() == 1) {
   // count(columnName) ==> Agg ( Scan )) ==> columnValueCount
 int index = aggCall.getArgList().get(0);
+
+if (proj != null) {
+  // project in the middle of Agg and Scan : Only when input of 
AggCall is a RexInputRef in Project, we find the index of Scan's field.
+  // For instance,
+  // Agg - count($0)
+  //  \
+  //  Proj - Exp={$1}
+  //\
+  //   Scan (col1, col2).
+  // return count of "col2" in Scan's metadata, if found.
+
+  if (proj.getProjects().get(index) instanceof RexInputRef) {
+index = ((RexInputRef) 
proj.getProjects().get(index)).getIndex();
--- End diff --

Doesn't this mean count(100) & count(1) still fail to pushdown?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4485 - MapR profile - switch to MapR 5.1...

[pwong-mapr]

GitHub user pwong-mapr opened a pull request:

https://github.com/apache/drill/pull/417

DRILL-4485 - MapR profile - switch to MapR 5.1.0, and improve compatibility 
with maprfs storage format and MapR DB storage plugin



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pwong-mapr/incubator-drill DRILL-4485-4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #417


commit fc076488cb88e1071ef403c300a8681d0b9c584c
Author: Patrick Wong 
Date:   2016-03-08T02:22:08Z

DRILL-4485 - MapR profile - switch to MapR 5.1.0, and improve compatibility 
with maprfs storage format and MapR DB storage plugin




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

GitHub user jinfengni opened a pull request:

https://github.com/apache/drill/pull/416

DRILL-4474: Ensure that ConvertCountToDirectScan does not push through 
project when nullable input of count is not RexInputRef



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinfengni/incubator-drill review/DRILL-4474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/416.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #416


commit 0a5f8fab786f931665d9d28ea67cf19ab37c07fb
Author: Jacques Nadeau 
Date:   2016-03-04T21:27:26Z

DRILL-4474: Ensure that ConvertCountToDirectScan only pushes through 
project when project is trivial.

commit ab00e6aa9563d79e62154ba1f3bbb71dba7d8036
Author: Jinfeng Ni 
Date:   2016-03-08T22:15:27Z

DRILL-4474: Ensure that ConvertCountToDirectScan does not push through 
project when nullable input of count is not RexInputRef




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4485 - switch to MapR 5.1.0

[pwong-mapr]

Github user pwong-mapr closed the pull request at:

https://github.com/apache/drill/pull/413


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4490) Count(*) function returns as optional instead of required

[Krystal (JIRA)]

Krystal created DRILL-4490:
--

 Summary: Count(*) function returns as optional instead of required
 Key: DRILL-4490
 URL: https://issues.apache.org/jira/browse/DRILL-4490
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types
Affects Versions: 1.6.0
Reporter: Krystal
Assignee: Sean Hsuan-Yi Chu


git.commit.id.abbrev=c8a7840

I have the following CTAS query:
create table test as select count(*) as col1 from cp.`tpch/orders.parquet`;

The schema of the test table shows col1 as optional:
message root {
  optional int64 col1;
}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/406#discussion_r5563
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
   return;
 }
 
+if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
--- End diff --

I have a patch, which works fine for Jacque's new unit test. It continues 
to use directScan for simple count query. The patch is pending pre-commit & 
unit test run. Will update results shortly. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Drill Hangout Today?

[Jason Altekruse]

Meeting notes - Hangout 3/8/2016

Parth, Aman, John O., Jason, Zelaine


- release

- 4474 - incorect direct scans, MapR will run regression tests

- Avro issues, one is ready, had a +1 there is just an update for a
related issue

- the other one Jason was having an issue with repro


- Union type

- Aman - what is needed to make union type complete

- Steven could give the best answer for known shortcomings

- Jason - I think mostly we just need more thorough testing

- Aman - we need to go through all of the operators to update for

  schema change

- Jason - Union type and handling schema change are really

 two separate issues, but we should discuss a path forward

 to making both work well in Drill

- John - handling messy JSON better would be really useful for users

- doesn't need to fix everything automatically, but giving a user
info

  about why something is failing and what the next step for getting
to

  further analysis is useful


- User experience - John O.

- partial JSON records

- JSON incosistent errors, sometimes not givng line numbers

- Feature request, show the record that failed to parse

- Just give users enough info so that they know what to fix

- Drill can try to have good defaults about how to handle

  abmiguities, but making a user choose anytime Drill cannot

  be sure is fine

- metadata cache issues

- permissions issues with authentication

On Tue, Mar 8, 2016 at 10:18 AM, Parth Chandra  wrote:

> Joining in a minute
>
> On Tue, Mar 8, 2016 at 10:17 AM, Jason Altekruse  >
> wrote:
>
> > For anyone else interested in joining the hangout here is the link.
> >
> > https://plus.google.com/hangouts/_/dremio.com/drillhangout?authuser=1
> >
> > On Tue, Mar 8, 2016 at 10:15 AM, Jason Altekruse <
> altekruseja...@gmail.com
> > >
> > wrote:
> >
> > > Yes, sorry I forgot to sign on.
> > >
> > > Can you try to join again?
> > >
> > > On Tue, Mar 8, 2016 at 10:10 AM, Zelaine Fong 
> > wrote:
> > >
> > >> Are we having one today?  We're trying to connect from the MapR end,
> but
> > >> not getting a response.
> > >>
> > >> -- Zelaine
> > >>
> > >
> > >
> >
>

[jira] [Created] (DRILL-4489) Add ValueVector tests from Drill

[Steven Phillips (JIRA)]

Steven Phillips created DRILL-4489:
--

 Summary: Add ValueVector tests from Drill
 Key: DRILL-4489
 URL: https://issues.apache.org/jira/browse/DRILL-4489
 Project: Apache Drill
  Issue Type: Bug
Reporter: Steven Phillips


There are some simple ValueVector tests that should be included in the Arrow 
project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: UnrecognizedPropertyException: Unrecognized field "config" (class org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable (4 known properties: "enabled", "formats", "connection"

[Abhishek Girish]

You mentioned you were able to successfully run queries now, right? I tried
on the latest build and on MapR Drill RPM - don't see any issues. So this
might have been a one-off issue. If you happen to reproduce it again, we
could investigate this further.

On Tue, Mar 8, 2016 at 10:11 AM, Khurram Faraaz 
wrote:

> I am running against MapR rpm. I did not build from source. This is the RPM
> that was used mapr-drill-1.6.0.201603072015-1.noarch.rpm
>
> On Tue, Mar 8, 2016 at 11:34 PM, Abhishek Girish  wrote:
>
> > Khurram,
> >
> > Can you confirm if this issue is specific to MapR RPMs or is seen with
> > latest builds as well?
> >
> > -Abhishek
> >
> > On Tue, Mar 8, 2016 at 9:22 AM, Khurram Faraaz 
> > wrote:
> >
> > > Thanks Jason. Here is what I did before I hit the Exception
> > >
> > > clush -g khurram rpm -e mapr-drill --noscripts
> > > clush -g khurram wget
> > >
> http://yum.qa.lab/opensource/mapr-drill-1.6.0.201603072015-1.noarch.rpm
> > > clush -g khurram rpm -i mapr-drill-1.6.0.201603072015-1.noarch.rpm
> > >
> > > cd /opt/mapr/zookeeper/zookeeper-3.4.5/bin
> > >
> > > ./zkCli.sh
> > >
> > > connect localhost:5181
> > >
> > > ls /drill
> > >
> > > rmr /drill/sys.options
> > >
> > > cd /opt/mapr/drill/drill-1.6.0/bin
> > > ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> > >
> > > Any query on sqlline would give that Exception.
> > >
> > > I then restarted warden, clush -g khurram service mapr-warden stop and
> > then
> > > start, and I am able to run queries now.
> > >
> > > Do we need a JIRA to track this problem ?
> > >
> > > - Khurram
> > >
> > > On Tue, Mar 8, 2016 at 9:40 PM, Jason Altekruse <
> > altekruseja...@gmail.com>
> > > wrote:
> > >
> > > > This exception should only occur if you start an older version of
> Drill
> > > > using a configuration (stored in zookeeper or your local temp
> > directory)
> > > > that was created by starting a version of Drill after 4383 was merged
> > > > (0842851c854595f140779e9ed09331dbb63f6623).
> > > >
> > > > This change added a new property to filesystem configuration to allow
> > > > passing custom options to the filesystem config. This can be used in
> > > place
> > > > of core-site.xml to set things like your AWS private keys, as well as
> > any
> > > > other properties normally provided to an implementation of the Hadoop
> > > > FileSystem API.
> > > >
> > > > Removing the new configuration should allow it to start up, but you
> > > > shouldn't be seeing this if you are running the build you mentioned.
> > Can
> > > > you verify that this version successfully built and that you are not
> > > > running an older version?
> > > >
> > > > - Jason
> > > >
> > > > P.S. I will be trying to get in a change soon that give a better
> error
> > in
> > > > this case, it should only happen with downgrades, which we generally
> > > don't
> > > > thoroughly test, but would still be good to fix. I'm sure there are
> > > several
> > > > bugs filed about these kinds of issues, this is one of them and I've
> > > > assigned it to myself, hoping to post a fix soon.
> > > >
> > > > https://issues.apache.org/jira/browse/DRILL-2048
> > > >
> > > >
> > > > On Tue, Mar 8, 2016 at 2:33 AM, Khurram Faraaz  >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am seeing an Exception on Drill 1.6.0 commit ID 447b093c (I am
> > using
> > > > the
> > > > > RPM)
> > > > >
> > > > > I did not see this Exception on earlier version of Drill 1.6.0
> commit
> > > ID
> > > > > 6d5f4983
> > > > >
> > > > > Could this be related to DRILL-4383
> > > > > 
> > > > >
> > > > > Drill version where we see the Exception is
> > > > >
> > > > > git.commit.id=447b093cd2b05bfeae001844a7e3573935e84389
> > > > > git.commit.message.short=DRILL-4332\: Makes vector comparison order
> > > > stable
> > > > > in test framework
> > > > >
> > > > > oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> > > > ERROR:
> > > > > UnrecognizedPropertyException: Unrecognized field "config" (class
> > > > > org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as
> > > > ignorable
> > > > > (4 known properties: "enabled", "formats", "connection",
> > "workspaces"])
> > > > >  at [Source: [B@2b88d9b2; line: 5, column: 18] (through reference
> > > chain:
> > > > > org.apache.drill.exec.store.dfs.FileSystemConfig["config"])
> > > > >
> > > > >
> > > > > [Error Id: 7fdc89ac-91ac-46eb-8201-8fe5e1acf278 on
> > > > centos-02.qa.lab:31010]
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> 

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[amansinha100]

Github user amansinha100 commented on the pull request:

https://github.com/apache/drill/pull/406#issuecomment-193971973
  
Agree with @jinfengni that the current fix can cause performance regression 
for simpler count queries.  I will change my review to -1 and let's see how to 
get the proper nullability check. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/406#discussion_r55428219
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
   return;
 }
 
+if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
--- End diff --

With the patch, the following query will not use directScan. 

{code}
select count(*) from cp.`tpch/nation.parquet`;
{code}

{code}
00-00Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, 
cumulative cost = {75.1 rows, 425.1 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 
169
00-01  Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): 
rowcount = 1.0, cumulative cost = {75.0 rows, 425.0 cpu, 0.0 io, 0.0 network, 
0.0 memory}, id = 168
00-02StreamAgg(group=[{}], EXPR$0=[COUNT()]) : rowType = 
RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {75.0 rows, 425.0 
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 167
00-03  Project($f0=[0]) : rowType = RecordType(INTEGER $f0): 
rowcount = 25.0, cumulative cost = {50.0 rows, 125.0 cpu, 0.0 io, 0.0 network, 
0.0 memory}, id = 166
00-04Scan(groupscan=[ParquetGroupScan 
[entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]], 
selectionRoot=classpath:/tpch/nation.parquet, numFiles=1, 
usedMetadataFile=false, columns=[]]]) : rowType = RecordType(): rowcount = 
25.0, cumulative cost = {25.0 rows, 25.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, 
id = 165
{code}

I debug a bit. Seems Line 115 is fine. But something is worng in the code 
Line 117 - 123. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4487: add unit test for DRILL-4449

[amansinha100]

Github user amansinha100 commented on the pull request:

https://github.com/apache/drill/pull/414#issuecomment-193960370
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/406#discussion_r55422075
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
   return;
 }
 
+if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
--- End diff --

We have a check whether the input to count() is nullable in Line 115. In 
theory, if the input is non-nullable, then count(non-nullalbe expression) = 
rowcount.

My guess is that the query (case expression) with incorrect result is 
caused by the wrong type resolution for the case expression. 

 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

[jinfengni]

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/406#discussion_r55421789
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java
 ---
@@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
   return;
 }
 
+if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
--- End diff --

I feel that this check might over-kill some optimization opportunity.  For 
example,

select count(100) 
from `parquetTable`;

In this case, count(100) is equal to rowcount in parquet table. However, 
the project is not a trial project, meaning the new code will disable the 
optimization.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-4488) Prefix "-" cause failure (NPE) in constant folding

[Sean Hsuan-Yi Chu (JIRA)]

Sean Hsuan-Yi Chu created DRILL-4488:


 Summary: Prefix "-" cause failure (NPE) in constant folding
 Key: DRILL-4488
 URL: https://issues.apache.org/jira/browse/DRILL-4488
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Sean Hsuan-Yi Chu
Assignee: Sean Hsuan-Yi Chu


For example, a query like this one:
{code}
SELECT -sqrt(5) as col
from cp.`tpch/nation.parquet`
{code}
gives NPE. 

The reason is because of the translation of prefix "-" to -1 .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4184: support variable length decimal fi...

[daveoshinsky]

Github user daveoshinsky commented on a diff in the pull request:

https://github.com/apache/drill/pull/372#discussion_r55417098
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableVarLengthValuesColumn.java
 ---
@@ -69,11 +73,16 @@ protected boolean readAndStoreValueSizeInformation() 
throws IOException {
 if ( currDefLevel == -1 ) {
   currDefLevel = pageReader.definitionLevels.readInteger();
 }
-if ( columnDescriptor.getMaxDefinitionLevel() > currDefLevel) {
+
+if (columnDescriptor.getMaxDefinitionLevel() > currDefLevel) {
   nullsRead++;
-  // set length of zero, each index in the vector defaults to null so 
no need to set the nullability
-  variableWidthVector.getMutator().setValueLengthSafe(
-  valuesReadInCurrentPass + pageReader.valuesReadyToRead, 0);
+  // set length of zero, each index in the vector defaults to null so 
no
+  // need to set the nullability
+  if (variableWidthVector == null) {
--- End diff --

Regarding the two variables variableWidthVector and fixedWidthVector that I 
added, here is my reasoning.  Either variableWidthVector is set if we have a 
VariableWidthVector, or fixedWidthVector is set if we have a FixedWidthVector 
(i.e., decimal).  Hence, variableWidthVector is non-null if and only if we are 
to invoke the pre-existing logic, that assumed a variable width vector.  When 
variableWidthVector is null (fixedWidthVector is non-null, but not currently 
used), we invoke the new logic to save the length information in 
decimalLengths.  If this is no good, please tell me why, and suggest an 
alternative.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: On improving project maintainence

[John Omernik]

I've been meaning to respond to this (Even have the spreadsheet still open)

I want to volunteer to do more list responding.  Not sure what I have for
time commitment, but in general, I want to be more helpful to user
questions etc.   We could also start to evaluate some of the response we
get and try to better incorporate those into the troubleshooting section of
the documentation.  I know that I go back to it to refresh my memory on
tips I can try to fix my problems.

Perhaps if we had list managers helping out, one of the responsibilities
could/would be to generalize responses and make contributions back to the
documentation or to the devs on how error messages could be improved? If
some of the "adhoc" knowledge could be baked into the docs, or even into
Drill itself it may start help both adoption and "expert" use of the
project.



On Tue, Mar 1, 2016 at 2:01 PM, Jason Altekruse 
wrote:

> Hello devs,
>
> I think everyone has noticed that there are some parts of project
> maintenance that have been lagging in the past few months.
>
> The good new is that the list has been really active, and I think that we
> really have been trying to get back to everyone. Despite a lot of responses
> from a lot of committers and contributors, there have been at least a fair
> number of threads that received no response, or didn't lead to a resolution
> of the issue for the user.
>
> On github there are currently 80ish open pull requests, while some are
> abandoned or replaced by other work, there are a number of instances of
> good contributions that are waiting for review.
>
> I don't have some magical prescription about how to solve this, but one
> small change we could make would be revive this document [1] for
> designating a primary list manager for each week. This role does not have
> to be terribly burdening, or even require a committer to fulfill it. Many
> of the questions on the list simply need to be marshalled into a JIRA with
> enough info for a reproduction of the bug, or in other cases just require a
> pointer to a doc page or existing JIRA on the thread to answer a question.
>
> As far as the outstanding reviews are concerned, it might make sense for
> the list manager to also try to make sure that contributions have an
> assigned reviewer when they are posted.
>
> Thoughts? Does it make sense to try to get something like this going, is
> there something that made this effort fade away the last time we tried it
> that we should change?
>
> [1]  -
>
> https://docs.google.com/spreadsheets/d/1bEQKk16Kktb1XeZwKD8xCuhaO8FtNfF1Cr2rcTv1a6M
>

[GitHub] drill pull request: DRILL-4474: Use varchar for default column whe...

[amansinha100]

Github user amansinha100 closed the pull request at:

https://github.com/apache/drill/pull/415


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Use varchar for default column whe...

[amansinha100]

Github user amansinha100 commented on the pull request:

https://github.com/apache/drill/pull/415#issuecomment-193901238
  
oops ... sorry, closing this and will reopen against the correct JIRA. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Drill Hangout Today?

[Parth Chandra]

Joining in a minute

On Tue, Mar 8, 2016 at 10:17 AM, Jason Altekruse 
wrote:

> For anyone else interested in joining the hangout here is the link.
>
> https://plus.google.com/hangouts/_/dremio.com/drillhangout?authuser=1
>
> On Tue, Mar 8, 2016 at 10:15 AM, Jason Altekruse  >
> wrote:
>
> > Yes, sorry I forgot to sign on.
> >
> > Can you try to join again?
> >
> > On Tue, Mar 8, 2016 at 10:10 AM, Zelaine Fong 
> wrote:
> >
> >> Are we having one today?  We're trying to connect from the MapR end, but
> >> not getting a response.
> >>
> >> -- Zelaine
> >>
> >
> >
>

Re: Drill Hangout Today?

[Jason Altekruse]

For anyone else interested in joining the hangout here is the link.

https://plus.google.com/hangouts/_/dremio.com/drillhangout?authuser=1

On Tue, Mar 8, 2016 at 10:15 AM, Jason Altekruse 
wrote:

> Yes, sorry I forgot to sign on.
>
> Can you try to join again?
>
> On Tue, Mar 8, 2016 at 10:10 AM, Zelaine Fong  wrote:
>
>> Are we having one today?  We're trying to connect from the MapR end, but
>> not getting a response.
>>
>> -- Zelaine
>>
>
>

Re: Drill Hangout Today?

[Jason Altekruse]

Yes, sorry I forgot to sign on.

Can you try to join again?

On Tue, Mar 8, 2016 at 10:10 AM, Zelaine Fong  wrote:

> Are we having one today?  We're trying to connect from the MapR end, but
> not getting a response.
>
> -- Zelaine
>

Re: UnrecognizedPropertyException: Unrecognized field "config" (class org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable (4 known properties: "enabled", "formats", "connection"

[Khurram Faraaz]

I am running against MapR rpm. I did not build from source. This is the RPM
that was used mapr-drill-1.6.0.201603072015-1.noarch.rpm

On Tue, Mar 8, 2016 at 11:34 PM, Abhishek Girish  wrote:

> Khurram,
>
> Can you confirm if this issue is specific to MapR RPMs or is seen with
> latest builds as well?
>
> -Abhishek
>
> On Tue, Mar 8, 2016 at 9:22 AM, Khurram Faraaz 
> wrote:
>
> > Thanks Jason. Here is what I did before I hit the Exception
> >
> > clush -g khurram rpm -e mapr-drill --noscripts
> > clush -g khurram wget
> > http://yum.qa.lab/opensource/mapr-drill-1.6.0.201603072015-1.noarch.rpm
> > clush -g khurram rpm -i mapr-drill-1.6.0.201603072015-1.noarch.rpm
> >
> > cd /opt/mapr/zookeeper/zookeeper-3.4.5/bin
> >
> > ./zkCli.sh
> >
> > connect localhost:5181
> >
> > ls /drill
> >
> > rmr /drill/sys.options
> >
> > cd /opt/mapr/drill/drill-1.6.0/bin
> > ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> >
> > Any query on sqlline would give that Exception.
> >
> > I then restarted warden, clush -g khurram service mapr-warden stop and
> then
> > start, and I am able to run queries now.
> >
> > Do we need a JIRA to track this problem ?
> >
> > - Khurram
> >
> > On Tue, Mar 8, 2016 at 9:40 PM, Jason Altekruse <
> altekruseja...@gmail.com>
> > wrote:
> >
> > > This exception should only occur if you start an older version of Drill
> > > using a configuration (stored in zookeeper or your local temp
> directory)
> > > that was created by starting a version of Drill after 4383 was merged
> > > (0842851c854595f140779e9ed09331dbb63f6623).
> > >
> > > This change added a new property to filesystem configuration to allow
> > > passing custom options to the filesystem config. This can be used in
> > place
> > > of core-site.xml to set things like your AWS private keys, as well as
> any
> > > other properties normally provided to an implementation of the Hadoop
> > > FileSystem API.
> > >
> > > Removing the new configuration should allow it to start up, but you
> > > shouldn't be seeing this if you are running the build you mentioned.
> Can
> > > you verify that this version successfully built and that you are not
> > > running an older version?
> > >
> > > - Jason
> > >
> > > P.S. I will be trying to get in a change soon that give a better error
> in
> > > this case, it should only happen with downgrades, which we generally
> > don't
> > > thoroughly test, but would still be good to fix. I'm sure there are
> > several
> > > bugs filed about these kinds of issues, this is one of them and I've
> > > assigned it to myself, hoping to post a fix soon.
> > >
> > > https://issues.apache.org/jira/browse/DRILL-2048
> > >
> > >
> > > On Tue, Mar 8, 2016 at 2:33 AM, Khurram Faraaz 
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am seeing an Exception on Drill 1.6.0 commit ID 447b093c (I am
> using
> > > the
> > > > RPM)
> > > >
> > > > I did not see this Exception on earlier version of Drill 1.6.0 commit
> > ID
> > > > 6d5f4983
> > > >
> > > > Could this be related to DRILL-4383
> > > > 
> > > >
> > > > Drill version where we see the Exception is
> > > >
> > > > git.commit.id=447b093cd2b05bfeae001844a7e3573935e84389
> > > > git.commit.message.short=DRILL-4332\: Makes vector comparison order
> > > stable
> > > > in test framework
> > > >
> > > > oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> > > ERROR:
> > > > UnrecognizedPropertyException: Unrecognized field "config" (class
> > > > org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as
> > > ignorable
> > > > (4 known properties: "enabled", "formats", "connection",
> "workspaces"])
> > > >  at [Source: [B@2b88d9b2; line: 5, column: 18] (through reference
> > chain:
> > > > org.apache.drill.exec.store.dfs.FileSystemConfig["config"])
> > > >
> > > >
> > > > [Error Id: 7fdc89ac-91ac-46eb-8201-8fe5e1acf278 on
> > > centos-02.qa.lab:31010]
> > > > at
> > > >
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
> > > > at
> > > >
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
> > > > at
> > > >
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> > > > at
> > > >
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> > > > at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
> > > > at
> > > oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
> > > > at
> > > >
> > > >
> > >
> >
> oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
> > > > at
> > > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
> > > > at
> > > >
> > > >
> > >
> >
> 

Drill Hangout Today?

[Zelaine Fong]

Are we having one today?  We're trying to connect from the MapR end, but
not getting a response.

-- Zelaine

Re: UnrecognizedPropertyException: Unrecognized field "config" (class org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable (4 known properties: "enabled", "formats", "connection"

[Abhishek Girish]

Khurram,

Can you confirm if this issue is specific to MapR RPMs or is seen with
latest builds as well?

-Abhishek

On Tue, Mar 8, 2016 at 9:22 AM, Khurram Faraaz  wrote:

> Thanks Jason. Here is what I did before I hit the Exception
>
> clush -g khurram rpm -e mapr-drill --noscripts
> clush -g khurram wget
> http://yum.qa.lab/opensource/mapr-drill-1.6.0.201603072015-1.noarch.rpm
> clush -g khurram rpm -i mapr-drill-1.6.0.201603072015-1.noarch.rpm
>
> cd /opt/mapr/zookeeper/zookeeper-3.4.5/bin
>
> ./zkCli.sh
>
> connect localhost:5181
>
> ls /drill
>
> rmr /drill/sys.options
>
> cd /opt/mapr/drill/drill-1.6.0/bin
> ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
>
> Any query on sqlline would give that Exception.
>
> I then restarted warden, clush -g khurram service mapr-warden stop and then
> start, and I am able to run queries now.
>
> Do we need a JIRA to track this problem ?
>
> - Khurram
>
> On Tue, Mar 8, 2016 at 9:40 PM, Jason Altekruse 
> wrote:
>
> > This exception should only occur if you start an older version of Drill
> > using a configuration (stored in zookeeper or your local temp directory)
> > that was created by starting a version of Drill after 4383 was merged
> > (0842851c854595f140779e9ed09331dbb63f6623).
> >
> > This change added a new property to filesystem configuration to allow
> > passing custom options to the filesystem config. This can be used in
> place
> > of core-site.xml to set things like your AWS private keys, as well as any
> > other properties normally provided to an implementation of the Hadoop
> > FileSystem API.
> >
> > Removing the new configuration should allow it to start up, but you
> > shouldn't be seeing this if you are running the build you mentioned. Can
> > you verify that this version successfully built and that you are not
> > running an older version?
> >
> > - Jason
> >
> > P.S. I will be trying to get in a change soon that give a better error in
> > this case, it should only happen with downgrades, which we generally
> don't
> > thoroughly test, but would still be good to fix. I'm sure there are
> several
> > bugs filed about these kinds of issues, this is one of them and I've
> > assigned it to myself, hoping to post a fix soon.
> >
> > https://issues.apache.org/jira/browse/DRILL-2048
> >
> >
> > On Tue, Mar 8, 2016 at 2:33 AM, Khurram Faraaz 
> > wrote:
> >
> > > Hi All,
> > >
> > > I am seeing an Exception on Drill 1.6.0 commit ID 447b093c (I am using
> > the
> > > RPM)
> > >
> > > I did not see this Exception on earlier version of Drill 1.6.0 commit
> ID
> > > 6d5f4983
> > >
> > > Could this be related to DRILL-4383
> > > 
> > >
> > > Drill version where we see the Exception is
> > >
> > > git.commit.id=447b093cd2b05bfeae001844a7e3573935e84389
> > > git.commit.message.short=DRILL-4332\: Makes vector comparison order
> > stable
> > > in test framework
> > >
> > > oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> > ERROR:
> > > UnrecognizedPropertyException: Unrecognized field "config" (class
> > > org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as
> > ignorable
> > > (4 known properties: "enabled", "formats", "connection", "workspaces"])
> > >  at [Source: [B@2b88d9b2; line: 5, column: 18] (through reference
> chain:
> > > org.apache.drill.exec.store.dfs.FileSystemConfig["config"])
> > >
> > >
> > > [Error Id: 7fdc89ac-91ac-46eb-8201-8fe5e1acf278 on
> > centos-02.qa.lab:31010]
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> > > at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
> > > at
> > oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
> > > at
> > >
> >
> oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
> > > at
> > >
> > >
> >
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
> > > at
> > >
> > >
> >
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> > > at
> > >
> > >
> >
> 

[GitHub] drill pull request: DRILL-4474: Use varchar for default column whe...

[jaltekruse]

Github user jaltekruse commented on the pull request:

https://github.com/apache/drill/pull/415#issuecomment-193884035
  
Could you also close this PR and open a new one? the JIRA number was wrong 
in your commit so this is posting to the JIRA about incorrect creation if 
direct scans. The correct number is 4479


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Use varchar for default column whe...

[jacques-n]

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/415#issuecomment-193880193
  
Can you generate the test file as part of the test rather than check in 
static?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for the 1.6 Release

[Parth Chandra]

OK, let's leave it out then.

On Tue, Mar 8, 2016 at 9:25 AM, Jason Altekruse 
wrote:

> To be honest I was expecting a longer review cycle so I hadn't run the unit
> tests before posting it for review. There were only very minor functional
> changes, so I wasn't thinking it would be an issue, and I was anticipating
> having to update the patch before merging it. I could update the test that
> is failing but I don't see much sense in trying to get it into the release
> because it only introduces new tests and some small core refactoring.
>
> I'm all for getting it merged so everyone can start using it, I just think
> it doesn't really matter if it happens on the release branch or back on
> master once we cut a release branch.
>
> I would rather try to focus on getting the Avro issues resolved, which is
> what I'm working on right now.
>
> - Jason
>
> On Tue, Mar 8, 2016 at 8:58 AM, Parth Chandra  wrote:
>
> > Sounds good Jason. Let's finalize this in the hangout.
> > Do you have the expected plans for the failing tests? If so can you
> update
> > those and put in a pull request and we'll merge and run the tests.
> > Any reason for the operator test framework to be punted? You have a +1 to
> > merge it.
> >
> >
> >
> > On Mon, Mar 7, 2016 at 9:33 PM, Khurram Faraaz 
> > wrote:
> >
> > > We should update the expected results (i.e. the expected query plan in
> > this
> > > case) and not mark them as Failing. We do not have a Failing test
> > directory
> > > today.
> > >
> > > If we are sure the expected query plans for these tests are incorrect
> > > (today), we should go ahead and update the expected results files with
> > the
> > > correct query plans.
> > >
> > > Functional/ctas/ctas_auto_partition/existing_partition_
> > > pruning/hierarchical/plan/count2_MD-185.q
> > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_2.q
> > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_4.q
> > >
> > > Thanks,
> > > Khurram
> > >
> > > On Tue, Mar 8, 2016 at 8:58 AM, Jason Altekruse <
> > altekruseja...@gmail.com>
> > > wrote:
> > >
> > > > The cutoff time sounds good to me. I don't think I'm too
> overburdened,
> > I
> > > > mostly tried putting together a merge branch today and had to kick a
> > few
> > > > things out as I found issues.
> > > >
> > > > As we are not running the full regression cluster at Dremio, it would
> > be
> > > > helpful if someone could merge Jacques patch (DRILL-4474) after
> running
> > > the
> > > > complete set of tests. Could someone at MapR pick up the test updates
> > and
> > > > merging this patch?
> > > >
> > > > These are the failures I was seeing so far, my fix was just going to
> be
> > > to
> > > > mark the tests as failing:
> > > >
> > > >
> > >
> >
> Functional/ctas/ctas_auto_partition/existing_partition_pruning/hierarchical/plan/count2_MD-185.q
> > > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_2.q
> > > > Functional/ctas/ctas_auto_partition/general/plan/drill3947_4.q
> > > >
> > > > I will just go ahead and merge the other two fixes as I had a clean
> > unit
> > > > test run and neither of the patches will have any impact on the
> > > regression
> > > > tests. One is a build fix, that only changes behavior when running
> the
> > > > release profile and the other is a unit test fix.
> > > >
> > > > I had punted the operator unit tests out of the release, so those can
> > be
> > > > removed from your list.
> > > >
> > > > I will start looking back at both of the Avro issues right after
> > merging.
> > > >
> > > > On Mon, Mar 7, 2016 at 6:32 PM, Parth Chandra  >
> > > > wrote:
> > > >
> > > > > The train is about to leave :). I'd like to propose a cutoff for
> > > > tomorrow 5
> > > > > pm.
> > > > > (This will be a full week after the initial mail so plenty of time
> > for
> > > > > everyone to get their issues aired.)
> > > > >
> > > > > Here are the open items (that I am aware of) -
> > > > >
> > > > > Waiting to be merged
> > > > > DRILL-4483/pr 411 (Fix text plan regression in query profiles)
> > (Venki)
> > > > > DRILL-4437 (and others)/pr 394 (Operator unit test framework).
> > (Jason)
> > > > > DRILL-4332/pr 389 (Make vector comparison order stable in test
> > > framework)
> > > > > (?)
> > > > > DRILL-4474/pr 406 (Ensure that ConvertCountToDirectScan only pushes
> > > > through
> > > > > project when project is trivial.) - Tests need to be fixed (Jason)
> > > > >
> > > > > Need review -
> > > > > DRILL-4375/pr 402 (Fix the maven release profile) - (Jason - patch
> > > > > reviewed. Merge)
> > > > > DRILL-4486/pr 412 (Fix expression serialization escaping) (Need are
> > > > > reviewer)
> > > > > DRILL-4485 - Update MapR profile to use MapR 5.1.0 (Parth/Aditya)
> > > > >
> > > > > Regression
> > > > > DRILL-4482 - Avro no longer selects data correctly from a
> > > sub-structure.
> > > > No
> > > > > patch available. (Jason)
> > > > >
> > > > > 

[GitHub] drill pull request: DRILL-4474: Use varchar for default column whe...

[amansinha100]

GitHub user amansinha100 opened a pull request:

https://github.com/apache/drill/pull/415

DRILL-4474: Use varchar for default column when all_text_mode is enab…

…led.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amansinha100/incubator-drill DRILL-4479

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #415


commit edfbf9bf0acd94fd0e8737f9162ca13281d00906
Author: Aman Sinha 
Date:   2016-03-08T17:27:32Z

DRILL-4474: Use varchar for default column when all_text_mode is enabled.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4487: add unit test for DRILL-4449

[adeneche]

GitHub user adeneche opened a pull request:

https://github.com/apache/drill/pull/414

DRILL-4487: add unit test for DRILL-4449

@amansinha100 can you please review ? thanks

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adeneche/incubator-drill DRILL-4487

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #414


commit b1f052d800bae05bbf36b3594fe3c171ea4cede4
Author: adeneche 
Date:   2016-03-08T15:54:31Z

DRILL-4487: add unit test for DRILL-4449




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Time for the 1.6 Release

[Parth Chandra]

Sounds good Jason. Let's finalize this in the hangout.
Do you have the expected plans for the failing tests? If so can you update
those and put in a pull request and we'll merge and run the tests.
Any reason for the operator test framework to be punted? You have a +1 to
merge it.



On Mon, Mar 7, 2016 at 9:33 PM, Khurram Faraaz  wrote:

> We should update the expected results (i.e. the expected query plan in this
> case) and not mark them as Failing. We do not have a Failing test directory
> today.
>
> If we are sure the expected query plans for these tests are incorrect
> (today), we should go ahead and update the expected results files with the
> correct query plans.
>
> Functional/ctas/ctas_auto_partition/existing_partition_
> pruning/hierarchical/plan/count2_MD-185.q
> Functional/ctas/ctas_auto_partition/general/plan/drill3947_2.q
> Functional/ctas/ctas_auto_partition/general/plan/drill3947_4.q
>
> Thanks,
> Khurram
>
> On Tue, Mar 8, 2016 at 8:58 AM, Jason Altekruse 
> wrote:
>
> > The cutoff time sounds good to me. I don't think I'm too overburdened, I
> > mostly tried putting together a merge branch today and had to kick a few
> > things out as I found issues.
> >
> > As we are not running the full regression cluster at Dremio, it would be
> > helpful if someone could merge Jacques patch (DRILL-4474) after running
> the
> > complete set of tests. Could someone at MapR pick up the test updates and
> > merging this patch?
> >
> > These are the failures I was seeing so far, my fix was just going to be
> to
> > mark the tests as failing:
> >
> >
> Functional/ctas/ctas_auto_partition/existing_partition_pruning/hierarchical/plan/count2_MD-185.q
> > Functional/ctas/ctas_auto_partition/general/plan/drill3947_2.q
> > Functional/ctas/ctas_auto_partition/general/plan/drill3947_4.q
> >
> > I will just go ahead and merge the other two fixes as I had a clean unit
> > test run and neither of the patches will have any impact on the
> regression
> > tests. One is a build fix, that only changes behavior when running the
> > release profile and the other is a unit test fix.
> >
> > I had punted the operator unit tests out of the release, so those can be
> > removed from your list.
> >
> > I will start looking back at both of the Avro issues right after merging.
> >
> > On Mon, Mar 7, 2016 at 6:32 PM, Parth Chandra 
> > wrote:
> >
> > > The train is about to leave :). I'd like to propose a cutoff for
> > tomorrow 5
> > > pm.
> > > (This will be a full week after the initial mail so plenty of time for
> > > everyone to get their issues aired.)
> > >
> > > Here are the open items (that I am aware of) -
> > >
> > > Waiting to be merged
> > > DRILL-4483/pr 411 (Fix text plan regression in query profiles) (Venki)
> > > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Jason)
> > > DRILL-4332/pr 389 (Make vector comparison order stable in test
> framework)
> > > (?)
> > > DRILL-4474/pr 406 (Ensure that ConvertCountToDirectScan only pushes
> > through
> > > project when project is trivial.) - Tests need to be fixed (Jason)
> > >
> > > Need review -
> > > DRILL-4375/pr 402 (Fix the maven release profile) - (Jason - patch
> > > reviewed. Merge)
> > > DRILL-4486/pr 412 (Fix expression serialization escaping) (Need are
> > > reviewer)
> > > DRILL-4485 - Update MapR profile to use MapR 5.1.0 (Parth/Aditya)
> > >
> > > Regression
> > > DRILL-4482 - Avro no longer selects data correctly from a
> sub-structure.
> > No
> > > patch available. (Jason)
> > >
> > > Dropped
> > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly
> > expose
> > > their types within Calcite.) Pushed out because of test failures
> > >
> > > Jason, at the moment it looks like you've volunteered for too much. Can
> > we
> > > get other folks to help out?
> > >
> > > Thanks
> > >
> > > Parth
> > >
> > >
> > > On Mon, Mar 7, 2016 at 5:32 PM, Jason Altekruse <
> > altekruseja...@gmail.com>
> > > wrote:
> > >
> > > > 4474 is actually revealing a few invalid tests in the Regression
> suite
> > > that
> > > > test for the current incorrect plans. The fix should be included in
> the
> > > > release, but I will post a PR on the regression suite to update the
> > tests
> > > > before I push it.
> > > >
> > > > On Mon, Mar 7, 2016 at 4:44 PM, Steven Phillips 
> > > wrote:
> > > >
> > > > > DRILL-4486 is a pretty simple fix. Without it, currently some regex
> > > > queries
> > > > > will fail.
> > > > >
> > > > > I think we should include it in the release.
> > > > >
> > > > >
> > > > > https://github.com/apache/drill/pull/412
> > > > >
> > > > > On Mon, Mar 7, 2016 at 2:15 PM, Jason Altekruse <
> > > > altekruseja...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > There is a small test issue with some of the refactoring that
> > > > accompanied
> > > > > > the operator unit tests. These don't change any user-facing
> > behavior,
> > > > so
> > > 

[jira] [Resolved] (DRILL-4313) C++ client - Improve method of drillbit selection from cluster

[Parth Chandra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Chandra resolved DRILL-4313.
--
Resolution: Fixed

Fixed in df0f0af3d963c1b65eb01c3141fe84532c53f5a5

> C++ client - Improve method of drillbit selection from cluster
> --
>
> Key: DRILL-4313
> URL: https://issues.apache.org/jira/browse/DRILL-4313
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Parth Chandra
>Assignee: Parth Chandra
> Fix For: 1.6.0
>
>
> The current C++ client handles multiple parallel queries over the same 
> connection, but that creates a bottleneck as the queries get sent to the same 
> drillbit.
> The client can manage this more effectively by choosing from a configurable 
> pool of connections and round robin queries to them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request: DRILL-4313: Improve method of picking a random...

[parthchandra]

Github user parthchandra commented on the pull request:

https://github.com/apache/drill/pull/396#issuecomment-193864509
  
Merged in df0f0af3d963c1b65eb01c3141fe84532c53f5a5


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4313: Improve method of picking a random...

[parthchandra]

Github user parthchandra closed the pull request at:

https://github.com/apache/drill/pull/396


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4332) tests in TestFrameworkTest fail in Java 8

[Jason Altekruse (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Altekruse resolved DRILL-4332.

   Resolution: Fixed
Fix Version/s: (was: Future)
   1.6.0

Fixed in 447b093cd2b05bfeae001844a7e3573935e84389

> tests in TestFrameworkTest fail in Java 8
> -
>
> Key: DRILL-4332
> URL: https://issues.apache.org/jira/browse/DRILL-4332
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Tools, Build & Test
>Affects Versions: 1.5.0
>Reporter: Deneche A. Hakim
>Assignee: Laurent Goujon
> Fix For: 1.6.0
>
>
> the following unit tests fail in Java 8:
> {noformat}
> TestFrameworkTest.testRepeatedColumnMatching
> TestFrameworkTest.testCSVVerificationOfOrder_checkFailure
> {noformat}
> The tests expect the query to fail with a specific error message. The message 
> generated by DrillTestWrapper.compareMergedVectors assumes a specific order 
> in a map keySet (which we shouldn't). In Java 8 it seems the order changed 
> which causes a slightly different error message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4486) Expression serializer incorrectly serializes escaped characters

[Jason Altekruse (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Altekruse resolved DRILL-4486.

   Resolution: Fixed
Fix Version/s: 1.6.0

Fixed in 80316f3f8bef866720f99e609fe758ec8e0c4612

> Expression serializer incorrectly serializes escaped characters
> ---
>
> Key: DRILL-4486
> URL: https://issues.apache.org/jira/browse/DRILL-4486
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Steven Phillips
>Assignee: Steven Phillips
> Fix For: 1.6.0
>
>
> the drill expression parser requires backslashes to be escaped. But the 
> ExpressionStringBuilder is not properly escaping them. This causes problems, 
> especially in the case of regex expressions run with parallel execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4375) Fix the maven release profile, broken by jdbc jar size enforcer added in DRILL-4291

[Jason Altekruse (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Altekruse resolved DRILL-4375.

   Resolution: Fixed
Fix Version/s: 1.6.0

Fixed in 1f29914fc5c7d1e36651ac28167804c4012501fe

> Fix the maven release profile, broken by jdbc jar size enforcer added in 
> DRILL-4291
> ---
>
> Key: DRILL-4375
> URL: https://issues.apache.org/jira/browse/DRILL-4375
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Jason Altekruse
>Assignee: Jason Altekruse
> Fix For: 1.6.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: UnrecognizedPropertyException: Unrecognized field "config" (class org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable (4 known properties: "enabled", "formats", "connection"

[Jason Altekruse]

This exception should only occur if you start an older version of Drill
using a configuration (stored in zookeeper or your local temp directory)
that was created by starting a version of Drill after 4383 was merged
(0842851c854595f140779e9ed09331dbb63f6623).

This change added a new property to filesystem configuration to allow
passing custom options to the filesystem config. This can be used in place
of core-site.xml to set things like your AWS private keys, as well as any
other properties normally provided to an implementation of the Hadoop
FileSystem API.

Removing the new configuration should allow it to start up, but you
shouldn't be seeing this if you are running the build you mentioned. Can
you verify that this version successfully built and that you are not
running an older version?

- Jason

P.S. I will be trying to get in a change soon that give a better error in
this case, it should only happen with downgrades, which we generally don't
thoroughly test, but would still be good to fix. I'm sure there are several
bugs filed about these kinds of issues, this is one of them and I've
assigned it to myself, hoping to post a fix soon.

https://issues.apache.org/jira/browse/DRILL-2048


On Tue, Mar 8, 2016 at 2:33 AM, Khurram Faraaz  wrote:

> Hi All,
>
> I am seeing an Exception on Drill 1.6.0 commit ID 447b093c (I am using the
> RPM)
>
> I did not see this Exception on earlier version of Drill 1.6.0 commit ID
> 6d5f4983
>
> Could this be related to DRILL-4383
> 
>
> Drill version where we see the Exception is
>
> git.commit.id=447b093cd2b05bfeae001844a7e3573935e84389
> git.commit.message.short=DRILL-4332\: Makes vector comparison order stable
> in test framework
>
> oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> UnrecognizedPropertyException: Unrecognized field "config" (class
> org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable
> (4 known properties: "enabled", "formats", "connection", "workspaces"])
>  at [Source: [B@2b88d9b2; line: 5, column: 18] (through reference chain:
> org.apache.drill.exec.store.dfs.FileSystemConfig["config"])
>
>
> [Error Id: 7fdc89ac-91ac-46eb-8201-8fe5e1acf278 on centos-02.qa.lab:31010]
> at
>
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
> at
>
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
> at
>
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> at
>
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
> at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
> at
>
> oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
> at
> oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
> at
>
> oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
> at
>
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
> at
>
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
> at
>
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
>
> oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
>
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
>
> oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
>
> oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
>
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 

UnrecognizedPropertyException: Unrecognized field "config" (class org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable (4 known properties: "enabled", "formats", "connection", "w

[Khurram Faraaz]

Hi All,

I am seeing an Exception on Drill 1.6.0 commit ID 447b093c (I am using the
RPM)

I did not see this Exception on earlier version of Drill 1.6.0 commit ID
6d5f4983

Could this be related to DRILL-4383


Drill version where we see the Exception is

git.commit.id=447b093cd2b05bfeae001844a7e3573935e84389
git.commit.message.short=DRILL-4332\: Makes vector comparison order stable
in test framework

oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
UnrecognizedPropertyException: Unrecognized field "config" (class
org.apache.drill.exec.store.dfs.FileSystemConfig), not marked as ignorable
(4 known properties: "enabled", "formats", "connection", "workspaces"])
 at [Source: [B@2b88d9b2; line: 5, column: 18] (through reference chain:
org.apache.drill.exec.store.dfs.FileSystemConfig["config"])


[Error Id: 7fdc89ac-91ac-46eb-8201-8fe5e1acf278 on centos-02.qa.lab:31010]
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
at
oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
at
oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
at
oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
at
oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
at
oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
at
oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
at
oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285)
at
oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at
oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
at
oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at

[GitHub] drill pull request: DRILL-4443: MIN/MAX on VARCHAR throw a NullPoi...

[adeneche]

Github user adeneche closed the pull request at:

https://github.com/apache/drill/pull/409


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---