[jira] [Created] (DRILL-3813) table from directory subtree having no descendent files fails with index error

[Daniel Barclay (Drill) (JIRA)]

Daniel Barclay (Drill) created DRILL-3813:
-

 Summary: table from directory subtree having no descendent files 
fails with index error
 Key: DRILL-3813
 URL: https://issues.apache.org/jira/browse/DRILL-3813
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser, Storage - Other
Reporter: Daniel Barclay (Drill)
Assignee: Aman Sinha


Trying to use as a table a directory subtree that has no descendent files (but 
zero or more descendent directories) yields what seems to be a partially 
handled index out-of-bounds condition.  

For example, with {{/tmp/empty_directory}} being an empty directory:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
grandchild directory, but not descendent files:

0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 

{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3813) table from directory subtree having no descendent files fails with index error

[Daniel Barclay (Drill) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Barclay (Drill) updated DRILL-3813:
--
Description: 
Trying to use as a table a directory subtree that has no descendent files (but 
zero or more descendent directories) yields what seems to be a partially 
handled index out-of-bounds condition.  

For example, with {{/tmp/empty_directory}} being an empty directory:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
grandchild directory, but not descendent files:

0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


A directory subtree having no files was expected to be taken as a table with no 
rows (and a null schema).


  was:
Trying to use as a table a directory subtree that has no descendent files (but 
zero or more descendent directories) yields what seems to be a partially 
handled index out-of-bounds condition.  

For example, with {{/tmp/empty_directory}} being an empty directory:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
grandchild directory, but not descendent files:

0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 

{noformat}



> table from directory subtree having no descendent files fails with index error
> --
>
> Key: DRILL-3813
> URL: https://issues.apache.org/jira/browse/DRILL-3813
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser, Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Aman Sinha
>
> Trying to use as a table a directory subtree that has no descendent files 
> (but zero or more descendent directories) yields what seems to be a partially 
> handled index out-of-bounds condition.  
> For example, with {{/tmp/empty_directory}} being an empty directory:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> 
> {noformat}
> Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
> grandchild directory, but not descendent files:
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> 
> {noformat}
> A directory subtree having no files was expected to be taken as a table with 
> no rows (and a null schema).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3814) Directory containing only unrecognized files reported as not found vs. taken as empty table

[Daniel Barclay (Drill) (JIRA)]

Daniel Barclay (Drill) created DRILL-3814:
-

 Summary: Directory containing only unrecognized files reported as 
not found vs. taken as empty table
 Key: DRILL-3814
 URL: https://issues.apache.org/jira/browse/DRILL-3814
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser, Storage - Other
Reporter: Daniel Barclay (Drill)
Assignee: Aman Sinha


A directory subtree all of whose descendent files have unrecognized extensions 
is reported as non-existent rather treated as a table with zero rows.

Is this intended? 

(The error message is the exact same error message that results if the user 
gets a directory name wrong and refers to a non-existent directory, making the 
message really confusing and misleading.)

For example, for directory {{/tmp/unrecognized_files_directory}} containing 
only file {{/tmp/unrecognized_files_directory/junk.junk}}:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM 
`dfs`.`tmp`.`unrecognized_files_directory`;
Sep 20, 2015 11:16:34 PM org.apache.calcite.sql.validate.SqlValidatorException 

SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 
'dfs.tmp.unrecognized_files_directory' not found
Sep 20, 2015 11:16:34 PM org.apache.calcite.runtime.CalciteException 
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column 
15 to line 1, column 19: Table 'dfs.tmp.unrecognized_files_directory' not found
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 19: Table 
'dfs.tmp.unrecognized_files_directory' not found


[Error Id: 0ce9ba05-7f62-4063-a2c0-7d2b4f1f7967 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}

Notice how that is the same message as for a non-existent directory:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_such_directory`;
Sep 20, 2015 11:17:12 PM org.apache.calcite.sql.validate.SqlValidatorException 

SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table 
'dfs.tmp.no_such_directory' not found
Sep 20, 2015 11:17:12 PM org.apache.calcite.runtime.CalciteException 
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column 
15 to line 1, column 19: Table 'dfs.tmp.no_such_directory' not found
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 19: Table 
'dfs.tmp.no_such_directory' not found


[Error Id: 49f423f1-5dfe-4435-8b72-78e0b80e on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-3781:
--
Attachment: 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch

> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch, 
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901621#comment-14901621
 ] 

Jinfeng Ni commented on DRILL-3781:
---

The fix for this problem is in Calcite-886. Here, the patch in Drill simply add 
unit testcase.

[~amansinha100], could you please review the patch? Thanks!



> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch, 
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Aman Sinha (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901646#comment-14901646
 ] 

Aman Sinha commented on DRILL-3781:
---

+1.  I also took a look at Calcite-886 and looks good to me. 

> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments: 
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch, 
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Aman Sinha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aman Sinha updated DRILL-3781:
--
Assignee: Jinfeng Ni  (was: Aman Sinha)

> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch, 
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3815) unknown suffixes .not_json and .json_not treated differently (multi-file case)

[Ted Dunning (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901691#comment-14901691
 ] 

Ted Dunning commented on DRILL-3815:


Daniel,

Did you trace this a bit to see where the extensions are being matched?

Could it be a naively constructed regex?  Kinda smells like that.



> unknown suffixes .not_json and .json_not treated differently (multi-file case)
> --
>
> Key: DRILL-3815
> URL: https://issues.apache.org/jira/browse/DRILL-3815
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Jacques Nadeau
>
> In scanning a directory subtree used as a table, unknown filename extensions 
> seem to be treated differently depending on whether they're similar to known 
> file extensions.  The behavior suggests that Drill checks whether a file name 
> _contains_ an extension's string rather than _ending_ with it. 
> For example, given these subtrees with almost identical leaf file names:
> {noformat}
> $ find /tmp/testext_xx_json/
> /tmp/testext_xx_json/
> /tmp/testext_xx_json/voter2.not_json
> /tmp/testext_xx_json/voter1.json
> $ find /tmp/testext_json_xx/
> /tmp/testext_json_xx/
> /tmp/testext_json_xx/voter1.json
> /tmp/testext_json_xx/voter2.json_not
> $ 
> {noformat}
> the results of trying to use them as tables differs:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT *   FROM `dfs.tmp`.`testext_xx_json`;
> Sep 21, 2015 11:41:50 AM 
> org.apache.calcite.sql.validate.SqlValidatorException 
> ...
> Error: VALIDATION ERROR: From line 1, column 17 to line 1, column 25: Table 
> 'dfs.tmp.testext_xx_json' not found
> [Error Id: 6fe41deb-0e39-43f6-beca-de27b39d276b on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> SELECT *   FROM `dfs.tmp`.`testext_json_xx`;
> +---+
> | onecf |
> +---+
> | {"name":"someName1"}  |
> | {"name":"someName2"}  |
> +---+
> 2 rows selected (0.149 seconds)
> {noformat}
> (Other probing seems to indicate that there is also some sensitivity to 
> whether the extension contains an underscore character.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901861#comment-14901861
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/163#discussion_r40050662
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java 
---
@@ -76,16 +74,17 @@ public void setLimit(long limit) {
 
   }
   /**
-   * Automatically allocate memory. This is used when an actual allocation 
happened to be larger than requested. This
-   * memory has already been used up so it must be accurately accounted 
for in future allocations.
+   * Automatically allocate memory. This is used when an actual allocation 
happened to be larger than requested, or when
+   * a buffer has it's ownership passed to another allocator.
+   * This memory has already been used up so it must be accurately 
accounted for in future allocations.
*
-   * @param size
+   * @param size extra allocated memory that needs to be accounted for
*/
   public boolean forceGet(long size) {
 if (get(size, this.applyFragmentLimit)) {
   return true;
 } else {
-  availableShared.addAndGet(size);
+  availableShared.addAndGet(-size);
--- End diff --

It is likely handled..I will defer to Chris on that. 


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901579#comment-14901579
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40037509
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
 ---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
   abstract boolean setLocalOption(OptionValue value);
 
+  /**
+   * Deletes the option with given name for this manager without falling 
back.
+   *
+   * @param type option type
+   * @return true iff the option was successfully deleted
--- End diff --

I will clarify the docs in both places.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3817) Refresh metadata does not work when used with sub schema

[Mehant Baid (JIRA)]

Mehant Baid created DRILL-3817:
--

 Summary: Refresh metadata does not work when used with sub schema  
 Key: DRILL-3817
 URL: https://issues.apache.org/jira/browse/DRILL-3817
 Project: Apache Drill
  Issue Type: Bug
Reporter: Mehant Baid
Assignee: Mehant Baid
 Fix For: 1.2.0


refresh table metadata dfs.tmp.`lineitem` does not work, hit the following 
exception

org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 
org.apache.calcite.sql.SqlBasicCall cannot be cast to 
org.apache.calcite.sql.SqlIdentifier

If the sub schema is removed it works.
refresh table metadata dfs.`/tmp/lineitem`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz closed DRILL-3621.
-

> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
> 
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong 
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY = 
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10  |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY = 
> 'DUMMY1';
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN 
> ('DUMMY1' , 'DUMMY10');
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY 
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN 
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
> [tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10, filter=null], 
> columns=[`row_key`]]])
>  |
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-3781:
--
Assignee: Aman Sinha  (was: Jinfeng Ni)

> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments: 
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch, 
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901757#comment-14901757
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/163#discussion_r40047181
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java 
---
@@ -76,16 +74,17 @@ public void setLimit(long limit) {
 
   }
   /**
-   * Automatically allocate memory. This is used when an actual allocation 
happened to be larger than requested. This
-   * memory has already been used up so it must be accurately accounted 
for in future allocations.
+   * Automatically allocate memory. This is used when an actual allocation 
happened to be larger than requested, or when
+   * a buffer has it's ownership passed to another allocator.
+   * This memory has already been used up so it must be accurately 
accounted for in future allocations.
*
-   * @param size
+   * @param size extra allocated memory that needs to be accounted for
*/
   public boolean forceGet(long size) {
 if (get(size, this.applyFragmentLimit)) {
   return true;
 } else {
-  availableShared.addAndGet(size);
+  availableShared.addAndGet(-size);
--- End diff --

It can become negative if the allocator takes ownership of a buffer and 
exceeds it's maximum allocated memory. Negative values are handled properly, at 
least that's my understanding


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901814#comment-14901814
 ] 

Khurram Faraaz commented on DRILL-3621:
---

Verified that Filter gets pushed down into the Scan operator on master commit 
id b525692e

case 1)

{code}
0: jdbc:drill:schema=dfs.tmp> select convert_from(row_key,'UTF8') from 
testrowkey WHERE ROW_KEY='DUMMY7' OR ROW_KEY BETWEEN 'DUMMY1' AND 'DUMMY10';
+--+
|  EXPR$0  |
+--+
| DUMMY1   |
| DUMMY10  |
| DUMMY7   |
+--+
3 rows selected (0.865 seconds)

explain plan for select convert_from(row_key,'UTF8') from testrowkey WHERE 
ROW_KEY='DUMMY7' OR ROW_KEY BETWEEN 'DUMMY1' AND 'DUMMY10';

00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
[tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY7\x00, filter=FilterList 
OR (2/2): [RowFilter (EQUAL, DUMMY7), FilterList AND (2/2): [RowFilter 
(GREATER_OR_EQUAL, DUMMY1), RowFilter (LESS_OR_EQUAL, DUMMY10)]]], 
columns=[`row_key`]]])

{code}

case 2)

{code}
0: jdbc:drill:schema=dfs.tmp> select convert_from(row_key,'UTF8') from 
testrowkey WHERE ROW_KEY  in ('DUMMY1' ,  'DUMMY10');
+--+
|  EXPR$0  |
+--+
| DUMMY1   |
| DUMMY10  |
+--+
2 rows selected (0.867 seconds)

00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
[tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10\x00, filter=FilterList 
OR (2/2): [RowFilter (EQUAL, DUMMY1), RowFilter (EQUAL, DUMMY10)]], 
columns=[`row_key`]]])

{code}

case 3)

{code}
: jdbc:drill:schema=dfs.tmp> select convert_from(row_key,'UTF8') from 
testrowkey WHERE ROW_KEY ='DUMMY1' OR ROW_KEY = 'DUMMY10';
+--+
|  EXPR$0  |
+--+
| DUMMY1   |
| DUMMY10  |
+--+
2 rows selected (0.854 seconds)

00-01  Project(EXPR$0=[CONVERT_FROMUTF8($0)])
00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
[tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10\x00, filter=FilterList 
OR (2/2): [RowFilter (EQUAL, DUMMY1), RowFilter (EQUAL, DUMMY10)]], 
columns=[`row_key`]]])

{code}

> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
> 
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong 
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY = 
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10  |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY = 
> 'DUMMY1';
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN 
> ('DUMMY1' , 'DUMMY10');
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY 
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN 
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan 

[jira] [Commented] (DRILL-3731) Partition Pruning not taking place when we have case statement within the filter

[Sean Hsuan-Yi Chu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901564#comment-14901564
 ] 

Sean Hsuan-Yi Chu commented on DRILL-3731:
--

Essentially, ReduceExpressionsRule.FILTER_INSTANCE should have simplified the 
expression before PruneScan rule takes place. 

This issue should be resolved from Calcite:

https://issues.apache.org/jira/browse/CALCITE-895

> Partition Pruning not taking place when we have case statement within the 
> filter
> 
>
> Key: DRILL-3731
> URL: https://issues.apache.org/jira/browse/DRILL-3731
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Rahul Challapalli
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: lineitempart.tgz
>
>
> Commit # 8f33c3d55228a98ea2a309490ecbfb7a180f012c
> Partition Pruning is not taking place in the below query
> {code}
> explain plan for select count(*) from 
> `/drill/testdata/partition_pruning/dfs/lineitempart` where case when 
> dir0=1991 then null else 1 end is null or dir0=1992;
> 00-00Screen
> 00-01  Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03  Project($f0=[0])
> 00-04SelectionVectorRemover
> 00-05  Filter(condition=[OR(IS NULL(CASE(=($0, 1991), null, 1)), 
> =($0, 1992))])
> 00-06Scan(groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/drill/testdata/partition_pruning/dfs/lineitempart, 
> numFiles=7, columns=[`dir0`], 
> files=[maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1991/lineitemaa.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1992/lineitemab.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1996/lineitemaf.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1994/lineitemad.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1993/lineitemac.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1997/lineitemag.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1995/lineitemae.tbl]]])
> {code}
> The planner should have pruned all partitions except 1991 & 1992
> I attached the dataset. Let me know if you need anything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3776) count(*) from empty text file does not return 0

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu updated DRILL-3776:
-
Fix Version/s: 1.4.0

> count(*) from empty text file does not return 0
> ---
>
> Key: DRILL-3776
> URL: https://issues.apache.org/jira/browse/DRILL-3776
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Text & CSV
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.4.0
>
>
> {code}
> select count(*) from `empty.csv`;
> {code}
> {code}
> select count(*) from `empty.json`;
> {code}
> Both return no record, no schema. But in postgres, it returns 0 for both 
> cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3776) count(*) from empty text file does not return 0

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu updated DRILL-3776:
-
Target Version/s:   (was: 1.2.0)

> count(*) from empty text file does not return 0
> ---
>
> Key: DRILL-3776
> URL: https://issues.apache.org/jira/browse/DRILL-3776
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Text & CSV
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.4.0
>
>
> {code}
> select count(*) from `empty.csv`;
> {code}
> {code}
> select count(*) from `empty.json`;
> {code}
> Both return no record, no schema. But in postgres, it returns 0 for both 
> cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3192) TestDrillbitResilience#cancelWhenQueryIdArrives hangs

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-3192:
---
Fix Version/s: (was: 1.2.0)
   1.3.0

> TestDrillbitResilience#cancelWhenQueryIdArrives hangs
> -
>
> Key: DRILL-3192
> URL: https://issues.apache.org/jira/browse/DRILL-3192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
>Priority: Critical
> Fix For: 1.3.0
>
>
> TestDrillbitResilience#cancelWhenQueryIdArrives (previously named 
> cancelBeforeAnyResultsArrive) hangs when the test is run multiple times. 
> -(Will add more information)-
> *Configuration: BIT_SERVER_RPC_THREADS = 1*
> The remote RPC thread with a cancel signal is waiting for the fragment to 
> start accepting external events. The fragment is waiting for a 
> ControlConnection to the Foreman node (through ReconnectingConnection). The 
> Foreman node is waiting for the remote node to accept the connection, which 
> happens through an RPC thread. Distributed deadlock.
> DRILL-3242 should solve this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901625#comment-14901625
 ] 

Jinfeng Ni commented on DRILL-3781:
---

The forked calcite version will be updated before merge to master,  after the 
patch completes the review /pre-commit regression run. 


> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments: 
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch, 
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3735) Directory pruning is not happening when number of files is larger than 64k

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901551#comment-14901551
 ] 

ASF GitHub Bot commented on DRILL-3735:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/156


> Directory pruning is not happening when number of files is larger than 64k
> --
>
> Key: DRILL-3735
> URL: https://issues.apache.org/jira/browse/DRILL-3735
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
>
> When the number of files is larger than 64k limit, directory pruning is not 
> happening. 
> We need to increase this limit further to handle most use cases.
> My proposal is to separate the code for directory pruning and partition 
> pruning. 
> Say in a parent directory there are 100 directories and 1 million files.
> If we only query the file from one directory, we should firstly read the 100 
> directories and narrow down to which directory; and then read the file paths 
> in that directory in memory and do the rest stuff.
> Current behavior is , Drill will read all the file paths of that 1 million 
> files in memory firstly, and then do directory pruning or partition pruning. 
> This is not performance efficient nor memory efficient. And also it can not 
> scale.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901697#comment-14901697
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/163#discussion_r40045182
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java 
---
@@ -76,16 +74,17 @@ public void setLimit(long limit) {
 
   }
   /**
-   * Automatically allocate memory. This is used when an actual allocation 
happened to be larger than requested. This
-   * memory has already been used up so it must be accurately accounted 
for in future allocations.
+   * Automatically allocate memory. This is used when an actual allocation 
happened to be larger than requested, or when
+   * a buffer has it's ownership passed to another allocator.
+   * This memory has already been used up so it must be accurately 
accounted for in future allocations.
*
-   * @param size
+   * @param size extra allocated memory that needs to be accounted for
*/
   public boolean forceGet(long size) {
 if (get(size, this.applyFragmentLimit)) {
   return true;
 } else {
-  availableShared.addAndGet(size);
+  availableShared.addAndGet(-size);
--- End diff --

Since this will be subtracting size bytes, should there be a check for 
availableShared >=0 ? I am not quite sure what's supposed to happen if this 
value drops below 0 at this point...


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3029) Wrong result with correlated not exists subquery

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-3029:
--
Fix Version/s: (was: 1.2.0)
   1.3.0

> Wrong result with correlated not exists subquery
> 
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where   not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1  |
> | 2  |
> | 3  |
> | 4  |
> | 5  |
> | 6  |
> | 7  |
> | 9  |
> | 10 |
> | null   |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where   not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-1162) 25 way join ended up in 0 results which is not expected

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-1162:
-
Attachment: oom_error.log

> 25 way join ended up in 0 results which is not expected
> ---
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Rahul Challapalli
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: error.log, oom_error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned 
> select count(*) from `lineitem1.parquet` a 
> inner join `part.parquet` j on a.l_partkey = j.p_partkey 
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey 
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey 
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey 
> = m.ps_suppkey 
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey 
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey 
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey 
> inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey 
> inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice 
> inner join `lineitem2.parquet` f on a.l_comment = f.l_comment 
> inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate 
> inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate 
> inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate 
> inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate 
> inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate 
> inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate 
> inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate 
> inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate 
> inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate 
> inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate 
> inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate 
> inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate 
> inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate;
> However when we remove the last 'inner join' and run the query it returns 
> '716372534'. Since the last inner join is similar to the one's before it, it 
> should match some records and return the data appropriately.
> The logs indicated that it actually returned 0 results. Attached the log file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-1162) 25 way join ended up in 0 results which is not expected

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-1162:
-
Assignee: Chris Westin  (was: Rahul Challapalli)

> 25 way join ended up in 0 results which is not expected
> ---
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Chris Westin
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: error.log, oom_error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned 
> select count(*) from `lineitem1.parquet` a 
> inner join `part.parquet` j on a.l_partkey = j.p_partkey 
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey 
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey 
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey 
> = m.ps_suppkey 
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey 
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey 
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey 
> inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey 
> inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice 
> inner join `lineitem2.parquet` f on a.l_comment = f.l_comment 
> inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate 
> inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate 
> inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate 
> inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate 
> inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate 
> inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate 
> inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate 
> inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate 
> inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate 
> inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate 
> inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate 
> inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate 
> inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate;
> However when we remove the last 'inner join' and run the query it returns 
> '716372534'. Since the last inner join is similar to the one's before it, it 
> should match some records and return the data appropriately.
> The logs indicated that it actually returned 0 results. Attached the log file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2808) Substr function throws weird message when wrong arguments are passed in

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901507#comment-14901507
 ] 

Victoria Markman commented on DRILL-2808:
-

Still in 1.2.0, tried with:

#Sun Sep 20 15:30:07 UTC 2015
git.commit.id.abbrev=092903d


> Substr function throws weird message when wrong arguments are passed in
> ---
>
> Key: DRILL-2808
> URL: https://issues.apache.org/jira/browse/DRILL-2808
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
>
> #Wed Apr 15 17:37:25 EDT 2015
> git.commit.id.abbrev=cb47df0
> {code}
> 0: jdbc:drill:schema=dfs> select substr(0, 1, 'abc') from sys.options limit 1;
> Query failed: SYSTEM ERROR: Unexpected exception during fragment 
> initialization: Internal error: Error while applying rule 
> ReduceExpressionsRule[Project], args 
> [rel#53990:ProjectRel.NONE.ANY([]).[](child=rel#53989:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
>  1, 'abc'))]
> [1f7e0c35-fc65-4a2f-b668-12e9baf5c9b8 on atsqa4-133.qa.lab:31010]
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> 2015-04-16 22:04:32,029 [2acfce0f-8af4-9786-4799-9180b9bac219:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - State change requested.  PENDING --> 
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception 
> during fragment initialization: Internal error: Error while applying rule 
> ReduceExpressionsRule[Project], args 
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
>  1, 'abc'))]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:211) 
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_71]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule[Project], args 
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
>  1, 'abc'))]
> at org.eigenbase.util.Util.newInternal(Util.java:750) 
> ~[optiq-core-0.9-drill-r21.jar:na]
> at 
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:246)
>  ~[optiq-core-0.9-drill-r21.jar:na]
> at 
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
>  ~[optiq-core-0.9-drill-r21.jar:na]
> at 
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165) 
> ~[optiq-core-0.9-drill-r21.jar:na]
> at 
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:278) 
> ~[optiq-core-0.9-drill-r21.jar:na]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:196)
>  ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:137)
>  ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:155)
>  ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:770) 
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:202) 
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> ... 3 common frames omitted
> Caused by: java.lang.RuntimeException: Error in evaluating function of 
> castBIGINT
> at 
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:323)
>  ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:147)
>  ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47)
>  ~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:251)
>  

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900825#comment-14900825
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39984380
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java
 ---
@@ -107,7 +117,7 @@ public SqlNode visitChild(
   }
   SqlNode newOperand = 
operand.accept(CompoundIdentifierConverter.this);
   enableComplex = localEnableComplex;
-  if (newOperand != operand) {
+  if (! newOperand.equalsDeep(operand, false)) {
--- End diff --

Is it necessary to call equalsDeep()?  If the expression has an identifier 
which is rewritten by CompoundIdentifierConverter, is it true that the new 
expression would be different in reference from then original one?



> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901021#comment-14901021
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997945
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 ---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
   }
 
   @Override
+  public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
--- End diff --

you can extract this try block as a static method `ensureOptionExists`, 
this same code is duplicated in multiple places


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-3463) Unit test of project pushdown in TestUnionAll should put more precisely plan attribute in plan verification.

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman closed DRILL-3463.
---

> Unit test of project pushdown in TestUnionAll should put more precisely plan 
> attribute  in plan verification. 
> --
>
> Key: DRILL-3463
> URL: https://issues.apache.org/jira/browse/DRILL-3463
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-3463-Unit-test-of-project-pushdown-in-TestUnio.patch
>
>
> As part of fix for DRILL-2802, it was discovered that several unit test cases 
> for project pushdown in TestUnionAll did not put the desired plan attributes 
> in to the expected plan result.
> To verify project pushdown is working properly, one simple way is to verify 
> that the the column list in the Scan operator contains the desired columns. 
> This should be the part of plan verification. However, the unit test cases in 
> TestUnionAll did not do that. In stead, it tries to match a pattern of 
> "Project -- Scan", which seems not serving the purpose it desired.
> For instance,
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*\n" +
> {code}
> should be replaced by 
> {code}
>  final String[] expectedPlan = {"UnionAll.*\n." +
>  "*Project.*\n" +
> ".*Scan.*columns=\\[`n_comment`, `n_nationkey`, `n_name`\\].*\n" 
> {code}
> if we want to verify the column 'n_comment', 'n_nationkey', 'n_name' are 
> pushed into Scan operator.
> To fix this, modify the expected plan result, such that it contains the plan 
> attributes that should be able to verify whether the project pushdown is 
> working or not.
> This will help catch project pushdown failure, and avoid causing more false 
> alarm in plan verification.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman updated DRILL-3621:

Assignee: Khurram Faraaz  (was: Jinfeng Ni)

> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
> 
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong 
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY = 
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10  |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY = 
> 'DUMMY1';
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN 
> ('DUMMY1' , 'DUMMY10');
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY 
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> |   RK|
> +-+
> | DUMMY1  |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT 
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN 
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec 
> [tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10, filter=null], 
> columns=[`row_key`]]])
>  |
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901000#comment-14901000
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39996854
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
 ---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
   abstract boolean setLocalOption(OptionValue value);
 
+  /**
+   * Deletes the option with given name for this manager without falling 
back.
+   *
+   * @param type option type
+   * @return true iff the option was successfully deleted
--- End diff --

this javadoc can have multiple meanings: _successfully deleted_ could 
either mean the implementation couldn't find the option, or it doesn't support 
the option type. It should actually return false if it doesn't support the 
option type


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901009#comment-14901009
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997204
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/InMemoryOptionManager.java
 ---
@@ -54,12 +56,32 @@ boolean setLocalOption(final OptionValue value) {
 return options.values();
   }
 
+  @Override
+  boolean deleteLocalOptions(final OptionType type) {
+if (supportsOption(type)) {
+  options.clear();
+  return true;
+} else {
+  return false;
+}
+  }
+
+  @Override
+  boolean deleteLocalOption(final String name, final OptionType type) {
+if (supportsOption(type)) {
+  options.remove(name);
+  return true;
+} else {
+  return false;
+}
+  }
+
   /**
-   * Check to see if implementations of this manager support the given 
option value (e.g. check for option type).
+   * Check to see if implementations of this manager support the given 
option type.
*
-   * @param value the option value
-   * @return true iff the option value is supported
+   * @param type option type
+   * @return true iff the type is supported
*/
-  abstract boolean supportsOption(OptionValue value);
+  abstract boolean supportsOption(OptionType type);
--- End diff --

you should probably rename this to `supportsOptionType` because we are 
effectively checking if the option type is supported. This will help the method 
be self explanatory


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3798) Cannot group by the functions without ()

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni resolved DRILL-3798.
---
Resolution: Duplicate

> Cannot group by the functions without ()
> 
>
> Key: DRILL-3798
> URL: https://issues.apache.org/jira/browse/DRILL-3798
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Jinfeng Ni
>
> Drill can not group-by the function without ().
> eg:
> {code}
> SELECT CURRENT_DATE 
> FROM hive.h1db.testdate
> group  by CURRENT_DATE;
>   Caused By (org.apache.calcite.sql.validate.SqlValidatorException) Column 
> 'CURRENT_DATE' not found in any table
> {code}
> Bad ones:
> {code}
> SELECT CURRENT_TIME 
> FROM hive.h1db.testdate
> group  by CURRENT_TIME;
> SELECT CURRENT_TIMESTAMP 
> FROM hive.h1db.testdate
> group  by CURRENT_TIMESTAMP;
> SELECT LOCALTIME 
> FROM hive.h1db.testdate
> group  by LOCALTIME;
> SELECT LOCALTIMESTAMP 
> FROM hive.h1db.testdate
> group  by LOCALTIMESTAMP;
> {code}
> Good ones:
> {code}
> SELECT NOW()
> FROM hive.h1db.testdate
> group  by NOW();
> SELECT TIMEOFDAY()
> FROM hive.h1db.testdate
> group  by TIMEOFDAY();
> SELECT UNIX_TIMESTAMP()
> FROM hive.h1db.testdate
> group  by UNIX_TIMESTAMP();
> SELECT PI()
> FROM hive.h1db.testdate
> group  by  PI();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni reassigned DRILL-3781:
-

Assignee: Jinfeng Ni  (was: Mehant Baid)

> Using CURRENT_DATE in a group by throws a column not found error for hive 
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files. 
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column 
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE  from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name 
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date  |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE  |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need 
> anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901010#comment-14901010
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997250
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionManager.java
 ---
@@ -31,8 +33,33 @@
   void setOption(OptionValue value);
 
   /**
+   * Deletes the option. Unfortunately, the type is required given the 
fallback structure of option managers.
--- End diff --

why _unfortunately_ ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901018#comment-14901018
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997752
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 ---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
   }
 
   @Override
+  public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
+  getValidator(name);
+} catch (final IllegalArgumentException e) {
+  throw UserException.validationError(e)
+.build(logger);
+}
+options.delete(name.toLowerCase());
+  }
+
+  @Override
+  public void deleteAllOptions(OptionType type) {
+assert type == OptionType.SYSTEM;
--- End diff --

same here


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901029#comment-14901029
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39998405
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
   ExecConstants.SLICE_TARGET_DEFAULT));
 testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name 
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
 .unOrdered()
 .baselineColumns("status")
 .baselineValues("DEFAULT")
 .build()
 .run();
   }
+
+  @Test
+  public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type = 
'SESSION';");
+testBuilder()
+  .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .baselineColumns("num_val")
+  .baselineValues((long) 10)
+  .build()
+  .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+  }
+
+  @Test
+  public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+  .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND 
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("bool_val")
+  .baselineValues(true)
+  .build()
+  .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+  }
+
+  @Test
+  public void resetAllSessionOptions() throws Exception {
--- End diff --

also change _SESSION_ and _SYSTEM_ options then reset all _SESSION_ options


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to 

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900989#comment-14900989
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39996054
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
 ---
@@ -49,45 +53,67 @@ public SetOptionHandler(QueryContext context) {
   }
 
   @Override
-  public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException, 
RelConversionException, IOException, ForemanSetupException {
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException, 
ForemanSetupException {
 final SqlSetOption option = unwrap(sqlNode, SqlSetOption.class);
-final String scope = option.getScope();
-final String name = option.getName();
 final SqlNode value = option.getValue();
-OptionValue.OptionType type;
-if (value instanceof SqlLiteral) {
+if (value != null && !(value instanceof SqlLiteral)) {
+  throw UserException.validationError()
+  .message("Drill does not support assigning non-literal values in 
SET statements.")
+  .build(logger);
+}
+
+final String scope = option.getScope();
+final OptionValue.OptionType type;
+if (scope == null) { // No scope mentioned assumed SESSION
+  type = OptionType.SESSION;
+} else {
   switch (scope.toLowerCase()) {
-case "session":
-  type = OptionValue.OptionType.SESSION;
-  break;
-case "system":
-  type = OptionValue.OptionType.SYSTEM;
-  break;
-//case "query":
-//  type = OptionValue.OptionType.QUERY;
-//  break;
-default:
-  throw new ValidationException("Invalid OPTION scope. Scope must 
be SESSION or SYSTEM.");
+  case "session":
+type = OptionType.SESSION;
+break;
+  case "system":
+type = OptionType.SYSTEM;
+break;
+  default:
+throw UserException.validationError()
+.message("Invalid OPTION scope %s. Scope must be SESSION or 
SYSTEM.", scope)
+.build(logger);
   }
+}
 
-  if (type == OptionType.SYSTEM) {
-// If the user authentication is enabled, make sure the user who 
is trying to change the system option has
-// administrative privileges.
-if (context.isUserAuthenticationEnabled() &&
-!ImpersonationUtil.hasAdminPrivileges(
-context.getQueryUserName(),
-
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
-
context.getOptions().getOption(ExecConstants.ADMIN_USER_GROUPS_KEY).string_val))
 {
-  throw UserException.permissionError()
-  .message("Not authorized to change SYSTEM options.")
-  .build(logger);
-}
+if (type == OptionType.SYSTEM) {
+  // If the user authentication is enabled, make sure the user who is 
trying to change the system option has
+  // administrative privileges.
+  if (context.isUserAuthenticationEnabled() &&
+  !ImpersonationUtil.hasAdminPrivileges(
+context.getQueryUserName(),
+
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
--- End diff --

you should reuse `context.getOptions()` as a local variable


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900994#comment-14900994
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39996432
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java
 ---
@@ -161,6 +171,7 @@ public SqlNode visitChild(
 rules.put(SqlJoin.class, R(D, D, D, D, D, E));
 rules.put(SqlOrderBy.class, R(D, E, D, D));
 rules.put(SqlDropTable.class, R(D));
+rules.put(SqlSetOption.class, R(D, D, D));
--- End diff --

can you add a comment explaining why this rule is needed ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901005#comment-14901005
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997032
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
 ---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
   abstract boolean setLocalOption(OptionValue value);
 
+  /**
+   * Deletes the option with given name for this manager without falling 
back.
+   *
+   * @param type option type
+   * @return true iff the option was successfully deleted
+   */
+  abstract boolean deleteLocalOptions(OptionType type);
--- End diff --

can you rename this to `deleteAllLocalOptions` ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901016#comment-14901016
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997731
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 ---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
   }
 
   @Override
+  public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
--- End diff --

can you please add an error message to the assertion ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901014#comment-14901014
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39997680
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -20,14 +20,21 @@
 import com.google.common.base.Predicate;
 import com.google.common.collect.Collections2;
 import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.drill.common.exceptions.UserException;
 import org.apache.drill.common.map.CaseInsensitiveMap;
 import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionValue.OptionType;
 
 import java.util.Collection;
 import java.util.Map;
 
 /**
- * {@link OptionManager} that holds options within {@link 
org.apache.drill.exec.rpc.user.UserSession} context.
+ * {@link OptionManager} that holds options within {@link 
org.apache.drill.exec.rpc.user.UserSession} context. Options
+ * set at the session level only apply to queries that you run during the 
current Drill connection. Session level
+ * settings override system level settings.
+ *
+ * NOTE that currently, the effects of deleting a short lived option (see 
{@link OptionValidator#isShortLived}) are
+ * undefined.
--- End diff --

can you please elaborate on the _undefined_ effects of deleting short lived 
options ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901027#comment-14901027
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39998259
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
   ExecConstants.SLICE_TARGET_DEFAULT));
 testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name 
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
 .unOrdered()
 .baselineColumns("status")
 .baselineValues("DEFAULT")
 .build()
 .run();
   }
+
+  @Test
+  public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type = 
'SESSION';");
+testBuilder()
+  .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .baselineColumns("num_val")
+  .baselineValues((long) 10)
+  .build()
+  .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+  }
+
+  @Test
+  public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+  .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND 
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("bool_val")
+  .baselineValues(true)
+  .build()
+  .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+  }
+
+  @Test
+  public void resetAllSessionOptions() throws Exception {
--- End diff --

resetting all _SYSTEM_ options is not covered here


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--

[jira] [Commented] (DRILL-2908) Support reading the Parquet int 96 type

[Parth Chandra (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900965#comment-14900965
 ] 

Parth Chandra commented on DRILL-2908:
--

Pull request: https://github.com/apache/drill/pull/162
[~adeneche] please review 

> Support reading the Parquet int 96 type
> ---
>
> Key: DRILL-2908
> URL: https://issues.apache.org/jira/browse/DRILL-2908
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Reporter: Jason Altekruse
>Assignee: Jason Altekruse
> Fix For: 1.2.0
>
>
> While Drill does not currently have an int96 type, it is supported by the 
> parquet format and we should be able to read files that contain columns of 
> this type. For now we will read the data into a varbinary and users will have 
> to use existing convert_from functions or write their own to interpret the 
> type of data actually stored. One example is the Impala timestamp format 
> which is encoded in an int96 column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901024#comment-14901024
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39998097
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControls.java
 ---
@@ -80,7 +80,7 @@
  * @param ttl  the number of queries for which this option should be 
valid
  */
 public ControlsOptionValidator(final String name, final String def, 
final int ttl) {
-  super(name, OptionValue.Kind.STRING, 
OptionValue.createString(OptionType.SESSION, name, def));
+  super(name, OptionValue.Kind.STRING, 
OptionValue.createString(OptionType.SYSTEM, name, def));
--- End diff --

can you add a comment explaining why this needs to be a _SYSTEM_ rather 
than a _SESSION_ option ? someone else might revert this change inadvertently.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901025#comment-14901025
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39998161
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
--- End diff --

you can remove `String.format` here too


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3813) table from directory subtree having no descendent files fails with index error

[Deneche A. Hakim (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900790#comment-14900790
 ] 

Deneche A. Hakim commented on DRILL-3813:
-

This could be related to DRILL-2618

> table from directory subtree having no descendent files fails with index error
> --
>
> Key: DRILL-3813
> URL: https://issues.apache.org/jira/browse/DRILL-3813
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser, Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Aman Sinha
>
> Trying to use as a table a directory subtree that has no descendent files 
> (but zero or more descendent directories) yields what seems to be a partially 
> handled index out-of-bounds condition.  
> For example, with {{/tmp/empty_directory}} being an empty directory:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> 
> {noformat}
> Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
> grandchild directory, but not descendent files:
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> 
> {noformat}
> A directory subtree having no files was expected to be taken as a table with 
> no rows (and a null schema).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3813) table from directory subtree having no descendent files fails with index error

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-3813:

Description: 
Trying to use as a table a directory subtree that has no descendent files (but 
zero or more descendent directories) yields what seems to be a partially 
handled index out-of-bounds condition.  

For example, with {{/tmp/empty_directory}} being an empty directory:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
grandchild directory, but not descendent files:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


A directory subtree having no files was expected to be taken as a table with no 
rows (and a null schema).


  was:
Trying to use as a table a directory subtree that has no descendent files (but 
zero or more descendent directories) yields what seems to be a partially 
handled index out-of-bounds condition.  

For example, with {{/tmp/empty_directory}} being an empty directory:

{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
grandchild directory, but not descendent files:

0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0


[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> 
{noformat}


A directory subtree having no files was expected to be taken as a table with no 
rows (and a null schema).



> table from directory subtree having no descendent files fails with index error
> --
>
> Key: DRILL-3813
> URL: https://issues.apache.org/jira/browse/DRILL-3813
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser, Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Aman Sinha
>
> Trying to use as a table a directory subtree that has no descendent files 
> (but zero or more descendent directories) yields what seems to be a partially 
> handled index out-of-bounds condition.  
> For example, with {{/tmp/empty_directory}} being an empty directory:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> 
> {noformat}
> Also, with {{/tmp/no_child_files_subtree}} having two child directories and a 
> grandchild directory, but not descendent files:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010] 
> (state=,code=0)
> 0: jdbc:drill:zk=local> 
> {noformat}
> A directory subtree having no files was expected to be taken as a table with 
> no rows (and a null schema).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3731) Partition Pruning not taking place when we have case statement within the filter

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901332#comment-14901332
 ] 

Rahul Challapalli commented on DRILL-3731:
--

[~jni] You are right. We can simplify the filter in this case. I constructed 
this just to test the behavior and it is not hard to imagine a real world 
scenario where a case statement is used in the filter which results in 
partition pruning not taking place.

> Partition Pruning not taking place when we have case statement within the 
> filter
> 
>
> Key: DRILL-3731
> URL: https://issues.apache.org/jira/browse/DRILL-3731
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Rahul Challapalli
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: lineitempart.tgz
>
>
> Commit # 8f33c3d55228a98ea2a309490ecbfb7a180f012c
> Partition Pruning is not taking place in the below query
> {code}
> explain plan for select count(*) from 
> `/drill/testdata/partition_pruning/dfs/lineitempart` where case when 
> dir0=1991 then null else 1 end is null or dir0=1992;
> 00-00Screen
> 00-01  Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03  Project($f0=[0])
> 00-04SelectionVectorRemover
> 00-05  Filter(condition=[OR(IS NULL(CASE(=($0, 1991), null, 1)), 
> =($0, 1992))])
> 00-06Scan(groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/drill/testdata/partition_pruning/dfs/lineitempart, 
> numFiles=7, columns=[`dir0`], 
> files=[maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1991/lineitemaa.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1992/lineitemab.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1996/lineitemaf.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1994/lineitemad.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1993/lineitemac.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1997/lineitemag.tbl,
>  
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1995/lineitemae.tbl]]])
> {code}
> The planner should have pruned all partitions except 1991 & 1992
> I attached the dataset. Let me know if you need anything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901366#comment-14901366
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40023540
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -20,14 +20,21 @@
 import com.google.common.base.Predicate;
 import com.google.common.collect.Collections2;
 import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.drill.common.exceptions.UserException;
 import org.apache.drill.common.map.CaseInsensitiveMap;
 import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionValue.OptionType;
 
 import java.util.Collection;
 import java.util.Map;
 
 /**
- * {@link OptionManager} that holds options within {@link 
org.apache.drill.exec.rpc.user.UserSession} context.
+ * {@link OptionManager} that holds options within {@link 
org.apache.drill.exec.rpc.user.UserSession} context. Options
+ * set at the session level only apply to queries that you run during the 
current Drill connection. Session level
+ * settings override system level settings.
+ *
+ * NOTE that currently, the effects of deleting a short lived option (see 
{@link OptionValidator#isShortLived}) are
+ * undefined.
--- End diff --

No, I meant could you explain what you mean by _undefined_ effects. I think 
you mean if we set a short lived session, for example we inject an exception, 
then try to delete it, depending on where the exception was injected the reset 
query could either succeed or the exception could actually be thrown in the 
reset query itself.
Just improve the comment to explain what you mean


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901321#comment-14901321
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40020795
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
   ExecConstants.SLICE_TARGET_DEFAULT));
 testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name 
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
 .unOrdered()
 .baselineColumns("status")
 .baselineValues("DEFAULT")
 .build()
 .run();
   }
+
+  @Test
+  public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type = 
'SESSION';");
+testBuilder()
+  .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .baselineColumns("num_val")
+  .baselineValues((long) 10)
+  .build()
+  .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+  }
+
+  @Test
+  public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+  .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND 
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("bool_val")
+  .baselineValues(true)
+  .build()
+  .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+  }
+
+  @Test
+  public void resetAllSessionOptions() throws Exception {
--- End diff --

changeSessionAndNotSystem is testing this.

Also, we cannot test resetting all system options because the unit tests 
have a "before" method that changes a system option.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to 

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901325#comment-14901325
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40020971
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControls.java
 ---
@@ -80,7 +80,7 @@
  * @param ttl  the number of queries for which this option should be 
valid
  */
 public ControlsOptionValidator(final String name, final String def, 
final int ttl) {
-  super(name, OptionValue.Kind.STRING, 
OptionValue.createString(OptionType.SESSION, name, def));
+  super(name, OptionValue.Kind.STRING, 
OptionValue.createString(OptionType.SYSTEM, name, def));
--- End diff --

I don't think this requires a comment in code, since this applies for all 
option validators.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901339#comment-14901339
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40021824
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 ---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
   }
 
   @Override
+  public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
--- End diff --

I left this as is because the callers could add context and handle the 
exception differently.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901343#comment-14901343
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40021955
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
   ExecConstants.SLICE_TARGET_DEFAULT));
 testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name 
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
 .unOrdered()
 .baselineColumns("status")
 .baselineValues("DEFAULT")
 .build()
 .run();
   }
+
+  @Test
+  public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type = 
'SESSION';");
+testBuilder()
+  .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .baselineColumns("num_val")
+  .baselineValues((long) 10)
+  .build()
+  .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+  }
+
+  @Test
+  public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+  .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND 
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("bool_val")
+  .baselineValues(true)
+  .build()
+  .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+  .unOrdered()
+  .baselineColumns("status")
+  .baselineValues("DEFAULT")
+  .build()
+  .run();
+  }
+
+  @Test
+  public void resetAllSessionOptions() throws Exception {
--- End diff --

my bad, I meant: we change the same option at the _SESSION_ and _SYSTEM_ 
level then we reset it at the _SESSION_ level and we make sure it's still 
changed at the _SYSTEM_ level.
`changeSessionAndNotSystem` kind of does that but with a `RESET ALL` and we 
need to make sure `ALTER SESSION RESET A` also works fine


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to 

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901349#comment-14901349
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40022217
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -20,14 +20,21 @@
 import com.google.common.base.Predicate;
 import com.google.common.collect.Collections2;
 import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.drill.common.exceptions.UserException;
 import org.apache.drill.common.map.CaseInsensitiveMap;
 import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionValue.OptionType;
 
 import java.util.Collection;
 import java.util.Map;
 
 /**
- * {@link OptionManager} that holds options within {@link 
org.apache.drill.exec.rpc.user.UserSession} context.
+ * {@link OptionManager} that holds options within {@link 
org.apache.drill.exec.rpc.user.UserSession} context. Options
+ * set at the session level only apply to queries that you run during the 
current Drill connection. Session level
+ * settings override system level settings.
+ *
+ * NOTE that currently, the effects of deleting a short lived option (see 
{@link OptionValidator#isShortLived}) are
+ * undefined.
--- End diff --

Should I add a TODO and create a ticket?
(personally I don't think this is worth a JIRA yet)


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901358#comment-14901358
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user cwestin commented on a diff in the pull request:

https://github.com/apache/drill/pull/163#discussion_r40023098
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java 
---
@@ -63,6 +63,29 @@
   private final static String planFile="/physical_allocator_test.json";
 
   @Test
+  public void testTransfer() throws Exception {
+final Properties props = new Properties() {
+  {
+put(ExecConstants.TOP_LEVEL_MAX_ALLOC, "100");
+put(ExecConstants.ERROR_ON_MEMORY_LEAK, "true");
+  }
+};
+final DrillConfig config = DrillConfig.create(props);
+BufferAllocator a = new TopLevelAllocator(config);
--- End diff --

I was surprised this compiled; in one of my recent patches, I had removed 
the "public" from TopLevelAllocator, but it seems to have come back. It was 
removed to force people to use RootAllocatorFactory.newRoot() instead of using 
the TopLevelAllocator constructor directly. Can you please remove the public 
from TopLevelAllocator again, and replace this constructor here and on the next 
line with RootAllocatorFactory.newRoot()?


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901354#comment-14901354
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40022907
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionManager.java
 ---
@@ -31,8 +33,33 @@
   void setOption(OptionValue value);
 
   /**
+   * Deletes the option. Unfortunately, the type is required given the 
fallback structure of option managers.
--- End diff --

*Unfortunately* because we assume *fallback structure of option managers*


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901356#comment-14901356
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40022994
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControls.java
 ---
@@ -80,7 +80,7 @@
  * @param ttl  the number of queries for which this option should be 
valid
  */
 public ControlsOptionValidator(final String name, final String def, 
final int ttl) {
-  super(name, OptionValue.Kind.STRING, 
OptionValue.createString(OptionType.SESSION, name, def));
+  super(name, OptionValue.Kind.STRING, 
OptionValue.createString(OptionType.SYSTEM, name, def));
--- End diff --

Ok, I see, `TypeValidator` always expects _SYSTEM_ OptionValues
Should we add an assertion to TypeValidator to make sure this doesn't 
happen again ? 


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901360#comment-14901360
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user cwestin commented on the pull request:

https://github.com/apache/drill/pull/163#issuecomment-142102955
  
Just one comment, otherwise +1 (non-binding).


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3816) weird file-extension recognition behavior in directory subtree scanning

[Daniel Barclay (Drill) (JIRA)]

Daniel Barclay (Drill) created DRILL-3816:
-

 Summary: weird file-extension recognition behavior in directory 
subtree scanning
 Key: DRILL-3816
 URL: https://issues.apache.org/jira/browse/DRILL-3816
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Other
Reporter: Daniel Barclay (Drill)
Assignee: Jacques Nadeau


In scanning of directory subtrees for files, recognition of known vs. unknown 
file extensions seems really screwy (not following any apparent pattern). 

For example:
- a suffix of {{.jsxon_not}}, as expected, is not recognized as a JSON file
- a suffix of {{.jsoxn_not}} unexpectedly _is_ taken as JSON
- a suffix of .{{jsonx_not}}, as expected, is not recognized as a JSON file

(Creating a directory containing only a non-empty JSON file ending with 
{{.json}} and another non-empty JSON file ending with one of the above suffixes 
sometimes reads both JSON files and sometimes reports a (presumably) expected 
error because of the mixed file extensions).)

The result sometimes seems to also depend on the rest of the filename, 
presumably related to the order of listing of files.  (It's not clear if it 
depends only on the order after filename sorting, or also depends on the order 
file names are listed the by OS.)

Here are more data points (using a JSON file named {{voter1.json}}): 

- with {{voter2.xjson_not}} - read, as JSON
- with {{voter2.jxson_not}} - read, as JSON
- with {{voter2.jsxon_not}} - causes expected error
- with {{voter2.jsoxn_not}} - read, as JSON
- with {{voter2.jsonx_not}} - causes expected error
- with {{voter2.json_xnot}} - read, as JSON
- with {{voter2.json_nxot}} - read, as JSON
- with {{voter2.json_noxt}} - read, as JSON
- with {{voter2.json_notx}} - read, as JSON
- with {{voter2.jsonxnot}}  - read, as JSON
- with {{voter2.jsonxot}}   - read, as JSON
- with {{voter2.jsoxot}}- causes expected error
- with {{voter2.jxsxoxn}}   - read, as JSON
- with {{voter2.xjxsxoxn}}  - read, as JSON
- with {{voter2.xjxsxoxnx}} - causes expected error
- with {{voter2.xjxxoxn}}   - read, as JSON
- with {{voter2.xjxxxn}- read, as JSON
- with {{voter2.n} - read, as JSON
- with {{voter2.}  - read, as JSON
- with {{voter2.xxx}}   - read, as JSON
- with {{voter2.xx}}- read, as JSON
- with {{voter2.x}} - read, as JSON
- with {{voter2.}}  - causes expected error
- with {{voter2.x - read, as JSON
- with {{voter2.xx- read, as JSON






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3816) weird file-extension recognition behavior in directory subtree scanning

[Daniel Barclay (Drill) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Barclay (Drill) updated DRILL-3816:
--
Description: 
In scanning of directory subtrees for files, recognition of known vs. unknown 
file extensions seems really screwy (not following any apparent pattern). 

For example:
- a suffix of {{.jsxon_not}}, as expected, is not recognized as a JSON file
- a suffix of {{.jsoxn_not}} unexpectedly _is_ taken as JSON
- a suffix of .{{jsonx_not}}, as expected, is not recognized as a JSON file

(Creating a directory containing only a non-empty JSON file ending with 
{{.json}} and another non-empty JSON file ending with one of the above suffixes 
sometimes reads both JSON files and sometimes reports a (presumably) expected 
error because of the mixed file extensions).)

The result sometimes seems to also depend on the rest of the filename, 
presumably related to the order of listing of files.  (It's not clear if it 
depends only on the order after filename sorting, or also depends on the order 
file names are listed the by OS.)

Here are more data points (using a JSON file named {{voter1.json}}): 

- with {{voter2.xjson_not}} - read, as JSON
- with {{voter2.jxson_not}} - read, as JSON
- with {{voter2.jsxon_not}} - causes expected error
- with {{voter2.jsoxn_not}} - read, as JSON
- with {{voter2.jsonx_not}} - causes expected error
- with {{voter2.json_xnot}} - read, as JSON
- with {{voter2.json_nxot}} - read, as JSON
- with {{voter2.json_noxt}} - read, as JSON
- with {{voter2.json_notx}} - read, as JSON
- with {{voter2.jsonxnot}}  - read, as JSON
- with {{voter2.jsonxot}}   - read, as JSON
- with {{voter2.jsoxot}}- causes expected error
- with {{voter2.jxsxoxn}}   - read, as JSON
- with {{voter2.xjxsxoxn}}  - read, as JSON
- with {{voter2.xjxsxoxnx}} - causes expected error
- with {{voter2.xjxxoxn}}   - read, as JSON
- with {{voter2.xjxxxn}}- read, as JSON
- with {{voter2.n}} - read, as JSON
- with {{voter2.}}  - read, as JSON
- with {{voter2.xxx}}   - read, as JSON
- with {{voter2.xx}}- read, as JSON
- with {{voter2.x}} - read, as JSON
- with {{voter2.}}  - causes expected error
- with {{voter2.x}} - read, as JSON
- with {{voter2.xx}}- read, as JSON




  was:
In scanning of directory subtrees for files, recognition of known vs. unknown 
file extensions seems really screwy (not following any apparent pattern). 

For example:
- a suffix of {{.jsxon_not}}, as expected, is not recognized as a JSON file
- a suffix of {{.jsoxn_not}} unexpectedly _is_ taken as JSON
- a suffix of .{{jsonx_not}}, as expected, is not recognized as a JSON file

(Creating a directory containing only a non-empty JSON file ending with 
{{.json}} and another non-empty JSON file ending with one of the above suffixes 
sometimes reads both JSON files and sometimes reports a (presumably) expected 
error because of the mixed file extensions).)

The result sometimes seems to also depend on the rest of the filename, 
presumably related to the order of listing of files.  (It's not clear if it 
depends only on the order after filename sorting, or also depends on the order 
file names are listed the by OS.)

Here are more data points (using a JSON file named {{voter1.json}}): 

- with {{voter2.xjson_not}} - read, as JSON
- with {{voter2.jxson_not}} - read, as JSON
- with {{voter2.jsxon_not}} - causes expected error
- with {{voter2.jsoxn_not}} - read, as JSON
- with {{voter2.jsonx_not}} - causes expected error
- with {{voter2.json_xnot}} - read, as JSON
- with {{voter2.json_nxot}} - read, as JSON
- with {{voter2.json_noxt}} - read, as JSON
- with {{voter2.json_notx}} - read, as JSON
- with {{voter2.jsonxnot}}  - read, as JSON
- with {{voter2.jsonxot}}   - read, as JSON
- with {{voter2.jsoxot}}- causes expected error
- with {{voter2.jxsxoxn}}   - read, as JSON
- with {{voter2.xjxsxoxn}}  - read, as JSON
- with {{voter2.xjxsxoxnx}} - causes expected error
- with {{voter2.xjxxoxn}}   - read, as JSON
- with {{voter2.xjxxxn}- read, as JSON
- with {{voter2.n} - read, as JSON
- with {{voter2.}  - read, as JSON
- with {{voter2.xxx}}   - read, as JSON
- with {{voter2.xx}}- read, as JSON
- with {{voter2.x}} - read, as JSON
- with {{voter2.}}  - causes expected error
- with {{voter2.x - read, as JSON
- with {{voter2.xx- read, as JSON





> weird file-extension recognition behavior in directory subtree scanning
> ---
>
> Key: DRILL-3816
> URL: https://issues.apache.org/jira/browse/DRILL-3816
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Jacques Nadeau
>
> In scanning of directory subtrees for 

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901237#comment-14901237
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

GitHub user adeneche opened a pull request:

https://github.com/apache/drill/pull/163

DRILL-3811: AtomicRemainder incorrectly accounts for transferred allo…

…cations

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adeneche/incubator-drill fix-acountor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/163.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #163


commit 94af90b8006ea95fdfa02cc4ddae5224ab31c586
Author: adeneche 
Date:   2015-09-20T19:11:55Z

DRILL-3811: AtomicRemainder incorrectly accounts for transferred allocations




> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901249#comment-14901249
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40014413
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java
 ---
@@ -161,6 +171,7 @@ public SqlNode visitChild(
 rules.put(SqlJoin.class, R(D, D, D, D, D, E));
 rules.put(SqlOrderBy.class, R(D, E, D, D));
 rules.put(SqlDropTable.class, R(D));
+rules.put(SqlSetOption.class, R(D, D, D));
--- End diff --

the comment doesn't explain why this specific rule has been added. It just 
explains how the rules work in general.
Why do we need this specific rule, what happens if I remove it ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3481) Query with Window Function fails with "SYSTEM ERROR: RpcException: Data not accepted downstream."

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-3481:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Query with Window Function fails with "SYSTEM ERROR: RpcException: Data not 
> accepted downstream."
> -
>
> Key: DRILL-3481
> URL: https://issues.apache.org/jira/browse/DRILL-3481
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Abhishek Girish
>Assignee: Deneche A. Hakim
>  Labels: window_function
> Fix For: 1.3.0
>
> Attachments: JSON Profile.txt, Physical Plan.txt
>
>
> I'm seeing an error when executing a simple WF query on the latest master. 
> Dataset: TPC-DS SF1000 - Parquet
> Git.Commit.ID: b6577fe (Jul 8 15)
> Options:
> {code}
> alter session set `planner.memory.max_query_memory_per_node` = 21474836480;
> {code}
> Query:
> {code:sql}
> SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk ORDER BY 
> ss.ss_store_sk)  FROM store_sales ss WHERE ss.ss_store_sk is not NULL LIMIT 
> 20;
> {code}
> Error:
> {code}
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: 
> RpcException: Data not accepted downstream.
> Fragment 2:1
> [Error Id: 49edceeb-8f10-474d-9cf2-adb4baa13bf4 on ucs-node7.perf.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at 
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-08 12:17:44,764 ucs-node7.perf.lab [BitClient-2] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested RUNNING --> 
> FAILED
> 2015-07-08 12:17:44,765 ucs-node7.perf.lab [BitClient-2] ERROR 
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping 
> future sends.
> 2015-07-08 12:17:44,765 ucs-node7.perf.lab [BitClient-2] ERROR 
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping 
> future sends.
> 2015-07-08 12:17:44,765 ucs-node7.perf.lab [BitClient-2] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED --> 
> FAILED
> 2015-07-08 12:17:44,768 ucs-node7.perf.lab [BitClient-2] ERROR 
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping 
> future sends.
> 2015-07-08 12:17:44,768 ucs-node7.perf.lab [BitClient-2] ERROR 
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping 
> future sends.
> 2015-07-08 12:17:44,769 ucs-node7.perf.lab [BitClient-2] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED --> 
> FAILED
> 2015-07-08 12:17:44,771 ucs-node7.perf.lab [BitClient-2] ERROR 
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping 
> future sends.
> 2015-07-08 12:17:44,771 ucs-node7.perf.lab [BitClient-2] ERROR 
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping 
> future sends.
> 2015-07-08 12:17:44,772 ucs-node7.perf.lab [BitClient-2] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED --> 
> FAILED
> 2015-07-08 12:17:44,790 ucs-node7.perf.lab 
> [2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:frag:2:1] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED --> 
> FINISHED
> 2015-07-08 12:17:44,814 ucs-node7.perf.lab 
> [2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:frag:2:1] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: RpcException: Data not 
> accepted downstream.
> Fragment 2:1
> [Error Id: 49edceeb-8f10-474d-9cf2-adb4baa13bf4 on ucs-node7.perf.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: RpcException: 
> Data not accepted downstream.
> Fragment 2:1
> [Error Id: 49edceeb-8f10-474d-9cf2-adb4baa13bf4 on ucs-node7.perf.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
>  ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
>  [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at 
> 

[jira] [Updated] (DRILL-3630) error message needs to be fixed LEAD , LAG window functions

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-3630:

Fix Version/s: (was: 1.2.0)
   1.3.0

> error message needs to be fixed LEAD , LAG window functions
> ---
>
> Key: DRILL-3630
> URL: https://issues.apache.org/jira/browse/DRILL-3630
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
> Environment: private-branch 
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Minor
>  Labels: window_function
> Fix For: 1.3.0
>
>
> The error message says LEAD expects ONE input argument, however if you see in 
> the other query below, lead( col9 , 1 ) does not throw any error (as this is 
> valid syntax and is supported). Another case is, lead(col9,1,1) where we 
> don't report any error being reported. We need to fix the error message and 
> handle inputs appropriately.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col9 , lead(col9,1,0,1) over(partition 
> by col7 order by col0) lead_col9 from FEWRWSPQQ_101;
> Error: PARSE ERROR: From line 1, column 15 to line 1, column 35: Invalid 
> number of arguments to function 'LEAD'. Was expecting 1 arguments
> [Error Id: 4a94487c-42e6-4672-978e-f011c997c537 on centos-03.qa.lab:31010] 
> (state=,code=0)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col9 , lead(col9,1) over(partition by 
> col7 order by col0) lead_col9 from FEWRWSPQQ_101;
> +---+---+
> | col9  | 
>   lead_col9   |
> +---+---+
> | OXCB  | 
> ZXCZ  |
> | ZXCZ  | 
> AXCZ  |
> | AXCZ  | 
> CXCB  |
> | CXCB  | 
> HXCZ  |
> | HXCZ  | 
> UXCB  |
> | UXCB  | 
> TXCD  |
> | TXCD  | 
> PXCD  |
> | PXCD  | 
> NXCB  |
> | NXCB  | 
> MXCB  |
> | MXCB  | 
> WXCB  |
> | WXCB  | null
>   |
> | KXCB  | 
> BXCD  |
> | BXCD  | 
> DXCB  |
> | DXCB  | 
> IXCD  |
> | IXCD  | 
> YXCB  |
> | YXCB  | 
> EXCZ  |
> | EXCZ  | 
> LXCB  |
> | LXCB  | 
> SXCB  |
> | SXCB  | 
> VXCZ  |
> | VXCZ  | 
> FXCB  |
> | FXCB  | 
> QXCZ  |
> | QXCZ  | null
>   |
> +---+---+
> 22 rows selected (0.245 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3733) erro message fix - NTILE function

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-3733:

Fix Version/s: (was: 1.2.0)
   1.3.0

> erro message fix - NTILE function
> -
>
> Key: DRILL-3733
> URL: https://issues.apache.org/jira/browse/DRILL-3733
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Minor
>  Labels: window_function
> Fix For: 1.3.0
>
>
> Can we have the message read this way
> "NTILE only accepts positive (non-zero) integer argument"
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col7 , col9 , ntile(0) over(order by 
> col0) lastVal_col9 from FEWRWSPQQ_101;
> Error: FUNCTION ERROR: NTILE only accepts positive integer argument
> Fragment 0:0
> [Error Id: 650d44a8-73d0-4675-8c18-67609f43a962 on centos-04.qa.lab:31010] 
> (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901255#comment-14901255
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40014502
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,124 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
   ExecConstants.SLICE_TARGET_DEFAULT));
 testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name 
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
 .unOrdered()
 .baselineColumns("status")
 .baselineValues("DEFAULT")
 .build()
 .run();
   }
+
+  @Test
+  public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type = 
'SESSION';");
+testBuilder()
+  .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .baselineColumns("num_val")
+  .baselineValues((long) 10)
+  .build()
+  .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+  }
+
+  @Test
+  public void setAndResetSystemOption() throws Exception {
--- End diff --

thanks for adding those unit tests


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3732) Drill leaks memory if external sort hits out of disk space exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901281#comment-14901281
 ] 

ASF GitHub Bot commented on DRILL-3732:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/147


> Drill leaks memory if external sort hits out of disk space exception
> 
>
> Key: DRILL-3732
> URL: https://issues.apache.org/jira/browse/DRILL-3732
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: drillbit.log
>
>
> Ran into it when running CTAS with partition by.
> Here is what reproduction looks like:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_4(ss_item_sk, 
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) 
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > case when columns[2] = '' then cast(null as 
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[3] = '' then cast(null as 
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[4] = '' then cast(null as 
> varchar(100)) else cast(columns[4] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[5] = '' then cast(null as 
> varchar(100)) else cast(columns[5] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as 
> varchar(100)) else cast(columns[0] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[8] = '' then cast(null as 
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . >  `store_sales.dat` ss 
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: IllegalStateException: Failure while closing accountor.  
> Expected private and shared pools to be set to initial values.  However, one 
> or more were not.  Stats are
>   zoneinitallocated   delta 
>   private 10009680512 319488 
>   shared  100010000.
> Fragment 1:21
> [Error Id: bd0d7d59-8693-476b-8671-70f0b2e7a176 on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> Setup:
> single node
> 8GB direct memory
> 4GB heap memory
>store_sales.dat is a file from TPCDS SF100
> drillbit.log attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-2359) ClassPathFileSystem.getFileStatus() should throw FileNotFoundException when path doesn't exist

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-2359:

Fix Version/s: (was: 1.2.0)
   1.3.0

> ClassPathFileSystem.getFileStatus() should throw FileNotFoundException when 
> path doesn't exist
> --
>
> Key: DRILL-2359
> URL: https://issues.apache.org/jira/browse/DRILL-2359
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 0.8.0
>Reporter: Deneche A. Hakim
>Assignee: Deneche A. Hakim
> Fix For: 1.3.0
>
> Attachments: DRILL-2359.1.patch.txt
>
>
> {{ClassPathFileSystem.getFileStatus(Path args0)}} throws {{IOException}} when 
> the path doesn't exist. It should throw {{FileNotFoundException}} instead. 
> One particular method that fails to work correctly because of this is 
> {{org.apache.hadoop.FileSystem.exists(Path f)}}:
> {code}
> public boolean exists(Path f) throws IOException {
> try {
>   return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
>   return false;
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3194) TestDrillbitResilience#memoryLeaksWhenFailed hangs

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-3194:

Fix Version/s: (was: 1.2.0)
   1.3.0

> TestDrillbitResilience#memoryLeaksWhenFailed hangs
> --
>
> Key: DRILL-3194
> URL: https://issues.apache.org/jira/browse/DRILL-3194
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Sudheesh Katkam
>Assignee: Deneche A. Hakim
> Fix For: 1.3.0
>
>
> TestDrillbitResilience#memoryLeaksWhenFailed hangs and fails when run 
> multiple times. This might be related to DRILL-3163.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3815) unknown suffixes .not_json and .json_not treated differently (multi-file case)

[Daniel Barclay (Drill) (JIRA)]

Daniel Barclay (Drill) created DRILL-3815:
-

 Summary: unknown suffixes .not_json and .json_not treated 
differently (multi-file case)
 Key: DRILL-3815
 URL: https://issues.apache.org/jira/browse/DRILL-3815
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Other
Reporter: Daniel Barclay (Drill)
Assignee: Jacques Nadeau


In scanning a directory subtree used as a table, unknown filename extensions 
seem to be treated differently depending on whether they're similar to known 
file extensions.  The behavior suggests that Drill checks whether a file name 
_contains_ an extension's string rather than _ending_ with it. 

For example, given these subtrees with almost identical leaf file names:

{noformat}
$ find /tmp/testext_xx_json/
/tmp/testext_xx_json/
/tmp/testext_xx_json/voter2.not_json
/tmp/testext_xx_json/voter1.json
$ find /tmp/testext_json_xx/
/tmp/testext_json_xx/
/tmp/testext_json_xx/voter1.json
/tmp/testext_json_xx/voter2.json_not
$ 
{noformat}

the results of trying to use them as tables differs:

{noformat}
0: jdbc:drill:zk=local> SELECT *   FROM `dfs.tmp`.`testext_xx_json`;
Sep 21, 2015 11:41:50 AM org.apache.calcite.sql.validate.SqlValidatorException 

...
Error: VALIDATION ERROR: From line 1, column 17 to line 1, column 25: Table 
'dfs.tmp.testext_xx_json' not found


[Error Id: 6fe41deb-0e39-43f6-beca-de27b39d276b on dev-linux2:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> SELECT *   FROM `dfs.tmp`.`testext_json_xx`;
+---+
| onecf |
+---+
| {"name":"someName1"}  |
| {"name":"someName2"}  |
+---+
2 rows selected (0.149 seconds)
{noformat}

(Other probing seems to indicate that there is also some sensitivity to whether 
the extension contains an underscore character.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901185#comment-14901185
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40010220
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
 ---
@@ -49,45 +53,67 @@ public SetOptionHandler(QueryContext context) {
   }
 
   @Override
-  public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException, 
RelConversionException, IOException, ForemanSetupException {
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException, 
ForemanSetupException {
 final SqlSetOption option = unwrap(sqlNode, SqlSetOption.class);
-final String scope = option.getScope();
-final String name = option.getName();
 final SqlNode value = option.getValue();
-OptionValue.OptionType type;
-if (value instanceof SqlLiteral) {
+if (value != null && !(value instanceof SqlLiteral)) {
+  throw UserException.validationError()
+  .message("Drill does not support assigning non-literal values in 
SET statements.")
+  .build(logger);
+}
+
+final String scope = option.getScope();
+final OptionValue.OptionType type;
+if (scope == null) { // No scope mentioned assumed SESSION
+  type = OptionType.SESSION;
+} else {
   switch (scope.toLowerCase()) {
-case "session":
-  type = OptionValue.OptionType.SESSION;
-  break;
-case "system":
-  type = OptionValue.OptionType.SYSTEM;
-  break;
-//case "query":
-//  type = OptionValue.OptionType.QUERY;
-//  break;
-default:
-  throw new ValidationException("Invalid OPTION scope. Scope must 
be SESSION or SYSTEM.");
+  case "session":
+type = OptionType.SESSION;
+break;
+  case "system":
+type = OptionType.SYSTEM;
+break;
+  default:
+throw UserException.validationError()
+.message("Invalid OPTION scope %s. Scope must be SESSION or 
SYSTEM.", scope)
+.build(logger);
   }
+}
 
-  if (type == OptionType.SYSTEM) {
-// If the user authentication is enabled, make sure the user who 
is trying to change the system option has
-// administrative privileges.
-if (context.isUserAuthenticationEnabled() &&
-!ImpersonationUtil.hasAdminPrivileges(
-context.getQueryUserName(),
-
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
-
context.getOptions().getOption(ExecConstants.ADMIN_USER_GROUPS_KEY).string_val))
 {
-  throw UserException.permissionError()
-  .message("Not authorized to change SYSTEM options.")
-  .build(logger);
-}
+if (type == OptionType.SYSTEM) {
+  // If the user authentication is enabled, make sure the user who is 
trying to change the system option has
+  // administrative privileges.
+  if (context.isUserAuthenticationEnabled() &&
+  !ImpersonationUtil.hasAdminPrivileges(
+context.getQueryUserName(),
+
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
--- End diff --

This is not a change I introduced (just difference in spacing), but I'll 
address this issue.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901214#comment-14901214
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40012463
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,124 @@ public void testOptions() throws Exception{
   @Test
   public void checkValidationException() throws Exception {
 thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';", 
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
   }
 
   @Test // DRILL-3122
   public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;", 
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
   ExecConstants.SLICE_TARGET_DEFAULT));
 testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name 
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
 .unOrdered()
 .baselineColumns("status")
 .baselineValues("DEFAULT")
 .build()
 .run();
   }
+
+  @Test
+  public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type = 
'SESSION';");
+testBuilder()
+  .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND 
type = 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .baselineColumns("num_val")
+  .baselineValues((long) 10)
+  .build()
+  .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+  .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type 
= 'SESSION'", SLICE_TARGET)
+  .unOrdered()
+  .expectsEmptyResultSet()
+  .build()
+  .run();
+  }
+
+  @Test
+  public void setAndResetSystemOption() throws Exception {
--- End diff --

changeSessionAndNotSystem and changeSystemAndNotSession are testing this.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901189#comment-14901189
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40010391
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java
 ---
@@ -161,6 +171,7 @@ public SqlNode visitChild(
 rules.put(SqlJoin.class, R(D, D, D, D, D, E));
 rules.put(SqlOrderBy.class, R(D, E, D, D));
 rules.put(SqlDropTable.class, R(D));
+rules.put(SqlSetOption.class, R(D, D, D));
--- End diff --

There is already a comment explaining the rules, and why they are needed. 
What more?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901384#comment-14901384
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/163#discussion_r40024717
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java 
---
@@ -63,6 +63,29 @@
   private final static String planFile="/physical_allocator_test.json";
 
   @Test
+  public void testTransfer() throws Exception {
+final Properties props = new Properties() {
+  {
+put(ExecConstants.TOP_LEVEL_MAX_ALLOC, "100");
+put(ExecConstants.ERROR_ON_MEMORY_LEAK, "true");
+  }
+};
+final DrillConfig config = DrillConfig.create(props);
+BufferAllocator a = new TopLevelAllocator(config);
--- End diff --

sure


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-1065) Provide a reset command to reset an option to its default value

[Sudheesh Katkam (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804598#comment-14804598
 ] 

Sudheesh Katkam edited comment on DRILL-1065 at 9/21/15 8:57 PM:
-

Also, adding:
+ support for
{code}
SET `option name`
RESET `option name`
RESET ALL
{code}
as discussed 
[here|http://mail-archives.apache.org/mod_mbox/calcite-dev/201507.mbox/%3ccampyv7c9ynatuk-2ufh_suddmc6d5-e4jd1_taqmnuoifz1...@mail.gmail.com%3E]
 with an assumption that the scope is SESSION.

+ allow for compound identifiers for option names


was (Author: sudheeshkatkam):
Also, adding:
SET `option name`
RESET `option name`
RESET ALL

as discussed 
[here|http://mail-archives.apache.org/mod_mbox/calcite-dev/201507.mbox/%3ccampyv7c9ynatuk-2ufh_suddmc6d5-e4jd1_taqmnuoifz1...@mail.gmail.com%3E]
 with an assumption that the scope is SESSION.

> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901390#comment-14901390
 ] 

Julian Hyde commented on DRILL-3180:


I don't often see non-join conditions into the ON clause, but I take 
[~magnusp]'s point. If that were an outer join, we would be able to push the 
date condition down to the salary table, whereas if it were in the WHERE clause 
we could not.

Anyway, the JDBC adapter's goal is not to format the SQL to follow any "best 
practice" or to look nice for humans to read. It is to communicate with the 
target DB's query optimizer, ideally in a form that the optimizer is unlikely 
to screw up, and most importantly to preserve semantics.

Sometimes there is a danger that Calcite will "over optimize" the query, e.g.

{code}select *
FROM mp.employees.`employees` e
INNER JOIN  (
  SELECT * FROM mp.employees.`salaries` s
  WHERE s.`to_date` > CURRENT_DATE) AS s
ON e.`EMP_NO` = s.`EMP_NO`{code}

is valid and efficient but the new query block might confuse optimizers like 
MySQL's.

> Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and 
> Netezza from Apache Drill
> ---
>
> Key: DRILL-3180
> URL: https://issues.apache.org/jira/browse/DRILL-3180
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.0.0
>Reporter: Magnus Pierre
>Assignee: Jacques Nadeau
>  Labels: Drill, JDBC, plugin
> Fix For: 1.2.0
>
> Attachments: patch.diff, pom.xml, storage-mpjdbc.zip
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> I have developed the base code for a JDBC storage-plugin for Apache Drill. 
> The code is primitive but consitutes a good starting point for further 
> coding. Today it provides primitive support for SELECT against RDBMS with 
> JDBC. 
> The goal is to provide complete SELECT support against RDBMS with push down 
> capabilities.
> Currently the code is using standard JDBC classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901431#comment-14901431
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40027080
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
 ---
@@ -102,6 +119,29 @@ public void setOption(OptionValue value) {
   }
 
   @Override
+  public void deleteOption(final String name, final OptionType type) {
+try {
+  SystemOptionManager.getValidator(name); // ensure the option exists
+} catch (final IllegalArgumentException e) {
+  throw UserException.validationError(e)
+.build(logger);
+}
+
+// fallback if unable to delete locally
+if (!deleteLocalOption(name, type)) {
--- End diff --

Implementations of OptionManager are required to be case insensitive to 
option names (mentioned in the OptionManager class docs). Calling 
```name.toLowerCase()``` in this one place is insufficient to ensure 
*implementation will always be case insensitive.*


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1162) 25 way join ended up in 0 results which is not expected

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901478#comment-14901478
 ] 

Rahul Challapalli commented on DRILL-1162:
--

git.commit.id.abbrev=3c89b30

On the current master, surprisingly the same query resulted in an OOM error on 
a 11 node drill cluster

{code}
0: jdbc:drill:zk=x.x.x.x:5181> select count(*) from `lineitem1.parquet` a 
. . . . . . . . . . . . . . . . . > inner join `part.parquet` j on a.l_partkey 
= j.p_partkey 
. . . . . . . . . . . . . . . . . > inner join `orders.parquet` k on 
a.l_orderkey = k.o_orderkey 
. . . . . . . . . . . . . . . . . > inner join `supplier.parquet` l on 
a.l_suppkey = l.s_suppkey 
. . . . . . . . . . . . . . . . . > inner join `partsupp.parquet` m on 
j.p_partkey = m.ps_partkey and l.s_suppkey = m.ps_suppkey 
. . . . . . . . . . . . . . . . . > inner join `customer.parquet` n on 
k.o_custkey = n.c_custkey 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` b on 
a.l_orderkey = b.l_orderkey 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` c on 
a.l_partkey = c.l_partkey 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` d on 
a.l_suppkey = d.l_suppkey 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` e on 
a.l_extendedprice = e.l_extendedprice 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` f on 
a.l_comment = f.l_comment 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` g on 
a.l_shipdate = g.l_shipdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` h on 
a.l_commitdate = h.l_commitdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` i on 
a.l_receiptdate = i.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` o on 
a.l_receiptdate = o.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` p on 
a.l_receiptdate = p.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` q on 
a.l_receiptdate = q.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` r on 
a.l_receiptdate = r.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` s on 
a.l_receiptdate = s.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` t on 
a.l_receiptdate = t.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` u on 
a.l_receiptdate = u.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` v on 
a.l_receiptdate = v.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` w on 
a.l_receiptdate = w.l_receiptdate 
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` x on 
a.l_receiptdate = x.l_receiptdate;
java.lang.RuntimeException: java.sql.SQLException: RESOURCE ERROR: One or more 
nodes ran out of memory while executing the query.

Fragment 0:0

[Error Id: 49e1b35d-a8a5-4243-a77d-058c2af81196 on pssc-69.qa.lab:31010]
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at 
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1583)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:738)
at sqlline.SqlLine.begin(SqlLine.java:612)
at sqlline.SqlLine.start(SqlLine.java:366)
at sqlline.SqlLine.main(SqlLine.java:259)
{code}

OOM did not happen in the past on a 4 node cluster. This should still be 
investigated. The data used is TPCH SF 0.01 data.

> 25 way join ended up in 0 results which is not expected
> ---
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Rahul Challapalli
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned 
> select count(*) from `lineitem1.parquet` a 
> inner join `part.parquet` j on a.l_partkey = j.p_partkey 
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey 
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey 
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey 
> = m.ps_suppkey 
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey 
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey 
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey 
> 

[jira] [Commented] (DRILL-3735) Directory pruning is not happening when number of files is larger than 64k

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901392#comment-14901392
 ] 

ASF GitHub Bot commented on DRILL-3735:
---

Github user jinfengni commented on the pull request:

https://github.com/apache/drill/pull/156#issuecomment-142106704
  
+1.   LGTM.




> Directory pruning is not happening when number of files is larger than 64k
> --
>
> Key: DRILL-3735
> URL: https://issues.apache.org/jira/browse/DRILL-3735
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
>
> When the number of files is larger than 64k limit, directory pruning is not 
> happening. 
> We need to increase this limit further to handle most use cases.
> My proposal is to separate the code for directory pruning and partition 
> pruning. 
> Say in a parent directory there are 100 directories and 1 million files.
> If we only query the file from one directory, we should firstly read the 100 
> directories and narrow down to which directory; and then read the file paths 
> in that directory in memory and do the rest stuff.
> Current behavior is , Drill will read all the file paths of that 1 million 
> files in memory firstly, and then do directory pruning or partition pruning. 
> This is not performance efficient nor memory efficient. And also it can not 
> scale.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901413#comment-14901413
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r40026355
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java
 ---
@@ -161,6 +171,7 @@ public SqlNode visitChild(
 rules.put(SqlJoin.class, R(D, D, D, D, D, E));
 rules.put(SqlOrderBy.class, R(D, E, D, D));
 rules.put(SqlDropTable.class, R(D));
+rules.put(SqlSetOption.class, R(D, D, D));
--- End diff --

I see the confusion. We now allow option names to be compound identifiers. 
Let me add this to the JIRA and the commit message.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901476#comment-14901476
 ] 

ASF GitHub Bot commented on DRILL-3811:
---

Github user adeneche commented on the pull request:

https://github.com/apache/drill/pull/163#issuecomment-142120660
  
TopLevelAllocator's constructor is not public but because the unit test is 
in the same package it was able to access the constructor anyway. Fixed the 
test to use RootAllocatorFactory instead


> AtomicRemainder incorrectly accounts for transferred allocations
> 
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int) 
> is called to account for the extra memory of the buffer, but when the 
> allocator exceeds it's maximum allocated memory it accounts for it 
> incorrectly. In the following code, {{availableShared.andAndGet(size)}} 
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
>   return true;
> } else {
>   availableShared.addAndGet(size);
>   if (parent != null) {
> parent.forceGet(size);
>   }
>   return false;
> }
>   }
> {code}
> I was able to reproduce the issue in a simple unit test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3029) Wrong result with correlated not exists subquery

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901485#comment-14901485
 ] 

Jinfeng Ni commented on DRILL-3029:
---

Talked to [~vicky]. We agree to set the fix version to 1.3.

> Wrong result with correlated not exists subquery
> 
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where   not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1  |
> | 2  |
> | 3  |
> | 4  |
> | 5  |
> | 6  |
> | 7  |
> | 9  |
> | 10 |
> | null   |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where   not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)