[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901862#comment-14901862
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user amansinha100 commented on the pull request:
https://github.com/apache/drill/pull/163#issuecomment-142170314
+1 .
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901861#comment-14901861
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/163#discussion_r40050662
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java
---
@@ -76,16 +74,17 @@ public void setLimit(long limit) {
}
/**
- * Automatically allocate memory. This is used when an actual allocation
happened to be larger than requested. This
- * memory has already been used up so it must be accurately accounted
for in future allocations.
+ * Automatically allocate memory. This is used when an actual allocation
happened to be larger than requested, or when
+ * a buffer has it's ownership passed to another allocator.
+ * This memory has already been used up so it must be accurately
accounted for in future allocations.
*
- * @param size
+ * @param size extra allocated memory that needs to be accounted for
*/
public boolean forceGet(long size) {
if (get(size, this.applyFragmentLimit)) {
return true;
} else {
- availableShared.addAndGet(size);
+ availableShared.addAndGet(-size);
--- End diff --
It is likely handled..I will defer to Chris on that.
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
[
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz closed DRILL-3621.
-
> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
>
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10 |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY1';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec
> [tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10, filter=null],
> columns=[`row_key`]]])
> |
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
[
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901814#comment-14901814
]
Khurram Faraaz commented on DRILL-3621:
---
Verified that Filter gets pushed down into the Scan operator on master commit
id b525692e
case 1)
{code}
0: jdbc:drill:schema=dfs.tmp> select convert_from(row_key,'UTF8') from
testrowkey WHERE ROW_KEY='DUMMY7' OR ROW_KEY BETWEEN 'DUMMY1' AND 'DUMMY10';
+--+
| EXPR$0 |
+--+
| DUMMY1 |
| DUMMY10 |
| DUMMY7 |
+--+
3 rows selected (0.865 seconds)
explain plan for select convert_from(row_key,'UTF8') from testrowkey WHERE
ROW_KEY='DUMMY7' OR ROW_KEY BETWEEN 'DUMMY1' AND 'DUMMY10';
00-01 Project(EXPR$0=[CONVERT_FROMUTF8($0)])
00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec
[tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY7\x00, filter=FilterList
OR (2/2): [RowFilter (EQUAL, DUMMY7), FilterList AND (2/2): [RowFilter
(GREATER_OR_EQUAL, DUMMY1), RowFilter (LESS_OR_EQUAL, DUMMY10)]]],
columns=[`row_key`]]])
{code}
case 2)
{code}
0: jdbc:drill:schema=dfs.tmp> select convert_from(row_key,'UTF8') from
testrowkey WHERE ROW_KEY in ('DUMMY1' , 'DUMMY10');
+--+
| EXPR$0 |
+--+
| DUMMY1 |
| DUMMY10 |
+--+
2 rows selected (0.867 seconds)
00-01 Project(EXPR$0=[CONVERT_FROMUTF8($0)])
00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec
[tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10\x00, filter=FilterList
OR (2/2): [RowFilter (EQUAL, DUMMY1), RowFilter (EQUAL, DUMMY10)]],
columns=[`row_key`]]])
{code}
case 3)
{code}
: jdbc:drill:schema=dfs.tmp> select convert_from(row_key,'UTF8') from
testrowkey WHERE ROW_KEY ='DUMMY1' OR ROW_KEY = 'DUMMY10';
+--+
| EXPR$0 |
+--+
| DUMMY1 |
| DUMMY10 |
+--+
2 rows selected (0.854 seconds)
00-01 Project(EXPR$0=[CONVERT_FROMUTF8($0)])
00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec
[tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10\x00, filter=FilterList
OR (2/2): [RowFilter (EQUAL, DUMMY1), RowFilter (EQUAL, DUMMY10)]],
columns=[`row_key`]]])
{code}
> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
>
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10 |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY1';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseSca
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901757#comment-14901757
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/163#discussion_r40047181
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java
---
@@ -76,16 +74,17 @@ public void setLimit(long limit) {
}
/**
- * Automatically allocate memory. This is used when an actual allocation
happened to be larger than requested. This
- * memory has already been used up so it must be accurately accounted
for in future allocations.
+ * Automatically allocate memory. This is used when an actual allocation
happened to be larger than requested, or when
+ * a buffer has it's ownership passed to another allocator.
+ * This memory has already been used up so it must be accurately
accounted for in future allocations.
*
- * @param size
+ * @param size extra allocated memory that needs to be accounted for
*/
public boolean forceGet(long size) {
if (get(size, this.applyFragmentLimit)) {
return true;
} else {
- availableShared.addAndGet(size);
+ availableShared.addAndGet(-size);
--- End diff --
It can become negative if the allocator takes ownership of a buffer and
exceeds it's maximum allocated memory. Negative values are handled properly, at
least that's my understanding
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901697#comment-14901697
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/163#discussion_r40045182
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/memory/AtomicRemainder.java
---
@@ -76,16 +74,17 @@ public void setLimit(long limit) {
}
/**
- * Automatically allocate memory. This is used when an actual allocation
happened to be larger than requested. This
- * memory has already been used up so it must be accurately accounted
for in future allocations.
+ * Automatically allocate memory. This is used when an actual allocation
happened to be larger than requested, or when
+ * a buffer has it's ownership passed to another allocator.
+ * This memory has already been used up so it must be accurately
accounted for in future allocations.
*
- * @param size
+ * @param size extra allocated memory that needs to be accounted for
*/
public boolean forceGet(long size) {
if (get(size, this.applyFragmentLimit)) {
return true;
} else {
- availableShared.addAndGet(size);
+ availableShared.addAndGet(-size);
--- End diff --
Since this will be subtracting size bytes, should there be a check for
availableShared >=0 ? I am not quite sure what's supposed to happen if this
value drops below 0 at this point...
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3815) unknown suffixes .not_json and .json_not treated differently (multi-file case)
[
https://issues.apache.org/jira/browse/DRILL-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901691#comment-14901691
]
Ted Dunning commented on DRILL-3815:
Daniel,
Did you trace this a bit to see where the extensions are being matched?
Could it be a naively constructed regex? Kinda smells like that.
> unknown suffixes .not_json and .json_not treated differently (multi-file case)
> --
>
> Key: DRILL-3815
> URL: https://issues.apache.org/jira/browse/DRILL-3815
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Jacques Nadeau
>
> In scanning a directory subtree used as a table, unknown filename extensions
> seem to be treated differently depending on whether they're similar to known
> file extensions. The behavior suggests that Drill checks whether a file name
> _contains_ an extension's string rather than _ending_ with it.
> For example, given these subtrees with almost identical leaf file names:
> {noformat}
> $ find /tmp/testext_xx_json/
> /tmp/testext_xx_json/
> /tmp/testext_xx_json/voter2.not_json
> /tmp/testext_xx_json/voter1.json
> $ find /tmp/testext_json_xx/
> /tmp/testext_json_xx/
> /tmp/testext_json_xx/voter1.json
> /tmp/testext_json_xx/voter2.json_not
> $
> {noformat}
> the results of trying to use them as tables differs:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs.tmp`.`testext_xx_json`;
> Sep 21, 2015 11:41:50 AM
> org.apache.calcite.sql.validate.SqlValidatorException
> ...
> Error: VALIDATION ERROR: From line 1, column 17 to line 1, column 25: Table
> 'dfs.tmp.testext_xx_json' not found
> [Error Id: 6fe41deb-0e39-43f6-beca-de27b39d276b on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs.tmp`.`testext_json_xx`;
> +---+
> | onecf |
> +---+
> | {"name":"someName1"} |
> | {"name":"someName2"} |
> +---+
> 2 rows selected (0.149 seconds)
> {noformat}
> (Other probing seems to indicate that there is also some sensitivity to
> whether the extension contains an underscore character.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Sinha updated DRILL-3781:
--
Assignee: Jinfeng Ni (was: Aman Sinha)
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch,
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901646#comment-14901646
]
Aman Sinha commented on DRILL-3781:
---
+1. I also took a look at Calcite-886 and looks good to me.
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments:
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch,
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901625#comment-14901625
]
Jinfeng Ni commented on DRILL-3781:
---
The forked calcite version will be updated before merge to master, after the
patch completes the review /pre-commit regression run.
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments:
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch,
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinfeng Ni updated DRILL-3781:
--
Assignee: Aman Sinha (was: Jinfeng Ni)
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments:
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch,
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901621#comment-14901621
]
Jinfeng Ni commented on DRILL-3781:
---
The fix for this problem is in Calcite-886. Here, the patch in Drill simply add
unit testcase.
[~amansinha100], could you please review the patch? Thanks!
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch,
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinfeng Ni updated DRILL-3781:
--
Attachment: 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0002-DRILL-3781-Group-by-system-function-in-schema-based-.patch,
> csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901579#comment-14901579
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40037509
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
abstract boolean setLocalOption(OptionValue value);
+ /**
+ * Deletes the option with given name for this manager without falling
back.
+ *
+ * @param type option type
+ * @return true iff the option was successfully deleted
--- End diff --
I will clarify the docs in both places.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3817) Refresh metadata does not work when used with sub schema
Mehant Baid created DRILL-3817: -- Summary: Refresh metadata does not work when used with sub schema Key: DRILL-3817 URL: https://issues.apache.org/jira/browse/DRILL-3817 Project: Apache Drill Issue Type: Bug Reporter: Mehant Baid Assignee: Mehant Baid Fix For: 1.2.0 refresh table metadata dfs.tmp.`lineitem` does not work, hit the following exception org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: org.apache.calcite.sql.SqlBasicCall cannot be cast to org.apache.calcite.sql.SqlIdentifier If the sub schema is removed it works. refresh table metadata dfs.`/tmp/lineitem` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3776) count(*) from empty text file does not return 0
[
https://issues.apache.org/jira/browse/DRILL-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu updated DRILL-3776:
-
Target Version/s: (was: 1.2.0)
> count(*) from empty text file does not return 0
> ---
>
> Key: DRILL-3776
> URL: https://issues.apache.org/jira/browse/DRILL-3776
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.4.0
>
>
> {code}
> select count(*) from `empty.csv`;
> {code}
> {code}
> select count(*) from `empty.json`;
> {code}
> Both return no record, no schema. But in postgres, it returns 0 for both
> cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3776) count(*) from empty text file does not return 0
[
https://issues.apache.org/jira/browse/DRILL-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu updated DRILL-3776:
-
Fix Version/s: 1.4.0
> count(*) from empty text file does not return 0
> ---
>
> Key: DRILL-3776
> URL: https://issues.apache.org/jira/browse/DRILL-3776
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.4.0
>
>
> {code}
> select count(*) from `empty.csv`;
> {code}
> {code}
> select count(*) from `empty.json`;
> {code}
> Both return no record, no schema. But in postgres, it returns 0 for both
> cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3731) Partition Pruning not taking place when we have case statement within the filter
[
https://issues.apache.org/jira/browse/DRILL-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901564#comment-14901564
]
Sean Hsuan-Yi Chu commented on DRILL-3731:
--
Essentially, ReduceExpressionsRule.FILTER_INSTANCE should have simplified the
expression before PruneScan rule takes place.
This issue should be resolved from Calcite:
https://issues.apache.org/jira/browse/CALCITE-895
> Partition Pruning not taking place when we have case statement within the
> filter
>
>
> Key: DRILL-3731
> URL: https://issues.apache.org/jira/browse/DRILL-3731
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Rahul Challapalli
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: lineitempart.tgz
>
>
> Commit # 8f33c3d55228a98ea2a309490ecbfb7a180f012c
> Partition Pruning is not taking place in the below query
> {code}
> explain plan for select count(*) from
> `/drill/testdata/partition_pruning/dfs/lineitempart` where case when
> dir0=1991 then null else 1 end is null or dir0=1992;
> 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03 Project($f0=[0])
> 00-04SelectionVectorRemover
> 00-05 Filter(condition=[OR(IS NULL(CASE(=($0, 1991), null, 1)),
> =($0, 1992))])
> 00-06Scan(groupscan=[EasyGroupScan
> [selectionRoot=maprfs:/drill/testdata/partition_pruning/dfs/lineitempart,
> numFiles=7, columns=[`dir0`],
> files=[maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1991/lineitemaa.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1992/lineitemab.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1996/lineitemaf.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1994/lineitemad.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1993/lineitemac.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1997/lineitemag.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1995/lineitemae.tbl]]])
> {code}
> The planner should have pruned all partitions except 1991 & 1992
> I attached the dataset. Let me know if you need anything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3735) Directory pruning is not happening when number of files is larger than 64k
[ https://issues.apache.org/jira/browse/DRILL-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901551#comment-14901551 ] ASF GitHub Bot commented on DRILL-3735: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/156 > Directory pruning is not happening when number of files is larger than 64k > -- > > Key: DRILL-3735 > URL: https://issues.apache.org/jira/browse/DRILL-3735 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.1.0 >Reporter: Hao Zhu >Assignee: Jinfeng Ni > Fix For: 1.2.0 > > > When the number of files is larger than 64k limit, directory pruning is not > happening. > We need to increase this limit further to handle most use cases. > My proposal is to separate the code for directory pruning and partition > pruning. > Say in a parent directory there are 100 directories and 1 million files. > If we only query the file from one directory, we should firstly read the 100 > directories and narrow down to which directory; and then read the file paths > in that directory in memory and do the rest stuff. > Current behavior is , Drill will read all the file paths of that 1 million > files in memory firstly, and then do directory pruning or partition pruning. > This is not performance efficient nor memory efficient. And also it can not > scale. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3192) TestDrillbitResilience#cancelWhenQueryIdArrives hangs
[ https://issues.apache.org/jira/browse/DRILL-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-3192: --- Fix Version/s: (was: 1.2.0) 1.3.0 > TestDrillbitResilience#cancelWhenQueryIdArrives hangs > - > > Key: DRILL-3192 > URL: https://issues.apache.org/jira/browse/DRILL-3192 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam >Priority: Critical > Fix For: 1.3.0 > > > TestDrillbitResilience#cancelWhenQueryIdArrives (previously named > cancelBeforeAnyResultsArrive) hangs when the test is run multiple times. > -(Will add more information)- > *Configuration: BIT_SERVER_RPC_THREADS = 1* > The remote RPC thread with a cancel signal is waiting for the fragment to > start accepting external events. The fragment is waiting for a > ControlConnection to the Foreman node (through ReconnectingConnection). The > Foreman node is waiting for the remote node to accept the connection, which > happens through an RPC thread. Distributed deadlock. > DRILL-3242 should solve this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2808) Substr function throws weird message when wrong arguments are passed in
[
https://issues.apache.org/jira/browse/DRILL-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901507#comment-14901507
]
Victoria Markman commented on DRILL-2808:
-
Still in 1.2.0, tried with:
#Sun Sep 20 15:30:07 UTC 2015
git.commit.id.abbrev=092903d
> Substr function throws weird message when wrong arguments are passed in
> ---
>
> Key: DRILL-2808
> URL: https://issues.apache.org/jira/browse/DRILL-2808
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
>
> #Wed Apr 15 17:37:25 EDT 2015
> git.commit.id.abbrev=cb47df0
> {code}
> 0: jdbc:drill:schema=dfs> select substr(0, 1, 'abc') from sys.options limit 1;
> Query failed: SYSTEM ERROR: Unexpected exception during fragment
> initialization: Internal error: Error while applying rule
> ReduceExpressionsRule[Project], args
> [rel#53990:ProjectRel.NONE.ANY([]).[](child=rel#53989:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> [1f7e0c35-fc65-4a2f-b668-12e9baf5c9b8 on atsqa4-133.qa.lab:31010]
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> 2015-04-16 22:04:32,029 [2acfce0f-8af4-9786-4799-9180b9bac219:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING -->
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> ReduceExpressionsRule[Project], args
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:211)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ReduceExpressionsRule[Project], args
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> at org.eigenbase.util.Util.newInternal(Util.java:750)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:246)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:278)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:196)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:137)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:155)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:770)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:202)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> ... 3 common frames omitted
> Caused by: java.lang.RuntimeException: Error in evaluating function of
> castBIGINT
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:323)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:147)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47)
> ~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:251)
> ~[drill-java-ex
[jira] [Updated] (DRILL-1162) 25 way join ended up in 0 results which is not expected
[ https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli updated DRILL-1162: - Assignee: Chris Westin (was: Rahul Challapalli) > 25 way join ended up in 0 results which is not expected > --- > > Key: DRILL-1162 > URL: https://issues.apache.org/jira/browse/DRILL-1162 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow, Query Planning & Optimization >Reporter: Rahul Challapalli >Assignee: Chris Westin >Priority: Critical > Fix For: 1.2.0 > > Attachments: error.log, oom_error.log > > > git.commit.id.abbrev=e5c2da0 > The below query results in 0 results being returned > select count(*) from `lineitem1.parquet` a > inner join `part.parquet` j on a.l_partkey = j.p_partkey > inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey > inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey > inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey > = m.ps_suppkey > inner join `customer.parquet` n on k.o_custkey = n.c_custkey > inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey > inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey > inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey > inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice > inner join `lineitem2.parquet` f on a.l_comment = f.l_comment > inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate > inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate > inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate > inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate > inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate > inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate > inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate > inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate > inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate > inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate > inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate > inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate > inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate; > However when we remove the last 'inner join' and run the query it returns > '716372534'. Since the last inner join is similar to the one's before it, it > should match some records and return the data appropriately. > The logs indicated that it actually returned 0 results. Attached the log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1162) 25 way join ended up in 0 results which is not expected
[ https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli updated DRILL-1162: - Attachment: oom_error.log > 25 way join ended up in 0 results which is not expected > --- > > Key: DRILL-1162 > URL: https://issues.apache.org/jira/browse/DRILL-1162 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow, Query Planning & Optimization >Reporter: Rahul Challapalli >Assignee: Rahul Challapalli >Priority: Critical > Fix For: 1.2.0 > > Attachments: error.log, oom_error.log > > > git.commit.id.abbrev=e5c2da0 > The below query results in 0 results being returned > select count(*) from `lineitem1.parquet` a > inner join `part.parquet` j on a.l_partkey = j.p_partkey > inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey > inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey > inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey > = m.ps_suppkey > inner join `customer.parquet` n on k.o_custkey = n.c_custkey > inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey > inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey > inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey > inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice > inner join `lineitem2.parquet` f on a.l_comment = f.l_comment > inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate > inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate > inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate > inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate > inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate > inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate > inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate > inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate > inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate > inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate > inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate > inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate > inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate; > However when we remove the last 'inner join' and run the query it returns > '716372534'. Since the last inner join is similar to the one's before it, it > should match some records and return the data appropriately. > The logs indicated that it actually returned 0 results. Attached the log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3029) Wrong result with correlated not exists subquery
[
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinfeng Ni updated DRILL-3029:
--
Fix Version/s: (was: 1.2.0)
1.3.0
> Wrong result with correlated not exists subquery
>
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 9 |
> | 10 |
> | null |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3029) Wrong result with correlated not exists subquery
[
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901485#comment-14901485
]
Jinfeng Ni commented on DRILL-3029:
---
Talked to [~vicky]. We agree to set the fix version to 1.3.
> Wrong result with correlated not exists subquery
>
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 9 |
> | 10 |
> | null |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1162) 25 way join ended up in 0 results which is not expected
[
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901478#comment-14901478
]
Rahul Challapalli commented on DRILL-1162:
--
git.commit.id.abbrev=3c89b30
On the current master, surprisingly the same query resulted in an OOM error on
a 11 node drill cluster
{code}
0: jdbc:drill:zk=x.x.x.x:5181> select count(*) from `lineitem1.parquet` a
. . . . . . . . . . . . . . . . . > inner join `part.parquet` j on a.l_partkey
= j.p_partkey
. . . . . . . . . . . . . . . . . > inner join `orders.parquet` k on
a.l_orderkey = k.o_orderkey
. . . . . . . . . . . . . . . . . > inner join `supplier.parquet` l on
a.l_suppkey = l.s_suppkey
. . . . . . . . . . . . . . . . . > inner join `partsupp.parquet` m on
j.p_partkey = m.ps_partkey and l.s_suppkey = m.ps_suppkey
. . . . . . . . . . . . . . . . . > inner join `customer.parquet` n on
k.o_custkey = n.c_custkey
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` b on
a.l_orderkey = b.l_orderkey
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` c on
a.l_partkey = c.l_partkey
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` d on
a.l_suppkey = d.l_suppkey
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` e on
a.l_extendedprice = e.l_extendedprice
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` f on
a.l_comment = f.l_comment
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` g on
a.l_shipdate = g.l_shipdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` h on
a.l_commitdate = h.l_commitdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` i on
a.l_receiptdate = i.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` o on
a.l_receiptdate = o.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` p on
a.l_receiptdate = p.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` q on
a.l_receiptdate = q.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` r on
a.l_receiptdate = r.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` s on
a.l_receiptdate = s.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` t on
a.l_receiptdate = t.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` u on
a.l_receiptdate = u.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` v on
a.l_receiptdate = v.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` w on
a.l_receiptdate = w.l_receiptdate
. . . . . . . . . . . . . . . . . > inner join `lineitem2.parquet` x on
a.l_receiptdate = x.l_receiptdate;
java.lang.RuntimeException: java.sql.SQLException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Fragment 0:0
[Error Id: 49e1b35d-a8a5-4243-a77d-058c2af81196 on pssc-69.qa.lab:31010]
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1583)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:738)
at sqlline.SqlLine.begin(SqlLine.java:612)
at sqlline.SqlLine.start(SqlLine.java:366)
at sqlline.SqlLine.main(SqlLine.java:259)
{code}
OOM did not happen in the past on a 4 node cluster. This should still be
investigated. The data used is TPCH SF 0.01 data.
> 25 way join ended up in 0 results which is not expected
> ---
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Rahul Challapalli
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned
> select count(*) from `lineitem1.parquet` a
> inner join `part.parquet` j on a.l_partkey = j.p_partkey
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey
> = m.ps_suppkey
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey
> inner join `lineitem2.parquet` c on a.l_partkey =
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901476#comment-14901476
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user adeneche commented on the pull request:
https://github.com/apache/drill/pull/163#issuecomment-142120660
TopLevelAllocator's constructor is not public but because the unit test is
in the same package it was able to access the constructor anyway. Fixed the
test to use RootAllocatorFactory instead
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901431#comment-14901431
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40027080
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -102,6 +119,29 @@ public void setOption(OptionValue value) {
}
@Override
+ public void deleteOption(final String name, final OptionType type) {
+try {
+ SystemOptionManager.getValidator(name); // ensure the option exists
+} catch (final IllegalArgumentException e) {
+ throw UserException.validationError(e)
+.build(logger);
+}
+
+// fallback if unable to delete locally
+if (!deleteLocalOption(name, type)) {
--- End diff --
Implementations of OptionManager are required to be case insensitive to
option names (mentioned in the OptionManager class docs). Calling
```name.toLowerCase()``` in this one place is insufficient to ensure
*implementation will always be case insensitive.*
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[ https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901413#comment-14901413 ] ASF GitHub Bot commented on DRILL-1065: --- Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/159#discussion_r40026355 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java --- @@ -161,6 +171,7 @@ public SqlNode visitChild( rules.put(SqlJoin.class, R(D, D, D, D, D, E)); rules.put(SqlOrderBy.class, R(D, E, D, D)); rules.put(SqlDropTable.class, R(D)); +rules.put(SqlSetOption.class, R(D, D, D)); --- End diff -- I see the confusion. We now allow option names to be compound identifiers. Let me add this to the JIRA and the commit message. > Provide a reset command to reset an option to its default value > --- > > Key: DRILL-1065 > URL: https://issues.apache.org/jira/browse/DRILL-1065 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: Aman Sinha >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.2.0 > > > Within a session, currently we set configuration options and it would be very > useful to have a 'reset' command to reset the value of an option to its > default system value: > ALTER SESSION RESET > If we don't want to add a new keyword for RESET, we could potentially > overload the SET command and allow the user to set to the 'default' value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3735) Directory pruning is not happening when number of files is larger than 64k
[ https://issues.apache.org/jira/browse/DRILL-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901392#comment-14901392 ] ASF GitHub Bot commented on DRILL-3735: --- Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/156#issuecomment-142106704 +1. LGTM. > Directory pruning is not happening when number of files is larger than 64k > -- > > Key: DRILL-3735 > URL: https://issues.apache.org/jira/browse/DRILL-3735 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.1.0 >Reporter: Hao Zhu >Assignee: Jinfeng Ni > Fix For: 1.2.0 > > > When the number of files is larger than 64k limit, directory pruning is not > happening. > We need to increase this limit further to handle most use cases. > My proposal is to separate the code for directory pruning and partition > pruning. > Say in a parent directory there are 100 directories and 1 million files. > If we only query the file from one directory, we should firstly read the 100 > directories and narrow down to which directory; and then read the file paths > in that directory in memory and do the rest stuff. > Current behavior is , Drill will read all the file paths of that 1 million > files in memory firstly, and then do directory pruning or partition pruning. > This is not performance efficient nor memory efficient. And also it can not > scale. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3180) Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and Netezza from Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901390#comment-14901390
]
Julian Hyde commented on DRILL-3180:
I don't often see non-join conditions into the ON clause, but I take
[~magnusp]'s point. If that were an outer join, we would be able to push the
date condition down to the salary table, whereas if it were in the WHERE clause
we could not.
Anyway, the JDBC adapter's goal is not to format the SQL to follow any "best
practice" or to look nice for humans to read. It is to communicate with the
target DB's query optimizer, ideally in a form that the optimizer is unlikely
to screw up, and most importantly to preserve semantics.
Sometimes there is a danger that Calcite will "over optimize" the query, e.g.
{code}select *
FROM mp.employees.`employees` e
INNER JOIN (
SELECT * FROM mp.employees.`salaries` s
WHERE s.`to_date` > CURRENT_DATE) AS s
ON e.`EMP_NO` = s.`EMP_NO`{code}
is valid and efficient but the new query block might confuse optimizers like
MySQL's.
> Apache Drill JDBC storage plugin to query rdbms systems such as MySQL and
> Netezza from Apache Drill
> ---
>
> Key: DRILL-3180
> URL: https://issues.apache.org/jira/browse/DRILL-3180
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 1.0.0
>Reporter: Magnus Pierre
>Assignee: Jacques Nadeau
> Labels: Drill, JDBC, plugin
> Fix For: 1.2.0
>
> Attachments: patch.diff, pom.xml, storage-mpjdbc.zip
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> I have developed the base code for a JDBC storage-plugin for Apache Drill.
> The code is primitive but consitutes a good starting point for further
> coding. Today it provides primitive support for SELECT against RDBMS with
> JDBC.
> The goal is to provide complete SELECT support against RDBMS with push down
> capabilities.
> Currently the code is using standard JDBC classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804598#comment-14804598
]
Sudheesh Katkam edited comment on DRILL-1065 at 9/21/15 8:57 PM:
-
Also, adding:
+ support for
{code}
SET `option name`
RESET `option name`
RESET ALL
{code}
as discussed
[here|http://mail-archives.apache.org/mod_mbox/calcite-dev/201507.mbox/%3ccampyv7c9ynatuk-2ufh_suddmc6d5-e4jd1_taqmnuoifz1...@mail.gmail.com%3E]
with an assumption that the scope is SESSION.
+ allow for compound identifiers for option names
was (Author: sudheeshkatkam):
Also, adding:
SET `option name`
RESET `option name`
RESET ALL
as discussed
[here|http://mail-archives.apache.org/mod_mbox/calcite-dev/201507.mbox/%3ccampyv7c9ynatuk-2ufh_suddmc6d5-e4jd1_taqmnuoifz1...@mail.gmail.com%3E]
with an assumption that the scope is SESSION.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901384#comment-14901384
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/163#discussion_r40024717
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java
---
@@ -63,6 +63,29 @@
private final static String planFile="/physical_allocator_test.json";
@Test
+ public void testTransfer() throws Exception {
+final Properties props = new Properties() {
+ {
+put(ExecConstants.TOP_LEVEL_MAX_ALLOC, "100");
+put(ExecConstants.ERROR_ON_MEMORY_LEAK, "true");
+ }
+};
+final DrillConfig config = DrillConfig.create(props);
+BufferAllocator a = new TopLevelAllocator(config);
--- End diff --
sure
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3807) Query with equality join and a FALSE condition fails to plan (possible regression)
[
https://issues.apache.org/jira/browse/DRILL-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Sinha updated DRILL-3807:
--
Assignee: Sean Hsuan-Yi Chu (was: Jinfeng Ni)
> Query with equality join and a FALSE condition fails to plan (possible
> regression)
> --
>
> Key: DRILL-3807
> URL: https://issues.apache.org/jira/browse/DRILL-3807
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Aman Sinha
>Assignee: Sean Hsuan-Yi Chu
>
> 1.2.0-SNAPSHOT behavior:
> {code}
> 0: jdbc:drill:zk=local> explain plan for select l.l_quantity from
> cp.`tpch/lineitem.parquet` l, cp.`tpch/part.parquet` p where l.l_partkey =
> p.p_partkey and (1 = 0);
> Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due
> to either a cartesian join or an inequality join
> [Error Id: f7466d86-b709-465e-bb49-d3c51ecf941b on 172.16.0.160:31010]
> (state=,code=0)
> {code}
> The simplification of ' l.l_partkey = p.p_partkey and (1 = 0)' to a False
> condition is valid and accordingly Drill fails to plan due to the cartesian
> join introduced by the False condition. However, in 1.1.0 apparently the
> 1=0 was converted to a LIMIT 0 which was pushed below the Join and the query
> successfully planned and executed:
> 1.1.0 behavior:
> {code}
> 0: jdbc:drill:zk=local> explain plan for select l.l_quantity from
> cp.`tpch/lineitem.parquet` l, cp.`tpch/part.parquet` p where l.l_partkey =
> p.p_partkey and (1 = 0);
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(l_quantity=[$1])
> 00-02HashJoin(condition=[=($0, $2)], joinType=[inner])
> 00-04 SelectionVectorRemover
> 00-05Limit(offset=[0], fetch=[0])
> 00-06 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/lineitem.parquet]],
> selectionRoot=classpath:/tpch/lineitem.parquet, numFiles=1,
> columns=[`l_partkey`, `l_quantity`]]])
> 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=classpath:/tpch/part.parquet]],
> selectionRoot=classpath:/tpch/part.parquet, numFiles=1,
> columns=[`p_partkey`]]])
> {code}
> [~cchang] and I looked at the commit history and it appears that the
> regression started somewhere between Aug 24 and Aug 28, which is the time
> when we rebased on Calcite 1.4.0. So we need to narrow down further the
> change that may have caused this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3757) Link to IntelliJ IDEA settings jar on the contributors guidelines page is broken.
[ https://issues.apache.org/jira/browse/DRILL-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun Chang closed DRILL-3757. - Assignee: Chun Chang (was: Kristine Hahn) verified link. > Link to IntelliJ IDEA settings jar on the contributors guidelines page is > broken. > - > > Key: DRILL-3757 > URL: https://issues.apache.org/jira/browse/DRILL-3757 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.1.0 >Reporter: Edmon Begoli >Assignee: Chun Chang >Priority: Minor > Labels: documentation > Fix For: 1.2.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Link > https://cwiki.apache.org/confluence/download/attachments/30757399/idea-settings.jar?version=1&modificationDate=1363022308000&api=v > on the page Apache Drill Contribution Guidelines is pointing to missing > settings jar: > https://drill.apache.org/docs/apache-drill-contribution-guidelines/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3816) weird file-extension recognition behavior in directory subtree scanning
[
https://issues.apache.org/jira/browse/DRILL-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Barclay (Drill) updated DRILL-3816:
--
Description:
In scanning of directory subtrees for files, recognition of known vs. unknown
file extensions seems really screwy (not following any apparent pattern).
For example:
- a suffix of {{.jsxon_not}}, as expected, is not recognized as a JSON file
- a suffix of {{.jsoxn_not}} unexpectedly _is_ taken as JSON
- a suffix of .{{jsonx_not}}, as expected, is not recognized as a JSON file
(Creating a directory containing only a non-empty JSON file ending with
{{.json}} and another non-empty JSON file ending with one of the above suffixes
sometimes reads both JSON files and sometimes reports a (presumably) expected
error because of the mixed file extensions).)
The result sometimes seems to also depend on the rest of the filename,
presumably related to the order of listing of files. (It's not clear if it
depends only on the order after filename sorting, or also depends on the order
file names are listed the by OS.)
Here are more data points (using a JSON file named {{voter1.json}}):
- with {{voter2.xjson_not}} - read, as JSON
- with {{voter2.jxson_not}} - read, as JSON
- with {{voter2.jsxon_not}} - causes expected error
- with {{voter2.jsoxn_not}} - read, as JSON
- with {{voter2.jsonx_not}} - causes expected error
- with {{voter2.json_xnot}} - read, as JSON
- with {{voter2.json_nxot}} - read, as JSON
- with {{voter2.json_noxt}} - read, as JSON
- with {{voter2.json_notx}} - read, as JSON
- with {{voter2.jsonxnot}} - read, as JSON
- with {{voter2.jsonxot}} - read, as JSON
- with {{voter2.jsoxot}}- causes expected error
- with {{voter2.jxsxoxn}} - read, as JSON
- with {{voter2.xjxsxoxn}} - read, as JSON
- with {{voter2.xjxsxoxnx}} - causes expected error
- with {{voter2.xjxxoxn}} - read, as JSON
- with {{voter2.xjxxxn}}- read, as JSON
- with {{voter2.n}} - read, as JSON
- with {{voter2.}} - read, as JSON
- with {{voter2.xxx}} - read, as JSON
- with {{voter2.xx}}- read, as JSON
- with {{voter2.x}} - read, as JSON
- with {{voter2.}} - causes expected error
- with {{voter2.x}} - read, as JSON
- with {{voter2.xx}}- read, as JSON
was:
In scanning of directory subtrees for files, recognition of known vs. unknown
file extensions seems really screwy (not following any apparent pattern).
For example:
- a suffix of {{.jsxon_not}}, as expected, is not recognized as a JSON file
- a suffix of {{.jsoxn_not}} unexpectedly _is_ taken as JSON
- a suffix of .{{jsonx_not}}, as expected, is not recognized as a JSON file
(Creating a directory containing only a non-empty JSON file ending with
{{.json}} and another non-empty JSON file ending with one of the above suffixes
sometimes reads both JSON files and sometimes reports a (presumably) expected
error because of the mixed file extensions).)
The result sometimes seems to also depend on the rest of the filename,
presumably related to the order of listing of files. (It's not clear if it
depends only on the order after filename sorting, or also depends on the order
file names are listed the by OS.)
Here are more data points (using a JSON file named {{voter1.json}}):
- with {{voter2.xjson_not}} - read, as JSON
- with {{voter2.jxson_not}} - read, as JSON
- with {{voter2.jsxon_not}} - causes expected error
- with {{voter2.jsoxn_not}} - read, as JSON
- with {{voter2.jsonx_not}} - causes expected error
- with {{voter2.json_xnot}} - read, as JSON
- with {{voter2.json_nxot}} - read, as JSON
- with {{voter2.json_noxt}} - read, as JSON
- with {{voter2.json_notx}} - read, as JSON
- with {{voter2.jsonxnot}} - read, as JSON
- with {{voter2.jsonxot}} - read, as JSON
- with {{voter2.jsoxot}}- causes expected error
- with {{voter2.jxsxoxn}} - read, as JSON
- with {{voter2.xjxsxoxn}} - read, as JSON
- with {{voter2.xjxsxoxnx}} - causes expected error
- with {{voter2.xjxxoxn}} - read, as JSON
- with {{voter2.xjxxxn}- read, as JSON
- with {{voter2.n} - read, as JSON
- with {{voter2.} - read, as JSON
- with {{voter2.xxx}} - read, as JSON
- with {{voter2.xx}}- read, as JSON
- with {{voter2.x}} - read, as JSON
- with {{voter2.}} - causes expected error
- with {{voter2.x - read, as JSON
- with {{voter2.xx- read, as JSON
> weird file-extension recognition behavior in directory subtree scanning
> ---
>
> Key: DRILL-3816
> URL: https://issues.apache.org/jira/browse/DRILL-3816
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Jacques Nadeau
>
> In scanning of directory subtrees for
[jira] [Created] (DRILL-3816) weird file-extension recognition behavior in directory subtree scanning
Daniel Barclay (Drill) created DRILL-3816:
-
Summary: weird file-extension recognition behavior in directory
subtree scanning
Key: DRILL-3816
URL: https://issues.apache.org/jira/browse/DRILL-3816
Project: Apache Drill
Issue Type: Bug
Components: Storage - Other
Reporter: Daniel Barclay (Drill)
Assignee: Jacques Nadeau
In scanning of directory subtrees for files, recognition of known vs. unknown
file extensions seems really screwy (not following any apparent pattern).
For example:
- a suffix of {{.jsxon_not}}, as expected, is not recognized as a JSON file
- a suffix of {{.jsoxn_not}} unexpectedly _is_ taken as JSON
- a suffix of .{{jsonx_not}}, as expected, is not recognized as a JSON file
(Creating a directory containing only a non-empty JSON file ending with
{{.json}} and another non-empty JSON file ending with one of the above suffixes
sometimes reads both JSON files and sometimes reports a (presumably) expected
error because of the mixed file extensions).)
The result sometimes seems to also depend on the rest of the filename,
presumably related to the order of listing of files. (It's not clear if it
depends only on the order after filename sorting, or also depends on the order
file names are listed the by OS.)
Here are more data points (using a JSON file named {{voter1.json}}):
- with {{voter2.xjson_not}} - read, as JSON
- with {{voter2.jxson_not}} - read, as JSON
- with {{voter2.jsxon_not}} - causes expected error
- with {{voter2.jsoxn_not}} - read, as JSON
- with {{voter2.jsonx_not}} - causes expected error
- with {{voter2.json_xnot}} - read, as JSON
- with {{voter2.json_nxot}} - read, as JSON
- with {{voter2.json_noxt}} - read, as JSON
- with {{voter2.json_notx}} - read, as JSON
- with {{voter2.jsonxnot}} - read, as JSON
- with {{voter2.jsonxot}} - read, as JSON
- with {{voter2.jsoxot}}- causes expected error
- with {{voter2.jxsxoxn}} - read, as JSON
- with {{voter2.xjxsxoxn}} - read, as JSON
- with {{voter2.xjxsxoxnx}} - causes expected error
- with {{voter2.xjxxoxn}} - read, as JSON
- with {{voter2.xjxxxn}- read, as JSON
- with {{voter2.n} - read, as JSON
- with {{voter2.} - read, as JSON
- with {{voter2.xxx}} - read, as JSON
- with {{voter2.xx}}- read, as JSON
- with {{voter2.x}} - read, as JSON
- with {{voter2.}} - causes expected error
- with {{voter2.x - read, as JSON
- with {{voter2.xx- read, as JSON
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901366#comment-14901366
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40023540
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
---
@@ -20,14 +20,21 @@
import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.drill.common.exceptions.UserException;
import org.apache.drill.common.map.CaseInsensitiveMap;
import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionValue.OptionType;
import java.util.Collection;
import java.util.Map;
/**
- * {@link OptionManager} that holds options within {@link
org.apache.drill.exec.rpc.user.UserSession} context.
+ * {@link OptionManager} that holds options within {@link
org.apache.drill.exec.rpc.user.UserSession} context. Options
+ * set at the session level only apply to queries that you run during the
current Drill connection. Session level
+ * settings override system level settings.
+ *
+ * NOTE that currently, the effects of deleting a short lived option (see
{@link OptionValidator#isShortLived}) are
+ * undefined.
--- End diff --
No, I meant could you explain what you mean by _undefined_ effects. I think
you mean if we set a short lived session, for example we inject an exception,
then try to delete it, depending on where the exception was injected the reset
query could either succeed or the exception could actually be thrown in the
reset query itself.
Just improve the comment to explain what you mean
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901360#comment-14901360
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user cwestin commented on the pull request:
https://github.com/apache/drill/pull/163#issuecomment-142102955
Just one comment, otherwise +1 (non-binding).
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901358#comment-14901358
]
ASF GitHub Bot commented on DRILL-3811:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/163#discussion_r40023098
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/memory/TestAllocators.java
---
@@ -63,6 +63,29 @@
private final static String planFile="/physical_allocator_test.json";
@Test
+ public void testTransfer() throws Exception {
+final Properties props = new Properties() {
+ {
+put(ExecConstants.TOP_LEVEL_MAX_ALLOC, "100");
+put(ExecConstants.ERROR_ON_MEMORY_LEAK, "true");
+ }
+};
+final DrillConfig config = DrillConfig.create(props);
+BufferAllocator a = new TopLevelAllocator(config);
--- End diff --
I was surprised this compiled; in one of my recent patches, I had removed
the "public" from TopLevelAllocator, but it seems to have come back. It was
removed to force people to use RootAllocatorFactory.newRoot() instead of using
the TopLevelAllocator constructor directly. Can you please remove the public
from TopLevelAllocator again, and replace this constructor here and on the next
line with RootAllocatorFactory.newRoot()?
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[ https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901354#comment-14901354 ] ASF GitHub Bot commented on DRILL-1065: --- Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/159#discussion_r40022907 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionManager.java --- @@ -31,8 +33,33 @@ void setOption(OptionValue value); /** + * Deletes the option. Unfortunately, the type is required given the fallback structure of option managers. --- End diff -- *Unfortunately* because we assume *fallback structure of option managers* > Provide a reset command to reset an option to its default value > --- > > Key: DRILL-1065 > URL: https://issues.apache.org/jira/browse/DRILL-1065 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: Aman Sinha >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.2.0 > > > Within a session, currently we set configuration options and it would be very > useful to have a 'reset' command to reset the value of an option to its > default system value: > ALTER SESSION RESET > If we don't want to add a new keyword for RESET, we could potentially > overload the SET command and allow the user to set to the 'default' value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901356#comment-14901356
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40022994
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControls.java
---
@@ -80,7 +80,7 @@
* @param ttl the number of queries for which this option should be
valid
*/
public ControlsOptionValidator(final String name, final String def,
final int ttl) {
- super(name, OptionValue.Kind.STRING,
OptionValue.createString(OptionType.SESSION, name, def));
+ super(name, OptionValue.Kind.STRING,
OptionValue.createString(OptionType.SYSTEM, name, def));
--- End diff --
Ok, I see, `TypeValidator` always expects _SYSTEM_ OptionValues
Should we add an assertion to TypeValidator to make sure this doesn't
happen again ?
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901349#comment-14901349
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40022217
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
---
@@ -20,14 +20,21 @@
import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.drill.common.exceptions.UserException;
import org.apache.drill.common.map.CaseInsensitiveMap;
import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionValue.OptionType;
import java.util.Collection;
import java.util.Map;
/**
- * {@link OptionManager} that holds options within {@link
org.apache.drill.exec.rpc.user.UserSession} context.
+ * {@link OptionManager} that holds options within {@link
org.apache.drill.exec.rpc.user.UserSession} context. Options
+ * set at the session level only apply to queries that you run during the
current Drill connection. Session level
+ * settings override system level settings.
+ *
+ * NOTE that currently, the effects of deleting a short lived option (see
{@link OptionValidator#isShortLived}) are
+ * undefined.
--- End diff --
Should I add a TODO and create a ticket?
(personally I don't think this is worth a JIRA yet)
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3707) Fix for DRILL-3616 can cause a NullPointerException in ExternalSort cleanup
[ https://issues.apache.org/jira/browse/DRILL-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun Chang closed DRILL-3707. - Assignee: Chun Chang (was: Deneche A. Hakim) The bug would cause NPE in functional automation test. Tests are passing. > Fix for DRILL-3616 can cause a NullPointerException in ExternalSort cleanup > --- > > Key: DRILL-3707 > URL: https://issues.apache.org/jira/browse/DRILL-3707 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Deneche A. Hakim >Assignee: Chun Chang > Fix For: 1.2.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901343#comment-14901343
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40021955
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
ExecConstants.SLICE_TARGET_DEFAULT));
testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
.unOrdered()
.baselineColumns("status")
.baselineValues("DEFAULT")
.build()
.run();
}
+
+ @Test
+ public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type =
'SESSION';");
+testBuilder()
+ .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .baselineColumns("num_val")
+ .baselineValues((long) 10)
+ .build()
+ .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+ }
+
+ @Test
+ public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+ .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("bool_val")
+ .baselineValues(true)
+ .build()
+ .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+ }
+
+ @Test
+ public void resetAllSessionOptions() throws Exception {
--- End diff --
my bad, I meant: we change the same option at the _SESSION_ and _SYSTEM_
level then we reset it at the _SESSION_ level and we make sure it's still
changed at the _SYSTEM_ level.
`changeSessionAndNotSystem` kind of does that but with a `RESET ALL` and we
need to make sure `ALTER SESSION RESET A` also works fine
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be ver
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901339#comment-14901339
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40021824
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
}
@Override
+ public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
--- End diff --
I left this as is because the callers could add context and handle the
exception differently.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3731) Partition Pruning not taking place when we have case statement within the filter
[
https://issues.apache.org/jira/browse/DRILL-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901332#comment-14901332
]
Rahul Challapalli commented on DRILL-3731:
--
[~jni] You are right. We can simplify the filter in this case. I constructed
this just to test the behavior and it is not hard to imagine a real world
scenario where a case statement is used in the filter which results in
partition pruning not taking place.
> Partition Pruning not taking place when we have case statement within the
> filter
>
>
> Key: DRILL-3731
> URL: https://issues.apache.org/jira/browse/DRILL-3731
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Rahul Challapalli
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: lineitempart.tgz
>
>
> Commit # 8f33c3d55228a98ea2a309490ecbfb7a180f012c
> Partition Pruning is not taking place in the below query
> {code}
> explain plan for select count(*) from
> `/drill/testdata/partition_pruning/dfs/lineitempart` where case when
> dir0=1991 then null else 1 end is null or dir0=1992;
> 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03 Project($f0=[0])
> 00-04SelectionVectorRemover
> 00-05 Filter(condition=[OR(IS NULL(CASE(=($0, 1991), null, 1)),
> =($0, 1992))])
> 00-06Scan(groupscan=[EasyGroupScan
> [selectionRoot=maprfs:/drill/testdata/partition_pruning/dfs/lineitempart,
> numFiles=7, columns=[`dir0`],
> files=[maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1991/lineitemaa.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1992/lineitemab.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1996/lineitemaf.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1994/lineitemad.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1993/lineitemac.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1997/lineitemag.tbl,
>
> maprfs:/drill/testdata/partition_pruning/dfs/lineitempart/1995/lineitemae.tbl]]])
> {code}
> The planner should have pruned all partitions except 1991 & 1992
> I attached the dataset. Let me know if you need anything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901325#comment-14901325
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40020971
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControls.java
---
@@ -80,7 +80,7 @@
* @param ttl the number of queries for which this option should be
valid
*/
public ControlsOptionValidator(final String name, final String def,
final int ttl) {
- super(name, OptionValue.Kind.STRING,
OptionValue.createString(OptionType.SESSION, name, def));
+ super(name, OptionValue.Kind.STRING,
OptionValue.createString(OptionType.SYSTEM, name, def));
--- End diff --
I don't think this requires a comment in code, since this applies for all
option validators.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901321#comment-14901321
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40020795
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
ExecConstants.SLICE_TARGET_DEFAULT));
testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
.unOrdered()
.baselineColumns("status")
.baselineValues("DEFAULT")
.build()
.run();
}
+
+ @Test
+ public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type =
'SESSION';");
+testBuilder()
+ .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .baselineColumns("num_val")
+ .baselineValues((long) 10)
+ .build()
+ .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+ }
+
+ @Test
+ public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+ .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("bool_val")
+ .baselineValues(true)
+ .build()
+ .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+ }
+
+ @Test
+ public void resetAllSessionOptions() throws Exception {
--- End diff --
changeSessionAndNotSystem is testing this.
Also, we cannot test resetting all system options because the unit tests
have a "before" method that changes a system option.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If
[jira] [Updated] (DRILL-3733) erro message fix - NTILE function
[
https://issues.apache.org/jira/browse/DRILL-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-3733:
Fix Version/s: (was: 1.2.0)
1.3.0
> erro message fix - NTILE function
> -
>
> Key: DRILL-3733
> URL: https://issues.apache.org/jira/browse/DRILL-3733
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Minor
> Labels: window_function
> Fix For: 1.3.0
>
>
> Can we have the message read this way
> "NTILE only accepts positive (non-zero) integer argument"
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col7 , col9 , ntile(0) over(order by
> col0) lastVal_col9 from FEWRWSPQQ_101;
> Error: FUNCTION ERROR: NTILE only accepts positive integer argument
> Fragment 0:0
> [Error Id: 650d44a8-73d0-4675-8c18-67609f43a962 on centos-04.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3630) error message needs to be fixed LEAD , LAG window functions
[
https://issues.apache.org/jira/browse/DRILL-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-3630:
Fix Version/s: (was: 1.2.0)
1.3.0
> error message needs to be fixed LEAD , LAG window functions
> ---
>
> Key: DRILL-3630
> URL: https://issues.apache.org/jira/browse/DRILL-3630
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
> Environment: private-branch
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Minor
> Labels: window_function
> Fix For: 1.3.0
>
>
> The error message says LEAD expects ONE input argument, however if you see in
> the other query below, lead( col9 , 1 ) does not throw any error (as this is
> valid syntax and is supported). Another case is, lead(col9,1,1) where we
> don't report any error being reported. We need to fix the error message and
> handle inputs appropriately.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col9 , lead(col9,1,0,1) over(partition
> by col7 order by col0) lead_col9 from FEWRWSPQQ_101;
> Error: PARSE ERROR: From line 1, column 15 to line 1, column 35: Invalid
> number of arguments to function 'LEAD'. Was expecting 1 arguments
> [Error Id: 4a94487c-42e6-4672-978e-f011c997c537 on centos-03.qa.lab:31010]
> (state=,code=0)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col9 , lead(col9,1) over(partition by
> col7 order by col0) lead_col9 from FEWRWSPQQ_101;
> +---+---+
> | col9 |
> lead_col9 |
> +---+---+
> | OXCB |
> ZXCZ |
> | ZXCZ |
> AXCZ |
> | AXCZ |
> CXCB |
> | CXCB |
> HXCZ |
> | HXCZ |
> UXCB |
> | UXCB |
> TXCD |
> | TXCD |
> PXCD |
> | PXCD |
> NXCB |
> | NXCB |
> MXCB |
> | MXCB |
> WXCB |
> | WXCB | null
> |
> | KXCB |
> BXCD |
> | BXCD |
> DXCB |
> | DXCB |
> IXCD |
> | IXCD |
> YXCB |
> | YXCB |
> EXCZ |
> | EXCZ |
> LXCB |
> | LXCB |
> SXCB |
> | SXCB |
> VXCZ |
> | VXCZ |
> FXCB |
> | FXCB |
> QXCZ |
> | QXCZ | null
> |
> +---+---+
> 22 rows selected (0.245 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3481) Query with Window Function fails with "SYSTEM ERROR: RpcException: Data not accepted downstream."
[
https://issues.apache.org/jira/browse/DRILL-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-3481:
Fix Version/s: (was: 1.2.0)
1.3.0
> Query with Window Function fails with "SYSTEM ERROR: RpcException: Data not
> accepted downstream."
> -
>
> Key: DRILL-3481
> URL: https://issues.apache.org/jira/browse/DRILL-3481
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Abhishek Girish
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.3.0
>
> Attachments: JSON Profile.txt, Physical Plan.txt
>
>
> I'm seeing an error when executing a simple WF query on the latest master.
> Dataset: TPC-DS SF1000 - Parquet
> Git.Commit.ID: b6577fe (Jul 8 15)
> Options:
> {code}
> alter session set `planner.memory.max_query_memory_per_node` = 21474836480;
> {code}
> Query:
> {code:sql}
> SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk ORDER BY
> ss.ss_store_sk) FROM store_sales ss WHERE ss.ss_store_sk is not NULL LIMIT
> 20;
> {code}
> Error:
> {code}
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> RpcException: Data not accepted downstream.
> Fragment 2:1
> [Error Id: 49edceeb-8f10-474d-9cf2-adb4baa13bf4 on ucs-node7.perf.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-08 12:17:44,764 ucs-node7.perf.lab [BitClient-2] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested RUNNING -->
> FAILED
> 2015-07-08 12:17:44,765 ucs-node7.perf.lab [BitClient-2] ERROR
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping
> future sends.
> 2015-07-08 12:17:44,765 ucs-node7.perf.lab [BitClient-2] ERROR
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping
> future sends.
> 2015-07-08 12:17:44,765 ucs-node7.perf.lab [BitClient-2] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED -->
> FAILED
> 2015-07-08 12:17:44,768 ucs-node7.perf.lab [BitClient-2] ERROR
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping
> future sends.
> 2015-07-08 12:17:44,768 ucs-node7.perf.lab [BitClient-2] ERROR
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping
> future sends.
> 2015-07-08 12:17:44,769 ucs-node7.perf.lab [BitClient-2] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED -->
> FAILED
> 2015-07-08 12:17:44,771 ucs-node7.perf.lab [BitClient-2] ERROR
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping
> future sends.
> 2015-07-08 12:17:44,771 ucs-node7.perf.lab [BitClient-2] ERROR
> o.a.drill.exec.ops.StatusHandler - Data not accepted downstream. Stopping
> future sends.
> 2015-07-08 12:17:44,772 ucs-node7.perf.lab [BitClient-2] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED -->
> FAILED
> 2015-07-08 12:17:44,790 ucs-node7.perf.lab
> [2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:frag:2:1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:2:1: State change requested FAILED -->
> FINISHED
> 2015-07-08 12:17:44,814 ucs-node7.perf.lab
> [2a628c9e-3ff4-cc1b-77d2-bab6d00a2d1d:frag:2:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: RpcException: Data not
> accepted downstream.
> Fragment 2:1
> [Error Id: 49edceeb-8f10-474d-9cf2-adb4baa13bf4 on ucs-node7.perf.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: RpcException:
> Data not accepted downstream.
> Fragment 2:1
> [Error Id: 49edceeb-8f10-474d-9cf2-adb4baa13bf4 on ucs-node7.perf.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.f
[jira] [Updated] (DRILL-3194) TestDrillbitResilience#memoryLeaksWhenFailed hangs
[ https://issues.apache.org/jira/browse/DRILL-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-3194: Fix Version/s: (was: 1.2.0) 1.3.0 > TestDrillbitResilience#memoryLeaksWhenFailed hangs > -- > > Key: DRILL-3194 > URL: https://issues.apache.org/jira/browse/DRILL-3194 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Sudheesh Katkam >Assignee: Deneche A. Hakim > Fix For: 1.3.0 > > > TestDrillbitResilience#memoryLeaksWhenFailed hangs and fails when run > multiple times. This might be related to DRILL-3163. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2359) ClassPathFileSystem.getFileStatus() should throw FileNotFoundException when path doesn't exist
[
https://issues.apache.org/jira/browse/DRILL-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-2359:
Fix Version/s: (was: 1.2.0)
1.3.0
> ClassPathFileSystem.getFileStatus() should throw FileNotFoundException when
> path doesn't exist
> --
>
> Key: DRILL-2359
> URL: https://issues.apache.org/jira/browse/DRILL-2359
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 0.8.0
>Reporter: Deneche A. Hakim
>Assignee: Deneche A. Hakim
> Fix For: 1.3.0
>
> Attachments: DRILL-2359.1.patch.txt
>
>
> {{ClassPathFileSystem.getFileStatus(Path args0)}} throws {{IOException}} when
> the path doesn't exist. It should throw {{FileNotFoundException}} instead.
> One particular method that fails to work correctly because of this is
> {{org.apache.hadoop.FileSystem.exists(Path f)}}:
> {code}
> public boolean exists(Path f) throws IOException {
> try {
> return getFileStatus(f) != null;
> } catch (FileNotFoundException e) {
> return false;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3732) Drill leaks memory if external sort hits out of disk space exception
[
https://issues.apache.org/jira/browse/DRILL-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901281#comment-14901281
]
ASF GitHub Bot commented on DRILL-3732:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/147
> Drill leaks memory if external sort hits out of disk space exception
>
>
> Key: DRILL-3732
> URL: https://issues.apache.org/jira/browse/DRILL-3732
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: drillbit.log
>
>
> Ran into it when running CTAS with partition by.
> Here is what reproduction looks like:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_4(ss_item_sk,
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk)
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > case when columns[2] = '' then cast(null as
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[3] = '' then cast(null as
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[4] = '' then cast(null as
> varchar(100)) else cast(columns[4] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[5] = '' then cast(null as
> varchar(100)) else cast(columns[5] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as
> varchar(100)) else cast(columns[0] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[8] = '' then cast(null as
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > `store_sales.dat` ss
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: IllegalStateException: Failure while closing accountor.
> Expected private and shared pools to be set to initial values. However, one
> or more were not. Stats are
> zoneinitallocated delta
> private 10009680512 319488
> shared 100010000.
> Fragment 1:21
> [Error Id: bd0d7d59-8693-476b-8671-70f0b2e7a176 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Setup:
> single node
> 8GB direct memory
> 4GB heap memory
>store_sales.dat is a file from TPCDS SF100
> drillbit.log attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901255#comment-14901255
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40014502
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,124 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
ExecConstants.SLICE_TARGET_DEFAULT));
testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
.unOrdered()
.baselineColumns("status")
.baselineValues("DEFAULT")
.build()
.run();
}
+
+ @Test
+ public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type =
'SESSION';");
+testBuilder()
+ .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .baselineColumns("num_val")
+ .baselineValues((long) 10)
+ .build()
+ .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+ }
+
+ @Test
+ public void setAndResetSystemOption() throws Exception {
--- End diff --
thanks for adding those unit tests
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[ https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901249#comment-14901249 ] ASF GitHub Bot commented on DRILL-1065: --- Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/159#discussion_r40014413 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java --- @@ -161,6 +171,7 @@ public SqlNode visitChild( rules.put(SqlJoin.class, R(D, D, D, D, D, E)); rules.put(SqlOrderBy.class, R(D, E, D, D)); rules.put(SqlDropTable.class, R(D)); +rules.put(SqlSetOption.class, R(D, D, D)); --- End diff -- the comment doesn't explain why this specific rule has been added. It just explains how the rules work in general. Why do we need this specific rule, what happens if I remove it ? > Provide a reset command to reset an option to its default value > --- > > Key: DRILL-1065 > URL: https://issues.apache.org/jira/browse/DRILL-1065 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: Aman Sinha >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.2.0 > > > Within a session, currently we set configuration options and it would be very > useful to have a 'reset' command to reset the value of an option to its > default system value: > ALTER SESSION RESET > If we don't want to add a new keyword for RESET, we could potentially > overload the SET command and allow the user to set to the 'default' value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3811) AtomicRemainder incorrectly accounts for transferred allocations
[
https://issues.apache.org/jira/browse/DRILL-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901237#comment-14901237
]
ASF GitHub Bot commented on DRILL-3811:
---
GitHub user adeneche opened a pull request:
https://github.com/apache/drill/pull/163
DRILL-3811: AtomicRemainder incorrectly accounts for transferred allo…
…cations
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/adeneche/incubator-drill fix-acountor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/163.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #163
commit 94af90b8006ea95fdfa02cc4ddae5224ab31c586
Author: adeneche
Date: 2015-09-20T19:11:55Z
DRILL-3811: AtomicRemainder incorrectly accounts for transferred allocations
> AtomicRemainder incorrectly accounts for transferred allocations
>
>
> Key: DRILL-3811
> URL: https://issues.apache.org/jira/browse/DRILL-3811
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Chris Westin
> Attachments: DRILL-3811.1.patch.txt
>
>
> when an allocator takes ownership of a buffer, AtomicRemainder.forceGet(int)
> is called to account for the extra memory of the buffer, but when the
> allocator exceeds it's maximum allocated memory it accounts for it
> incorrectly. In the following code, {{availableShared.andAndGet(size)}}
> should actually receive {{-size}}:
> {code}
> public boolean forceGet(long size) {
> if (get(size, this.applyFragmentLimit)) {
> return true;
> } else {
> availableShared.addAndGet(size);
> if (parent != null) {
> parent.forceGet(size);
> }
> return false;
> }
> }
> {code}
> I was able to reproduce the issue in a simple unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901214#comment-14901214
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40012463
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,124 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
ExecConstants.SLICE_TARGET_DEFAULT));
testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
.unOrdered()
.baselineColumns("status")
.baselineValues("DEFAULT")
.build()
.run();
}
+
+ @Test
+ public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type =
'SESSION';");
+testBuilder()
+ .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .baselineColumns("num_val")
+ .baselineValues((long) 10)
+ .build()
+ .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+ }
+
+ @Test
+ public void setAndResetSystemOption() throws Exception {
--- End diff --
changeSessionAndNotSystem and changeSystemAndNotSession are testing this.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3815) unknown suffixes .not_json and .json_not treated differently (multi-file case)
Daniel Barclay (Drill) created DRILL-3815:
-
Summary: unknown suffixes .not_json and .json_not treated
differently (multi-file case)
Key: DRILL-3815
URL: https://issues.apache.org/jira/browse/DRILL-3815
Project: Apache Drill
Issue Type: Bug
Components: Storage - Other
Reporter: Daniel Barclay (Drill)
Assignee: Jacques Nadeau
In scanning a directory subtree used as a table, unknown filename extensions
seem to be treated differently depending on whether they're similar to known
file extensions. The behavior suggests that Drill checks whether a file name
_contains_ an extension's string rather than _ending_ with it.
For example, given these subtrees with almost identical leaf file names:
{noformat}
$ find /tmp/testext_xx_json/
/tmp/testext_xx_json/
/tmp/testext_xx_json/voter2.not_json
/tmp/testext_xx_json/voter1.json
$ find /tmp/testext_json_xx/
/tmp/testext_json_xx/
/tmp/testext_json_xx/voter1.json
/tmp/testext_json_xx/voter2.json_not
$
{noformat}
the results of trying to use them as tables differs:
{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs.tmp`.`testext_xx_json`;
Sep 21, 2015 11:41:50 AM org.apache.calcite.sql.validate.SqlValidatorException
...
Error: VALIDATION ERROR: From line 1, column 17 to line 1, column 25: Table
'dfs.tmp.testext_xx_json' not found
[Error Id: 6fe41deb-0e39-43f6-beca-de27b39d276b on dev-linux2:31010]
(state=,code=0)
0: jdbc:drill:zk=local> SELECT * FROM `dfs.tmp`.`testext_json_xx`;
+---+
| onecf |
+---+
| {"name":"someName1"} |
| {"name":"someName2"} |
+---+
2 rows selected (0.149 seconds)
{noformat}
(Other probing seems to indicate that there is also some sensitivity to whether
the extension contains an underscore character.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[ https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901189#comment-14901189 ] ASF GitHub Bot commented on DRILL-1065: --- Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/159#discussion_r40010391 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java --- @@ -161,6 +171,7 @@ public SqlNode visitChild( rules.put(SqlJoin.class, R(D, D, D, D, D, E)); rules.put(SqlOrderBy.class, R(D, E, D, D)); rules.put(SqlDropTable.class, R(D)); +rules.put(SqlSetOption.class, R(D, D, D)); --- End diff -- There is already a comment explaining the rules, and why they are needed. What more? > Provide a reset command to reset an option to its default value > --- > > Key: DRILL-1065 > URL: https://issues.apache.org/jira/browse/DRILL-1065 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: Aman Sinha >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.2.0 > > > Within a session, currently we set configuration options and it would be very > useful to have a 'reset' command to reset the value of an option to its > default system value: > ALTER SESSION RESET > If we don't want to add a new keyword for RESET, we could potentially > overload the SET command and allow the user to set to the 'default' value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901185#comment-14901185
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r40010220
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
---
@@ -49,45 +53,67 @@ public SetOptionHandler(QueryContext context) {
}
@Override
- public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException,
RelConversionException, IOException, ForemanSetupException {
+ public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException,
ForemanSetupException {
final SqlSetOption option = unwrap(sqlNode, SqlSetOption.class);
-final String scope = option.getScope();
-final String name = option.getName();
final SqlNode value = option.getValue();
-OptionValue.OptionType type;
-if (value instanceof SqlLiteral) {
+if (value != null && !(value instanceof SqlLiteral)) {
+ throw UserException.validationError()
+ .message("Drill does not support assigning non-literal values in
SET statements.")
+ .build(logger);
+}
+
+final String scope = option.getScope();
+final OptionValue.OptionType type;
+if (scope == null) { // No scope mentioned assumed SESSION
+ type = OptionType.SESSION;
+} else {
switch (scope.toLowerCase()) {
-case "session":
- type = OptionValue.OptionType.SESSION;
- break;
-case "system":
- type = OptionValue.OptionType.SYSTEM;
- break;
-//case "query":
-// type = OptionValue.OptionType.QUERY;
-// break;
-default:
- throw new ValidationException("Invalid OPTION scope. Scope must
be SESSION or SYSTEM.");
+ case "session":
+type = OptionType.SESSION;
+break;
+ case "system":
+type = OptionType.SYSTEM;
+break;
+ default:
+throw UserException.validationError()
+.message("Invalid OPTION scope %s. Scope must be SESSION or
SYSTEM.", scope)
+.build(logger);
}
+}
- if (type == OptionType.SYSTEM) {
-// If the user authentication is enabled, make sure the user who
is trying to change the system option has
-// administrative privileges.
-if (context.isUserAuthenticationEnabled() &&
-!ImpersonationUtil.hasAdminPrivileges(
-context.getQueryUserName(),
-
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
-
context.getOptions().getOption(ExecConstants.ADMIN_USER_GROUPS_KEY).string_val))
{
- throw UserException.permissionError()
- .message("Not authorized to change SYSTEM options.")
- .build(logger);
-}
+if (type == OptionType.SYSTEM) {
+ // If the user authentication is enabled, make sure the user who is
trying to change the system option has
+ // administrative privileges.
+ if (context.isUserAuthenticationEnabled() &&
+ !ImpersonationUtil.hasAdminPrivileges(
+context.getQueryUserName(),
+
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
--- End diff --
This is not a change I introduced (just difference in spacing), but I'll
address this issue.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3463) Unit test of project pushdown in TestUnionAll should put more precisely plan attribute in plan verification.
[
https://issues.apache.org/jira/browse/DRILL-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3463.
---
> Unit test of project pushdown in TestUnionAll should put more precisely plan
> attribute in plan verification.
> --
>
> Key: DRILL-3463
> URL: https://issues.apache.org/jira/browse/DRILL-3463
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3463-Unit-test-of-project-pushdown-in-TestUnio.patch
>
>
> As part of fix for DRILL-2802, it was discovered that several unit test cases
> for project pushdown in TestUnionAll did not put the desired plan attributes
> in to the expected plan result.
> To verify project pushdown is working properly, one simple way is to verify
> that the the column list in the Scan operator contains the desired columns.
> This should be the part of plan verification. However, the unit test cases in
> TestUnionAll did not do that. In stead, it tries to match a pattern of
> "Project -- Scan", which seems not serving the purpose it desired.
> For instance,
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*\n" +
> {code}
> should be replaced by
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*columns=\\[`n_comment`, `n_nationkey`, `n_name`\\].*\n"
> {code}
> if we want to verify the column 'n_comment', 'n_nationkey', 'n_name' are
> pushed into Scan operator.
> To fix this, modify the expected plan result, such that it contains the plan
> attributes that should be able to verify whether the project pushdown is
> working or not.
> This will help catch project pushdown failure, and avoid causing more false
> alarm in plan verification.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3463) Unit test of project pushdown in TestUnionAll should put more precisely plan attribute in plan verification.
[
https://issues.apache.org/jira/browse/DRILL-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901045#comment-14901045
]
Victoria Markman commented on DRILL-3463:
-
Verified changes in the expected result of the unit test are correct.
> Unit test of project pushdown in TestUnionAll should put more precisely plan
> attribute in plan verification.
> --
>
> Key: DRILL-3463
> URL: https://issues.apache.org/jira/browse/DRILL-3463
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3463-Unit-test-of-project-pushdown-in-TestUnio.patch
>
>
> As part of fix for DRILL-2802, it was discovered that several unit test cases
> for project pushdown in TestUnionAll did not put the desired plan attributes
> in to the expected plan result.
> To verify project pushdown is working properly, one simple way is to verify
> that the the column list in the Scan operator contains the desired columns.
> This should be the part of plan verification. However, the unit test cases in
> TestUnionAll did not do that. In stead, it tries to match a pattern of
> "Project -- Scan", which seems not serving the purpose it desired.
> For instance,
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*\n" +
> {code}
> should be replaced by
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*columns=\\[`n_comment`, `n_nationkey`, `n_name`\\].*\n"
> {code}
> if we want to verify the column 'n_comment', 'n_nationkey', 'n_name' are
> pushed into Scan operator.
> To fix this, modify the expected plan result, such that it contains the plan
> attributes that should be able to verify whether the project pushdown is
> working or not.
> This will help catch project pushdown failure, and avoid causing more false
> alarm in plan verification.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
[
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3621:
Assignee: Khurram Faraaz (was: Jinfeng Ni)
> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
>
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10 |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY1';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec
> [tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10, filter=null],
> columns=[`row_key`]]])
> |
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901029#comment-14901029
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39998405
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
ExecConstants.SLICE_TARGET_DEFAULT));
testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
.unOrdered()
.baselineColumns("status")
.baselineValues("DEFAULT")
.build()
.run();
}
+
+ @Test
+ public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type =
'SESSION';");
+testBuilder()
+ .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .baselineColumns("num_val")
+ .baselineValues((long) 10)
+ .build()
+ .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+ }
+
+ @Test
+ public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+ .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("bool_val")
+ .baselineValues(true)
+ .build()
+ .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+ }
+
+ @Test
+ public void resetAllSessionOptions() throws Exception {
--- End diff --
also change _SESSION_ and _SYSTEM_ options then reset all _SESSION_ options
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901027#comment-14901027
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39998259
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
ExecConstants.SLICE_TARGET_DEFAULT));
testBuilder()
-.sqlQuery(String.format("SELECT status FROM sys.options WHERE name
= '%s' AND type = 'SESSION'", ExecConstants.SLICE_TARGET))
+.sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
.unOrdered()
.baselineColumns("status")
.baselineValues("DEFAULT")
.build()
.run();
}
+
+ @Test
+ public void setAndResetSessionOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+
+// change option
+test("SET `%s` = %d;", SLICE_TARGET, 10);
+// check changed
+test("SELECT status, type, name FROM sys.options WHERE type =
'SESSION';");
+testBuilder()
+ .sqlQuery("SELECT num_val FROM sys.options WHERE name = '%s' AND
type = 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .baselineColumns("num_val")
+ .baselineValues((long) 10)
+ .build()
+ .run();
+
+// reset option
+test("RESET `%s`;", SLICE_TARGET);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SESSION'", SLICE_TARGET)
+ .unOrdered()
+ .expectsEmptyResultSet()
+ .build()
+ .run();
+ }
+
+ @Test
+ public void setAndResetSystemOption() throws Exception {
+// check unchanged
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+
+// change option
+test("ALTER system SET `%s` = %b;", ENABLE_VERBOSE_ERRORS_KEY, true);
+// check changed
+testBuilder()
+ .sqlQuery("SELECT bool_val FROM sys.options WHERE name = '%s' AND
type = 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("bool_val")
+ .baselineValues(true)
+ .build()
+ .run();
+
+// reset option
+test("ALTER system RESET `%s`;", ENABLE_VERBOSE_ERRORS_KEY);
+// check reverted
+testBuilder()
+ .sqlQuery("SELECT status FROM sys.options WHERE name = '%s' AND type
= 'SYSTEM'", ENABLE_VERBOSE_ERRORS_KEY)
+ .unOrdered()
+ .baselineColumns("status")
+ .baselineValues("DEFAULT")
+ .build()
+ .run();
+ }
+
+ @Test
+ public void resetAllSessionOptions() throws Exception {
--- End diff --
resetting all _SYSTEM_ options is not covered here
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'defau
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901025#comment-14901025
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39998161
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/server/TestOptions.java ---
@@ -46,19 +48,207 @@ public void testOptions() throws Exception{
@Test
public void checkValidationException() throws Exception {
thrownException.expect(new UserExceptionMatcher(VALIDATION));
-test(String.format("ALTER session SET `%s` = '%s';",
ExecConstants.SLICE_TARGET, "fail"));
+test("ALTER session SET `%s` = '%s';", SLICE_TARGET, "fail");
}
@Test // DRILL-3122
public void checkChangedColumn() throws Exception {
-test(String.format("ALTER session SET `%s` = %d;",
ExecConstants.SLICE_TARGET,
+test(String.format("ALTER session SET `%s` = %d;", SLICE_TARGET,
--- End diff --
you can remove `String.format` here too
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901024#comment-14901024
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39998097
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExecutionControls.java
---
@@ -80,7 +80,7 @@
* @param ttl the number of queries for which this option should be
valid
*/
public ControlsOptionValidator(final String name, final String def,
final int ttl) {
- super(name, OptionValue.Kind.STRING,
OptionValue.createString(OptionType.SESSION, name, def));
+ super(name, OptionValue.Kind.STRING,
OptionValue.createString(OptionType.SYSTEM, name, def));
--- End diff --
can you add a comment explaining why this needs to be a _SYSTEM_ rather
than a _SESSION_ option ? someone else might revert this change inadvertently.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901021#comment-14901021
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39997945
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
}
@Override
+ public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
--- End diff --
you can extract this try block as a static method `ensureOptionExists`,
this same code is duplicated in multiple places
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901018#comment-14901018
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39997752
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
}
@Override
+ public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
+ getValidator(name);
+} catch (final IllegalArgumentException e) {
+ throw UserException.validationError(e)
+.build(logger);
+}
+options.delete(name.toLowerCase());
+ }
+
+ @Override
+ public void deleteAllOptions(OptionType type) {
+assert type == OptionType.SYSTEM;
--- End diff --
same here
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901016#comment-14901016
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39997731
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
---
@@ -241,6 +243,30 @@ public void setOption(final OptionValue value) {
}
@Override
+ public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
--- End diff --
can you please add an error message to the assertion ?
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901014#comment-14901014
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39997680
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
---
@@ -20,14 +20,21 @@
import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.drill.common.exceptions.UserException;
import org.apache.drill.common.map.CaseInsensitiveMap;
import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.options.OptionValue.OptionType;
import java.util.Collection;
import java.util.Map;
/**
- * {@link OptionManager} that holds options within {@link
org.apache.drill.exec.rpc.user.UserSession} context.
+ * {@link OptionManager} that holds options within {@link
org.apache.drill.exec.rpc.user.UserSession} context. Options
+ * set at the session level only apply to queries that you run during the
current Drill connection. Session level
+ * settings override system level settings.
+ *
+ * NOTE that currently, the effects of deleting a short lived option (see
{@link OptionValidator#isShortLived}) are
+ * undefined.
--- End diff --
can you please elaborate on the _undefined_ effects of deleting short lived
options ?
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-3798) Cannot group by the functions without ()
[
https://issues.apache.org/jira/browse/DRILL-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinfeng Ni resolved DRILL-3798.
---
Resolution: Duplicate
> Cannot group by the functions without ()
>
>
> Key: DRILL-3798
> URL: https://issues.apache.org/jira/browse/DRILL-3798
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Jinfeng Ni
>
> Drill can not group-by the function without ().
> eg:
> {code}
> SELECT CURRENT_DATE
> FROM hive.h1db.testdate
> group by CURRENT_DATE;
> Caused By (org.apache.calcite.sql.validate.SqlValidatorException) Column
> 'CURRENT_DATE' not found in any table
> {code}
> Bad ones:
> {code}
> SELECT CURRENT_TIME
> FROM hive.h1db.testdate
> group by CURRENT_TIME;
> SELECT CURRENT_TIMESTAMP
> FROM hive.h1db.testdate
> group by CURRENT_TIMESTAMP;
> SELECT LOCALTIME
> FROM hive.h1db.testdate
> group by LOCALTIME;
> SELECT LOCALTIMESTAMP
> FROM hive.h1db.testdate
> group by LOCALTIMESTAMP;
> {code}
> Good ones:
> {code}
> SELECT NOW()
> FROM hive.h1db.testdate
> group by NOW();
> SELECT TIMEOFDAY()
> FROM hive.h1db.testdate
> group by TIMEOFDAY();
> SELECT UNIX_TIMESTAMP()
> FROM hive.h1db.testdate
> group by UNIX_TIMESTAMP();
> SELECT PI()
> FROM hive.h1db.testdate
> group by PI();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901009#comment-14901009
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39997204
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/InMemoryOptionManager.java
---
@@ -54,12 +56,32 @@ boolean setLocalOption(final OptionValue value) {
return options.values();
}
+ @Override
+ boolean deleteLocalOptions(final OptionType type) {
+if (supportsOption(type)) {
+ options.clear();
+ return true;
+} else {
+ return false;
+}
+ }
+
+ @Override
+ boolean deleteLocalOption(final String name, final OptionType type) {
+if (supportsOption(type)) {
+ options.remove(name);
+ return true;
+} else {
+ return false;
+}
+ }
+
/**
- * Check to see if implementations of this manager support the given
option value (e.g. check for option type).
+ * Check to see if implementations of this manager support the given
option type.
*
- * @param value the option value
- * @return true iff the option value is supported
+ * @param type option type
+ * @return true iff the type is supported
*/
- abstract boolean supportsOption(OptionValue value);
+ abstract boolean supportsOption(OptionType type);
--- End diff --
you should probably rename this to `supportsOptionType` because we are
effectively checking if the option type is supported. This will help the method
be self explanatory
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (DRILL-3781) Using CURRENT_DATE in a group by throws a column not found error for hive tables and csv files
[
https://issues.apache.org/jira/browse/DRILL-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinfeng Ni reassigned DRILL-3781:
-
Assignee: Jinfeng Ni (was: Mehant Baid)
> Using CURRENT_DATE in a group by throws a column not found error for hive
> tables and csv files
> --
>
> Key: DRILL-3781
> URL: https://issues.apache.org/jira/browse/DRILL-3781
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: csv-error.log, hive_error.log
>
>
> Commit # : e43155d8eabb6fc2d0fa4c68c25d6e7c59bf4521
> Using CURRENT_DATE in a group by seems to failing against hive and csv files.
> With parquet and json, there seems to be no issues.
> Query against a hive table :
> {code}
> select CURRENT_DATE from student_hive group by CURRENT_DATE;
> Error: PARSE ERROR: From line 1, column 48 to line 1, column 59: Column
> 'CURRENT_DATE' not found in any table
> [Error Id: e7d7df50-c5e8-4eda-990a-050b9a2b188e on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> Query against csv files :
> {code}
> select CURRENT_DATE from `temp.tbl` group by CURRENT_DATE;
> Error: DATA_READ ERROR: Selected column 'CURRENT_DATE' must have name
> 'columns' or must be plain '*'
> File Path maprfs:///drill/testdata/temp.tbl
> Fragment 0:0
> [Error Id: 1856f171-966e-4078-bbea-7ff3e9e22e15 on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> A similar query against json and parquet seems to be working fine
> {code}
> select current_date from `a.json` group by current_date;
> +---+
> | current_date |
> +---+
> | 2015-09-14|
> +---+
> select CURRENT_DATE from cp.`tpch/lineitem.parquet` group by CURRENT_DATE;
> +---+
> | CURRENT_DATE |
> +---+
> | 2015-09-14|
> +---+
> {code}
> I attached the log files for the failing conditions. Let me know if you need
> anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[ https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901010#comment-14901010 ] ASF GitHub Bot commented on DRILL-1065: --- Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/159#discussion_r39997250 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionManager.java --- @@ -31,8 +33,33 @@ void setOption(OptionValue value); /** + * Deletes the option. Unfortunately, the type is required given the fallback structure of option managers. --- End diff -- why _unfortunately_ ? > Provide a reset command to reset an option to its default value > --- > > Key: DRILL-1065 > URL: https://issues.apache.org/jira/browse/DRILL-1065 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: Aman Sinha >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.2.0 > > > Within a session, currently we set configuration options and it would be very > useful to have a 'reset' command to reset the value of an option to its > default system value: > ALTER SESSION RESET > If we don't want to add a new keyword for RESET, we could potentially > overload the SET command and allow the user to set to the 'default' value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900993#comment-14900993
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39996293
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
---
@@ -49,45 +53,67 @@ public SetOptionHandler(QueryContext context) {
}
@Override
- public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException,
RelConversionException, IOException, ForemanSetupException {
+ public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException,
ForemanSetupException {
final SqlSetOption option = unwrap(sqlNode, SqlSetOption.class);
-final String scope = option.getScope();
-final String name = option.getName();
final SqlNode value = option.getValue();
-OptionValue.OptionType type;
-if (value instanceof SqlLiteral) {
+if (value != null && !(value instanceof SqlLiteral)) {
+ throw UserException.validationError()
+ .message("Drill does not support assigning non-literal values in
SET statements.")
+ .build(logger);
+}
+
+final String scope = option.getScope();
+final OptionValue.OptionType type;
+if (scope == null) { // No scope mentioned assumed SESSION
+ type = OptionType.SESSION;
+} else {
switch (scope.toLowerCase()) {
-case "session":
- type = OptionValue.OptionType.SESSION;
- break;
-case "system":
- type = OptionValue.OptionType.SYSTEM;
- break;
-//case "query":
-// type = OptionValue.OptionType.QUERY;
-// break;
-default:
- throw new ValidationException("Invalid OPTION scope. Scope must
be SESSION or SYSTEM.");
+ case "session":
+type = OptionType.SESSION;
+break;
+ case "system":
+type = OptionType.SYSTEM;
+break;
+ default:
+throw UserException.validationError()
+.message("Invalid OPTION scope %s. Scope must be SESSION or
SYSTEM.", scope)
+.build(logger);
}
+}
- if (type == OptionType.SYSTEM) {
-// If the user authentication is enabled, make sure the user who
is trying to change the system option has
-// administrative privileges.
-if (context.isUserAuthenticationEnabled() &&
-!ImpersonationUtil.hasAdminPrivileges(
-context.getQueryUserName(),
-
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
-
context.getOptions().getOption(ExecConstants.ADMIN_USER_GROUPS_KEY).string_val))
{
- throw UserException.permissionError()
- .message("Not authorized to change SYSTEM options.")
- .build(logger);
-}
+if (type == OptionType.SYSTEM) {
+ // If the user authentication is enabled, make sure the user who is
trying to change the system option has
+ // administrative privileges.
+ if (context.isUserAuthenticationEnabled() &&
+ !ImpersonationUtil.hasAdminPrivileges(
+context.getQueryUserName(),
+
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
+
context.getOptions().getOption(ExecConstants.ADMIN_USER_GROUPS_KEY).string_val))
{
+throw UserException.permissionError()
+.message("Not authorized to change SYSTEM options.")
+.build(logger);
}
+}
+
+// Currently, we use one part identifiers.
+final SqlIdentifier nameIdentifier = option.getName();
+if (!nameIdentifier.isSimple()) {
+ throw UserException.validationError()
+.message("Drill does not support multi-part identifier for an
option name (%s).",
+ nameIdentifier.toString())
--- End diff --
this is not important, but you can just pass nameIdentifier and .toString()
will be called implicitly
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901004#comment-14901004
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39996968
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
abstract boolean setLocalOption(OptionValue value);
+ /**
+ * Deletes the option with given name for this manager without falling
back.
+ *
+ * @param type option type
+ * @return true iff the option was successfully deleted
--- End diff --
actually it's less confusing here because we only pass an `OptionType` to
the method but it's more prevalent for `deleteLocalOption()`
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901005#comment-14901005
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39997032
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
abstract boolean setLocalOption(OptionValue value);
+ /**
+ * Deletes the option with given name for this manager without falling
back.
+ *
+ * @param type option type
+ * @return true iff the option was successfully deleted
+ */
+ abstract boolean deleteLocalOptions(OptionType type);
--- End diff --
can you rename this to `deleteAllLocalOptions` ?
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901001#comment-14901001
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39996879
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
abstract boolean setLocalOption(OptionValue value);
+ /**
+ * Deletes the option with given name for this manager without falling
back.
+ *
+ * @param type option type
+ * @return true iff the option was successfully deleted
+ */
+ abstract boolean deleteLocalOptions(OptionType type);
+
+ /**
+ * Deletes the option with given name for this manager without falling
back.
+ *
+ * @param name option name
+ * @param type option type
+ * @return true iff the option was successfully deleted
--- End diff --
same here
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901000#comment-14901000
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39996854
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -83,14 +84,30 @@ public OptionValue getOption(final String name) {
*/
abstract boolean setLocalOption(OptionValue value);
+ /**
+ * Deletes the option with given name for this manager without falling
back.
+ *
+ * @param type option type
+ * @return true iff the option was successfully deleted
--- End diff --
this javadoc can have multiple meanings: _successfully deleted_ could
either mean the implementation couldn't find the option, or it doesn't support
the option type. It should actually return false if it doesn't support the
option type
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900997#comment-14900997
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39996569
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/FallbackOptionManager.java
---
@@ -102,6 +119,29 @@ public void setOption(OptionValue value) {
}
@Override
+ public void deleteOption(final String name, final OptionType type) {
+try {
+ SystemOptionManager.getValidator(name); // ensure the option exists
+} catch (final IllegalArgumentException e) {
+ throw UserException.validationError(e)
+.build(logger);
+}
+
+// fallback if unable to delete locally
+if (!deleteLocalOption(name, type)) {
--- End diff --
you should call `name.toLowerCase()` here to make sure implementation will
always be case insensitive.
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[ https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900994#comment-14900994 ] ASF GitHub Bot commented on DRILL-1065: --- Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/159#discussion_r39996432 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java --- @@ -161,6 +171,7 @@ public SqlNode visitChild( rules.put(SqlJoin.class, R(D, D, D, D, D, E)); rules.put(SqlOrderBy.class, R(D, E, D, D)); rules.put(SqlDropTable.class, R(D)); +rules.put(SqlSetOption.class, R(D, D, D)); --- End diff -- can you add a comment explaining why this rule is needed ? > Provide a reset command to reset an option to its default value > --- > > Key: DRILL-1065 > URL: https://issues.apache.org/jira/browse/DRILL-1065 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: Aman Sinha >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.2.0 > > > Within a session, currently we set configuration options and it would be very > useful to have a 'reset' command to reset the value of an option to its > default system value: > ALTER SESSION RESET > If we don't want to add a new keyword for RESET, we could potentially > overload the SET command and allow the user to set to the 'default' value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900989#comment-14900989
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39996054
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/SetOptionHandler.java
---
@@ -49,45 +53,67 @@ public SetOptionHandler(QueryContext context) {
}
@Override
- public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException,
RelConversionException, IOException, ForemanSetupException {
+ public PhysicalPlan getPlan(SqlNode sqlNode) throws ValidationException,
ForemanSetupException {
final SqlSetOption option = unwrap(sqlNode, SqlSetOption.class);
-final String scope = option.getScope();
-final String name = option.getName();
final SqlNode value = option.getValue();
-OptionValue.OptionType type;
-if (value instanceof SqlLiteral) {
+if (value != null && !(value instanceof SqlLiteral)) {
+ throw UserException.validationError()
+ .message("Drill does not support assigning non-literal values in
SET statements.")
+ .build(logger);
+}
+
+final String scope = option.getScope();
+final OptionValue.OptionType type;
+if (scope == null) { // No scope mentioned assumed SESSION
+ type = OptionType.SESSION;
+} else {
switch (scope.toLowerCase()) {
-case "session":
- type = OptionValue.OptionType.SESSION;
- break;
-case "system":
- type = OptionValue.OptionType.SYSTEM;
- break;
-//case "query":
-// type = OptionValue.OptionType.QUERY;
-// break;
-default:
- throw new ValidationException("Invalid OPTION scope. Scope must
be SESSION or SYSTEM.");
+ case "session":
+type = OptionType.SESSION;
+break;
+ case "system":
+type = OptionType.SYSTEM;
+break;
+ default:
+throw UserException.validationError()
+.message("Invalid OPTION scope %s. Scope must be SESSION or
SYSTEM.", scope)
+.build(logger);
}
+}
- if (type == OptionType.SYSTEM) {
-// If the user authentication is enabled, make sure the user who
is trying to change the system option has
-// administrative privileges.
-if (context.isUserAuthenticationEnabled() &&
-!ImpersonationUtil.hasAdminPrivileges(
-context.getQueryUserName(),
-
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
-
context.getOptions().getOption(ExecConstants.ADMIN_USER_GROUPS_KEY).string_val))
{
- throw UserException.permissionError()
- .message("Not authorized to change SYSTEM options.")
- .build(logger);
-}
+if (type == OptionType.SYSTEM) {
+ // If the user authentication is enabled, make sure the user who is
trying to change the system option has
+ // administrative privileges.
+ if (context.isUserAuthenticationEnabled() &&
+ !ImpersonationUtil.hasAdminPrivileges(
+context.getQueryUserName(),
+
context.getOptions().getOption(ExecConstants.ADMIN_USERS_KEY).string_val,
--- End diff --
you should reuse `context.getOptions()` as a local variable
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2908) Support reading the Parquet int 96 type
[ https://issues.apache.org/jira/browse/DRILL-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900965#comment-14900965 ] Parth Chandra commented on DRILL-2908: -- Pull request: https://github.com/apache/drill/pull/162 [~adeneche] please review > Support reading the Parquet int 96 type > --- > > Key: DRILL-2908 > URL: https://issues.apache.org/jira/browse/DRILL-2908 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Reporter: Jason Altekruse >Assignee: Jason Altekruse > Fix For: 1.2.0 > > > While Drill does not currently have an int96 type, it is supported by the > parquet format and we should be able to read files that contain columns of > this type. For now we will read the data into a varbinary and users will have > to use existing convert_from functions or write their own to interpret the > type of data actually stored. One example is the Impala timestamp format > which is encoded in an int96 column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value
[
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900825#comment-14900825
]
ASF GitHub Bot commented on DRILL-1065:
---
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/159#discussion_r39984380
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/CompoundIdentifierConverter.java
---
@@ -107,7 +117,7 @@ public SqlNode visitChild(
}
SqlNode newOperand =
operand.accept(CompoundIdentifierConverter.this);
enableComplex = localEnableComplex;
- if (newOperand != operand) {
+ if (! newOperand.equalsDeep(operand, false)) {
--- End diff --
Is it necessary to call equalsDeep()? If the expression has an identifier
which is rewritten by CompoundIdentifierConverter, is it true that the new
expression would be different in reference from then original one?
> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very
> useful to have a 'reset' command to reset the value of an option to its
> default system value:
> ALTER SESSION RESET
> If we don't want to add a new keyword for RESET, we could potentially
> overload the SET command and allow the user to set to the 'default' value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3813) table from directory subtree having no descendent files fails with index error
[
https://issues.apache.org/jira/browse/DRILL-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900824#comment-14900824
]
Deneche A. Hakim commented on DRILL-3813:
-
enabling {{exec.errors.verbose}} confirms it's the same issue:
{noformat}
0: jdbc:drill:zk=local> select * from dfs.tmp.empty;
Error: VALIDATION ERROR: Index: 0, Size: 0
[Error Id: 694cec43-8529-4f59-876a-8084576b380e on 172.30.1.109:31010]
(org.apache.calcite.tools.ValidationException)
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
org.apache.calcite.prepare.PlannerImpl.validate():179
org.apache.calcite.prepare.PlannerImpl.validateAndGetType():188
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():447
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():190
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():159
...
Caused By (java.lang.IndexOutOfBoundsException) Index: 0, Size: 0
java.util.ArrayList.rangeCheck():635
java.util.ArrayList.get():411
org.apache.drill.exec.store.dfs.FileSelection.getFirstPath():100
org.apache.drill.exec.store.dfs.BasicFormatMatcher.isReadable():75
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create():338
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create():153
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry():96
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get():90
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable():276
org.apache.calcite.jdbc.SimpleCalciteSchema.getTable():83
...
{noformat}
> table from directory subtree having no descendent files fails with index error
> --
>
> Key: DRILL-3813
> URL: https://issues.apache.org/jira/browse/DRILL-3813
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser, Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Aman Sinha
>
> Trying to use as a table a directory subtree that has no descendent files
> (but zero or more descendent directories) yields what seems to be a partially
> handled index out-of-bounds condition.
> For example, with {{/tmp/empty_directory}} being an empty directory:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> Also, with {{/tmp/no_child_files_subtree}} having two child directories and a
> grandchild directory, but not descendent files:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> A directory subtree having no files was expected to be taken as a table with
> no rows (and a null schema).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3813) table from directory subtree having no descendent files fails with index error
[
https://issues.apache.org/jira/browse/DRILL-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-3813:
Description:
Trying to use as a table a directory subtree that has no descendent files (but
zero or more descendent directories) yields what seems to be a partially
handled index out-of-bounds condition.
For example, with {{/tmp/empty_directory}} being an empty directory:
{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0
[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010]
(state=,code=0)
0: jdbc:drill:zk=local>
{noformat}
Also, with {{/tmp/no_child_files_subtree}} having two child directories and a
grandchild directory, but not descendent files:
{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0
[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010]
(state=,code=0)
0: jdbc:drill:zk=local>
{noformat}
A directory subtree having no files was expected to be taken as a table with no
rows (and a null schema).
was:
Trying to use as a table a directory subtree that has no descendent files (but
zero or more descendent directories) yields what seems to be a partially
handled index out-of-bounds condition.
For example, with {{/tmp/empty_directory}} being an empty directory:
{noformat}
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
Error: VALIDATION ERROR: Index: 0, Size: 0
[Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010]
(state=,code=0)
0: jdbc:drill:zk=local>
{noformat}
Also, with {{/tmp/no_child_files_subtree}} having two child directories and a
grandchild directory, but not descendent files:
0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
Error: VALIDATION ERROR: Index: 0, Size: 0
[Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010]
(state=,code=0)
0: jdbc:drill:zk=local>
{noformat}
A directory subtree having no files was expected to be taken as a table with no
rows (and a null schema).
> table from directory subtree having no descendent files fails with index error
> --
>
> Key: DRILL-3813
> URL: https://issues.apache.org/jira/browse/DRILL-3813
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser, Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Aman Sinha
>
> Trying to use as a table a directory subtree that has no descendent files
> (but zero or more descendent directories) yields what seems to be a partially
> handled index out-of-bounds condition.
> For example, with {{/tmp/empty_directory}} being an empty directory:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> Also, with {{/tmp/no_child_files_subtree}} having two child directories and a
> grandchild directory, but not descendent files:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> A directory subtree having no files was expected to be taken as a table with
> no rows (and a null schema).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3813) table from directory subtree having no descendent files fails with index error
[
https://issues.apache.org/jira/browse/DRILL-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900790#comment-14900790
]
Deneche A. Hakim commented on DRILL-3813:
-
This could be related to DRILL-2618
> table from directory subtree having no descendent files fails with index error
> --
>
> Key: DRILL-3813
> URL: https://issues.apache.org/jira/browse/DRILL-3813
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser, Storage - Other
>Reporter: Daniel Barclay (Drill)
>Assignee: Aman Sinha
>
> Trying to use as a table a directory subtree that has no descendent files
> (but zero or more descendent directories) yields what seems to be a partially
> handled index out-of-bounds condition.
> For example, with {{/tmp/empty_directory}} being an empty directory:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`empty_directory`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 747425c9-5350-4813-9f0d-ecf580e15101 on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> Also, with {{/tmp/no_child_files_subtree}} having two child directories and a
> grandchild directory, but not descendent files:
> 0: jdbc:drill:zk=local> SELECT * FROM `dfs`.`tmp`.`no_child_files_subtree`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: abc90424-8434-4403-b44b-0ba69ef43151 on dev-linux2:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
> {noformat}
> A directory subtree having no files was expected to be taken as a table with
> no rows (and a null schema).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
