[jira] [Created] (DRILL-6701) Non-admin users can access options page via restapi
Krystal created DRILL-6701: -- Summary: Non-admin users can access options page via restapi Key: DRILL-6701 URL: https://issues.apache.org/jira/browse/DRILL-6701 Project: Apache Drill Issue Type: Bug Components: Security Affects Versions: 1.14.0 Reporter: Krystal Only admin users should be able to access drill's options page. However via restapi, a non-admin user can access the options page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6700) Graceful shutdown from command line against parquet tables fails
Krystal created DRILL-6700: -- Summary: Graceful shutdown from command line against parquet tables fails Key: DRILL-6700 URL: https://issues.apache.org/jira/browse/DRILL-6700 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Reporter: Krystal Attachments: drillbit.log Trigger a graceful shutdown from the command line while a query against parquet table is running would fail with the following error: "Error: SYSTEM ERROR: IOException: Filesystem closed Fragment 5:11" This does not occur if graceful shutdown is triggered from the webUI. Attached is the drillbit.log. [^drillbit.log] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6695) Graceful shutdown removes spill directory before query finished
[ https://issues.apache.org/jira/browse/DRILL-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584120#comment-16584120 ] Krystal commented on DRILL-6695: Additional info: the problem does not exists if graceful shutdown is triggered from a non-foreman. > Graceful shutdown removes spill directory before query finished > > > Key: DRILL-6695 > URL: https://issues.apache.org/jira/browse/DRILL-6695 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.14.0 > Environment: Ran the following query from sqlline: > select a.columns[0], b.columns[1], a.columns[2], b.columns[3], a.columns[4] > from `test` a join `test` b on a.columns[0]=b.columns[0] and > a.columns[4]=b.columns[4] order by a.columns[0] limit 1000; > While the query was running, initiated a graceful shutdown from command line > on the foreman node. The query failed with the following error message: > Error: RESOURCE ERROR: Hash Join failed to open spill file: > /tmp/drill/spill/248a054a-ee63-e795-a44e-d9205df8e9b8_HashJoin_3-2-0/spill7_outer > Fragment 3:0 > Looks like somehow the spill directory gets deleted while query is still > running when graceful_shutdown is initiated. > > >Reporter: Krystal >Priority: Major > Attachments: drillbit.log > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6695) Graceful shutdown removes spill directory before query finished
[ https://issues.apache.org/jira/browse/DRILL-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583151#comment-16583151 ] Krystal commented on DRILL-6695: Forgot to mention that this problem does not occur if graceful shutdown is initiated from the webUI. > Graceful shutdown removes spill directory before query finished > > > Key: DRILL-6695 > URL: https://issues.apache.org/jira/browse/DRILL-6695 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.14.0 > Environment: Ran the following query from sqlline: > select a.columns[0], b.columns[1], a.columns[2], b.columns[3], a.columns[4] > from `test` a join `test` b on a.columns[0]=b.columns[0] and > a.columns[4]=b.columns[4] order by a.columns[0] limit 1000; > While the query was running, initiated a graceful shutdown from command line > on the foreman node. The query failed with the following error message: > Error: RESOURCE ERROR: Hash Join failed to open spill file: > /tmp/drill/spill/248a054a-ee63-e795-a44e-d9205df8e9b8_HashJoin_3-2-0/spill7_outer > Fragment 3:0 > Looks like somehow the spill directory gets deleted while query is still > running when graceful_shutdown is initiated. > > >Reporter: Krystal >Priority: Major > Attachments: drillbit.log > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6695) Graceful shutdown removes spill directory before query finished
[ https://issues.apache.org/jira/browse/DRILL-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-6695: --- Attachment: drillbit.log > Graceful shutdown removes spill directory before query finished > > > Key: DRILL-6695 > URL: https://issues.apache.org/jira/browse/DRILL-6695 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.14.0 > Environment: Ran the following query from sqlline: > select a.columns[0], b.columns[1], a.columns[2], b.columns[3], a.columns[4] > from `test` a join `test` b on a.columns[0]=b.columns[0] and > a.columns[4]=b.columns[4] order by a.columns[0] limit 1000; > While the query was running, initiated a graceful shutdown from command line > on the foreman node. The query failed with the following error message: > Error: RESOURCE ERROR: Hash Join failed to open spill file: > /tmp/drill/spill/248a054a-ee63-e795-a44e-d9205df8e9b8_HashJoin_3-2-0/spill7_outer > Fragment 3:0 > Looks like somehow the spill directory gets deleted while query is still > running when graceful_shutdown is initiated. > > >Reporter: Krystal >Priority: Major > Attachments: drillbit.log > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6695) Graceful shutdown removes spill directory before query finished
[ https://issues.apache.org/jira/browse/DRILL-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583147#comment-16583147 ] Krystal commented on DRILL-6695: Here is the log file: [^drillbit.log] > Graceful shutdown removes spill directory before query finished > > > Key: DRILL-6695 > URL: https://issues.apache.org/jira/browse/DRILL-6695 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.14.0 > Environment: Ran the following query from sqlline: > select a.columns[0], b.columns[1], a.columns[2], b.columns[3], a.columns[4] > from `test` a join `test` b on a.columns[0]=b.columns[0] and > a.columns[4]=b.columns[4] order by a.columns[0] limit 1000; > While the query was running, initiated a graceful shutdown from command line > on the foreman node. The query failed with the following error message: > Error: RESOURCE ERROR: Hash Join failed to open spill file: > /tmp/drill/spill/248a054a-ee63-e795-a44e-d9205df8e9b8_HashJoin_3-2-0/spill7_outer > Fragment 3:0 > Looks like somehow the spill directory gets deleted while query is still > running when graceful_shutdown is initiated. > > >Reporter: Krystal >Priority: Major > Attachments: drillbit.log > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6695) Graceful shutdown removes spill directory before query finished
Krystal created DRILL-6695: -- Summary: Graceful shutdown removes spill directory before query finished Key: DRILL-6695 URL: https://issues.apache.org/jira/browse/DRILL-6695 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.14.0 Environment: Ran the following query from sqlline: select a.columns[0], b.columns[1], a.columns[2], b.columns[3], a.columns[4] from `test` a join `test` b on a.columns[0]=b.columns[0] and a.columns[4]=b.columns[4] order by a.columns[0] limit 1000; While the query was running, initiated a graceful shutdown from command line on the foreman node. The query failed with the following error message: Error: RESOURCE ERROR: Hash Join failed to open spill file: /tmp/drill/spill/248a054a-ee63-e795-a44e-d9205df8e9b8_HashJoin_3-2-0/spill7_outer Fragment 3:0 Looks like somehow the spill directory gets deleted while query is still running when graceful_shutdown is initiated. Reporter: Krystal -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6690) Non-admin users can access threads page using restAPI
[
https://issues.apache.org/jira/browse/DRILL-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal updated DRILL-6690:
---
Description:
Through restapi, non-admin users can access drill threads data:
{code:java}
curl -b ~/.drill_cookies -k -H "Content-Type: application/json" -X GET
https://10.10.10.000:8047/status/threads
"Reference Handler" id=2 state=WAITING
- waiting on <0x15c20b08> (a java.lang.ref.Reference$Lock)
- locked <0x15c20b08> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
{code}
was:
Through restapi, non-admin users can access drill threads data:
{code:java}
curl -b ~/.drill_cookies -k -H "Content-Type: application/json" -X GET
https://10.10.30.206:8047/status/threads
"Reference Handler" id=2 state=WAITING
- waiting on <0x15c20b08> (a java.lang.ref.Reference$Lock)
- locked <0x15c20b08> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
{code}
> Non-admin users can access threads page using restAPI
> -
>
> Key: DRILL-6690
> URL: https://issues.apache.org/jira/browse/DRILL-6690
> Project: Apache Drill
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.14.0
>Reporter: Krystal
>Priority: Major
>
> Through restapi, non-admin users can access drill threads data:
> {code:java}
> curl -b ~/.drill_cookies -k -H "Content-Type: application/json" -X GET
> https://10.10.10.000:8047/status/threads
> "Reference Handler" id=2 state=WAITING
> - waiting on <0x15c20b08> (a java.lang.ref.Reference$Lock)
> - locked <0x15c20b08> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6690) Non-admin users can access threads page using restAPI
[
https://issues.apache.org/jira/browse/DRILL-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal updated DRILL-6690:
---
Description:
Through restapi, non-admin users can access drill threads data:
{code:java}
curl -b ~/.drill_cookies -k -H "Content-Type: application/json" -X GET
https://10.10.30.206:8047/status/threads
"Reference Handler" id=2 state=WAITING
- waiting on <0x15c20b08> (a java.lang.ref.Reference$Lock)
- locked <0x15c20b08> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
{code}
was:
Through restapi, non-admin users can access drill metrics data:
{code:java}
[root@mfs41 ~]# curl -b ~/.drill_cookies -k -H "Content-Type: application/json"
-X GET https://10.10.10.000:8047/status/metrics
{"version":"4.0.0","gauges":{"G1-Old-Generation.count":{"value":0},"G1-Old-Generation.time":{"value":0},"G1-Young-Generation.count":{"value":8},"G1-Young-Generation.time":{"value":329},"blocked.count":{"value":0},"count":{"value":28},"daemon.count":{"value":19},...{code}
> Non-admin users can access threads page using restAPI
> -
>
> Key: DRILL-6690
> URL: https://issues.apache.org/jira/browse/DRILL-6690
> Project: Apache Drill
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.14.0
>Reporter: Krystal
>Priority: Major
>
> Through restapi, non-admin users can access drill threads data:
> {code:java}
> curl -b ~/.drill_cookies -k -H "Content-Type: application/json" -X GET
> https://10.10.30.206:8047/status/threads
> "Reference Handler" id=2 state=WAITING
> - waiting on <0x15c20b08> (a java.lang.ref.Reference$Lock)
> - locked <0x15c20b08> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6690) Non-admin users can access threads page using restAPI
[
https://issues.apache.org/jira/browse/DRILL-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal updated DRILL-6690:
---
Summary: Non-admin users can access threads page using restAPI (was:
Non-admin users can access metrics page using restAPI)
> Non-admin users can access threads page using restAPI
> -
>
> Key: DRILL-6690
> URL: https://issues.apache.org/jira/browse/DRILL-6690
> Project: Apache Drill
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.14.0
>Reporter: Krystal
>Priority: Major
>
> Through restapi, non-admin users can access drill metrics data:
> {code:java}
> [root@mfs41 ~]# curl -b ~/.drill_cookies -k -H "Content-Type:
> application/json" -X GET https://10.10.10.000:8047/status/metrics
> {"version":"4.0.0","gauges":{"G1-Old-Generation.count":{"value":0},"G1-Old-Generation.time":{"value":0},"G1-Young-Generation.count":{"value":8},"G1-Young-Generation.time":{"value":329},"blocked.count":{"value":0},"count":{"value":28},"daemon.count":{"value":19},...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6690) Non-admin users can access metrics page using restAPI
[
https://issues.apache.org/jira/browse/DRILL-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal updated DRILL-6690:
---
Description:
Through restapi, non-admin users can access drill metrics data:
{code:java}
[root@mfs41 ~]# curl -b ~/.drill_cookies -k -H "Content-Type: application/json"
-X GET https://10.10.10.000:8047/status/metrics
{"version":"4.0.0","gauges":{"G1-Old-Generation.count":{"value":0},"G1-Old-Generation.time":{"value":0},"G1-Young-Generation.count":{"value":8},"G1-Young-Generation.time":{"value":329},"blocked.count":{"value":0},"count":{"value":28},"daemon.count":{"value":19},...{code}
was:
Through restapi, non-admin users can access drill metrics data:
{code:java}
[root@mfs41 ~]# curl -b ~/.drill_cookies -k -H "Content-Type: application/json"
-X GET https://10.10.30.206:8047/status/metrics
{"version":"4.0.0","gauges":{"G1-Old-Generation.count":{"value":0},"G1-Old-Generation.time":{"value":0},"G1-Young-Generation.count":{"value":8},"G1-Young-Generation.time":{"value":329},"blocked.count":{"value":0},"count":{"value":28},"daemon.count":{"value":19},...{code}
> Non-admin users can access metrics page using restAPI
> -
>
> Key: DRILL-6690
> URL: https://issues.apache.org/jira/browse/DRILL-6690
> Project: Apache Drill
> Issue Type: Bug
> Components: Security
>Affects Versions: 1.14.0
>Reporter: Krystal
>Priority: Major
>
> Through restapi, non-admin users can access drill metrics data:
> {code:java}
> [root@mfs41 ~]# curl -b ~/.drill_cookies -k -H "Content-Type:
> application/json" -X GET https://10.10.10.000:8047/status/metrics
> {"version":"4.0.0","gauges":{"G1-Old-Generation.count":{"value":0},"G1-Old-Generation.time":{"value":0},"G1-Young-Generation.count":{"value":8},"G1-Young-Generation.time":{"value":329},"blocked.count":{"value":0},"count":{"value":28},"daemon.count":{"value":19},...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6690) Non-admin users can access metrics page using restAPI
Krystal created DRILL-6690:
--
Summary: Non-admin users can access metrics page using restAPI
Key: DRILL-6690
URL: https://issues.apache.org/jira/browse/DRILL-6690
Project: Apache Drill
Issue Type: Bug
Components: Security
Affects Versions: 1.14.0
Reporter: Krystal
Through restapi, non-admin users can access drill metrics data:
{code:java}
[root@mfs41 ~]# curl -b ~/.drill_cookies -k -H "Content-Type: application/json"
-X GET https://10.10.30.206:8047/status/metrics
{"version":"4.0.0","gauges":{"G1-Old-Generation.count":{"value":0},"G1-Old-Generation.time":{"value":0},"G1-Young-Generation.count":{"value":8},"G1-Young-Generation.time":{"value":329},"blocked.count":{"value":0},"count":{"value":28},"daemon.count":{"value":19},...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (DRILL-6039) drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit
[ https://issues.apache.org/jira/browse/DRILL-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal reopened DRILL-6039: > drillbit.sh graceful_stop does not wait for fragments to complete before > stopping the drillbit > -- > > Key: DRILL-6039 > URL: https://issues.apache.org/jira/browse/DRILL-6039 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.3.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > git.commit.id.abbrev=eb0c403 > I have 3-nodes cluster with drillbits running on each node. I kicked off a > long running query. In the middle of the query, I did a "./drillbit.sh > graceful_stop" on one of the non-foreman node. The node was stopped within a > few seconds and the query failed with error: > Error: SYSTEM ERROR: IOException: Filesystem closed > Fragment 4:15 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6039) drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit
[
https://issues.apache.org/jira/browse/DRILL-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514293#comment-16514293
]
Krystal commented on DRILL-6039:
The "drillbit.sh graceful_stop" from command line against parquet files still
fails - does not wait for fragments to finish. Interesting thing is the
problem does not occur when shutting down the drillbit from the WebUI. The
drillbit.log does not show any memory leaks. Here is the stack trace:
{code:java}
Error: SYSTEM ERROR: IOException: Filesystem closed
Fragment 1:10
(org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
record reader.
Message: Failure in setting up reader
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 l_orderkey;
optional int64 l_partkey;
optional int64 l_suppkey;
optional int32 l_linenumber;
optional double l_quantity;
optional double l_extendedprice;
optional double l_discount;
optional double l_tax;
optional binary l_returnflag (UTF8);
optional binary l_linestatus (UTF8);
optional int32 l_shipdate (DATE);
optional int32 l_commitdate (DATE);
optional int32 l_receiptdate (DATE);
optional binary l_shipinstruct (UTF8);
optional binary l_shipmode (UTF8);
optional binary l_comment (UTF8);
}
, metadata: {drill.version=1.7.0-SNAPSHOT}}, blocks: [BlockMetaData{9785551,
1338587809 [ColumnMetaData{SNAPPY [l_orderkey] INT64 [PLAIN, BIT_PACKED, RLE],
4}, ColumnMetaData{SNAPPY [l_partkey] INT64 [PLAIN, BIT_PACKED, RLE],
15273019}, ColumnMetaData{SNAPPY [l_suppkey] INT64 [PLAIN, BIT_PACKED, RLE],
73277460}, ColumnMetaData{SNAPPY [l_linenumber] INT32 [PLAIN, BIT_PACKED,
RLE], 124321400}, ColumnMetaData{SNAPPY [l_quantity] DOUBLE [PLAIN,
BIT_PACKED, RLE], 132087986}, ColumnMetaData{SNAPPY [l_extendedprice] DOUBLE
[PLAIN, BIT_PACKED, RLE], 151838465}, ColumnMetaData{SNAPPY [l_discount] DOUBLE
[PLAIN, BIT_PACKED, RLE], 208270450}, ColumnMetaData{SNAPPY [l_tax] DOUBLE
[PLAIN, BIT_PACKED, RLE], 227351535}, ColumnMetaData{SNAPPY [l_returnflag]
BINARY [PLAIN, BIT_PACKED, RLE], 245574230}, ColumnMetaData{SNAPPY
[l_linestatus] BINARY [PLAIN, BIT_PACKED, RLE], 254814472},
ColumnMetaData{SNAPPY [l_shipdate] INT32 [PLAIN, BIT_PACKED, RLE], 260500185},
ColumnMetaData{SNAPPY [l_commitdate] INT32 [PLAIN, BIT_PACKED, RLE],
290097700}, ColumnMetaData{SNAPPY [l_receiptdate] INT32 [PLAIN, BIT_PACKED,
RLE], 319358270}, ColumnMetaData{SNAPPY [l_shipinstruct] BINARY [PLAIN,
BIT_PACKED, RLE], 348982057}, ColumnMetaData{SNAPPY [l_shipmode] BINARY
[PLAIN, BIT_PACKED, RLE], 370125048}, ColumnMetaData{SNAPPY [l_comment] BINARY
[PLAIN, BIT_PACKED, RLE], 392116052}]}]}
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException():316
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.setup():300
org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas():335
org.apache.drill.exec.physical.impl.ScanBatch.internalNext():222
org.apache.drill.exec.physical.impl.ScanBatch.next():274
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():80
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.test.generated.StreamingAggregatorGen42.doWork():187
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():194
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():233
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1633
org.apache.drill.exec.work.fragment.FragmentExecutor.run
[jira] [Closed] (DRILL-3584) Drill Kerberos HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-3584. -- Resolution: Fixed > Drill Kerberos HDFS Support / Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Critical > Labels: security > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472764#comment-16472764 ] Krystal commented on DRILL-3584: Tested the steps mentioned by Sudheesh against cloudera cdh 5.13 vm and got drill working with Kerberos enabled. > Drill Kerberos HDFS Support / Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Critical > Labels: security > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6292) Improve error message when NTLM token is sent to server using SPNEGO
[
https://issues.apache.org/jira/browse/DRILL-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448476#comment-16448476
]
Krystal commented on DRILL-6292:
We can check that if the header of the response starts with "Negotiate TIR"
then it's doing spnego over NTLM instead of over Kerberos.
> Improve error message when NTLM token is sent to server using SPNEGO
>
>
> Key: DRILL-6292
> URL: https://issues.apache.org/jira/browse/DRILL-6292
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - Java
>Reporter: Krystal
>Priority: Major
>
> The following error message is logged in drillbit.log file when the brower
> sends NTLM token instead of SPNEGO token.
> {color:#00}WARN o.a.d.e.s.r.a.DrillSpnegoLoginService - Caught
> GSSException trying to authenticate the client org.ietf.jgss.GSSException:
> Defective token detected (Mechanism level: GSSHeader did not find the right
> tag){color}
> Should have a clearer error that indicates the token sent is of NTLM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6306) Should not be able to run queries against disabled storage plugins
Krystal created DRILL-6306: -- Summary: Should not be able to run queries against disabled storage plugins Key: DRILL-6306 URL: https://issues.apache.org/jira/browse/DRILL-6306 Project: Apache Drill Issue Type: Bug Components: Storage - Other Affects Versions: 1.13.0 Reporter: Krystal Currently, queries against disabled storage plugins are returning data. This should not be the case. Queries against disabled storage plugins should fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6292) Improve error message when NTLM token is sent to server using SPNEGO
Krystal created DRILL-6292:
--
Summary: Improve error message when NTLM token is sent to server
using SPNEGO
Key: DRILL-6292
URL: https://issues.apache.org/jira/browse/DRILL-6292
Project: Apache Drill
Issue Type: Improvement
Components: Client - Java
Reporter: Krystal
The following error message is logged in drillbit.log file when the brower
sends NTLM token instead of SPNEGO token.
{color:#00}WARN o.a.d.e.s.r.a.DrillSpnegoLoginService - Caught GSSException
trying to authenticate the client org.ietf.jgss.GSSException: Defective token
detected (Mechanism level: GSSHeader did not find the right tag){color}
Should have a clearer error that indicates the token sent is of NTLM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6021) Show shutdown button when authentication is not enabled
[
https://issues.apache.org/jira/browse/DRILL-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-6021.
--
Verified that bug is fixed.
> Show shutdown button when authentication is not enabled
> ---
>
> Key: DRILL-6021
> URL: https://issues.apache.org/jira/browse/DRILL-6021
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> After DRILL-6017 {{shouldShowAdminInfo}} is used to decide if shutdown button
> should be displayed on index page. But this option is set to true when
> authentication is enabled and user is an admin. When authentication is not
> enabled, user by default is admin. So with this fix without authentication,
> shutdown button is absent but should be present.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6044) Shutdown button does not work from WebUI
[ https://issues.apache.org/jira/browse/DRILL-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-6044. -- Verified that bug is fixed. > Shutdown button does not work from WebUI > > > Key: DRILL-6044 > URL: https://issues.apache.org/jira/browse/DRILL-6044 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 1.13.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Critical > Labels: ready-to-commit > Fix For: 1.13.0 > > Attachments: Screen Shot 2017-12-19 at 10.51.16 AM.png > > > git.commit.id.abbrev=eb0c403 > Nothing happens when click on the SHUTDOWN button from the WebUI. The > browser's debugger showed that the request failed due to access control > checks (see attached screen shot). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh
[ https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397308#comment-16397308 ] Krystal commented on DRILL-6040: git.commit.id.abbrev=cac2882 Verified that bug is fixed. /opt/drill/bin/drillbit.sh Usage: drillbit.sh [--config|--site ] (start|stop|status|restart|run|graceful_stop) [args] > Need to add usage for graceful_stop to drillbit.sh > -- > > Key: DRILL-6040 > URL: https://issues.apache.org/jira/browse/DRILL-6040 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.13.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > > git.commit.id.abbrev=eb0c403 > Usage for graceful_stop is missing from drillbit.sh. > ./drillbit.sh > Usage: drillbit.sh [--config|--site ] > (start|stop|status|restart|run) [args] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh
[ https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-6040. -- Verified that bug is fixed. > Need to add usage for graceful_stop to drillbit.sh > -- > > Key: DRILL-6040 > URL: https://issues.apache.org/jira/browse/DRILL-6040 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.13.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > > git.commit.id.abbrev=eb0c403 > Usage for graceful_stop is missing from drillbit.sh. > ./drillbit.sh > Usage: drillbit.sh [--config|--site ] > (start|stop|status|restart|run) [args] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6017) Fix for SHUTDOWN button being visible for non Admin users
[ https://issues.apache.org/jira/browse/DRILL-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-6017. -- Verified that only Admin users can access the SHUTDOWN button in a secured cluster. > Fix for SHUTDOWN button being visible for non Admin users > - > > Key: DRILL-6017 > URL: https://issues.apache.org/jira/browse/DRILL-6017 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Arina Ielchiieva >Assignee: Karthikeyan Manivannan >Priority: Blocker > Labels: ready-to-commit > Fix For: 1.12.0 > > > DRILL-4286 introduces shutdown button on index page but when authorization is > enabled it should be visible only to admin users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6006) Label current is missing on Web UI since the same drillbits with different status are considered to be different
[
https://issues.apache.org/jira/browse/DRILL-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-6006.
--
Verified that bug is fixed.
> Label current is missing on Web UI since the same drillbits with different
> status are considered to be different
>
>
> Key: DRILL-6006
> URL: https://issues.apache.org/jira/browse/DRILL-6006
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.12.0
>
>
> After DRILL-4286 label current is missing on Web UI indicating current
> drillbit.
> This happens because when creating DrillbitInfo object, we check [if current
> drillbit is the same as
> available|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java#L197].
> But {{dbContext.getEndpoint()}} returns drillbit with status {{startup}} and
> {{work.getContext().getAvailableBits()}} returns drillbit with status
> {{online}}, thus isCurrent variable is set as false but should be true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6019) Only admin should be able to access shutdown resources
[ https://issues.apache.org/jira/browse/DRILL-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-6019. -- Verified that bug is fixed. > Only admin should be able to access shutdown resources > -- > > Key: DRILL-6019 > URL: https://issues.apache.org/jira/browse/DRILL-6019 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Blocker > Labels: ready-to-commit > Fix For: 1.12.0 > > > DRILL-4286 introduces graceful shutdown but only admin should be able to > access shutdown resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6044) Shutdown button does not work from WebUI
[ https://issues.apache.org/jira/browse/DRILL-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-6044: --- Attachment: Screen Shot 2017-12-19 at 10.51.16 AM.png > Shutdown button does not work from WebUI > > > Key: DRILL-6044 > URL: https://issues.apache.org/jira/browse/DRILL-6044 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 1.13.0 >Reporter: Krystal > Attachments: Screen Shot 2017-12-19 at 10.51.16 AM.png > > > git.commit.id.abbrev=eb0c403 > Nothing happens when click on the SHUTDOWN button from the WebUI. The > browser's debugger showed that the request failed due to access control > checks (see attached screen shot). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-6044) Shutdown button does not work from WebUI
Krystal created DRILL-6044: -- Summary: Shutdown button does not work from WebUI Key: DRILL-6044 URL: https://issues.apache.org/jira/browse/DRILL-6044 Project: Apache Drill Issue Type: Bug Components: Client - HTTP Affects Versions: 1.13.0 Reporter: Krystal git.commit.id.abbrev=eb0c403 Nothing happens when click on the SHUTDOWN button from the WebUI. The browser's debugger showed that the request failed due to access control checks (see attached screen shot). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh
Krystal created DRILL-6040: -- Summary: Need to add usage for graceful_stop to drillbit.sh Key: DRILL-6040 URL: https://issues.apache.org/jira/browse/DRILL-6040 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.13.0 Reporter: Krystal git.commit.id.abbrev=eb0c403 Usage for graceful_stop is missing from drillbit.sh. ./drillbit.sh Usage: drillbit.sh [--config|--site ] (start|stop|status|restart|run) [args] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-6039) drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit
Krystal created DRILL-6039: -- Summary: drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit Key: DRILL-6039 URL: https://issues.apache.org/jira/browse/DRILL-6039 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.3.0 Reporter: Krystal git.commit.id.abbrev=eb0c403 I have 3-nodes cluster with drillbits running on each node. I kicked off a long running query. In the middle of the query, I did a "./drillbit.sh graceful_stop" on one of the non-foreman node. The node was stopped within a few seconds and the query failed with error: Error: SYSTEM ERROR: IOException: Filesystem closed Fragment 4:15 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5996) Unable to re-run queries from Profiles tab with impersonation and without authentication
Krystal created DRILL-5996: -- Summary: Unable to re-run queries from Profiles tab with impersonation and without authentication Key: DRILL-5996 URL: https://issues.apache.org/jira/browse/DRILL-5996 Project: Apache Drill Issue Type: Bug Components: Client - CLI Affects Versions: 1.12.0 Reporter: Krystal On drillbit, enable impersonation without authentication. From WebUI, click on a query from Profiles link -> click on the "Edit Query" tab -> click on the "Re-Run Query" button. The query would fail with the following error: "HTTP ERROR 412 Problem accessing /query. Reason: User-Name header is not set" We should also provide a username dialog box for this tab. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5873) Drill C++ Client should throw proper/complete error message for the ODBC driver to consume
Krystal created DRILL-5873: -- Summary: Drill C++ Client should throw proper/complete error message for the ODBC driver to consume Key: DRILL-5873 URL: https://issues.apache.org/jira/browse/DRILL-5873 Project: Apache Drill Issue Type: Bug Components: Client - C++ Reporter: Krystal Assignee: Parth Chandra The Drill C++ Client should throw a proper/complete error message for the driver to utilize. The ODBC driver is directly outputting the exception message thrown by the client by calling the getError() API after the connect() API has failed with an error status. For the Java client, similar logic is hard coded at https://github.com/apache/drill/blob/1.11.0/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java#L247. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (DRILL-5034) Select timestamp from hive generated parquet always return in UTC
[
https://issues.apache.org/jira/browse/DRILL-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-5034.
--
Verified
> Select timestamp from hive generated parquet always return in UTC
> -
>
> Key: DRILL-5034
> URL: https://issues.apache.org/jira/browse/DRILL-5034
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Krystal
>Assignee: Vitalii Diravka
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.10.0
>
>
> commit id: 5cea9afa6278e21574c6a982ae5c3d82085ef904
> Reading timestamp data against a hive parquet table from drill automatically
> converts the timestamp data to UTC.
> {code}
> SELECT TIMEOFDAY() FROM (VALUES(1));
> +--+
> |EXPR$0|
> +--+
> | 2016-11-10 12:33:26.547 America/Los_Angeles |
> +--+
> {code}
> data schema:
> {code}
> message hive_schema {
> optional int32 voter_id;
> optional binary name (UTF8);
> optional int32 age;
> optional binary registration (UTF8);
> optional fixed_len_byte_array(3) contributions (DECIMAL(6,2));
> optional int32 voterzone;
> optional int96 create_timestamp;
> optional int32 create_date (DATE);
> }
> {code}
> Using drill-1.8, the returned timestamps match the table data:
> {code}
> select convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from
> `/user/hive/warehouse/voter_hive_parquet` limit 5;
> ++
> | EXPR$0 |
> ++
> | 2016-10-23 20:03:58.0 |
> | null |
> | 2016-09-09 12:01:18.0 |
> | 2017-03-06 20:35:55.0 |
> | 2017-01-20 22:32:43.0 |
> ++
> 5 rows selected (1.032 seconds)
> {code}
> If the user timzone is changed to UTC, then the timestamp data is returned in
> UTC time.
> Using drill-1.9, the returned timestamps got converted to UTC eventhough the
> user timezone is in PST.
> {code}
> select convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from
> dfs.`/user/hive/warehouse/voter_hive_parquet` limit 5;
> ++
> | EXPR$0 |
> ++
> | 2016-10-24 03:03:58.0 |
> | null |
> | 2016-09-09 19:01:18.0 |
> | 2017-03-07 04:35:55.0 |
> | 2017-01-21 06:32:43.0 |
> ++
> {code}
> {code}
> alter session set `store.parquet.reader.int96_as_timestamp`=true;
> +---+---+
> | ok | summary |
> +---+---+
> | true | store.parquet.reader.int96_as_timestamp updated. |
> +---+---+
> select create_timestamp from dfs.`/user/hive/warehouse/voter_hive_parquet`
> limit 5;
> ++
> |create_timestamp|
> ++
> | 2016-10-24 03:03:58.0 |
> | null |
> | 2016-09-09 19:01:18.0 |
> | 2017-03-07 04:35:55.0 |
> | 2017-01-21 06:32:43.0 |
> ++
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-4864) Add ANSI format for date/time functions
[ https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4864. -- Verified bug and added test cases to automation. > Add ANSI format for date/time functions > --- > > Key: DRILL-4864 > URL: https://issues.apache.org/jira/browse/DRILL-4864 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Serhii Harnyk >Assignee: Serhii Harnyk > Labels: doc-impacting > Fix For: 1.10.0 > > > The TO_DATE() is exposing the Joda string formatting conventions into the SQL > layer. This is not following SQL conventions used by ANSI and many other > database engines on the market. > Add new UDFs: > * sql_to_date(String, Format), > * sql_to_time(String, Format), > * sql_to_timestamp(String, Format) > that requires Postgres datetime format. > Table of supported Postgres patterns > ||Pattern name||Postgres format > |Full name of day|day > |Day of year|ddd > |Day of month|dd > |Day of week|d > |Name of month|month > |Abr name of month|mon > |Full era name|ee > |Name of day|dy > |Time zone|tz > |Hour 12 |hh > |Hour 12 |hh12 > |Hour 24|hh24 > |Minute of hour|mi > |Second of minute|ss > |Millisecond of minute|ms > |Week of year|ww > |Month|mm > |Halfday am|am > |Year | y > |ref.| > https://www.postgresql.org/docs/8.2/static/functions-formatting.html | > Table of acceptable Postgres pattern modifiers, which may be used in Format > string > ||Description||Pattern|| > |fill mode (suppress padding blanks and zeroes)|fm | > |fixed format global option (see usage notes)|fx | > |translation mode (print localized day and month names based on > lc_messages)|tm | > |spell mode (not yet implemented)|sp| > |ref.| > https://www.postgresql.org/docs/8.2/static/functions-formatting.html| -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5448) ODBC client crashed when user does not have access to text formatted hive table
Krystal created DRILL-5448: -- Summary: ODBC client crashed when user does not have access to text formatted hive table Key: DRILL-5448 URL: https://issues.apache.org/jira/browse/DRILL-5448 Project: Apache Drill Issue Type: Bug Components: Client - C++ Affects Versions: 1.10.0, 1.11.0 Reporter: Krystal While many connections are connecting to the drillbit, odbc client crashed with "Segmentation Fault" while executing a select from a hive table in text format that the user does not have access to. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5437) Ansi format date/time functions fail when escaping double quotes
Krystal created DRILL-5437:
--
Summary: Ansi format date/time functions fail when escaping double
quotes
Key: DRILL-5437
URL: https://issues.apache.org/jira/browse/DRILL-5437
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.11.0
Reporter: Krystal
The following date/time functions fails when escaping double quotes:
sql_to_time
sql_to_date
sql_to_timestamp
select sql_to_date('"2017 March"', '\" Month"\') from (values(1));
Error: FUNCTION ERROR: Error parsing date-time "2017 March" in sql_to_date
function
This works from postgres:
postgres=# select to_date('"2017 March"', '\" Month\"');
to_date
2017-03-01
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-4919) Fix select count(1) / count(*) on csv with header
[
https://issues.apache.org/jira/browse/DRILL-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-4919.
--
Verified bug.
> Fix select count(1) / count(*) on csv with header
> -
>
> Key: DRILL-4919
> URL: https://issues.apache.org/jira/browse/DRILL-4919
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.8.0
>Reporter: F Méthot
>Assignee: Arina Ielchiieva
>Priority: Minor
> Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> This happens since 1.8
> Dataset (I used extended char for display purpose) test.csvh:
> a,b,c,d\n
> 1,2,3,4\n
> 5,6,7,8\n
> Storage config:
> "csvh": {
> "type": "text",
> "extensions" : [
> "csvh"
>],
>"extractHeader": true,
>"delimiter": ","
> }
> select count(1) from dfs.`test.csvh`
> Error: UNSUPPORTED_OPERATION ERROR: With extractHeader enabled, only header
> names are supported
> coumn name columns
> column index
> Fragment 0:0
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-4974) NPE in FindPartitionConditions.analyzeCall() for 'holistic' expressions
[ https://issues.apache.org/jira/browse/DRILL-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4974. -- Verified and test cases added to automation. > NPE in FindPartitionConditions.analyzeCall() for 'holistic' expressions > --- > > Key: DRILL-4974 > URL: https://issues.apache.org/jira/browse/DRILL-4974 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.6.0, 1.7.0, 1.8.0 >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan > Fix For: 1.9.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > The following query can cause an NPE in FindPartitionConditions.analyzeCall() > if the fileSize column is a partitioned column. > SELECT fileSize FROM dfs.`/drill-data/data/` WHERE compoundId LIKE > 'FOO-1234567%' > This is because, the LIKE is treated as a holistic expression in > FindPartitionConditions.analyzeCall(), causing opStack to be empty, thus > causing opStack.peek() to return a NULL value. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (DRILL-4974) NPE in FindPartitionConditions.analyzeCall() for 'holistic' expressions
[ https://issues.apache.org/jira/browse/DRILL-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-4974: --- Reviewer: Krystal (was: Chun Chang) > NPE in FindPartitionConditions.analyzeCall() for 'holistic' expressions > --- > > Key: DRILL-4974 > URL: https://issues.apache.org/jira/browse/DRILL-4974 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.6.0, 1.7.0, 1.8.0 >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan > Fix For: 1.9.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > The following query can cause an NPE in FindPartitionConditions.analyzeCall() > if the fileSize column is a partitioned column. > SELECT fileSize FROM dfs.`/drill-data/data/` WHERE compoundId LIKE > 'FOO-1234567%' > This is because, the LIKE is treated as a holistic expression in > FindPartitionConditions.analyzeCall(), causing opStack to be empty, thus > causing opStack.peek() to return a NULL value. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-5121) A memory leak is observed when exact case is not specified for a column in a filter condition
[ https://issues.apache.org/jira/browse/DRILL-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-5121. -- Verified and test cases added to automation. > A memory leak is observed when exact case is not specified for a column in a > filter condition > - > > Key: DRILL-5121 > URL: https://issues.apache.org/jira/browse/DRILL-5121 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.6.0, 1.8.0 >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan > Labels: ready-to-commit > Fix For: 1.10.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > When the query SELECT XYZ from dfs.`/tmp/foo' where xYZ like "abc", is > executed on a setup where /tmp/foo has 2 Parquet files, 1.parquet and > 2.parquet, where 1.parquet has the column XYZ but 2.parquet does not, then > there is a memory leak. > This seems to happen because xYZ seem to be treated as a new column. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (DRILL-5121) A memory leak is observed when exact case is not specified for a column in a filter condition
[ https://issues.apache.org/jira/browse/DRILL-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-5121: --- Reviewer: Krystal (was: Chun Chang) > A memory leak is observed when exact case is not specified for a column in a > filter condition > - > > Key: DRILL-5121 > URL: https://issues.apache.org/jira/browse/DRILL-5121 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.6.0, 1.8.0 >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan > Labels: ready-to-commit > Fix For: 1.10.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > When the query SELECT XYZ from dfs.`/tmp/foo' where xYZ like "abc", is > executed on a setup where /tmp/foo has 2 Parquet files, 1.parquet and > 2.parquet, where 1.parquet has the column XYZ but 2.parquet does not, then > there is a memory leak. > This seems to happen because xYZ seem to be treated as a new column. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-5091) JDBC unit test fail on Java 8
[
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-5091.
--
Ran the TestJdbcTest against java 8 and there were no failures:
Results :
Tests run: 28, Failures: 0, Errors: 0, Skipped: 7
> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting
> in a null object and NPE.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-5043) Function that returns a unique id per session/connection similar to MySQL's CONNECTION_ID()
[
https://issues.apache.org/jira/browse/DRILL-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-5043.
--
Verified that session_id function works as expected via the following tests:
1. Each sqlline session returns different session_id.
2. In the same session, multiple "select session_id" queries return the same
result.
3. For table with column name of "session_id", using table alias with column in
the select clause returns data for column.
4. Ran tests from sqlline and Web UI; with and without impersonation.
> Function that returns a unique id per session/connection similar to MySQL's
> CONNECTION_ID()
> ---
>
> Key: DRILL-5043
> URL: https://issues.apache.org/jira/browse/DRILL-5043
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Drill
>Affects Versions: 1.8.0
>Reporter: Nagarajan Chinnasamy
>Assignee: Arina Ielchiieva
>Priority: Minor
> Labels: CONNECTION_ID, SESSION, UDF, doc-impacting
> Fix For: 1.10.0
>
> Attachments: 01_session_id_sqlline.png,
> 02_session_id_webconsole_query.png, 03_session_id_webconsole_result.png
>
>
> Design and implement a function that returns a unique id per
> session/connection similar to MySQL's CONNECTION_ID().
> *Implementation details*
> function *session_id* will be added. Function returns current session unique
> id represented as string. Parameter {code:java} boolean isNiladic{code} will
> be added to UDF FunctionTemplate to indicate if a function is niladic (a
> function to be called without any parameters and parentheses)
> Please note, this function will override columns that have the same name.
> Table alias should be used to retrieve column value from table.
> Example:
> {code:sql}select session_id from // returns the value of niladic
> function session_id {code}
> {code:sql}select t1.session_id from t1 // returns session_id column
> value from table {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-5081) Excessive info level logging introduced in DRILL-4203
[ https://issues.apache.org/jira/browse/DRILL-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-5081. -- Verified that bug is fixed. > Excessive info level logging introduced in DRILL-4203 > - > > Key: DRILL-5081 > URL: https://issues.apache.org/jira/browse/DRILL-5081 > Project: Apache Drill > Issue Type: Bug >Reporter: Sudheesh Katkam >Assignee: Vitalii Diravka > Fix For: 1.10.0 > > > Excessive info level logging introduced in > [8461d10|https://github.com/apache/drill/commit/8461d10b4fd6ce56361d1d826bb3a38b6dc8473c]. > A line is printed for every row group being read, and for every metadata > file. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-5097) Using store.parquet.reader.int96_as_timestamp gives IOOB whereas convert_from works
[
https://issues.apache.org/jira/browse/DRILL-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-5097.
--
Verified that bug is fixed. Automation will be added as part of DRILL-5034
> Using store.parquet.reader.int96_as_timestamp gives IOOB whereas convert_from
> works
> ---
>
> Key: DRILL-5097
> URL: https://issues.apache.org/jira/browse/DRILL-5097
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types, Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Labels: ready-to-commit
> Fix For: 1.10.0
>
> Attachments: data.snappy.parquet
>
>
> Using store.parquet.reader.int96_as_timestamp gives IOOB whereas convert_from
> works.
> The below query succeeds:
> {code}
> select c, convert_from(d, 'TIMESTAMP_IMPALA') from
> dfs.`/drill/testdata/parquet_timestamp/spark_generated/d3`;
> {code}
> The below query fails:
> {code}
> 0: jdbc:drill:zk=10.10.100.190:5181> alter session set
> `store.parquet.reader.int96_as_timestamp` = true;
> +---+---+
> | ok | summary |
> +---+---+
> | true | store.parquet.reader.int96_as_timestamp updated. |
> +---+---+
> 1 row selected (0.231 seconds)
> 0: jdbc:drill:zk=10.10.100.190:5181> select c, d from
> dfs.`/drill/testdata/parquet_timestamp/spark_generated/d3`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: readerIndex: 0, writerIndex:
> 131076 (expected: 0 <= readerIndex <= writerIndex <= capacity(131072))
> Fragment 0:0
> [Error Id: bd94f477-7c01-420f-8920-06263212177b on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5389) select 2 int96 using convert_from(col, 'TIMESTAMP_IMPALA') function fails
Krystal created DRILL-5389: -- Summary: select 2 int96 using convert_from(col, 'TIMESTAMP_IMPALA') function fails Key: DRILL-5389 URL: https://issues.apache.org/jira/browse/DRILL-5389 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.9.0, 1.10.0 Reporter: Krystal I have a table containing 2 int96 time stamp columns. If I select one column at a time, it works. select convert_from(create_timestamp1, 'TIMESTAMP_IMPALA') from dfs.`/user/hive/warehouse/hive1_parquet` where voter_id=3; ++ | EXPR$0 | ++ | 2017-04-14 02:27:55.0 | ++ select convert_from(create_timestamp2, 'TIMESTAMP_IMPALA') from dfs.`/user/hive/warehouse/hive1_parquet` where voter_id=3; ++ | EXPR$0 | ++ | 2017-05-30 19:30:11.0 | ++ However, if I include both columns on the same select, it fails: select convert_from(create_timestamp1, 'TIMESTAMP_IMPALA'), convert_from(create_timestamp2, 'TIMESTAMP_IMPALA') from dfs.`/user/hive/warehouse/hive1_parquet` where voter_id=3; Error: SYSTEM ERROR: ArrayIndexOutOfBoundsException: 0 This is reproducible in drill-1.9 also. In drill-1.10, setting store.parquet.reader.int96_as_timestamp`=true, the same query works fine. select create_timestamp1,create_timestamp2 from dfs.`/user/hive/warehouse/hive1_parquet` where voter_id=3; +++ | create_timestamp1| create_timestamp2| +++ | 2017-04-14 02:27:55.0 | 2017-05-30 19:30:11.0 | +++ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5381) convert_from(col, 'TIMESTAMP_IMPALA') returns incorrect timestamp if there are multiple nulls
Krystal created DRILL-5381: -- Summary: convert_from(col, 'TIMESTAMP_IMPALA') returns incorrect timestamp if there are multiple nulls Key: DRILL-5381 URL: https://issues.apache.org/jira/browse/DRILL-5381 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Affects Versions: 1.9.0, 1.8.0, 1.10.0 Reporter: Krystal In drill-1.10, setting `store.parquet.reader.int96_as_timestamp`=true returns expected data: select voter_id,create_timestamp from dfs.`/user/hive/warehouse/voter_hive_parquet` limit 15; +---++ | voter_id |create_timestamp| +---++ | 1 | 2016-10-23 20:03:58.0 | | 2 | null | | 3 | 2016-09-09 12:01:18.0 | | 4 | 2017-03-06 20:35:55.0 | | 5 | 2017-01-20 22:32:43.0 | | 6 | 2016-10-22 05:46:12.0 | | 7 | 2016-09-19 10:21:29.0 | | 8 | null | | 9 | 2016-07-23 13:39:02.0 | | 10| 2017-01-28 17:27:19.0 | | 11| 2016-10-23 10:55:44.0 | | 12| 2016-06-07 22:44:03.0 | | 13| 2016-05-04 13:59:20.0 | | 14| 2016-11-08 17:20:14.0 | | 15| 2016-05-14 11:23:53.0 | +---++ However, setting `store.parquet.reader.int96_as_timestamp`=false returns incorrect timestamp when it encounters the second "null" value. select voter_id,convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from dfs.`/user/hive/warehouse/voter_hive_parquet` limit 15; +---++ | voter_id | EXPR$1 | +---++ | 1 | 2016-10-23 20:03:58.0 | | 2 | null | | 3 | 2016-09-09 12:01:18.0 | | 4 | 2017-03-06 20:35:55.0 | | 5 | 2017-01-20 22:32:43.0 | | 6 | 2016-10-22 05:46:12.0 | | 7 | 2016-09-19 10:21:29.0 | | 8 | 2016-07-23 13:39:02.0 | | 9 | 2016-10-23 10:55:44.0 | | 10| 2016-06-07 22:44:03.0 | | 11| 2016-05-04 13:59:20.0 | | 12| 2016-11-08 17:20:14.0 | | 13| 2016-05-14 11:23:53.0 | | 14| 2016-06-20 16:18:51.0 | | 15| 2016-09-09 10:02:28.0 | +---++ Notice that the timestamp for voter_id=9 shifts to voter_id=8 which suppose to have value of "null". The rest of the timestamps after voter_id=7 are incorrect. This issue is also reproducible on both drill-1.8.0 and drill-1.9.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-4373) Drill and Hive have incompatible timestamp representations in parquet
[ https://issues.apache.org/jira/browse/DRILL-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937318#comment-15937318 ] Krystal commented on DRILL-4373: [~rkins] For #1, I will check. For #2, I tested that and it works as expected. For #3 and #4, TIMESTAMP_IMPALA_LOCALTIMEZONE function is removed as part of DRILL-5034. > Drill and Hive have incompatible timestamp representations in parquet > - > > Key: DRILL-4373 > URL: https://issues.apache.org/jira/browse/DRILL-4373 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Storage - Parquet >Affects Versions: 1.8.0 >Reporter: Rahul Challapalli >Assignee: Vitalii Diravka > Labels: doc-impacting > Fix For: 1.10.0 > > > git.commit.id.abbrev=83d460c > I created a parquet file with a timestamp type using Drill. Now if I define a > hive table on top of the parquet file and use "timestamp" as the column type, > drill fails to read the hive table through the hive storage plugin > Implementation: > Added int96 to timestamp converter for both parquet readers and controling it > by system / session option "store.parquet.int96_as_timestamp". > The value of the option is false by default for the proper work of the old > query scripts with the "convert_from TIMESTAMP_IMPALA" function. > When the option is true using of that function is unnesessary and can lead to > the query fail. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-4373) Drill and Hive have incompatible timestamp representations in parquet
[ https://issues.apache.org/jira/browse/DRILL-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4373. -- Verified as part of DRILL-5034 > Drill and Hive have incompatible timestamp representations in parquet > - > > Key: DRILL-4373 > URL: https://issues.apache.org/jira/browse/DRILL-4373 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Storage - Parquet >Affects Versions: 1.8.0 >Reporter: Rahul Challapalli >Assignee: Vitalii Diravka > Labels: doc-impacting > Fix For: 1.10.0 > > > git.commit.id.abbrev=83d460c > I created a parquet file with a timestamp type using Drill. Now if I define a > hive table on top of the parquet file and use "timestamp" as the column type, > drill fails to read the hive table through the hive storage plugin > Implementation: > Added int96 to timestamp converter for both parquet readers and controling it > by system / session option "store.parquet.int96_as_timestamp". > The value of the option is false by default for the proper work of the old > query scripts with the "convert_from TIMESTAMP_IMPALA" function. > When the option is true using of that function is unnesessary and can lead to > the query fail. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-5034) Select timestamp from hive generated parquet always return in UTC
[
https://issues.apache.org/jira/browse/DRILL-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935204#comment-15935204
]
Krystal commented on DRILL-5034:
Verified that int96 timestamps return correct result for queries against
various user timezones such as UTC, PST, and EST. Tested with alter session
set `store.parquet.reader.int96_as_timestamp`=true/false.
> Select timestamp from hive generated parquet always return in UTC
> -
>
> Key: DRILL-5034
> URL: https://issues.apache.org/jira/browse/DRILL-5034
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.9.0
>Reporter: Krystal
>Assignee: Vitalii Diravka
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.10.0
>
>
> commit id: 5cea9afa6278e21574c6a982ae5c3d82085ef904
> Reading timestamp data against a hive parquet table from drill automatically
> converts the timestamp data to UTC.
> {code}
> SELECT TIMEOFDAY() FROM (VALUES(1));
> +--+
> |EXPR$0|
> +--+
> | 2016-11-10 12:33:26.547 America/Los_Angeles |
> +--+
> {code}
> data schema:
> {code}
> message hive_schema {
> optional int32 voter_id;
> optional binary name (UTF8);
> optional int32 age;
> optional binary registration (UTF8);
> optional fixed_len_byte_array(3) contributions (DECIMAL(6,2));
> optional int32 voterzone;
> optional int96 create_timestamp;
> optional int32 create_date (DATE);
> }
> {code}
> Using drill-1.8, the returned timestamps match the table data:
> {code}
> select convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from
> `/user/hive/warehouse/voter_hive_parquet` limit 5;
> ++
> | EXPR$0 |
> ++
> | 2016-10-23 20:03:58.0 |
> | null |
> | 2016-09-09 12:01:18.0 |
> | 2017-03-06 20:35:55.0 |
> | 2017-01-20 22:32:43.0 |
> ++
> 5 rows selected (1.032 seconds)
> {code}
> If the user timzone is changed to UTC, then the timestamp data is returned in
> UTC time.
> Using drill-1.9, the returned timestamps got converted to UTC eventhough the
> user timezone is in PST.
> {code}
> select convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from
> dfs.`/user/hive/warehouse/voter_hive_parquet` limit 5;
> ++
> | EXPR$0 |
> ++
> | 2016-10-24 03:03:58.0 |
> | null |
> | 2016-09-09 19:01:18.0 |
> | 2017-03-07 04:35:55.0 |
> | 2017-01-21 06:32:43.0 |
> ++
> {code}
> {code}
> alter session set `store.parquet.reader.int96_as_timestamp`=true;
> +---+---+
> | ok | summary |
> +---+---+
> | true | store.parquet.reader.int96_as_timestamp updated. |
> +---+---+
> select create_timestamp from dfs.`/user/hive/warehouse/voter_hive_parquet`
> limit 5;
> ++
> |create_timestamp|
> ++
> | 2016-10-24 03:03:58.0 |
> | null |
> | 2016-09-09 19:01:18.0 |
> | 2017-03-07 04:35:55.0 |
> | 2017-01-21 06:32:43.0 |
> ++
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-867) tpcds queries 6, 9 and 10 fail to plan
[
https://issues.apache.org/jira/browse/DRILL-867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-867.
-
Bug was verified.
> tpcds queries 6, 9 and 10 fail to plan
> --
>
> Key: DRILL-867
> URL: https://issues.apache.org/jira/browse/DRILL-867
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Reporter: Krystal
>Priority: Minor
> Fix For: Future
>
>
> git.commit.id.abbrev=e1e5ea0
> git.commit.time=29.05.2014 @ 15\:32\:29 PDT
> query 6:
> {code}
> select * from (select a.ca_state state, count(*) cnt
> from customer_address a
> ,customer c
> ,store_sales s
> ,date_dim d
> ,item i
> where a.ca_address_sk = c.c_current_addr_sk
> and c.c_customer_sk = s.ss_customer_sk
> and s.ss_sold_date_sk = d.d_date_sk
> and s.ss_item_sk = i.i_item_sk
> and d.d_month_seq =
>(select distinct (d.d_month_seq)
> from date_dim d
>where d.d_year = 1998
> and d.d_moy = 5 )
> and i.i_current_price > 1.2 *
> (select avg(j.i_current_price)
>from item j
>where j.i_category = i.i_category)
> group by a.ca_state
> having count(*) >= 10
> order by cnt
> ) limit 100;
> {code}
> query 7:
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-573) SUBSTR/SUBSTRING functions of a number requires explicit cast
[ https://issues.apache.org/jira/browse/DRILL-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-573. - Bug was verified. > SUBSTR/SUBSTRING functions of a number requires explicit cast > - > > Key: DRILL-573 > URL: https://issues.apache.org/jira/browse/DRILL-573 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.0.0 >Reporter: Krystal > Fix For: 0.4.0 > > > Unlike Oracle, drill requires explicit cast a number to string when used in a > substr or substring function. > Here is the test data: > select * from voter where voter_id=11; > voter_id | name | age | registration | contributions | voterzone | > create_time > +--+-+--+---+---+- > 11 | | 58 | republican |578.08 | 16161 | 2014-08-27 > 06:35:33 > In drill, an explicit cast is required: > 0: jdbc:drill:schema=dfs> select substr(cast(contributions as varchar(8)), 3, > 5) from voter where voter_id=11; > ++ > | EXPR$0 | > ++ > | 8.08 | > ++ > If explicit cast is not given, drill returns result in some form of binary > data: > 0: jdbc:drill:schema=dfs> select substr(contributions, 3, 5) from voter where > voter_id=11; > ++ > | EXPR$0 | > ++ > | 㠮〸 | > ++ > In Oracle, the cast is implicit: > SQL> select substr(contributions, 3, 5) from voter where voter_id=11; > SUBST > - > 8.08 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-710) SIGN function should return an int value
[ https://issues.apache.org/jira/browse/DRILL-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-710. - bug was verified. > SIGN function should return an int value > > > Key: DRILL-710 > URL: https://issues.apache.org/jira/browse/DRILL-710 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 0.1.0-m1 >Reporter: Krystal > Fix For: 0.4.0 > > Attachments: DRILL-710.patch > > > The SIGN(n) function should return int values of 0, 1, or -1 regardless of > number data type of n. In drill, the return value is 1.0/-1.0/0.0 if n is a > decimal value and 1/-1/0 if n is an int. In oracle, postgres, and mysql, the > return value is an int regardless of numeric data type passed to n. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-789) Left outer join returns "null" values for columns from the right table
[ https://issues.apache.org/jira/browse/DRILL-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-789. - Bug was verified. > Left outer join returns "null" values for columns from the right table > -- > > Key: DRILL-789 > URL: https://issues.apache.org/jira/browse/DRILL-789 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Krystal > Fix For: 0.4.0 > > Attachments: DRILL-789.patch > > > git.commit.id.abbrev=5d7e3d3 > 0: jdbc:drill:schema=dfs> select voter.name voter_name, voter.registration > registration, student.name student_name, student.gpa gpa from voter left > outer join student on (student.name = voter.name) where voter.age < 30; > ++--+--++ > | voter_name | registration | student_name |gpa | > ++--+--++ > | [B@6ca2652 | [B@4199d4f9 | null | null | > | [B@5a6d4914 | [B@3cd8ee6d | null | null | > | [B@460d5550 | [B@155c1b1e | null | null | > | [B@51f85986 | [B@7bd9675 | null | null | > | [B@2fe0df4b | [B@5463cd7b | null | null | > | [B@64477185 | [B@e6e0632 | null | null | -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-575) RPAD/LPAD functions fails when padded string is not specified
[
https://issues.apache.org/jira/browse/DRILL-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-575.
-
Bug was verified.
> RPAD/LPAD functions fails when padded string is not specified
> -
>
> Key: DRILL-575
> URL: https://issues.apache.org/jira/browse/DRILL-575
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.0.0
>Reporter: Krystal
> Fix For: 0.4.0
>
> Attachments: DRILL-575.patch
>
>
> In oracle and postgres, rpad and lpad functions defaults to spaces when str2
> is not specified - rpad(str1, length [, str2]). For example in oracle:
> SQL> select rpad(name,20) from voter where voter_id=10;
> RPAD(NAME,20)
>
> tom underhill
> SQL> select lpad(name,20) from voter where voter_id=10;
> LPAD(NAME,20)
>
>tom underhill
> In drill, the queries fail because the str2 parameter is required:
> 0: jdbc:drill:schema=dfs> relect lpad(name, 20) from voter where voter_id=10;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "dde5cf1c-1d05-46c4-902a-c4bd6621c065"
> endpoint {
> address: "qa-node57.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while parsing sql. < SqlParseException:[ Non-query
> expression encountered in illegal context ] < EigenbaseException:[ Non-query
> expression encountered in illegal context ]"
> ]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-574) RPAD function returns different result compared to other rdbms when length is less than string
[ https://issues.apache.org/jira/browse/DRILL-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-574. - bug was verified. > RPAD function returns different result compared to other rdbms when length is > less than string > -- > > Key: DRILL-574 > URL: https://issues.apache.org/jira/browse/DRILL-574 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.0.0 >Reporter: Krystal > Fix For: 0.4.0 > > Attachments: DRILL-574.patch > > > The rpad function - rpad(str1, length [, str2]) returns different result > compared to postgres, oracle and mysql when length is less than the lenght of > str1. For example: > select * from voter where voter_id=10; > voter_id | name | age | registration | contributions | voterzone | >create_time > +---+-+--+---+---+- > 10 | tom underhill | 75 | socialist|525.33 | 18592 | > 2015-01-24 07:27:05 > 0: jdbc:drill:schema=dfs> select rpad(name, 8, 'A') from voter where > voter_id=10; > ++ > | EXPR$0 | > ++ > | nderhill | > ++ > The above result from drill truncates the name from right to left. For the > other rdbms: > SQL> select rpad(name, 8, 'A') from voter where voter_id=10; > RPAD(NAM > > tom unde > Looks like truncation occurs from left to right eventhough it's rpad. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-503) Cast fail for float and double
[
https://issues.apache.org/jira/browse/DRILL-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-503.
-
Bug was verified.
> Cast fail for float and double
> --
>
> Key: DRILL-503
> URL: https://issues.apache.org/jira/browse/DRILL-503
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Krystal
> Fix For: 0.4.0
>
>
> Cast failed for the following queries:
> 0: jdbc:drill:schema=dfs> select cast(gpa as double) from dfs.`student`;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "9cb5c45d-cf98-44f9-8e14-46f537cf2fa2"
> endpoint {
> address: "qa-node57.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while setting up query. < FragmentSetupException:[ Failure
> while trying to convert fragment into json. ] < JsonMappingException:[ (was
> java.lang.NullPointerException) (through reference chain:
> org.apache.drill.exec.physical.config.Screen[\"child\"]->org.apache.drill.exec.physical.config.Project[\"exprs\"]->java.util.ArrayList[0]->org.apache.drill.common.logical.data.NamedExpression[\"expr\"])
> ] < NullPointerException"
> ]
> Error: exception while executing query (state=,code=0)
> -
> 0: jdbc:drill:schema=dfs> select cast(gpa as float8) from dfs.`student`;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "1f5f85ef-1d78-4d87-bdea-feebf7521aff"
> endpoint {
> address: "qa-node57.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while parsing sql. < ValidationException:[
> org.eigenbase.util.EigenbaseContextException: From line 1, column 20 to line
> 1, column 25 ] < EigenbaseContextException:[ From line 1, column 20 to line
> 1, column 25 ] < SqlValidatorException:[ Unknown datatype name \'float8\' ]"
> ]
> Error: exception while executing query (state=,code=0)
>
> 0: jdbc:drill:schema=dfs> select cast(gpa as float) from dfs.`student`;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "1fc8859f-b6ac-4789-9265-1d2810b72abc"
> endpoint {
> address: "qa-node57.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while setting up query. < FragmentSetupException:[ Failure
> while trying to convert fragment into json. ] < JsonMappingException:[ (was
> java.lang.NullPointerException) (through reference chain:
> org.apache.drill.exec.physical.config.Screen[\"child\"]->org.apache.drill.exec.physical.config.Project[\"exprs\"]->java.util.ArrayList[0]->org.apache.drill.common.logical.data.NamedExpression[\"expr\"])
> ] < NullPointerException"
> ]
> Error: exception while executing query (state=,code=0)
> --
> 0: jdbc:drill:schema=dfs> select cast(gpa as float8) from dfs.`student`;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "8af67797-d174-4b84-b9b7-e6fe9413908b"
> endpoint {
> address: "qa-node57.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while parsing sql. < ValidationException:[
> org.eigenbase.util.EigenbaseContextException: From line 1, column 20 to line
> 1, column 25 ] < EigenbaseContextException:[ From line 1, column 20 to line
> 1, column 25 ] < SqlValidatorException:[ Unknown datatype name \'float8\' ]"
> ]
> Error: exception while executing query (state=,code=0)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-673) Select against M7 table fails due to missing hbase jars in classpaths
[
https://issues.apache.org/jira/browse/DRILL-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-673.
-
Closing bug.
> Select against M7 table fails due to missing hbase jars in classpaths
> -
>
> Key: DRILL-673
> URL: https://issues.apache.org/jira/browse/DRILL-673
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 0.1.0-m1
>Reporter: Krystal
> Fix For: 0.4.0
>
>
> I have an m7 table at this location:
> hbase(main):014:0> list '/test/tables'
> TABLE
>
> /test/tables/m7voter
> From sqlline, I ran the following query:
> 0: jdbc:drill:schema=hbase> select * from `/test/tables/m7voter` limit 5;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "4e7d2b40-6511-436b-a459-ab0667cd5f74"
> endpoint {
> address: "qa-node64.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while parsing sql. < ValidationException:[
> org.eigenbase.util.EigenbaseContextException: From line 1, column 15 to line
> 1, column 36 ] < EigenbaseContextException:[ From line 1, column 15 to line
> 1, column 36 ] < SqlValidatorException:[ Table \'/test/tables/m7voter\' not
> found ]"
> ]
> Error: exception while executing query (state=,code=0)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-758) to_timestamp(unix time) not returning correct result
[ https://issues.apache.org/jira/browse/DRILL-758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-758. - Bug was verified. > to_timestamp(unix time) not returning correct result > > > Key: DRILL-758 > URL: https://issues.apache.org/jira/browse/DRILL-758 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Reporter: Krystal > Fix For: 0.4.0 > > Attachments: DRILL-758.patch > > > Ran the following to_timestamp query passing it a unix timestamp: > select to_timestamp(1284352323) from voter where voter_id=10; > The value returned looks like unix epoch time: > 1970-01-15T20:45:52.323-08:00 > Any unix time used returned the same year (1970) and month (01). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-772) Null values are not filtered out in query result
[ https://issues.apache.org/jira/browse/DRILL-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-772. - Bug was verified. > Null values are not filtered out in query result > > > Key: DRILL-772 > URL: https://issues.apache.org/jira/browse/DRILL-772 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Krystal > Fix For: 0.4.0 > > > git.commit.id.abbrev=70fab8c > 0: jdbc:drill:schema=dfs> select voter_id, name, age from voter where age not > between 20 and 80; > ++++ > | voter_id |name|age | > ++++ > | 3 | [B@183c94e6 | 18 | > | 17 | [B@31539478 | null | > | 22 | [B@66386b94 | 19 | > | 57 | [B@21ef5fa | 19 | > | 59 | [B@3ea6f12c | 18 | > | 70 | [B@42239fd6 | 18 | > | 85 | [B@30d2bc86 | 19 | > | 92 | [B@5b571a74 | null | > | 109| [B@3389e485 | 18 | > | 111| [B@21f9f232 | 18 | > The output above returned age with "null" values. In oracle and postgres, > the rows with "null" value for age are filtered out. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-5208) Finding path to java executable should be deterministic
[ https://issues.apache.org/jira/browse/DRILL-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-5208. -- commit id: 3dfb497293a177164a158b246000ee8291ef258c On my two nodes where the order of the paths to the java executable are different: find -L $JAVA_HOME -name java -type f /usr/local/java/jdk1.7.0_67/jre/bin/java /usr/local/java/jdk1.7.0_67/bin/java find -L $JAVA_HOME -name java -type f /usr/local/java/jdk1.7.0_67/bin/java /usr/local/java/jdk1.7.0_67/jre/bin/java Now, the paths of the java executable on the two nodes are pointing to the same location: /usr/local/java/jdk1.7.0_67//bin/java > Finding path to java executable should be deterministic > --- > > Key: DRILL-5208 > URL: https://issues.apache.org/jira/browse/DRILL-5208 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.10.0 >Reporter: Krystal >Assignee: Paul Rogers >Priority: Minor > Labels: ready-to-commit > Fix For: 1.10.0 > > > Command to find JAVA in drill-config.sh is not deterministic. > drill-config.sh uses the following command to find JAVA: > JAVA=`find -L "$JAVA_HOME" -name $JAVA_BIN -type f | head -n 1` > On one of my node the following command returned 2 entries: > find -L $JAVA_HOME -name java -type f > /usr/local/java/jdk1.7.0_67/jre/bin/java > /usr/local/java/jdk1.7.0_67/bin/java > On another node, the same command returned entries in different order: > find -L $JAVA_HOME -name java -type f > /usr/local/java/jdk1.7.0_67/bin/java > /usr/local/java/jdk1.7.0_67/jre/bin/java > The complete command picks the first one returned which may not be the same > on each node: > find -L $JAVA_HOME -name java -type f | head -n 1 > /usr/local/java/jdk1.7.0_67/jre/bin/java > If JAVA_HOME is found, we should just append the "bin/java" to the path" > JAVA=$JAVA_HOME/bin/java -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (DRILL-520) ceiling/ceil and floor functions return decimal value instead of an integer
[ https://issues.apache.org/jira/browse/DRILL-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876705#comment-15876705 ] Krystal commented on DRILL-520: --- Tried the tests on the latest 1.10 master and the results are still the same as originally reported. > ceiling/ceil and floor functions return decimal value instead of an integer > --- > > Key: DRILL-520 > URL: https://issues.apache.org/jira/browse/DRILL-520 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 1.0.0 >Reporter: Krystal >Assignee: Jinfeng Ni >Priority: Critical > Fix For: Future > > Attachments: DRILL-520.patch > > > Ran the following queries in drill: > 0: jdbc:drill:schema=dfs> select ceiling(55.8) from dfs.`student` where > rownum=11; > ++ > | EXPR$0 | > ++ > | 56.0 | > ++ > 0: jdbc:drill:schema=dfs> select floor(55.8) from dfs.`student` where > rownum=11; > ++ > | EXPR$0 | > ++ > | 55.0 | > ++ > The same queries executed from oracle, postgres and mysql returned integer > values of 56 and 55. > Found the following description of the two functions from > http://users.atw.hu/sqlnut/sqlnut2-chp-4-sect-4.html : > Ceil/Ceiling: > Rounds a noninteger value upwards to the next greatest integer. Returns an > integer value unchanged. > Floor: > Rounds a noninteger value downwards to the next least integer. Returns an > integer value unchanged. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-515) Result from int divided by in drill not consistent with other databases.
[
https://issues.apache.org/jira/browse/DRILL-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-515.
-
Close as won't fix.
> Result from int divided by in drill not consistent with other databases.
> -
>
> Key: DRILL-515
> URL: https://issues.apache.org/jira/browse/DRILL-515
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.0.0
>Reporter: Krystal
> Fix For: 0.4.0
>
>
> Here is the schema of voter parquet table:
> message pig_schema {
> optional int32 rownum;
> optional binary name;
> optional int32 age;
> optional binary registration;
> optional double contributions;
> optional int32 voterzone;
> optional binary create_time;
> }
> Row data from voter table:
> 0: jdbc:drill:schema=dfs> select rownum, age from dfs.`voter` where rownum=10;
> +++
> | rownum |age |
> +++
> | 10 | 75 |
> +++
> Division query from drill:
> 0: jdbc:drill:schema=dfs> select (age/rownum) from dfs.`voter` where
> rownum=10;
> ++
> | EXPR$0 |
> ++
> | 7 |
> ++
> If hard code the value, 7.5 is returned:
> 0: jdbc:drill:schema=dfs> select (75/10) from dfs.`voter` where rownum=10;
> ++
> | EXPR$0 |
> ++
> | 7.5|
> ++
> Here is the result from postgres:
> postgres=# \d voter
> Table "public.voter"
> Column |Type | Modifiers
> ---+-+---
> rownum| integer |
> name | character varying(50) |
> age | integer |
> registration | character varying(20) |
> contributions | numeric(7,2)|
> voterzone | smallint|
> create_time | timestamp without time zone |
> postgres=# select (age/rownum) from voter where rownum=10;
> ?column?
> --
> 7
> (1 row)
> postgres=# select (75/10) from voter where rownum=10;
> ?column?
> --
> 7
> (1 row)
> From Oracle:
> SQL> describe voter;
> NameNull?Type
> -
>
> VOTER_ID NUMBER
> NAME VARCHAR2(50)
> AGE NUMBER
> REGISTRATION VARCHAR2(20)
> CONTRIBUTIONSNUMBER(5,2)
> VOTERZONENUMBER
> CREATE_TIME TIMESTAMP(6)
> Note that oracle does not have an integer data type.
> SQL> select (age/voter_id) from voter where voter_id=10;
> (AGE/VOTER_ID)
> --
> 7.5
> SQL> select (75/10) as result from voter where voter_id=10;
> RESULT
> --
>7.5
> From mysql database:
> mysql> describe voter;
> +---+-+--+-+-+---+
> | Field | Type| Null | Key | Default | Extra |
> +---+-+--+-+-+---+
> | rownum| int(11) | YES | | NULL| |
> | name | varchar(50) | YES | | NULL| |
> | age | tinyint(4) | YES | | NULL| |
> | registration | varchar(15) | YES | | NULL| |
> | contributions | float | YES | | NULL| |
> | voterzone | smallint(6) | YES | | NULL| |
> +---+-+--+-+-+---+
> mysql> select (age/rownum) from voter where rownum=10;
> +--+
> | (age/rownum) |
> +--+
> | 7.5000 |
> +--+
> mysql> select (75/10);
> +-+
> | (75/10) |
> +-+
> | 7.5000 |
> +-+
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (DRILL-777) Parse error for modulo function "%"
[
https://issues.apache.org/jira/browse/DRILL-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-777.
-
Close as won't fix as drill do not support % modulo function.
> Parse error for modulo function "%"
>
>
> Key: DRILL-777
> URL: https://issues.apache.org/jira/browse/DRILL-777
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: Future
>Reporter: Krystal
> Fix For: Future
>
>
> git.commit.id.abbrev=70fab8c
> 0: jdbc:drill:schema=dfs> select (age%rownum) from voter where voter_id=10;
> Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
> running query.[error_id: "f1e9fcef-57f0-4d60-ab57-b5b73b33ee1c"
> endpoint {
> address: "qa-node64.qa.lab"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> }
> error_type: 0
> message: "Failure while parsing sql. < SqlParseException:[ Lexical error at
> line 1, column 12. Encountered: "%" (37), after : "" ] < TokenMgrError:[
> Lexical error at line 1, column 12. Encountered: "%" (37), after : "" ]"
> ]
> Stack trace:
> org.apache.drill.exec.planner.sql.parser.impl.TokenMgrError: Lexical error at
> line 1, column 12. Encountered: "%" (37), after : ""
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImplTokenManager.getNextToken(DrillParserImplTokenManager.java:13785)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_scan_token(DrillParserImpl.java:16417)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3_300(DrillParserImpl.java:15265)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_42(DrillParserImpl.java:15275)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3_376(DrillParserImpl.java:11721)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_83(DrillParserImpl.java:11738)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_124(DrillParserImpl.java:12018)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_114(DrillParserImpl.java:11190)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_74(DrillParserImpl.java:11342)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3_199(DrillParserImpl.java:11896)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_69(DrillParserImpl.java:11879)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_70(DrillParserImpl.java:12291)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3R_25(DrillParserImpl.java:12380)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_3_169(DrillParserImpl.java:12469)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_2_169(DrillParserImpl.java:6391)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.LeafQueryOrExpr(DrillParserImpl.java:2386)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.QueryOrExpr(DrillParserImpl.java:2309)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.OrderedQueryOrExpr(DrillParserImpl.java:436)
>
> ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> org.apache
[jira] [Closed] (DRILL-681) Drill returns big int result not matching expected result
[ https://issues.apache.org/jira/browse/DRILL-681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-681. - Close bug as invalid. This is not an issue. > Drill returns big int result not matching expected result > - > > Key: DRILL-681 > URL: https://issues.apache.org/jira/browse/DRILL-681 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Affects Versions: 0.1.0-m1 >Reporter: Krystal > Fix For: 0.4.0 > > > I have several functions that take integers (int, big int). The output from > drill displays the numbers with E to the x power. In some cases with huge > integer results, it truncates the data. Here are some examples: > select variance(voterzone) from voter; > drill: 7.56958727143133E7 > postgres: 75695872.714313313313 > select var_samp(studentnum) from student; > drill: 8.521370486035167E22 > postgres: 85213704860351887237202 > select variance(studentnum) from student; > drill: 8.521370486035167E22 > postgres: 85213704860351887237202 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (DRILL-857) localtime function returns incorrect result
[ https://issues.apache.org/jira/browse/DRILL-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-857. - Close as a duplicate of DRILL-585. Issue is fixed. > localtime function returns incorrect result > --- > > Key: DRILL-857 > URL: https://issues.apache.org/jira/browse/DRILL-857 > Project: Apache Drill > Issue Type: Bug > Components: Client - CLI >Reporter: Krystal > Fix For: 0.4.0 > > > git.commit.id.abbrev=01bf849 > 0: jdbc:drill:schema=dfs> select localtime from voter where voter_id=10; > ++ > | localtime | > ++ > | 1970-01-01T08:02:38.332-08:00 | -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5225) Needs better error message for table function when having incorrect table path
Krystal created DRILL-5225: -- Summary: Needs better error message for table function when having incorrect table path Key: DRILL-5225 URL: https://issues.apache.org/jira/browse/DRILL-5225 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.9.0, 1.10.0 Reporter: Krystal When schema is missing from table path or full path to table is not given, the error message displayed is very misleading. For example, the query below runs successfully with correct table path: select columns[0],columns[1],columns[2] from table(`dfs.drillTestDir`.`/table_function/header.csv`(type=>'text',lineDelimiter=>'\r\n',fieldDelimiter=>',',skipFirstLine=>true)); +-+-+-+ | EXPR$0 | EXPR$1 | EXPR$2 | +-+-+-+ | 1 | aaa | bbb | | 2 | ccc | ddd | | 3 | eee | null| | 4 | fff | ggg | +-+-+-+ However if the part of the path is left out, for example the schema, then a very misleading error is thrown: SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text', lineDelimiter=>'\r\n')) 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query: select columns[0] from table(`table_function/cr_lf.csv`(type=>'text', lineDelimiter=>'\r\n')) [30027]Query execution error. Details:[ SYSTEM ERROR: SqlValidatorException: No match found for function signature table_function/cr_lf.csv(type => , lineDelimiter => ) Error should indicate that table `table_function/cr_lf.csv` is not found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-5222) C++ client unable to parse queries with table function
[
https://issues.apache.org/jira/browse/DRILL-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-5222.
--
Resolution: Not A Bug
> C++ client unable to parse queries with table function
> --
>
> Key: DRILL-5222
> URL: https://issues.apache.org/jira/browse/DRILL-5222
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> The following query failed from was odbc and custom C++ client app:
> SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query:
> select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> [30027]Query execution error. Details:[
> SYSTEM ERROR: SqlValidatorException: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> Here is the stack trace:
> {code}
>SYSTEM ERROR: SqlValidatorException: No match found for function
> signature table_function/cr_lf.csv(type => , lineDelimiter => )
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.drill.exec.exception.FunctionNotFoundException) No
> match found for function signature table_function/cr_lf.csv(type => ,
> lineDelimiter => )
> org.apache.drill.exec.planner.sql.SqlConverter.validate():170
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():606
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():192
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():122
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():96
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1017
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.calcite.runtime.CalciteContextException) From line 1,
> column 45 to line 1, column 107: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
> sun.reflect.NativeConstructorAccessorImpl.newInstance():57
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
> java.lang.reflect.Constructor.newInstance():526
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex():405
> org.apache.calcite.sql.SqlUtil.newContextException():765
> org.apache.calcite.sql.SqlUtil.newContextException():753
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():3974
>
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction():1583
> org.apache.calcite.sql.SqlFunction.deriveType():278
> org.apache.calcite.sql.SqlFunction.deriveType():222
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4337
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4324
> org.apache.calcite.sql.SqlCall.accept():130
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl():1501
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():53
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
> org.apache.calcite.sql.validate.Sele
[jira] [Commented] (DRILL-5222) C++ client unable to parse queries with table function
[
https://issues.apache.org/jira/browse/DRILL-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838469#comment-15838469
]
Krystal commented on DRILL-5222:
Sorry, it's my bad. I ran a query against a view created with the table
function using relative path (without the schema name) instead of the full
path. The ODBC session was not in the schema but my sqlline/UI session was;
thus resulting in the error. Need a better error message in this case against
the table function; will open a new bug for this.
> C++ client unable to parse queries with table function
> --
>
> Key: DRILL-5222
> URL: https://issues.apache.org/jira/browse/DRILL-5222
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> The following query failed from was odbc and custom C++ client app:
> SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query:
> select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> [30027]Query execution error. Details:[
> SYSTEM ERROR: SqlValidatorException: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> Here is the stack trace:
> {code}
>SYSTEM ERROR: SqlValidatorException: No match found for function
> signature table_function/cr_lf.csv(type => , lineDelimiter => )
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.drill.exec.exception.FunctionNotFoundException) No
> match found for function signature table_function/cr_lf.csv(type => ,
> lineDelimiter => )
> org.apache.drill.exec.planner.sql.SqlConverter.validate():170
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():606
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():192
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():122
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():96
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1017
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.calcite.runtime.CalciteContextException) From line 1,
> column 45 to line 1, column 107: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
> sun.reflect.NativeConstructorAccessorImpl.newInstance():57
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
> java.lang.reflect.Constructor.newInstance():526
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex():405
> org.apache.calcite.sql.SqlUtil.newContextException():765
> org.apache.calcite.sql.SqlUtil.newContextException():753
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():3974
>
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction():1583
> org.apache.calcite.sql.SqlFunction.deriveType():278
> org.apache.calcite.sql.SqlFunction.deriveType():222
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4337
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4324
> org.apache.calcite.sql.SqlCall.accept():130
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl():1501
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():53
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.va
[jira] [Commented] (DRILL-5222) C++ client unable to parse queries with table function
[
https://issues.apache.org/jira/browse/DRILL-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838303#comment-15838303
]
Krystal commented on DRILL-5222:
The query runs fine from sqlline and the UI.
> C++ client unable to parse queries with table function
> --
>
> Key: DRILL-5222
> URL: https://issues.apache.org/jira/browse/DRILL-5222
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - C++
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> The following query failed from was odbc and custom C++ client app:
> SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> 1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query:
> select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
> lineDelimiter=>'\r\n'))
> [30027]Query execution error. Details:[
> SYSTEM ERROR: SqlValidatorException: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> Here is the stack trace:
> {code}
>SYSTEM ERROR: SqlValidatorException: No match found for function
> signature table_function/cr_lf.csv(type => , lineDelimiter => )
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.drill.exec.exception.FunctionNotFoundException) No
> match found for function signature table_function/cr_lf.csv(type => ,
> lineDelimiter => )
> org.apache.drill.exec.planner.sql.SqlConverter.validate():170
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():606
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():192
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():122
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():96
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1017
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.calcite.runtime.CalciteContextException) From line 1,
> column 45 to line 1, column 107: No match found for function signature
> table_function/cr_lf.csv(type => , lineDelimiter => )
> sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
> sun.reflect.NativeConstructorAccessorImpl.newInstance():57
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
> java.lang.reflect.Constructor.newInstance():526
> org.apache.calcite.runtime.Resources$ExInstWithCause.ex():405
> org.apache.calcite.sql.SqlUtil.newContextException():765
> org.apache.calcite.sql.SqlUtil.newContextException():753
> org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():3974
>
> org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction():1583
> org.apache.calcite.sql.SqlFunction.deriveType():278
> org.apache.calcite.sql.SqlFunction.deriveType():222
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4337
>
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4324
> org.apache.calcite.sql.SqlCall.accept():130
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl():1501
> org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():53
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
> org.apache.calcite.sql.validate.AbstractNamespace.validate():86
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
> org.apache.calcite.sql.validate.Sq
[jira] [Created] (DRILL-5222) C++ client unable to parse queries with table function
Krystal created DRILL-5222:
--
Summary: C++ client unable to parse queries with table function
Key: DRILL-5222
URL: https://issues.apache.org/jira/browse/DRILL-5222
Project: Apache Drill
Issue Type: Bug
Components: Client - C++
Affects Versions: 1.10.0
Reporter: Krystal
The following query failed from was odbc and custom C++ client app:
SQL>select columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
lineDelimiter=>'\r\n'))
1: SQLPrepare = [MapR][Drill] (1040) Drill failed to execute the query: select
columns[0] from table(`table_function/cr_lf.csv`(type=>'text',
lineDelimiter=>'\r\n'))
[30027]Query execution error. Details:[
SYSTEM ERROR: SqlValidatorException: No match found for function signature
table_function/cr_lf.csv(type => , lineDelimiter => )
Here is the stack trace:
{code}
SYSTEM ERROR: SqlValidatorException: No match found for function
signature table_function/cr_lf.csv(type => , lineDelimiter => )
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
during fragment initialization: No match found for function signature
table_function/cr_lf.csv(type => , lineDelimiter => )
org.apache.drill.exec.work.foreman.Foreman.run():281
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (org.apache.drill.exec.exception.FunctionNotFoundException) No
match found for function signature table_function/cr_lf.csv(type => ,
lineDelimiter => )
org.apache.drill.exec.planner.sql.SqlConverter.validate():170
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode():606
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():192
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():122
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():96
org.apache.drill.exec.work.foreman.Foreman.runSQL():1017
org.apache.drill.exec.work.foreman.Foreman.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (org.apache.calcite.runtime.CalciteContextException) From line 1,
column 45 to line 1, column 107: No match found for function signature
table_function/cr_lf.csv(type => , lineDelimiter => )
sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2
sun.reflect.NativeConstructorAccessorImpl.newInstance():57
sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45
java.lang.reflect.Constructor.newInstance():526
org.apache.calcite.runtime.Resources$ExInstWithCause.ex():405
org.apache.calcite.sql.SqlUtil.newContextException():765
org.apache.calcite.sql.SqlUtil.newContextException():753
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError():3974
org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction():1583
org.apache.calcite.sql.SqlFunction.deriveType():278
org.apache.calcite.sql.SqlFunction.deriveType():222
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4337
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit():4324
org.apache.calcite.sql.SqlCall.accept():130
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl():1501
org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():53
org.apache.calcite.sql.validate.AbstractNamespace.validate():86
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
org.apache.calcite.sql.validate.AbstractNamespace.validate():86
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2806
org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2791
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3014
org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
org.apache.calcite.sql.validate.AbstractNamespace.validate():86
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():883
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():869
org.apache.calcite.sql.SqlSelect.validate():210
org.apache.calcite.sql.validate
[jira] [Updated] (DRILL-5208) Finding path to java executable should be deterministic
[ https://issues.apache.org/jira/browse/DRILL-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-5208: --- Assignee: Paul Rogers > Finding path to java executable should be deterministic > --- > > Key: DRILL-5208 > URL: https://issues.apache.org/jira/browse/DRILL-5208 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Affects Versions: 1.10.0 >Reporter: Krystal >Assignee: Paul Rogers >Priority: Minor > > Command to find JAVA in drill-config.sh is not deterministic. > drill-config.sh uses the following command to find JAVA: > JAVA=`find -L "$JAVA_HOME" -name $JAVA_BIN -type f | head -n 1` > On one of my node the following command returned 2 entries: > find -L $JAVA_HOME -name java -type f > /usr/local/java/jdk1.7.0_67/jre/bin/java > /usr/local/java/jdk1.7.0_67/bin/java > On another node, the same command returned entries in different order: > find -L $JAVA_HOME -name java -type f > /usr/local/java/jdk1.7.0_67/bin/java > /usr/local/java/jdk1.7.0_67/jre/bin/java > The complete command picks the first one returned which may not be the same > on each node: > find -L $JAVA_HOME -name java -type f | head -n 1 > /usr/local/java/jdk1.7.0_67/jre/bin/java > If JAVA_HOME is found, we should just append the "bin/java" to the path" > JAVA=$JAVA_HOME/bin/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5208) Finding path to java executable should be deterministic
Krystal created DRILL-5208: -- Summary: Finding path to java executable should be deterministic Key: DRILL-5208 URL: https://issues.apache.org/jira/browse/DRILL-5208 Project: Apache Drill Issue Type: Bug Components: Tools, Build & Test Affects Versions: 1.10.0 Reporter: Krystal Priority: Minor Command to find JAVA in drill-config.sh is not deterministic. drill-config.sh uses the following command to find JAVA: JAVA=`find -L "$JAVA_HOME" -name $JAVA_BIN -type f | head -n 1` On one of my node the following command returned 2 entries: find -L $JAVA_HOME -name java -type f /usr/local/java/jdk1.7.0_67/jre/bin/java /usr/local/java/jdk1.7.0_67/bin/java On another node, the same command returned entries in different order: find -L $JAVA_HOME -name java -type f /usr/local/java/jdk1.7.0_67/bin/java /usr/local/java/jdk1.7.0_67/jre/bin/java The complete command picks the first one returned which may not be the same on each node: find -L $JAVA_HOME -name java -type f | head -n 1 /usr/local/java/jdk1.7.0_67/jre/bin/java If JAVA_HOME is found, we should just append the "bin/java" to the path" JAVA=$JAVA_HOME/bin/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-5176) Rest API for stats.json gives HTTP 404 Not Found error
[
https://issues.apache.org/jira/browse/DRILL-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-5176.
--
Verified that rest api for cluster.json returned expected data.
> Rest API for stats.json gives HTTP 404 Not Found error
> ---
>
> Key: DRILL-5176
> URL: https://issues.apache.org/jira/browse/DRILL-5176
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> commit id: bbcf4b765e6946a8b6c7110372c4e1cadbfefa44
> rest api for stats.json now returns "HTTP 404 Not Found" error.
> curl -X GET -i http://:8047/stats.json
> HTTP/1.1 500 Internal Server Error
> Content-Type: application/json
> Content-Length: 43
> Server: Jetty(9.1.5.v20140505)
> {
> "errorMessage" : "HTTP 404 Not Found"
> }
> The following data is returned for the same rest api in previous drill
> version:
> curl -X GET -i http://:8047/stats.json
> HTTP/1.1 200 OK
> Content-Type: application/json
> Content-Length: 398
> Server: Jetty(9.1.5.v20140505)
> [ {
> "name" : "Number of Drill Bits",
> "value" : 1
> }, {
> "name" : "Bit #0",
> "value" : "abc.qa.lab initialized"
> }, {
> "name" : "Data Port Address",
> "value" : "abc.qa.lab:31012"
> }, {
> "name" : "User Port Address",
> "value" : "abc.qa.lab:31010"
> }, {
> "name" : "Control Port Address",
> "value" : "abc.qa.lab:31011"
> }, {
> "name" : "Maximum Direct Memory",
> "value" : 8589934592
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-5176) Rest API for stats.json gives HTTP 404 Not Found error
[
https://issues.apache.org/jira/browse/DRILL-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15801916#comment-15801916
]
Krystal edited comment on DRILL-5176 at 1/5/17 5:19 PM:
Thanks Arina for the clarification. Marking jira as resolved.
Verified that rest api for cluster.json returned expected data.
was (Author: knguyen):
Thanks Arina for the clarification. Marking jira as resolved.
> Rest API for stats.json gives HTTP 404 Not Found error
> ---
>
> Key: DRILL-5176
> URL: https://issues.apache.org/jira/browse/DRILL-5176
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> commit id: bbcf4b765e6946a8b6c7110372c4e1cadbfefa44
> rest api for stats.json now returns "HTTP 404 Not Found" error.
> curl -X GET -i http://:8047/stats.json
> HTTP/1.1 500 Internal Server Error
> Content-Type: application/json
> Content-Length: 43
> Server: Jetty(9.1.5.v20140505)
> {
> "errorMessage" : "HTTP 404 Not Found"
> }
> The following data is returned for the same rest api in previous drill
> version:
> curl -X GET -i http://:8047/stats.json
> HTTP/1.1 200 OK
> Content-Type: application/json
> Content-Length: 398
> Server: Jetty(9.1.5.v20140505)
> [ {
> "name" : "Number of Drill Bits",
> "value" : 1
> }, {
> "name" : "Bit #0",
> "value" : "abc.qa.lab initialized"
> }, {
> "name" : "Data Port Address",
> "value" : "abc.qa.lab:31012"
> }, {
> "name" : "User Port Address",
> "value" : "abc.qa.lab:31010"
> }, {
> "name" : "Control Port Address",
> "value" : "abc.qa.lab:31011"
> }, {
> "name" : "Maximum Direct Memory",
> "value" : 8589934592
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-5176) Rest API for stats.json gives HTTP 404 Not Found error
[
https://issues.apache.org/jira/browse/DRILL-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal resolved DRILL-5176.
Resolution: Not A Problem
Thanks Arina for the clarification. Marking jira as resolved.
> Rest API for stats.json gives HTTP 404 Not Found error
> ---
>
> Key: DRILL-5176
> URL: https://issues.apache.org/jira/browse/DRILL-5176
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.10.0
>Reporter: Krystal
>
> commit id: bbcf4b765e6946a8b6c7110372c4e1cadbfefa44
> rest api for stats.json now returns "HTTP 404 Not Found" error.
> curl -X GET -i http://:8047/stats.json
> HTTP/1.1 500 Internal Server Error
> Content-Type: application/json
> Content-Length: 43
> Server: Jetty(9.1.5.v20140505)
> {
> "errorMessage" : "HTTP 404 Not Found"
> }
> The following data is returned for the same rest api in previous drill
> version:
> curl -X GET -i http://:8047/stats.json
> HTTP/1.1 200 OK
> Content-Type: application/json
> Content-Length: 398
> Server: Jetty(9.1.5.v20140505)
> [ {
> "name" : "Number of Drill Bits",
> "value" : 1
> }, {
> "name" : "Bit #0",
> "value" : "abc.qa.lab initialized"
> }, {
> "name" : "Data Port Address",
> "value" : "abc.qa.lab:31012"
> }, {
> "name" : "User Port Address",
> "value" : "abc.qa.lab:31010"
> }, {
> "name" : "Control Port Address",
> "value" : "abc.qa.lab:31011"
> }, {
> "name" : "Maximum Direct Memory",
> "value" : 8589934592
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5176) Rest API for stats.json gives HTTP 404 Not Found error
Krystal created DRILL-5176:
--
Summary: Rest API for stats.json gives HTTP 404 Not Found error
Key: DRILL-5176
URL: https://issues.apache.org/jira/browse/DRILL-5176
Project: Apache Drill
Issue Type: Bug
Components: Client - HTTP
Affects Versions: 1.10.0
Reporter: Krystal
commit id: bbcf4b765e6946a8b6c7110372c4e1cadbfefa44
rest api for stats.json now returns "HTTP 404 Not Found" error.
curl -X GET -i http://:8047/stats.json
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Content-Length: 43
Server: Jetty(9.1.5.v20140505)
{
"errorMessage" : "HTTP 404 Not Found"
}
The following data is returned for the same rest api in previous drill version:
curl -X GET -i http://:8047/stats.json
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 398
Server: Jetty(9.1.5.v20140505)
[ {
"name" : "Number of Drill Bits",
"value" : 1
}, {
"name" : "Bit #0",
"value" : "abc.qa.lab initialized"
}, {
"name" : "Data Port Address",
"value" : "abc.qa.lab:31012"
}, {
"name" : "User Port Address",
"value" : "abc.qa.lab:31010"
}, {
"name" : "Control Port Address",
"value" : "abc.qa.lab:31011"
}, {
"name" : "Maximum Direct Memory",
"value" : 8589934592
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[ https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4604. -- git.commit.id=cf2b7c70ea47e604364ccb5e8076bbd7eae310c3 Verified that the WebUI displays expected warnings and versions of each drill-bits based on the current drillbit. > Generate warning on Web UI if drillbits version mismatch is detected > > > Key: DRILL-4604 > URL: https://issues.apache.org/jira/browse/DRILL-4604 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Labels: doc-impacting, ready-to-commit > Fix For: 1.10.0 > > Attachments: NEW_matching_drillbits.JPG, > NEW_mismatching_drillbits.JPG, index_page.JPG, index_page_mismatch.JPG, > screenshots_with_different_states.docx > > > Display drillbit version on web UI. If any of drillbits version doesn't match > with current drillbit, generate warning. > Screenshots - NEW_matching_drillbits.JPG, NEW_mismatching_drillbits.JPG -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5042) Settings in drill-env.sh are not overriding the same settings in distrib-env.sh
Krystal created DRILL-5042: -- Summary: Settings in drill-env.sh are not overriding the same settings in distrib-env.sh Key: DRILL-5042 URL: https://issues.apache.org/jira/browse/DRILL-5042 Project: Apache Drill Issue Type: Bug Components: Tools, Build & Test Affects Versions: 1.9.0 Reporter: Krystal Assignee: Paul Rogers commit id: d925f738d006caedf8635db1024825e355b5dada currently, settings from drill-env.sh are not overriding those from distrib-env.sh. Instead, the ones from distrib-env.sh are overriding the same setting from drill-env.sh. The correct behavior is that the settings from drill-env.sh should override the same settings in distrib-env.sh. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4986) Allow users to customize the Drill log file name
[
https://issues.apache.org/jira/browse/DRILL-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-4986.
--
Verified that drill log file name can be customized.
In drill-env.sh, the folloiwng is set:
export DRILL_LOG_NAME="drillbit-$HOSTNAME"
After drillbit is restarted,log files containng the drillbit-.out and
.log are created in the log directory.
> Allow users to customize the Drill log file name
>
>
> Key: DRILL-4986
> URL: https://issues.apache.org/jira/browse/DRILL-4986
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.9.0
>
>
> From John Omernik on the user list:
> I am getting my head around the new Drill config files in 1.8, and I have
> been setting DRILL_LOG_PREFIX in previous versions. Based on what I am
> seeing in drill-config.sh though, it looks like I no longer can set this as
> instead of using a "If set use what's set otherwise default to" methodology
> like other variables, drill-config.sh just exports a static value... is
> this correct?
> My goal is to have logs all in a single directory but have prefixes based
> on the host names... "centralized" logging if you will :)
> Suggestion:
> {code}
> export DRILL_LOG_PREFIX="$DRILL_LOG_DIR/drillbit"
> {code}
> So looking at this, I think that DRILL_LOG_PREFIX is a misnomer, it should
> be DRILL_LOG_PATH = Path to a directory, DRILL_LOG_PREFIX = A prefix to
> prepend to drill log files, thus, in reality, DRILL_LOG_PREFIX should not
> be set to $DRILL_LOG_DIR/drillbit, instead the default should be "
> {code}
> export DRILL_LOG_PREFIX=${DRILL_LOG_PREFIX:-"drillbit"}
> {code}
> and then the next line that makes the drill log path should be:
> {code}
> export DRILLBIT_LOG_PATH="${DRILL_LOG_DIR}/${DRILL_LOG_PREFIX}.log"
> {code}
> That way the Prefix is just a file prefix, and then in the drill-env.sh I
> could set
> {code}
> export DRILL_LOG_PREFIX="drillbit-$HOSTNAME"
> {code}
> and have all my logs in one folder, but not be overwritten by easy other
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4921) Scripts drill_config.sh, drillbit.sh, and drill-embedded fail when accessed via a symbolic link
[ https://issues.apache.org/jira/browse/DRILL-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4921. -- Verified on Mac and Linux that drill starts up successfully using symbollic links. > Scripts drill_config.sh, drillbit.sh, and drill-embedded fail when accessed > via a symbolic link > > > Key: DRILL-4921 > URL: https://issues.apache.org/jira/browse/DRILL-4921 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.8.0 > Environment: The drill-embedded on the Mac; the other files on Linux >Reporter: Boaz Ben-Zvi >Assignee: Boaz Ben-Zvi >Priority: Minor > Fix For: 1.9.0 > > > Several of the drill... scripts under $DRILL_HOME/bin use "pwd" to produce > the local path of that script. However "pwd" defaults to "logical" (i.e. the > same as "pwd -L"); so if accessed via a symbolic link, that link is used > verbatim in the path, which can produce wrong paths (e.g., when followed by > "cd .."). > For example, creating a symbolic link and using it (on the Mac): > $ cd ~/drill > $ ln -s $DRILL_HOME/bin > $ bin/drill-embedded > ERROR: Drill config file missing: > /Users/boazben-zvi/drill/conf/drill-override.conf -- Wrong config dir? > Similarly on Linux the CLASS_PATH gets set wrong (when running "drillbit.sh > start" via a symlink). > Solution: need to replace all the "pwd" in all the scripts with "pwd -P" > which produces the Physical path. (Or replace a preceding "cd" with "cd -P" > which does the same). > Relevant scripts: > = > $ cd bin; grep pwd * > drillbit.sh:bin=`cd "$bin">/dev/null; pwd` > drillbit.sh: echo "cwd:" `pwd` > drill-conf:bin=`cd "$bin">/dev/null; pwd` > drill-config.sh:home=`cd "$bin/..">/dev/null; pwd` > drill-config.sh: DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" > drill-config.sh:JAVA_HOME="$( cd -P "$( dirname "$SOURCE" )" && cd .. && > pwd )" > drill-embedded:bin=`cd "$bin">/dev/null; pwd` > drill-localhost:bin=`cd "$bin">/dev/null; pwd` > submit_plan:bin=`cd "$bin">/dev/null; pwd` > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4792) Include session options used for a query as part of the profile
[ https://issues.apache.org/jira/browse/DRILL-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4792. -- Verified that feature is working as expected. There is a minor issue with the formatting of the jsonOptions in the query's Json Profile. Open DRILL-5036 for this issue. > Include session options used for a query as part of the profile > --- > > Key: DRILL-4792 > URL: https://issues.apache.org/jira/browse/DRILL-4792 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Labels: doc-impacting > Fix For: 1.9.0 > > Attachments: no_session_options.JPG, session_options_block.JPG, > session_options_collapsed.JPG, session_options_json.JPG > > > Include session options used for a query as part of the profile. > This will be very useful for debugging/diagnostics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5036) The session options used for a query is not nicely formated in the JSON Profile
Krystal created DRILL-5036:
--
Summary: The session options used for a query is not nicely
formated in the JSON Profile
Key: DRILL-5036
URL: https://issues.apache.org/jira/browse/DRILL-5036
Project: Apache Drill
Issue Type: Bug
Components: Client - HTTP
Affects Versions: 1.9.0
Reporter: Krystal
Priority: Minor
The optionsJson section in the Json Profile from the WebUI and profile json
file shows the sessions used for a query displays escape characters and newline
characters:
"optionsJson": "[ {\n \"kind\" : \"BOOLEAN\",\n \"type\" : \"SESSION\",\n
\"name\" : \"planner.enable_decimal_data_type\",\n \"bool_val\" : false\n},
{\n \"kind\" : \"BOOLEAN\",\n \"type\" : \"SESSION\",\n \"name\" :
\"store.parquet.reader.int96_as_timestamp\",\n \"bool_val\" : true\n}, {\n
\"kind\" : \"LONG\",\n \"type\" : \"SESSION\",\n \"name\" :
\"web.logs.max_lines\",\n \"num_val\" : 5000\n} ]"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (DRILL-4373) Drill and Hive have incompatible timestamp representations in parquet
[ https://issues.apache.org/jira/browse/DRILL-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal reopened DRILL-4373: Re-open jira as this breaks existing behavior as described in https://issues.apache.org/jira/browse/DRILL-5034. > Drill and Hive have incompatible timestamp representations in parquet > - > > Key: DRILL-4373 > URL: https://issues.apache.org/jira/browse/DRILL-4373 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive, Storage - Parquet >Affects Versions: 1.8.0 >Reporter: Rahul Challapalli >Assignee: Vitalii Diravka > Labels: doc-impacting > Fix For: 1.9.0 > > > git.commit.id.abbrev=83d460c > I created a parquet file with a timestamp type using Drill. Now if I define a > hive table on top of the parquet file and use "timestamp" as the column type, > drill fails to read the hive table through the hive storage plugin > Implementation: > Added int96 to timestamp converter for both parquet readers and controling it > by system / session option "store.parquet.int96_as_timestamp". > The value of the option is false by default for the proper work of the old > query scripts with the "convert_from TIMESTAMP_IMPALA" function. > When the option is true using of that function is unnesessary and can lead to > the query fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5034) Select timestamp from hive generated parquet always return in UTC
Krystal created DRILL-5034:
--
Summary: Select timestamp from hive generated parquet always
return in UTC
Key: DRILL-5034
URL: https://issues.apache.org/jira/browse/DRILL-5034
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.9.0
Reporter: Krystal
commit id: 5cea9afa6278e21574c6a982ae5c3d82085ef904
Reading timestamp data against a hive parquet table from drill automatically
converts the timestamp data to UTC.
SELECT TIMEOFDAY() FROM (VALUES(1));
+--+
|EXPR$0|
+--+
| 2016-11-10 12:33:26.547 America/Los_Angeles |
+--+
data schema:
message hive_schema {
optional int32 voter_id;
optional binary name (UTF8);
optional int32 age;
optional binary registration (UTF8);
optional fixed_len_byte_array(3) contributions (DECIMAL(6,2));
optional int32 voterzone;
optional int96 create_timestamp;
optional int32 create_date (DATE);
}
Using drill-1.8, the returned timestamps match the table data:
select convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from
`/user/hive/warehouse/voter_hive_parquet` limit 5;
++
| EXPR$0 |
++
| 2016-10-23 20:03:58.0 |
| null |
| 2016-09-09 12:01:18.0 |
| 2017-03-06 20:35:55.0 |
| 2017-01-20 22:32:43.0 |
++
5 rows selected (1.032 seconds)
If the user timzone is changed to UTC, then the timestamp data is returned in
UTC time.
Using drill-1.9, the returned timestamps got converted to UTC eventhough the
user timezone is in PST.
select convert_from(create_timestamp, 'TIMESTAMP_IMPALA') from
dfs.`/user/hive/warehouse/voter_hive_parquet` limit 5;
++
| EXPR$0 |
++
| 2016-10-24 03:03:58.0 |
| null |
| 2016-09-09 19:01:18.0 |
| 2017-03-07 04:35:55.0 |
| 2017-01-21 06:32:43.0 |
++
alter session set `store.parquet.reader.int96_as_timestamp`=true;
+---+---+
| ok | summary |
+---+---+
| true | store.parquet.reader.int96_as_timestamp updated. |
+---+---+
select create_timestamp from dfs.`/user/hive/warehouse/voter_hive_parquet`
limit 5;
++
|create_timestamp|
++
| 2016-10-24 03:03:58.0 |
| null |
| 2016-09-09 19:01:18.0 |
| 2017-03-07 04:35:55.0 |
| 2017-01-21 06:32:43.0 |
++
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4870) drill-config.sh sets JAVA_HOME incorrectly for the Mac
[ https://issues.apache.org/jira/browse/DRILL-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4870. -- Verified on Mac the without setting JAVA_HOME, drill does set it and sqlline in embedded mode starts successfully. > drill-config.sh sets JAVA_HOME incorrectly for the Mac > -- > > Key: DRILL-4870 > URL: https://issues.apache.org/jira/browse/DRILL-4870 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0 > Environment: MacOS with unset JAVA_HOME >Reporter: Paul Rogers >Assignee: Chunhui Shi >Priority: Minor > Fix For: 1.9.0 > > > It turns out that drill-config.sh is both improperly and unnecessarily > setting the JAVA_HOME envrironment variable. That setting should be removed. > In the Drill 1.7 version, drill-config.sh checks if the JAVA_HOME environment > variable is set. If not, it sets JAVA_HOME based on its guess as to the > proper value. > In the 1.7 version, the veriable was set, but not exported, so the variable > was never actually used. > The recent script fixes for 1.8 "fixed" the export problem. The fix works > fine on Linux. But, the Java install on the Mac has a different structure > than that on Linux. The value that drill-config.sh guesses is fine for Linux, > wrong for the Mac. > When we export the (wrong) JAVA_HOME, Mac users who have not set JAVA_HOME > will get the following error when using a Drill script: > ./drill-embedded > Unable to locate an executable at > "/System/Library/Frameworks/JavaVM.framework/Versions/A/bin/java" > Mac users who do set JAVA_HOME will not encounter the problem (because > drill-config.sh does not change an existing value.) > It seems likely that someone in the past ecountered the same problem and > removed the export of DRILL_HOME as an attempt to fix the problem. > As it turns out, Java does know how to set JAVA_HOME properly if not set. So, > setting JAVA_HOME is unnecessary. > The proper fix is to remove JAVA_HOME setting from drill-config.sh. > The workaround for any 1.8 user who encounters the problem is to edit their > $DRILL_HOME/bin/drill-config.sh file and delete this line near the end of the > file: > export JAVA_HOME -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4964. -- commit id: 190d5d46d8ec21efa31ee6dcb3d0d031efde4280 Verified that after restarting hive metastore, hive tables and databases are still accessible from sqlline without restarting drill. Also, show databases and show table returned expected result. > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > Fix For: 1.9.0 > > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store
[jira] [Closed] (DRILL-4911) SimpleParallelizer should avoid plan serialization for logging purpose when debug logging is not enabled.
[ https://issues.apache.org/jira/browse/DRILL-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4911. -- Per Jinfeng's previous comment, closing bug as fixed. > SimpleParallelizer should avoid plan serialization for logging purpose when > debug logging is not enabled. > - > > Key: DRILL-4911 > URL: https://issues.apache.org/jira/browse/DRILL-4911 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni >Priority: Minor > Fix For: 1.9.0 > > > SimpleParallelizer and subclass have code to put serialized plan in debug > log. However, in case when the physical plan fragment is very large (table is > huge with large # of partitions, files, etc), the overhead of serialization > is not negligible. The code seems to always do serialization, regardless > whether the debug flag is turned on. > To reduce the overhead, SimpleParallelizer and subclass should only do plan > serialization for logging purpose when debug logging is turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4972) Drillbit shuts down immediately after starting if embedded web server is disabled
[ https://issues.apache.org/jira/browse/DRILL-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4972. -- commit id: 190d5d46d8ec21efa31ee6dcb3d0d031efde4280 Duplicated failure on drill-1.8. With drill-1.9, using the same re-production, drillbit started up successfully. > Drillbit shuts down immediately after starting if embedded web server is > disabled > - > > Key: DRILL-4972 > URL: https://issues.apache.org/jira/browse/DRILL-4972 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.8.0 >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam >Priority: Critical > Fix For: 1.9.0 > > > Disable embedded web server by setting "drill.exec.http.enabled" to false. > Now when drillbit is started, it shuts down immediately after starting. > JVM exits when the only threads running are all daemon threads. Turns out all > threads in a drillbit, other than the thread pool started by the web server, > are daemon. So I suggest WorkManager#StatusThread be made non-daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-3178) csv reader should allow newlines inside quotes
[ https://issues.apache.org/jira/browse/DRILL-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-3178. -- Verified and added test cases to automation suite. > csv reader should allow newlines inside quotes > --- > > Key: DRILL-3178 > URL: https://issues.apache.org/jira/browse/DRILL-3178 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text & CSV >Affects Versions: 1.0.0 > Environment: Ubuntu Trusty 14.04.2 LTS >Reporter: Neal McBurnett >Assignee: F Méthot > Fix For: 1.9.0 > > Attachments: drill-3178.patch > > > When reading a csv file which contains newlines within quoted strings, e.g. > via > select * from dfs.`/tmp/q.csv`; > Drill 1.0 says: > Error: SYSTEM ERROR: com.univocity.parsers.common.TextParsingException: > Error processing input: Cannot use newline character within quoted string > But many tools produce csv files with newlines in quoted strings. Drill > should be able to handle them. > Workaround: the csvquote program (https://github.com/dbro/csvquote) can > encode embedded commas and newlines, and even decode them later if desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4499) Remove unused classes
[ https://issues.apache.org/jira/browse/DRILL-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4499. -- Recent nightly functional runs that include this fix ran successfully. > Remove unused classes > - > > Key: DRILL-4499 > URL: https://issues.apache.org/jira/browse/DRILL-4499 > Project: Apache Drill > Issue Type: Task >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > Fix For: 1.8.0 > > > List of unused classes that I've tracked over time: > exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestPrune.java > exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/MethodMap.java > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionBody.java > exec/java-exec/src/main/java/org/apache/drill/exec/ops/Multitimer.java > exec/java-exec/src/main/java/org/apache/drill/exec/ops/QuerySetup.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/AvailabilityListener.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlCommand.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/SendProgress.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/SendProgress.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/DrillUser.java > exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordRecorder.java > exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/PartialWork.java > exec/java-exec/src/main/java/org/apache/drill/exec/util/AtomicState.java > exec/java-exec/src/main/java/org/apache/drill/exec/work/RecordOutputStream.java > exec/java-exec/src/main/java/org/apache/drill/exec/work/ResourceRequest.java > exec/java-exec/src/main/java/org/apache/drill/exec/work/RootNodeDriver.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3178) csv reader should allow newlines inside quotes
[ https://issues.apache.org/jira/browse/DRILL-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626977#comment-15626977 ] Krystal commented on DRILL-3178: commit id: 83513daf0903e0d94fcaad7b1ae4e8ad6272b494 Using data from comment #2, verified that data gets returned as expected. select * from `drill-3178/drill3178.csv`; +--+ | columns | +--+ | ["1","line1"]| | ["2","line2\n"] | | ["3","line3"]| +--+ 3 rows selected (0.158 seconds) select columns[0], columns[1] from `drill-3178/drill3178.csv`; +-+-+ | EXPR$0 | EXPR$1 | +-+-+ | 1 | line1 | | 2 | line2 | | 3 | line3 | +-+-+ select columns[0],columns[1] from `drill-3178/drill3178.csv` where columns[0] > 1 order by columns[1] desc; +-+-+ | EXPR$0 | EXPR$1 | +-+-+ | 3 | line3 | | 2 | line2 | +-+-+ 2 rows selected (0.373 seconds) > csv reader should allow newlines inside quotes > --- > > Key: DRILL-3178 > URL: https://issues.apache.org/jira/browse/DRILL-3178 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text & CSV >Affects Versions: 1.0.0 > Environment: Ubuntu Trusty 14.04.2 LTS >Reporter: Neal McBurnett >Assignee: F Méthot > Fix For: 1.9.0 > > Attachments: drill-3178.patch > > > When reading a csv file which contains newlines within quoted strings, e.g. > via > select * from dfs.`/tmp/q.csv`; > Drill 1.0 says: > Error: SYSTEM ERROR: com.univocity.parsers.common.TextParsingException: > Error processing input: Cannot use newline character within quoted string > But many tools produce csv files with newlines in quoted strings. Drill > should be able to handle them. > Workaround: the csvquote program (https://github.com/dbro/csvquote) can > encode embedded commas and newlines, and even decode them later if desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4949) Need better handling of empty parquet files
Krystal created DRILL-4949:
--
Summary: Need better handling of empty parquet files
Key: DRILL-4949
URL: https://issues.apache.org/jira/browse/DRILL-4949
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.9.0
Reporter: Krystal
I have an empty parquet file created from hive. When I tried to query against
this table I got "IllegalArgumentException".
{code}
select * from `test_dir/voter_empty`;
Error: SYSTEM ERROR: IllegalArgumentException: MinorFragmentId 0 has no read
entries assigned
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
during fragment initialization: MinorFragmentId 0 has no read entries assigned
org.apache.drill.exec.work.foreman.Foreman.run():281
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.IllegalArgumentException) MinorFragmentId 0 has no read
entries assigned
com.google.common.base.Preconditions.checkArgument():122
org.apache.drill.exec.store.parquet.ParquetGroupScan.getSpecificScan():824
org.apache.drill.exec.store.parquet.ParquetGroupScan.getSpecificScan():101
org.apache.drill.exec.planner.fragment.Materializer.visitGroupScan():68
org.apache.drill.exec.planner.fragment.Materializer.visitGroupScan():35
org.apache.drill.exec.physical.base.AbstractGroupScan.accept():63
org.apache.drill.exec.planner.fragment.Materializer.visitOp():102
org.apache.drill.exec.planner.fragment.Materializer.visitOp():35
org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitProject():79
org.apache.drill.exec.physical.config.Project.accept():51
org.apache.drill.exec.planner.fragment.Materializer.visitStore():82
org.apache.drill.exec.planner.fragment.Materializer.visitStore():35
org.apache.drill.exec.physical.base.AbstractPhysicalVisitor.visitScreen():202
org.apache.drill.exec.physical.config.Screen.accept():98
org.apache.drill.exec.planner.fragment.SimpleParallelizer.generateWorkUnit():283
org.apache.drill.exec.planner.fragment.SimpleParallelizer.getFragments():127
org.apache.drill.exec.work.foreman.Foreman.getQueryWorkUnit():596
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan():426
org.apache.drill.exec.work.foreman.Foreman.runSQL():1010
org.apache.drill.exec.work.foreman.Foreman.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745 (state=,code=0)
{code}
Either drill should block the query and display a user friendly error message
or allow the query to run and return empty result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4879) Hive default storage plugin template shows "null" for fresh install
[
https://issues.apache.org/jira/browse/DRILL-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15469013#comment-15469013
]
Krystal commented on DRILL-4879:
Looks like commitId 8bbf6589ad746b8949c962875f0ea2de59a13f79 (DRILL-4819:
Update MapR version to 5.2.0) is the cause of the bug.
> Hive default storage plugin template shows "null" for fresh install
> ---
>
> Key: DRILL-4879
> URL: https://issues.apache.org/jira/browse/DRILL-4879
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Krystal
>Assignee: Sudheesh Katkam
>
> Deleted the /drill folder from zookeeper; then install drill-1.8.
> [root@mfs41 drill-1.8.0]# cat git.properties
> #Generated by Git-Commit-Id-Plugin
> #Thu Sep 01 20:58:58 UTC 2016
> git.commit.id.abbrev=3b5bedd
> git.commit.user.email=prog...@maprtech.com
> git.commit.message.full=One additional fix fo the DoY unit tests.\n
> git.commit.id=3b5beddce3c1850eaba57aea2d8e7b7d22e641d6
> git.commit.message.short=One additional fix fo the DoY unit tests.
> git.commit.user.name=Paul Rogers
> From the UI, open up the Hive storage plugin. The content of the plugin
> showed as "null" instead of the default template. Below is the snippit of
> the error in drillbit.log that gets generated when I clicked on the "Hive"
> storage plugin:
> {code}
> 2016-09-06 15:08:19,581 [qtp1078495730-50] INFO
> o.a.d.e.server.rest.StorageResources - Failure while trying to access storage
> config: hive
> org.apache.drill.common.exceptions.ExecutionSetupException: Failure setting
> up new storage plugin configuration for config
> org.apache.drill.exec.store.hive.HiveStoragePluginConfig@4adf869f
> at
> org.apache.drill.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:325)
> ~[drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:213)
> ~[drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:263)
> ~[drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.server.rest.StorageResources.getStoragePluginJSON(StorageResources.java:101)
> [drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.server.rest.StorageResources.getStoragePlugin(StorageResources.java:115)
> [drill-java-exec-1.8.0.jar:1.8.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_45]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
> [jersey-server-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[jira] [Created] (DRILL-4879) Hive default storage plugin template shows "null" for fresh install
Krystal created DRILL-4879:
--
Summary: Hive default storage plugin template shows "null" for
fresh install
Key: DRILL-4879
URL: https://issues.apache.org/jira/browse/DRILL-4879
Project: Apache Drill
Issue Type: Bug
Components: Storage - Hive
Affects Versions: 1.8.0, 1.9.0
Reporter: Krystal
Deleted the /drill folder from zookeeper; then install drill-1.8.
[root@mfs41 drill-1.8.0]# cat git.properties
#Generated by Git-Commit-Id-Plugin
#Thu Sep 01 20:58:58 UTC 2016
git.commit.id.abbrev=3b5bedd
git.commit.user.email=prog...@maprtech.com
git.commit.message.full=One additional fix fo the DoY unit tests.\n
git.commit.id=3b5beddce3c1850eaba57aea2d8e7b7d22e641d6
git.commit.message.short=One additional fix fo the DoY unit tests.
git.commit.user.name=Paul Rogers
>From the UI, open up the Hive storage plugin. The content of the plugin
>showed as "null" instead of the default template. Below is the snippit of the
>error in drillbit.log that gets generated when I clicked on the "Hive" storage
>plugin:
{code}
2016-09-06 15:08:19,581 [qtp1078495730-50] INFO
o.a.d.e.server.rest.StorageResources - Failure while trying to access storage
config: hive
org.apache.drill.common.exceptions.ExecutionSetupException: Failure setting up
new storage plugin configuration for config
org.apache.drill.exec.store.hive.HiveStoragePluginConfig@4adf869f
at
org.apache.drill.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:325)
~[drill-java-exec-1.8.0.jar:1.8.0]
at
org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:213)
~[drill-java-exec-1.8.0.jar:1.8.0]
at
org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:263)
~[drill-java-exec-1.8.0.jar:1.8.0]
at
org.apache.drill.exec.server.rest.StorageResources.getStoragePluginJSON(StorageResources.java:101)
[drill-java-exec-1.8.0.jar:1.8.0]
at
org.apache.drill.exec.server.rest.StorageResources.getStoragePlugin(StorageResources.java:115)
[drill-java-exec-1.8.0.jar:1.8.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_45]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_45]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[jersey-common-2.8.jar:na]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
[jersey-common-2.8.jar:na]
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
[jersey-container-servlet-core-2.8.jar:na]
[jira] [Closed] (DRILL-4175) IOBE may occur in Calcite RexProgramBuilder when queries are submitted concurrently
[
https://issues.apache.org/jira/browse/DRILL-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal closed DRILL-4175.
--
Verified that bug is fixed.
> IOBE may occur in Calcite RexProgramBuilder when queries are submitted
> concurrently
> ---
>
> Key: DRILL-4175
> URL: https://issues.apache.org/jira/browse/DRILL-4175
> Project: Apache Drill
> Issue Type: Bug
> Environment: distribution
>Reporter: huntersjm
> Fix For: 1.8.0
>
>
> I queryed a sql just like `selelct v from table limit 1`,I get a error:
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IndexOutOfBoundsException: Index: 68, Size: 67
> After debug, I found there is a bug in calcite parse:
> first we look line 72 in org.apache.calcite.rex.RexProgramBuilder
> {noformat}
>registerInternal(RexInputRef.of(i, fields), false);
> {noformat}
> there we get RexInputRef from RexInputRef.of, and it has a method named
> createName(int idex), here NAMES is SelfPopulatingList.class.
> SelfPopulatingList.class describe as Thread-safe list, but in fact it is
> Thread-unsafe. when NAMES.get(index) is called distributed, it gets a error.
> We hope SelfPopulatingList.class to be {$0 $1 $2 $n}, but when it called
> distributed, it may be {$0,$1...$29,$30...$59,$30,$31...$59...}.
> We see method registerInternal
> {noformat}
> private RexLocalRef registerInternal(RexNode expr, boolean force) {
> expr = simplify(expr);
> RexLocalRef ref;
> final Pair key;
> if (expr instanceof RexLocalRef) {
> key = null;
> ref = (RexLocalRef) expr;
> } else {
> key = RexUtil.makeKey(expr);
> ref = exprMap.get(key);
> }
> if (ref == null) {
> if (validating) {
> validate(
> expr,
> exprList.size());
> }
> {noformat}
> Here makeKey(expr) hope to get different key, however it get same key, so
> addExpr(expr) called less, in this method
> {noformat}
> RexLocalRef ref;
> final int index = exprList.size();
> exprList.add(expr);
> ref =
> new RexLocalRef(
> index,
> expr.getType());
> localRefList.add(ref);
> return ref;
> {noformat}
> localRefList get error size, so in line 939,
> {noformat}
> final RexLocalRef ref = localRefList.get(index);
> {noformat}
> throw IndexOutOfBoundsException
> bugfix:
> We can't change origin code of calcite before they fix this bug, so we can
> init NAMEs in RexLocalRef on start. Just add
> {noformat}
> RexInputRef.createName(2048);
> {noformat}
> on Bootstrap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4175) IOBE may occur in Calcite RexProgramBuilder when queries are submitted concurrently
[
https://issues.apache.org/jira/browse/DRILL-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krystal updated DRILL-4175:
---
Reviewer: Krystal (was: Rahul Challapalli)
> IOBE may occur in Calcite RexProgramBuilder when queries are submitted
> concurrently
> ---
>
> Key: DRILL-4175
> URL: https://issues.apache.org/jira/browse/DRILL-4175
> Project: Apache Drill
> Issue Type: Bug
> Environment: distribution
>Reporter: huntersjm
> Fix For: 1.8.0
>
>
> I queryed a sql just like `selelct v from table limit 1`,I get a error:
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IndexOutOfBoundsException: Index: 68, Size: 67
> After debug, I found there is a bug in calcite parse:
> first we look line 72 in org.apache.calcite.rex.RexProgramBuilder
> {noformat}
>registerInternal(RexInputRef.of(i, fields), false);
> {noformat}
> there we get RexInputRef from RexInputRef.of, and it has a method named
> createName(int idex), here NAMES is SelfPopulatingList.class.
> SelfPopulatingList.class describe as Thread-safe list, but in fact it is
> Thread-unsafe. when NAMES.get(index) is called distributed, it gets a error.
> We hope SelfPopulatingList.class to be {$0 $1 $2 $n}, but when it called
> distributed, it may be {$0,$1...$29,$30...$59,$30,$31...$59...}.
> We see method registerInternal
> {noformat}
> private RexLocalRef registerInternal(RexNode expr, boolean force) {
> expr = simplify(expr);
> RexLocalRef ref;
> final Pair key;
> if (expr instanceof RexLocalRef) {
> key = null;
> ref = (RexLocalRef) expr;
> } else {
> key = RexUtil.makeKey(expr);
> ref = exprMap.get(key);
> }
> if (ref == null) {
> if (validating) {
> validate(
> expr,
> exprList.size());
> }
> {noformat}
> Here makeKey(expr) hope to get different key, however it get same key, so
> addExpr(expr) called less, in this method
> {noformat}
> RexLocalRef ref;
> final int index = exprList.size();
> exprList.add(expr);
> ref =
> new RexLocalRef(
> index,
> expr.getType());
> localRefList.add(ref);
> return ref;
> {noformat}
> localRefList get error size, so in line 939,
> {noformat}
> final RexLocalRef ref = localRefList.get(index);
> {noformat}
> throw IndexOutOfBoundsException
> bugfix:
> We can't change origin code of calcite before they fix this bug, so we can
> init NAMEs in RexLocalRef on start. Just add
> {noformat}
> RexInputRef.createName(2048);
> {noformat}
> on Bootstrap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads
[
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437703#comment-15437703
]
Krystal commented on DRILL-4766:
Used the following steps to re-produce problem without the fix:
1. Added 1 schemas/tables to hive.
2. Lowered the rpc bit timeout (drillbit heart beat) to 10 seconds in
drill-override.conf (rpc.bit.timeout: 10).
3. Ran the following query:
select DISTINCT TABLE_TYPE from INFORMATION_SCHEMA.`TABLES` WHERE TABLE_TYPE
LIKE '%';
4. Cancelled the query.
5. Kicked off several queries from different clients.
The queries from Step 5 that connect to the same foreman as the cancelled query
consistently fail with error:
{code}
ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
Connection: /10.10.100.123:60884 <--> /10.10.100.123:31011 (control client).
Closing connection.
java.io.IOException: syscall:read(...)() failed: Connection reset by peer
2016-08-09 14:50:10,285 [BitServer-3] ERROR
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL
fragment query_id {
part1: 2906422873165293041
part2: 4085948525804402223
}
major_fragment_id: 2
minor_fragment_id: 1
on endpoint address: "qa-node111"
user_port: 31010
control_port: 31011
data_port: 31012
with org.apache.drill.exec.rpc.ChannelClosedException: Channel closed
/10.10.100.123:60884 <--> /10.10.100.123:31011..
2016-08-09 14:50:10,286 [BitServer-3] ERROR
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL
fragment query_id {
part1: 2906422873165293041
part2: 4085948525804402223
}
{code}
Executed the same steps against build with the fix and did not encounter any
failures.
> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.8.0
>
>
> Currently, rpc thread can block when trying to deliver a cancel or early
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios.
> FragmentExecutor could be improved to avoid blocking rpc threads as well
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads
[ https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal closed DRILL-4766. -- Verified fix. > FragmentExecutor should use EventProcessor and avoid blocking rpc threads > - > > Key: DRILL-4766 > URL: https://issues.apache.org/jira/browse/DRILL-4766 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Affects Versions: 1.7.0 >Reporter: Deneche A. Hakim >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.8.0 > > > Currently, rpc thread can block when trying to deliver a cancel or early > termination message to a blocked fragment executor. > Foreman already uses an EventProcessor to avoid such scenarios. > FragmentExecutor could be improved to avoid blocking rpc threads as well -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads
[ https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-4766: --- Reviewer: Krystal (was: Rahul Challapalli) > FragmentExecutor should use EventProcessor and avoid blocking rpc threads > - > > Key: DRILL-4766 > URL: https://issues.apache.org/jira/browse/DRILL-4766 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Affects Versions: 1.7.0 >Reporter: Deneche A. Hakim >Assignee: Sudheesh Katkam >Priority: Minor > Fix For: 1.8.0 > > > Currently, rpc thread can block when trying to deliver a cancel or early > termination message to a blocked fragment executor. > Foreman already uses an EventProcessor to avoid such scenarios. > FragmentExecutor could be improved to avoid blocking rpc threads as well -- This message was sent by Atlassian JIRA (v6.3.4#6332)
