[jira] [Created] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
Hari Sekhon created DRILL-6457:
--
Summary: Sqlline - infer Kerberos principal dynamically to be able
to use individual keytabs across Drill nodes and still use ZooKeeper connection
string for High Availability
Key: DRILL-6457
URL: https://issues.apache.org/jira/browse/DRILL-6457
Project: Apache Drill
Issue Type: Improvement
Components: Client - CLI, Client - JDBC, Security, Tools, Build &
Test
Affects Versions: 1.13.0
Environment: MapR 6
Reporter: Hari Sekhon
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos errors due to the mismatch between the explicit sqlline kerberos
principal and zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Description:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,hsot3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiList
[jira] [Updated] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Description:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error due to the mismatch between the explicit sqlline kerberos
principal and zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
was:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,hsot3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed
[jira] [Commented] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496484#comment-16496484
]
Hari Sekhon commented on DRILL-6457:
For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
org.apache.drill.exec.rpc.AbstractHandshakeHandle
[jira] [Comment Edited] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availabil
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496484#comment-16496484
]
Hari Sekhon edited comment on DRILL-6457 at 5/31/18 12:45 PM:
--
For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
[jira] [Issue Comment Deleted] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Av
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Comment: was deleted
(was: For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
org.apache.drill.exec.rpc.AbstractHandshakeHandler.decode(AbstractHandshake
[jira] [Commented] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496490#comment-16496490
]
Hari Sekhon commented on DRILL-6457:
For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
org.apache.drill.exec.rpc.AbstractHandshakeHandle
[jira] [Updated] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Description:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error:
{code:java}
Caused by: KrbException: Identifier doesn't match expected value{code}
due to the mismatch between the explicit sqlline kerberos principal and
zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
was:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error due to the mismatch between the explicit sqlline kerberos
principal and zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
> Sqlline - infer Kerberos principal dynamically to be able to use individual
> keytabs across Drill nodes and still use ZooKeeper connection string for High
> Availability
> --
>
> Key: DRILL-6457
> URL: https://issues.apache.org/jira/browse/DRILL-6457
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - CLI, Client - JDBC, Security, Tools, Build
> & Test
>Affects Versions: 1.13.0
> Environment: MapR 6
>Reporter: Hari Sekhon
>Priority: Major
>
> Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
> connection string, eg:
> {code:java}
> zk=;auth=kerberos;principal=mapr/@REALM{code}
> When Drill nodes are configured with individual keytabs containing the node's
> fqdn and configured like so:
> {code:java}
> security: { auth.principal: mapr/_HOST@REALM }{code}
> then the ZooKeeper connection string from sqlline does not work and results
> in GSS Kerberos error:
> {code:java}
> Caused by: KrbException: Identifier doesn't match expected value{code}
> due to the mismatch between the explicit sqlline kerberos principal and
> zookeeper's random drillbit's principal.
> For the connection to work in this case requires something more like:
> {code:java}
> drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname
> -f)@REALM{code}
> but this lacks the high availability of using the ZooKeeper connection string
> to connect to any available node
> Hence it would be good if there was a way for sqlline arguments to be able to
> either infer the correct kerberos principal to match the host that zookeeper
> tells it to connect to or else accept a more generic parameter such as:
> {code:java}
> zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
> I've tested the above but it doesn't work showing that sqlline is not using a
> dynamic kerberos principal to match the host it is connecting to.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5796) Filter pruning for multi rowgroup parquet file
[ https://issues.apache.org/jira/browse/DRILL-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496589#comment-16496589 ] ASF GitHub Bot commented on DRILL-5796: --- jbimbert opened a new pull request #1298: DRILL-5796: Filter pruning for multi rowgroup parquet file URL: https://github.com/apache/drill/pull/1298 In ParquetFilterPredicate, replaced canDrop with ROWS_MATCH enum to keep inside rowgroup the filter result information. This information allows to prune the filter when all rows match. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Filter pruning for multi rowgroup parquet file > -- > > Key: DRILL-5796 > URL: https://issues.apache.org/jira/browse/DRILL-5796 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Reporter: Damien Profeta >Priority: Major > > Today, filter pruning use the file name as the partitioning key. This means > you can remove a partition only if the whole file is for the same partition. > With parquet, you can prune the filter if the rowgroup make a partition of > your dataset as the unit of work if the rowgroup not the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6450) Visualized plans for profiles querying JDBC sources is broken
[
https://issues.apache.org/jira/browse/DRILL-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-6450:
Labels: ready-to-commit (was: )
> Visualized plans for profiles querying JDBC sources is broken
> -
>
> Key: DRILL-6450
> URL: https://issues.apache.org/jira/browse/DRILL-6450
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> When viewing a profile for a query against a JDBC source, the visualized plan
> is not rendered. This is because the generated SQL pushed down to the JDBC
> source has a line break injected just before the {{FROM}} clause.
> The workaround is to strip away any injected newlines at least for the SQL
> defined in the text plan, so that the backend Javascript can render it
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[ https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496612#comment-16496612 ] ASF GitHub Bot commented on DRILL-6456: --- HanumathRao opened a new pull request #1299: DRILL-6456: Planner shouldn't create any exchanges on the right side … URL: https://github.com/apache/drill/pull/1299 …of Lateral Join. @amansinha100 Can you please review this PR. This PR includes fix to not generate the Exchanges on the right side of LateralJoin. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Planner shouldn't create any exchanges on the right side of Lateral Join. > - > > Key: DRILL-6456 > URL: https://issues.apache.org/jira/browse/DRILL-6456 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.14.0 >Reporter: Hanumath Rao Maduri >Assignee: Hanumath Rao Maduri >Priority: Major > Fix For: 1.14.0 > > > Currently, there is no restriction placed on right side of the LateralJoin. > This is causing planner to generate an Exchange when there are operators like > (Agg, Limit, Sort etc). > Due to this unnest operator cannot retrieve the row from lateral's left side > to process the pipeline further. Enhance the planner to not generate > exchanges on the right side of the LateralJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6455) JDBC Scan Operator does not appear in profile
[
https://issues.apache.org/jira/browse/DRILL-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496596#comment-16496596
]
ASF GitHub Bot commented on DRILL-6455:
---
kkhatua commented on a change in pull request #1297: DRILL-6455: Add missing
JDBC Scan Operator for profiles
URL: https://github.com/apache/drill/pull/1297#discussion_r192111061
##
File path:
protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java
##
@@ -24327,11 +24336,11 @@ public Builder clearStatus() {
"$\022\021\n\rPCAP_SUB_SCAN\020%\022\022\n\016KAFKA_SUB_SCAN\020&" +
"\022\021\n\rKUDU_SUB_SCAN\020\'\022\013\n\007FLATTEN\020(\022\020\n\014LATE" +
"RAL_JOIN\020)\022\n\n\006UNNEST\020*\022,\n(HIVE_DRILL_NAT" +
- "IVE_PARQUET_ROW_GROUP_SCAN\020+*g\n\nSaslStat" +
-
"us\022\020\n\014SASL_UNKNOWN\020\000\022\016\n\nSASL_START\020\001\022\024\n\020"
+
-
"SASL_IN_PROGRESS\020\002\022\020\n\014SASL_SUCCESS\020\003\022\017\n\013" +
- "SASL_FAILED\020\004B.\n\033org.apache.drill.exec.p" +
- "rotoB\rUserBitSharedH\001"
+ "IVE_PARQUET_ROW_GROUP_SCAN\020+\022\r\n\tJDBC_SCA" +
Review comment:
This came out of compiling the protobuf, which is needed. This change looks
arbitrary because the generated code in this segment appears to be ordered
alphabetically.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> JDBC Scan Operator does not appear in profile
> -
>
> Key: DRILL-6455
> URL: https://issues.apache.org/jira/browse/DRILL-6455
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Critical
> Fix For: 1.14.0
>
>
> It seems that the Operator is not defined, though it appears in the text plan.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5735: Reviewer: John Omernik (was: Sorabh Hamirwasia) > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5735: Labels: ready-to-commit (was: ) > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4326) JDBC Storage Plugin for PostgreSQL does not work
[
https://issues.apache.org/jira/browse/DRILL-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-4326:
Component/s: (was: Storage - Other)
Storage - JDBC
> JDBC Storage Plugin for PostgreSQL does not work
>
>
> Key: DRILL-4326
> URL: https://issues.apache.org/jira/browse/DRILL-4326
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
> Environment: Mac OS X JDK 1.8 PostgreSQL 9.4.4 PostgreSQL JDBC jars
> (postgresql-9.2-1004-jdbc4.jar, postgresql-9.1-901-1.jdbc4.jar, )
>Reporter: Akon Dey
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.5.0
>
>
> Queries with the JDBC Storage Plugin for PostgreSQL fail with DATA_READ ERROR.
> The JDBC Storage Plugin settings in use are:
> {code}
> {
> "type": "jdbc",
> "driver": "org.postgresql.Driver",
> "url": "jdbc:postgresql://127.0.0.1/test",
> "username": "akon",
> "password": null,
> "enabled": false
> }
> {code}
> Please refer to the following stack for further details:
> {noformat}
> Akons-MacBook-Pro:drill akon$
> ./distribution/target/apache-drill-1.5.0-SNAPSHOT/apache-drill-1.5.0-SNAPSHOT/bin/drill-embedded
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
> support was removed in 8.0
> Jan 29, 2016 9:17:18 AM org.glassfish.jersey.server.ApplicationHandler
> initialize
> INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29
> 01:25:26...
> apache drill 1.5.0-SNAPSHOT
> "a little sql for your nosql"
> 0: jdbc:drill:zk=local> !verbose
> verbose: on
> 0: jdbc:drill:zk=local> use pgdb;
> +---+---+
> | ok | summary |
> +---+---+
> | true | Default schema changed to [pgdb] |
> +---+---+
> 1 row selected (0.753 seconds)
> 0: jdbc:drill:zk=local> select * from ips;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the
> SQL query.
> sql SELECT *
> FROM "test"."ips"
> plugin pgdb
> Fragment 0:0
> [Error Id: 26ada06d-e08d-456a-9289-0dec2089b018 on 10.200.104.128:31010]
> (state=,code=0)
> java.sql.SQLException: DATA_READ ERROR: The JDBC storage plugin failed while
> trying setup the SQL query.
> sql SELECT *
> FROM "test"."ips"
> plugin pgdb
> Fragment 0:0
> [Error Id: 26ada06d-e08d-456a-9289-0dec2089b018 on 10.200.104.128:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1923)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:73)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at
> net.hydromatic.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:338)
> at
> net.hydromatic.avatica.AvaticaStatement.execute(AvaticaStatement.java:69)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:101)
> at sqlline.Commands.execute(Commands.java:841)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:746)
> at sqlline.SqlLine.begin(SqlLine.java:621)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: DATA_READ
> ERROR: The JDBC storage plugin failed while trying setup the SQL query.
> sql SELECT *
> FROM "test"."ips"
> plugin pgdb
> Fragment 0:0
> [Error Id: 26ada06d-e08d-456a-9289-0dec2089b018 on 10.200.104.128:31010]
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:119)
> at
> org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:113)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67)
> at org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374)
> at
> org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
> at
> org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252)
> at
> org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
[jira] [Updated] (DRILL-5764) upgrade netty version to higher
[ https://issues.apache.org/jira/browse/DRILL-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5764: Description: DRILL-4326 > upgrade netty version to higher > --- > > Key: DRILL-5764 > URL: https://issues.apache.org/jira/browse/DRILL-5764 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Affects Versions: 1.11.0 > Environment: As current netty 4.0.27.Final has a bug > (https://github.com/netty/netty/issues/3539) to epoll method, we now abandon > the epoll method at linux platform (see TransportCheck). Now the epoll bug > has been fixed at 4.0.28 version , the arrow project also upgrade the netty > version to 4.0.41. I wonder we can upgrade the netty version and allow the > epoll method at linux platform. >Reporter: weijie.tong >Priority: Major > Fix For: 1.12.0 > > > DRILL-4326 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5764) upgrade netty version to higher
[ https://issues.apache.org/jira/browse/DRILL-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5764: Environment: (was: As current netty 4.0.27.Final has a bug (https://github.com/netty/netty/issues/3539) to epoll method, we now abandon the epoll method at linux platform (see TransportCheck). Now the epoll bug has been fixed at 4.0.28 version , the arrow project also upgrade the netty version to 4.0.41. I wonder we can upgrade the netty version and allow the epoll method at linux platform. ) > upgrade netty version to higher > --- > > Key: DRILL-5764 > URL: https://issues.apache.org/jira/browse/DRILL-5764 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Affects Versions: 1.11.0 >Reporter: weijie.tong >Priority: Major > Fix For: 1.12.0 > > > As current netty 4.0.27.Final has a bug > (https://github.com/netty/netty/issues/3539) to epoll method, we now abandon > the epoll method at linux platform (see TransportCheck). Now the epoll bug > has been fixed at 4.0.28 version , the arrow project also upgrade the netty > version to 4.0.41. I wonder we can upgrade the netty version and allow the > epoll method at linux platform. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5764) upgrade netty version to higher
[ https://issues.apache.org/jira/browse/DRILL-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5764: Description: As current netty 4.0.27.Final has a bug (https://github.com/netty/netty/issues/3539) to epoll method, we now abandon the epoll method at linux platform (see TransportCheck). Now the epoll bug has been fixed at 4.0.28 version , the arrow project also upgrade the netty version to 4.0.41. I wonder we can upgrade the netty version and allow the epoll method at linux platform. (was: DRILL-4326) > upgrade netty version to higher > --- > > Key: DRILL-5764 > URL: https://issues.apache.org/jira/browse/DRILL-5764 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Affects Versions: 1.11.0 > Environment: As current netty 4.0.27.Final has a bug > (https://github.com/netty/netty/issues/3539) to epoll method, we now abandon > the epoll method at linux platform (see TransportCheck). Now the epoll bug > has been fixed at 4.0.28 version , the arrow project also upgrade the netty > version to 4.0.41. I wonder we can upgrade the netty version and allow the > epoll method at linux platform. >Reporter: weijie.tong >Priority: Major > Fix For: 1.12.0 > > > As current netty 4.0.27.Final has a bug > (https://github.com/netty/netty/issues/3539) to epoll method, we now abandon > the epoll method at linux platform (see TransportCheck). Now the epoll bug > has been fixed at 4.0.28 version , the arrow project also upgrade the netty > version to 4.0.41. I wonder we can upgrade the netty version and allow the > epoll method at linux platform. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-651) Implement Log functions
[ https://issues.apache.org/jira/browse/DRILL-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua resolved DRILL-651. Resolution: Fixed Fix Version/s: (was: Future) This is already available: https://drill.apache.org/docs/math-and-trig/#table-of-math-functions > Implement Log functions > --- > > Key: DRILL-651 > URL: https://issues.apache.org/jira/browse/DRILL-651 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 0.1.0-m1 >Reporter: Ajay Bhat >Priority: Minor > > Implement Logarithm functions -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-2035) Add ability to cancel multiple queries
[ https://issues.apache.org/jira/browse/DRILL-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-2035: Component/s: Web Server > Add ability to cancel multiple queries > -- > > Key: DRILL-2035 > URL: https://issues.apache.org/jira/browse/DRILL-2035 > Project: Apache Drill > Issue Type: New Feature > Components: Client - HTTP, Web Server >Reporter: Neeraja >Priority: Major > Fix For: Future > > > Currently Drill UI allows canceling one query at a time. > This could be cumbersome to manage for scenarios using with BI tools which > generate multiple queries for a single action in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-2035) Add ability to cancel multiple queries
[ https://issues.apache.org/jira/browse/DRILL-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua reassigned DRILL-2035: --- Assignee: Kunal Khatua > Add ability to cancel multiple queries > -- > > Key: DRILL-2035 > URL: https://issues.apache.org/jira/browse/DRILL-2035 > Project: Apache Drill > Issue Type: New Feature > Components: Client - HTTP, Web Server >Reporter: Neeraja >Assignee: Kunal Khatua >Priority: Major > Fix For: Future > > > Currently Drill UI allows canceling one query at a time. > This could be cumbersome to manage for scenarios using with BI tools which > generate multiple queries for a single action in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-2610) Local File System Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua resolved DRILL-2610.
-
Resolution: Fixed
Fix Version/s: (was: Future)
Creating a storage plugin with {{"connection": "file:///"}} will automatically
point to the local file system.
{code:language=json}
{
"type": "file",
"enabled": true,
"connection": "file:///",
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": "csv",
"allowAccessOutsideWorkspace": false
}
},
"formats": {
"psv": {
"type": "text",
"extensions": [
"tbl"
],
"delimiter": "|"
},
"dsv": {
"type": "text",
"extensions": [
"dat"
],
"delimiter": "|"
},
"csv": {
"type": "text",
"extensions": [
"csv"
],
"delimiter": ","
},
"tsv": {
"type": "text",
"extensions": [
"tsv"
],
"delimiter": "\t"
},
"parquet": {
"type": "parquet"
},
"json": {
"type": "json",
"extensions": [
"json",
"drill"
]
},
"abc": {
"type": "json",
"extensions": [
"abc"
]
},
"maprdb": {
"type": "maprdb"
}
}
}
{code}
> Local File System Storage Plugin
>
>
> Key: DRILL-2610
> URL: https://issues.apache.org/jira/browse/DRILL-2610
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Affects Versions: 0.8.0
>Reporter: Sudheesh Katkam
>Priority: Major
>
> Create a storage plugin to query files on the local file system on the nodes
> in the cluster. For example, users should be able to query log files in
> /var/log/drill/ on all nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-3878) Support XML Querying (selects/projections, no writing)
[ https://issues.apache.org/jira/browse/DRILL-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-3878: Component/s: Storage - Other > Support XML Querying (selects/projections, no writing) > -- > > Key: DRILL-3878 > URL: https://issues.apache.org/jira/browse/DRILL-3878 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: Future >Reporter: Edmon Begoli >Priority: Major > Labels: features > Fix For: Future > > Original Estimate: 3,360h > Remaining Estimate: 3,360h > > Support querying of the XML documents (as read-only selects, > Writing should be implemented as a different feature that brings its own set > of challenges.) > To consider is reading of the trivial, schema-less, XML documents, > DTD-oriented ones and also of schema-defined ones. > Also, we should consider direct querying vs. using converter tools to change > the representation from XML to JSON, CSV, etc. > Design and Implementation discussion, notes, ideas and implementation > suggestions should be captured here: > https://docs.google.com/document/d/1oS-cObSaTlAmuW_XghDLmHbBEorLl0z-axaHnjy7vg0/edit?usp=sharing > > (no vandalism, please) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-4282) Allow to customize address registered in zookeeper
[ https://issues.apache.org/jira/browse/DRILL-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua resolved DRILL-4282. - Resolution: Incomplete Need more information. > Allow to customize address registered in zookeeper > --- > > Key: DRILL-4282 > URL: https://issues.apache.org/jira/browse/DRILL-4282 > Project: Apache Drill > Issue Type: New Feature > Components: Execution - RPC >Reporter: johnny >Priority: Major > > the useIP property in ServiceEngine is harecoded to false ,drill should allow > to customize it. > User should be able to specify the address which register in the zookeeper > to make drill cluster able to work with some cloud system like kubernetes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4650) Excel file (.xsl) and Microsoft Access file (.accdb) problem
[ https://issues.apache.org/jira/browse/DRILL-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-4650: Component/s: Storage - Other > Excel file (.xsl) and Microsoft Access file (.accdb) problem > - > > Key: DRILL-4650 > URL: https://issues.apache.org/jira/browse/DRILL-4650 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.6.0 >Reporter: Sanjiv Kumar >Priority: Major > > I am trying to query from excel file(.xsl file) and ms access file (.accdb), > but i am unable to query from these files in drill. Is there any way to query > these files. Or any Storage Plugin for query these excel and ms access files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6223) Drill fails on Schema changes
[ https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6223: - Fix Version/s: (was: 1.14.0) > Drill fails on Schema changes > -- > > Key: DRILL-6223 > URL: https://issues.apache.org/jira/browse/DRILL-6223 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.10.0, 1.12.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > > Drill Query Failing when selecting all columns from a Complex Nested Data > File (Parquet) Set). There are differences in Schema among the files: > * The Parquet files exhibit differences both at the first level and within > nested data types > * A select * will not cause an exception but using a limit clause will > * Note also this issue seems to happen only when multiple Drillbit minor > fragments are involved (concurrency higher than one) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6101) Optimize Implicit Columns Processing
[ https://issues.apache.org/jira/browse/DRILL-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6101: - Fix Version/s: (was: 1.14.0) > Optimize Implicit Columns Processing > > > Key: DRILL-6101 > URL: https://issues.apache.org/jira/browse/DRILL-6101 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.12.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Critical > > Problem Description - > * Apache Drill allows users to specify columns even for SELECT STAR queries > * From my discussion with [~paul-rogers], Apache Calcite has a limitation > where the, extra columns are not provided > * The workaround has been to always include all implicit columns for SELECT > STAR queries > * Unfortunately, the current implementation is very inefficient as implicit > column values get duplicated; this leads to substantial performance > degradation when the number of rows are large > Suggested Optimization - > * The NullableVarChar vector should be enhanced to efficiently store > duplicate values > * This will not only address the current Calcite limitations (for SELECT > STAR queries) but also optimize all queries with implicit columns > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496897#comment-16496897
]
ASF GitHub Bot commented on DRILL-6456:
---
amansinha100 commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192175763
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
##
@@ -45,18 +49,22 @@ public static Prel removeExcessiveEchanges(Prel prel, long
targetSliceSize) {
public Prel visitExchange(ExchangePrel prel, MajorFragmentStat parent)
throws RuntimeException {
parent.add(prel);
MajorFragmentStat newFrag = new MajorFragmentStat();
+newFrag.setRightSideOfLateral(parent.isRightSideOfLateral());
Prel newChild = ((Prel) prel.getInput()).accept(this, newFrag);
-
-if (newFrag.isSingular() && parent.isSingular() &&
-// if one of them has strict distribution or none, we can remove the
exchange
-(!newFrag.isDistributionStrict() || !parent.isDistributionStrict())
-) {
+if (canRemoveExchange(parent, newFrag)) {
return newChild;
} else {
return (Prel) prel.copy(prel.getTraitSet(),
Collections.singletonList((RelNode) newChild));
}
}
+ private boolean canRemoveExchange(MajorFragmentStat parentFrag,
MajorFragmentStat childFrag) {
+return (childFrag.isSingular() && parentFrag.isSingular() &&
Review comment:
For better readability, Since the lateral check is independent of the
others, can you split it into a separate statement ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496898#comment-16496898
]
ASF GitHub Bot commented on DRILL-6456:
---
amansinha100 commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192176210
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
##
@@ -70,6 +78,40 @@ public Prel visitScan(ScanPrel prel, MajorFragmentStat s)
throws RuntimeExceptio
return prel;
}
+ @Override
+ public Prel visitCorrelate(CorrelatePrel prel, MajorFragmentStat s) throws
RuntimeException {
+List children = Lists.newArrayList();
+s.add(prel);
+
+for (Prel p : prel) {
+ s.add(p);
+}
+
+// Traverse the left side of the Lateral join first. Left side of the
+// Lateral shouldn't have any restrictions on Exchanges.
+children.add(((Prel)prel.getInput(0)).accept(this, s));
+// Save the outermost Lateral join so as to unset the flag later.
+if (topMostLateralJoin.size() == 0) {
+ topMostLateralJoin.add(prel);
+}
+
+// Right side of the Lateral shouldn't have any Exchanges. Hence set the
+// flag so that visitExchange removes the exchanges.
+s.setRightSideOfLateral(true);
+children.add(((Prel)prel.getInput(1)).accept(this, s));
+if (topMostLateralJoin.contains(prel)) {
+ topMostLateralJoin.remove(prel);
+ s.setRightSideOfLateral(false);
+}
+return (Prel) prel.copy(prel.getTraitSet(), children);
+ }
+
+ @Override
+ public Prel visitUnnest(UnnestPrel prel, MajorFragmentStat s) throws
RuntimeException {
+s.addUnnest(prel);
Review comment:
Is a specific unnest visitor needed here ? I don't see where the unnest
node is being consumed in the visitor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496899#comment-16496899
]
ASF GitHub Bot commented on DRILL-6456:
---
amansinha100 commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192179757
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
##
@@ -18,19 +18,23 @@
package org.apache.drill.exec.planner.physical.visitor;
import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
-
+import java.util.Set;
import org.apache.drill.exec.planner.fragment.DistributionAffinity;
+import org.apache.drill.exec.planner.physical.CorrelatePrel;
import org.apache.drill.exec.planner.physical.ExchangePrel;
import org.apache.drill.exec.planner.physical.Prel;
import org.apache.drill.exec.planner.physical.ScanPrel;
import org.apache.drill.exec.planner.physical.ScreenPrel;
import org.apache.calcite.rel.RelNode;
import com.google.common.collect.Lists;
+import org.apache.drill.exec.planner.physical.UnnestPrel;
public class ExcessiveExchangeIdentifier extends BasePrelVisitor {
private final long targetSliceSize;
+ private Set topMostLateralJoin = new HashSet<>();
Review comment:
Is this supposed to be a set of just 1 LJ ? (the name 'topMost' implies
there is only 1) but why is a collection needed then ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496907#comment-16496907
]
ASF GitHub Bot commented on DRILL-6456:
---
HanumathRao commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192181489
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
##
@@ -45,18 +49,22 @@ public static Prel removeExcessiveEchanges(Prel prel, long
targetSliceSize) {
public Prel visitExchange(ExchangePrel prel, MajorFragmentStat parent)
throws RuntimeException {
parent.add(prel);
MajorFragmentStat newFrag = new MajorFragmentStat();
+newFrag.setRightSideOfLateral(parent.isRightSideOfLateral());
Prel newChild = ((Prel) prel.getInput()).accept(this, newFrag);
-
-if (newFrag.isSingular() && parent.isSingular() &&
-// if one of them has strict distribution or none, we can remove the
exchange
-(!newFrag.isDistributionStrict() || !parent.isDistributionStrict())
-) {
+if (canRemoveExchange(parent, newFrag)) {
return newChild;
} else {
return (Prel) prel.copy(prel.getTraitSet(),
Collections.singletonList((RelNode) newChild));
}
}
+ private boolean canRemoveExchange(MajorFragmentStat parentFrag,
MajorFragmentStat childFrag) {
+return (childFrag.isSingular() && parentFrag.isSingular() &&
Review comment:
Sure. Will do it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496906#comment-16496906
]
ASF GitHub Bot commented on DRILL-6456:
---
HanumathRao commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192181436
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
##
@@ -18,19 +18,23 @@
package org.apache.drill.exec.planner.physical.visitor;
import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
-
+import java.util.Set;
import org.apache.drill.exec.planner.fragment.DistributionAffinity;
+import org.apache.drill.exec.planner.physical.CorrelatePrel;
import org.apache.drill.exec.planner.physical.ExchangePrel;
import org.apache.drill.exec.planner.physical.Prel;
import org.apache.drill.exec.planner.physical.ScanPrel;
import org.apache.drill.exec.planner.physical.ScreenPrel;
import org.apache.calcite.rel.RelNode;
import com.google.common.collect.Lists;
+import org.apache.drill.exec.planner.physical.UnnestPrel;
public class ExcessiveExchangeIdentifier extends BasePrelVisitor {
private final long targetSliceSize;
+ private Set topMostLateralJoin = new HashSet<>();
Review comment:
Agree, this is not required. I can as well use outerMost correlate Prel and
check for null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496908#comment-16496908
]
ASF GitHub Bot commented on DRILL-6456:
---
HanumathRao commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192181853
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/ExcessiveExchangeIdentifier.java
##
@@ -70,6 +78,40 @@ public Prel visitScan(ScanPrel prel, MajorFragmentStat s)
throws RuntimeExceptio
return prel;
}
+ @Override
+ public Prel visitCorrelate(CorrelatePrel prel, MajorFragmentStat s) throws
RuntimeException {
+List children = Lists.newArrayList();
+s.add(prel);
+
+for (Prel p : prel) {
+ s.add(p);
+}
+
+// Traverse the left side of the Lateral join first. Left side of the
+// Lateral shouldn't have any restrictions on Exchanges.
+children.add(((Prel)prel.getInput(0)).accept(this, s));
+// Save the outermost Lateral join so as to unset the flag later.
+if (topMostLateralJoin.size() == 0) {
+ topMostLateralJoin.add(prel);
+}
+
+// Right side of the Lateral shouldn't have any Exchanges. Hence set the
+// flag so that visitExchange removes the exchanges.
+s.setRightSideOfLateral(true);
+children.add(((Prel)prel.getInput(1)).accept(this, s));
+if (topMostLateralJoin.contains(prel)) {
+ topMostLateralJoin.remove(prel);
+ s.setRightSideOfLateral(false);
+}
+return (Prel) prel.copy(prel.getTraitSet(), children);
+ }
+
+ @Override
+ public Prel visitUnnest(UnnestPrel prel, MajorFragmentStat s) throws
RuntimeException {
+s.addUnnest(prel);
Review comment:
Unnest visitor needs no specialization, but I thought Unnest is more like a
scan (i.e a leaf node) hence I just thought of introducing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5995) ArangoDB storage engine
[ https://issues.apache.org/jira/browse/DRILL-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5995: Component/s: Storage - Other > ArangoDB storage engine > --- > > Key: DRILL-5995 > URL: https://issues.apache.org/jira/browse/DRILL-5995 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Pinco Palla >Priority: Major > > Any timeline on ArangoDB storage engine availability in Drill? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6458) NPE when error while applying rule ReduceExpressionsRule_Project
[
https://issues.apache.org/jira/browse/DRILL-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Wu updated DRILL-6458:
--
Issue Type: Bug (was: Task)
> NPE when error while applying rule ReduceExpressionsRule_Project
>
>
> Key: DRILL-6458
> URL: https://issues.apache.org/jira/browse/DRILL-6458
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Rob Wu
>Priority: Minor
>
> *Query*: select 1 +( -( 21.0)) from (values(1))
>
>
> {code:java}
> SYSTEM ERROR: NullPointerException
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> ReduceExpressionsRule_Project, args
> [rel#113:LogicalProject.NONE.ANY([]).[](input=rel#112:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=+($1,
> -(20.0)))]
> org.apache.drill.exec.work.foreman.Foreman.run():281
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule ReduceExpressionsRule_Project, args
> [rel#113:LogicalProject.NONE.ANY([]).[](input=rel#112:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=+($1,
> -(20.0)))]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
> org.apache.calcite.tools.Programs$RuleSetProgram.run():303
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():404
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():343
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():240
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():290
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():168
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():123
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():97
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1008
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (java.lang.NullPointerException) null
>
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall():132
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall():80
> org.apache.calcite.rex.RexCall.accept():107
> org.apache.drill.exec.planner.logical.DrillOptiq.toDrill():77
> org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce():124
> org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions():499
> org.apache.calcite.rel.rules.ReduceExpressionsRule$1.onMatch():241
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
> org.apache.calcite.tools.Programs$RuleSetProgram.run():303
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():404
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():343
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():240
>
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():290
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():168
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():123
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():97
> org.apache.drill.exec.work.foreman.Foreman.runSQL():1008
> org.apache.drill.exec.work.foreman.Foreman.run():264
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6458) NPE when error while applying rule ReduceExpressionsRule_Project
Rob Wu created DRILL-6458:
-
Summary: NPE when error while applying rule
ReduceExpressionsRule_Project
Key: DRILL-6458
URL: https://issues.apache.org/jira/browse/DRILL-6458
Project: Apache Drill
Issue Type: Task
Reporter: Rob Wu
*Query*: select 1 +( -( 21.0)) from (values(1))
{code:java}
SYSTEM ERROR: NullPointerException
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
during fragment initialization: Internal error: Error while applying rule
ReduceExpressionsRule_Project, args
[rel#113:LogicalProject.NONE.ANY([]).[](input=rel#112:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=+($1,
-(20.0)))]
org.apache.drill.exec.work.foreman.Foreman.run():281
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.AssertionError) Internal error: Error while applying
rule ReduceExpressionsRule_Project, args
[rel#113:LogicalProject.NONE.ANY([]).[](input=rel#112:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=+($1,
-(20.0)))]
org.apache.calcite.util.Util.newInternal():792
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
org.apache.calcite.tools.Programs$RuleSetProgram.run():303
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():404
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():343
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():240
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():290
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():168
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():123
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():97
org.apache.drill.exec.work.foreman.Foreman.runSQL():1008
org.apache.drill.exec.work.foreman.Foreman.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.NullPointerException) null
org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall():132
org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall():80
org.apache.calcite.rex.RexCall.accept():107
org.apache.drill.exec.planner.logical.DrillOptiq.toDrill():77
org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce():124
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions():499
org.apache.calcite.rel.rules.ReduceExpressionsRule$1.onMatch():241
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():228
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():808
org.apache.calcite.tools.Programs$RuleSetProgram.run():303
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():404
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform():343
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():240
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel():290
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():168
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan():123
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():97
org.apache.drill.exec.work.foreman.Foreman.runSQL():1008
org.apache.drill.exec.work.foreman.Foreman.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5365) FileNotFoundException when reading a parquet file
[ https://issues.apache.org/jira/browse/DRILL-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497093#comment-16497093 ] ASF GitHub Bot commented on DRILL-5365: --- ilooner commented on issue #1296: DRILL-5365: Prevent plugin config from changing default fs. Make DrillFileSystem Immutable. URL: https://github.com/apache/drill/pull/1296#issuecomment-393663248 @vdiravka @paul-rogers Please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > FileNotFoundException when reading a parquet file > - > > Key: DRILL-5365 > URL: https://issues.apache.org/jira/browse/DRILL-5365 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Affects Versions: 1.10.0 >Reporter: Chun Chang >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > The parquet file is generated through the following CTAS. > To reproduce the issue: 1) two or more nodes cluster; 2) enable > impersonation; 3) set "fs.default.name": "file:///" in hive storage plugin; > 4) restart drillbits; 5) as a regular user, on node A, drop the table/file; > 6) ctas from a large enough hive table as source to recreate the table/file; > 7) query the table from node A should work; 8) query from node B as same user > should reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5365) FileNotFoundException when reading a parquet file
[ https://issues.apache.org/jira/browse/DRILL-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-5365: -- Reviewer: Vitalii Diravka (was: Paul Rogers) > FileNotFoundException when reading a parquet file > - > > Key: DRILL-5365 > URL: https://issues.apache.org/jira/browse/DRILL-5365 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Affects Versions: 1.10.0 >Reporter: Chun Chang >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > The parquet file is generated through the following CTAS. > To reproduce the issue: 1) two or more nodes cluster; 2) enable > impersonation; 3) set "fs.default.name": "file:///" in hive storage plugin; > 4) restart drillbits; 5) as a regular user, on node A, drop the table/file; > 6) ctas from a large enough hive table as source to recreate the table/file; > 7) query the table from node A should work; 8) query from node B as same user > should reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6340) Output Batch Control in Project using the RecordBatchSizer
[ https://issues.apache.org/jira/browse/DRILL-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6340: - Reviewer: Padma Penumarthy > Output Batch Control in Project using the RecordBatchSizer > -- > > Key: DRILL-6340 > URL: https://issues.apache.org/jira/browse/DRILL-6340 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Fix For: 1.14.0 > > > This bug is for tracking the changes required to implement Output Batch > Sizing in Project using the RecordBatchSizer. The challenge in doing this > mainly lies in dealing with expressions that produce variable-length columns. > The following doc talks about some of the design approaches for dealing with > such variable-length columns. > [https://docs.google.com/document/d/1h0WsQsen6xqqAyyYSrtiAniQpVZGmQNQqC1I2DJaxAA/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6340) Output Batch Control in Project using the RecordBatchSizer
[ https://issues.apache.org/jira/browse/DRILL-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6340: - Fix Version/s: 1.14.0 > Output Batch Control in Project using the RecordBatchSizer > -- > > Key: DRILL-6340 > URL: https://issues.apache.org/jira/browse/DRILL-6340 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan >Priority: Major > Fix For: 1.14.0 > > > This bug is for tracking the changes required to implement Output Batch > Sizing in Project using the RecordBatchSizer. The challenge in doing this > mainly lies in dealing with expressions that produce variable-length columns. > The following doc talks about some of the design approaches for dealing with > such variable-length columns. > [https://docs.google.com/document/d/1h0WsQsen6xqqAyyYSrtiAniQpVZGmQNQqC1I2DJaxAA/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5984) Support for Symlinked Table Paths to be used in Drill Queries.
[ https://issues.apache.org/jira/browse/DRILL-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497108#comment-16497108 ] Kunal Khatua commented on DRILL-5984: - This is interesting because for even simple text files, it is not able to read the entire file, but only the first line. Upvoting this so that it gets picked up. > Support for Symlinked Table Paths to be used in Drill Queries. > -- > > Key: DRILL-5984 > URL: https://issues.apache.org/jira/browse/DRILL-5984 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.11.0 > Environment: OS : CentOS 7.1 > MapR-DB Version: 5.2.2 >Reporter: Saravanabavagugan Vengadasundaram >Priority: Major > > MapR-FS supports symlinks and hence MapR-DB table paths support symlinks as > well. As part of the project I work on, we use symlinks as a means of > communication to talk to the physical file. An employee table in MapR-DB will > be represented as "/tables/Employee/Entity_1233232" and there will be a > symlink called "/tables/Employee/Entity" pointing to the actual physical > table. Currently, drill does not understand queries having the symlink path > but only executes queries having the actual physical table path. So every > time, I need to find out the actual physical path of the table and frame my > query. It would be nice to have this feature in next version of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4897) NumberFormatException in Drill SQL while casting to BIGINT when its actually a number
[ https://issues.apache.org/jira/browse/DRILL-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-4897: - Fix Version/s: (was: 1.14.0) 1.15.0 > NumberFormatException in Drill SQL while casting to BIGINT when its actually > a number > - > > Key: DRILL-4897 > URL: https://issues.apache.org/jira/browse/DRILL-4897 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Reporter: Srihari Karanth >Assignee: Karthikeyan Manivannan >Priority: Blocker > Fix For: 1.15.0 > > > In the following SQL, drill cribs when trying to convert a number which is in > varchar >select cast (case IsNumeric(Delta_Radio_Delay) > when 0 then 0 else Delta_Radio_Delay end as BIGINT) > from datasource.`./sometable` > where Delta_Radio_Delay='4294967294'; > BIGINT should be able to take very large number. I dont understand how it > throws the below error: > 0: jdbc:drill:> select cast (case IsNumeric(Delta_Radio_Delay) > when 0 then 0 else Delta_Radio_Delay end as BIGINT) > from datasource.`./sometable` > where Delta_Radio_Delay='4294967294'; > Error: SYSTEM ERROR: NumberFormatException: 4294967294 > Fragment 1:29 > [Error Id: a63bb113-271f-4d8b-8194-2c9728543200 on cluster-3:31010] > (state=,code=0) > How can i modify SQL to fix this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5700) nohup support for sqlline
[
https://issues.apache.org/jira/browse/DRILL-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497122#comment-16497122
]
Kunal Khatua commented on DRILL-5700:
-
[~arjun_kr] can you open a PR for this?
> nohup support for sqlline
> --
>
> Key: DRILL-5700
> URL: https://issues.apache.org/jira/browse/DRILL-5700
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - CLI
>Reporter: Arjun
>Priority: Minor
>
> Sqlline script does support nohup mode for execution. On execution, it
> remains stopped until it is made fore ground.
> {code:java}
> [mapr@node1 ~]$ cat test.sql
> select * from sys.drillbits
> [mapr@node1 ~]$
> [mapr@node1 ~]$ nohup sqlline -u "jdbc:drill:" -n mapr -p mapr -f test.sql &
> [1] 24019
> [mapr@node1 ~]$ nohup: ignoring input and appending output to `nohup.out'
> [1]+ Stopped nohup sqlline -u "jdbc:drill:zk=node1:5181" -n
> mapr -p mapr -f test.sql
> [mapr@node1 ~]$
> [mapr@node1 ~]$ fg
> nohup sqlline -u "jdbc:drill:zk=node1:5181" -n mapr -p mapr -f test.sql
> [mapr@node1 ~]$
> [mapr@node1 ~]$ cat nohup.out
> 0: jdbc:drill:zk=node1:5181> Closing:
> org.apache.drill.jdbc.impl.DrillConnectionImpl
> output of ps: S
> 1/1 select * from sys.drillbits;
> +--++---++--+--+
> | hostname | user_port | control_port | data_port | current |
> version |
> +--++---++--+--+
> | node1 | 31010 | 31011 | 31012 | true | 1.10.0 |
> +--++---++--+--+
> 1 row selected (0.354 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.10.0
> "drill baby drill"
> [mapr@node1 ~]$
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6236) batch sizing for hash join
[ https://issues.apache.org/jira/browse/DRILL-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Padma Penumarthy updated DRILL-6236: Labels: ready-to-commit (was: ) > batch sizing for hash join > -- > > Key: DRILL-6236 > URL: https://issues.apache.org/jira/browse/DRILL-6236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > limit output batch size for hash join based on memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5626) Drill pushing invalid SQL to ORACLE when joined table with equi and theta join term used
[ https://issues.apache.org/jira/browse/DRILL-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5626: Component/s: Storage - JDBC > Drill pushing invalid SQL to ORACLE when joined table with equi and theta > join term used > > > Key: DRILL-5626 > URL: https://issues.apache.org/jira/browse/DRILL-5626 > Project: Apache Drill > Issue Type: Bug > Components: Server, Storage - JDBC >Affects Versions: 1.10.0 > Environment: Drill 1.10 > ORACLE 12C2 via JDBC >Reporter: N Campbell >Priority: Major > > Plugin defined to access ORACLE 12C2 via JDBC > Following query sent to Drill > select TJOIN1.RNUM, TJOIN1.C1, TJOIN1.C2, TJOIN2.C2 as C2J2 from > certora.DBCERT.TJOIN1 inner join certora.DBCERT.TJOIN2 on ( TJOIN1.C1 = > TJOIN2.C1 and TJOIN2.C1 between 15 and 25 ) > Fails with DATA_READ ERROR: The JDBC storage plugin failed while trying setup > the SQL query. > SELECT "C1", "C2", "C1" >= 15 "$f3", "C1" <= 25 "$f4" > FROM "DBCERT"."TJOIN2" > ORACLE DDL > CREATE TABLE "DBCERT"."TJOIN1" >( "RNUM" NUMBER(*,0), > "C1" NUMBER(*,0), > "C2" NUMBER(*,0) >) ; > GRANT SELECT ON "DBCERT"."TJOIN1" TO PUBLIC; > > -- DDL for Table TJOIN2 > > CREATE TABLE "DBCERT"."TJOIN2" >( "RNUM" NUMBER(*,0), > "C1" NUMBER(*,0), > "C2" CHAR(2 BYTE) >) ; > GRANT SELECT ON "DBCERT"."TJOIN2" TO PUBLIC; > Other examples > select TJOIN1.RNUM, TJOIN1.C1, TJOIN1.C2, TJOIN2.C2 as C2J2 from > certora.DBCERT.TJOIN1 inner join certora.DBCERT.TJOIN2 on ( TJOIN1.C1 = > TJOIN2.C1 and TJOIN2.C2 like 'A%' ) > DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL > query. > sql SELECT "C1", "C2", "C2" LIKE 'A%' "$f3" > FROM "DBCERT"."TJOIN2" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5626) Drill pushing invalid SQL to ORACLE when joined table with equi and theta join term used
[ https://issues.apache.org/jira/browse/DRILL-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5626: Component/s: (was: Server) > Drill pushing invalid SQL to ORACLE when joined table with equi and theta > join term used > > > Key: DRILL-5626 > URL: https://issues.apache.org/jira/browse/DRILL-5626 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JDBC >Affects Versions: 1.10.0 > Environment: Drill 1.10 > ORACLE 12C2 via JDBC >Reporter: N Campbell >Priority: Major > > Plugin defined to access ORACLE 12C2 via JDBC > Following query sent to Drill > select TJOIN1.RNUM, TJOIN1.C1, TJOIN1.C2, TJOIN2.C2 as C2J2 from > certora.DBCERT.TJOIN1 inner join certora.DBCERT.TJOIN2 on ( TJOIN1.C1 = > TJOIN2.C1 and TJOIN2.C1 between 15 and 25 ) > Fails with DATA_READ ERROR: The JDBC storage plugin failed while trying setup > the SQL query. > SELECT "C1", "C2", "C1" >= 15 "$f3", "C1" <= 25 "$f4" > FROM "DBCERT"."TJOIN2" > ORACLE DDL > CREATE TABLE "DBCERT"."TJOIN1" >( "RNUM" NUMBER(*,0), > "C1" NUMBER(*,0), > "C2" NUMBER(*,0) >) ; > GRANT SELECT ON "DBCERT"."TJOIN1" TO PUBLIC; > > -- DDL for Table TJOIN2 > > CREATE TABLE "DBCERT"."TJOIN2" >( "RNUM" NUMBER(*,0), > "C1" NUMBER(*,0), > "C2" CHAR(2 BYTE) >) ; > GRANT SELECT ON "DBCERT"."TJOIN2" TO PUBLIC; > Other examples > select TJOIN1.RNUM, TJOIN1.C1, TJOIN1.C2, TJOIN2.C2 as C2J2 from > certora.DBCERT.TJOIN1 inner join certora.DBCERT.TJOIN2 on ( TJOIN1.C1 = > TJOIN2.C1 and TJOIN2.C2 like 'A%' ) > DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL > query. > sql SELECT "C1", "C2", "C2" LIKE 'A%' "$f3" > FROM "DBCERT"."TJOIN2" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-5762) CSV having header column by name "suffix" fails to load
[
https://issues.apache.org/jira/browse/DRILL-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua resolved DRILL-5762.
-
Resolution: Invalid
Since {{suffix}} is a reserved key word, the only workaround is to use a
different column name.
> CSV having header column by name "suffix" fails to load
> ---
>
> Key: DRILL-5762
> URL: https://issues.apache.org/jira/browse/DRILL-5762
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Affects Versions: 1.11.0
>Reporter: Praveen Yadav
>Priority: Major
>
> Trying select * query on the below csv file using apache drill 1.11.0.
> {code:none}
> id,email,first_name,last_name,middle_name,suffix,work_phone,mobile_phone,gender,picture,speciality,taxonomy_code,education_details,experience_details,keywords,doctor_npi,wait_time,created_tstamp,created_by,last_updated_tstamp,last_updated_by,is_deleted
> 1,x...@gmail.com,X,,,Dr,912225711234,,M,assets/images/doctorIcon.png,Primary
> Care Physician,Primary Care Doctor,M.D,3 years,Primary Care
> Doctor,12349765,10,2015-04-22 17:20:48.0,,2015-12-16 12:06:27.0,,N
> 2,x...@gmail.com,,,,Dr,913345311234,,M,assets/images/doctorIcon.png,Eye
> Doctor,EYE Care Doctor,MD,5 years,,16456076,20,2015-04-30
> 11:07:57.0,,2015-11-07 08:49:57.0,,N
> {code}
> I get this error :
> {noformat}
> Error: DATA_READ ERROR: Error processing input: , line=1, char=286. Content
> parsed: [ ]
> Failure while reading file file:somepath/file.csv. Happened at or shortly
> before byte position 286.
> Fragment 0:0
> [Error Id: 1fff3645-e788-4ec3-b678-bea86a39003c on praveens-mbp.lan:31010]
> (state=,code=0)
> {noformat}
> Solution:
> Replacing column name "suffix" with any other text fixed the error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5046) Add documentation for directory based partition pruning
[ https://issues.apache.org/jira/browse/DRILL-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5046: Fix Version/s: 1.15.0 > Add documentation for directory based partition pruning > --- > > Key: DRILL-5046 > URL: https://issues.apache.org/jira/browse/DRILL-5046 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Rahul Challapalli >Assignee: Bridget Bevens >Priority: Major > Fix For: 1.15.0 > > > Drill's documentation for partition pruning should cover the below 2 features > 1. Directory based partition pruning > 2. Partition pruning based on auto-partitioned parquet files > The first one seems to be missing from our documentation. At the very least > we should cover > a. How we can leverage this feature to avoid full table scans > b. How this feature works in-conjunction with metadata cache pruning > c. A few examples which involve using wildcards for one of the sub-directories -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-5030) Drill SSL Docs have Bad Link to Oracle Website
[ https://issues.apache.org/jira/browse/DRILL-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua reassigned DRILL-5030: --- Assignee: Bridget Bevens > Drill SSL Docs have Bad Link to Oracle Website > -- > > Key: DRILL-5030 > URL: https://issues.apache.org/jira/browse/DRILL-5030 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.8.0 >Reporter: John Omernik >Assignee: Bridget Bevens >Priority: Major > Fix For: 1.14.0 > > > When going to setup custom SSL certs on Drill, I found that the link to the > oracle website was broken on this page: > https://drill.apache.org/docs/configuring-web-console-and-rest-api-security/ > at: > As cluster administrator, you can set the following SSL configuration > parameters in the conf/drill-override.conf file, as described in the Java > product documentation: > Obviously fixing the link is one option, another would be to provide > instructions for SSL certs directly in the drill docs so we are not reliant > on Oracle's website. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5030) Drill SSL Docs have Bad Link to Oracle Website
[ https://issues.apache.org/jira/browse/DRILL-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5030: Fix Version/s: 1.14.0 > Drill SSL Docs have Bad Link to Oracle Website > -- > > Key: DRILL-5030 > URL: https://issues.apache.org/jira/browse/DRILL-5030 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.8.0 >Reporter: John Omernik >Assignee: Bridget Bevens >Priority: Major > Fix For: 1.14.0 > > > When going to setup custom SSL certs on Drill, I found that the link to the > oracle website was broken on this page: > https://drill.apache.org/docs/configuring-web-console-and-rest-api-security/ > at: > As cluster administrator, you can set the following SSL configuration > parameters in the conf/drill-override.conf file, as described in the Java > product documentation: > Obviously fixing the link is one option, another would be to provide > instructions for SSL certs directly in the drill docs so we are not reliant > on Oracle's website. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-4973) Sqlline history
[ https://issues.apache.org/jira/browse/DRILL-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497165#comment-16497165 ] Kunal Khatua edited comment on DRILL-4973 at 5/31/18 8:57 PM: -- [~aengelbrecht] you want the ability to retain more than 500 commands? was (Author: kkhatua): [~aengelbrecht] you want the ability to retain more than 50?? > Sqlline history > --- > > Key: DRILL-4973 > URL: https://issues.apache.org/jira/browse/DRILL-4973 > Project: Apache Drill > Issue Type: Improvement > Components: Client - CLI >Reporter: Andries Engelbrecht >Priority: Minor > > Currently the history on sqlline stops working after 500 queries have been > logged in the users .sqlline/history file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4973) Sqlline history
[ https://issues.apache.org/jira/browse/DRILL-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497165#comment-16497165 ] Kunal Khatua commented on DRILL-4973: - [~aengelbrecht] you want the ability to retain more than 50?? > Sqlline history > --- > > Key: DRILL-4973 > URL: https://issues.apache.org/jira/browse/DRILL-4973 > Project: Apache Drill > Issue Type: Improvement > Components: Client - CLI >Reporter: Andries Engelbrecht >Priority: Minor > > Currently the history on sqlline stops working after 500 queries have been > logged in the users .sqlline/history file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-5189) There's no documentation for the typeof() function
[ https://issues.apache.org/jira/browse/DRILL-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua reassigned DRILL-5189: --- Assignee: Bridget Bevens > There's no documentation for the typeof() function > -- > > Key: DRILL-5189 > URL: https://issues.apache.org/jira/browse/DRILL-5189 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Chris Westin >Assignee: Bridget Bevens >Priority: Major > > I looked through the documentation at https://drill.apache.org/docs/ under > SQL Reference > SQL Functions > ... and could not find any reference to > typeof(). Google searches only turned up a reference to DRILL-4204. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5201) Query bug: null values in result of a conditioned query
[ https://issues.apache.org/jira/browse/DRILL-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497180#comment-16497180 ] Kunal Khatua commented on DRILL-5201: - Thanks, [~mehran]. Could you mark the issue as _RESOLVED_ with the resolution of _Not Reproducible_? > Query bug: null values in result of a conditioned query > - > > Key: DRILL-5201 > URL: https://issues.apache.org/jira/browse/DRILL-5201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.9.0 >Reporter: mehran >Assignee: Jinfeng Ni >Priority: Critical > Fix For: 1.14.0 > > Attachments: 20.tar.gz > > > I have a table with 300 columns. > When I select a query , some null columns are also returned. > This is the query that I used. > "select a,b from dfs.root.`/2016/1/17` where a=1" > For example in this query some columns are returned that "a" value is null. > I think null values are problems. Because when I add a condition to it like > "select a,b from dfs.root.`/2016/1/17` where a=1 and a IS NOT NULL" > there is not any change and still NULL values are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5281) JdbcSchema throws exception when detecting nullable for columns
[ https://issues.apache.org/jira/browse/DRILL-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5281: Fix Version/s: 1.14.0 > JdbcSchema throws exception when detecting nullable for columns > --- > > Key: DRILL-5281 > URL: https://issues.apache.org/jira/browse/DRILL-5281 > Project: Apache Drill > Issue Type: Bug >Reporter: Usman Shahid >Priority: Major > Fix For: 1.14.0 > > > Technically, it's a calcite bug i.e. > [CALCITE-1572|https://issues.apache.org/jira/browse/CALCITE-1572] which has > been recently fixed in the original Calcite source. Drill, however uses a > specific port of [Calcite by MapR|https://github.com/mapr/incubator-calcite] > and does not include most of the fixes from the original calcite source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5281) JdbcSchema throws exception when detecting nullable for columns
[ https://issues.apache.org/jira/browse/DRILL-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497183#comment-16497183 ] Kunal Khatua commented on DRILL-5281: - [~usmanshahid92] can you verify if the issue still persists? Drill have upgraded to Calcite 1.15 > JdbcSchema throws exception when detecting nullable for columns > --- > > Key: DRILL-5281 > URL: https://issues.apache.org/jira/browse/DRILL-5281 > Project: Apache Drill > Issue Type: Bug >Reporter: Usman Shahid >Priority: Major > Fix For: 1.14.0 > > > Technically, it's a calcite bug i.e. > [CALCITE-1572|https://issues.apache.org/jira/browse/CALCITE-1572] which has > been recently fixed in the original Calcite source. Drill, however uses a > specific port of [Calcite by MapR|https://github.com/mapr/incubator-calcite] > and does not include most of the fixes from the original calcite source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5435) Using Limit causes Memory Leaked Error since 1.10
[ https://issues.apache.org/jira/browse/DRILL-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497195#comment-16497195 ] Kunal Khatua commented on DRILL-5435: - [~kyrill007] can you share more details of the issue? We hadn't heard back from [~fmethot] or [~andreatbonanno], so I assumed that the issue was fixed. > Using Limit causes Memory Leaked Error since 1.10 > - > > Key: DRILL-5435 > URL: https://issues.apache.org/jira/browse/DRILL-5435 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: 1.10.0 >Reporter: F Méthot >Assignee: Parth Chandra >Priority: Major > > Here is the details I can provide: > We migrated our production system from Drill 1.9 to 1.10 just 5 days ago. > (220 nodes cluster) > Our log show there was some 900+ queries ran without problem in first 4 days. > (similar queries, that never use the `limit` clause) > Yesterday we started doing simple adhoc select * ... limit 10 queries (like > we often do, that was our first use of limit with 1.10) > and we got a `Memory was leaked` exception below. > Also, once we get the error, Most of all subsequent user queries fails with > Channel Close Exception. We need to restart Drill to bring it back to normal. > A day later, I used a similar select * limit 10 queries, and the same thing > happen, had to restart Drill. > In the exception, it was refering to a file (1_0_0.parquet) > I moved that file to smaller test cluster (12 nodes) and got the error on the > first attempt. but I am no longer able to reproduce the issue on that file. > Between the 12 and 220 nodes cluster, a different Column name and Row Group > Start was listed in the error. > The parquet file was generated by Drill 1.10. > I tried the same file with a local drill-embedded 1.9 and 1.10 and had no > issue. > Here is the error (manually typed), if you think of anything obvious, let us > know. > AsyncPageReader - User Error Occured: Exception Occurred while reading from > disk (can not read class o.a.parquet.format.PageHeader: java.io.IOException: > input stream is closed.) > File:/1_0_0.parquet > Column: StringColXYZ > Row Group Start: 115215476 > [Error Id: ] > at UserException.java:544) > at > o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.handleAndThrowException(AsynvPageReader.java:199) > at > o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.access(AsynvPageReader.java:81) > at > o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:483) > at > o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:392) > at > o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:392) > ... > Caused by: java.io.IOException: can not read class > org.apache.parquet.format.PageHeader: java.io.IOException: Input Stream is > closed. >at o.a.parquet.format.Util.read(Util.java:216) >at o.a.parquet.format.Util.readPageHeader(Util.java:65) >at > o.a.drill.exec.store.parquet.columnreaders.AsyncPageReader(AsyncPageReaderTask:430) > Caused by: parquet.org.apache.thrift.transport.TTransportException: Input > stream is closed >at ...read(TIOStreamTransport.java:129) >at TTransport.readAll(TTransport.java:84) >at TCompactProtocol.readByte(TCompactProtocol.java:474) >at TCompactProtocol.readFieldBegin(TCompactProtocol.java:481) >at InterningProtocol.readFieldBegin(InterningProtocol.java:158) >at o.a.parquet.format.PageHeader.read(PageHeader.java:828) >at o.a.parquet.format.Util.read(Util.java:213) > Fragment 0:0 > [Error id: ...] > o.a.drill.common.exception.UserException: SYSTEM ERROR: > IllegalStateException: Memory was leaked by query. Memory leaked: (524288) > Allocator(op:0:0:4:ParquetRowGroupScan) 100/524288/39919616/100 > at o.a.d.common.exceptions.UserException (UserException.java:544) > at > o.a.d.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) > at o.a.d.exec.work.fragment.FragmentExecutor.cleanup( > FragmentExecutor.java:160) > at o.a.d.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262) > ... > Caused by: IllegalStateException: Memory was leaked by query. Memory leaked: > (524288) > at o.a.d.exec.memory.BaseAllocator.close(BaseAllocator.java:502) > at o.a.d.exec.ops.OperatorContextImpl(OperatorContextImpl.java:149) > at o.a.d.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422) > at o.a.d.exec.ops.FragmentContext.close(FragmentContext.java:411) > at > o.a.d.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318) > at > o.a.d.exec.work.fragment.F
[jira] [Commented] (DRILL-5487) Vector corruption in CSV with headers and truncated last row
[
https://issues.apache.org/jira/browse/DRILL-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497201#comment-16497201
]
Kunal Khatua commented on DRILL-5487:
-
[~paul-rogers] was this fixed? It's a fairly old bug, but I vaguely remember
discussing this and that you had a solution.
> Vector corruption in CSV with headers and truncated last row
>
>
> Key: DRILL-5487
> URL: https://issues.apache.org/jira/browse/DRILL-5487
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Priority: Major
>
> The CSV format plugin allows two ways of reading data:
> * As named columns
> * As a single array, called {{columns}}, that holds all columns for a row
> The named columns feature will corrupt the offset vectors if the last row of
> the file is truncated: leaves off one or more columns.
> To illustrate the CSV data corruption, I created a CSV file, test4.csv, of
> the following form:
> {code}
> h,u
> abc,def
> ghi
> {code}
> Note that the file is truncated: the command and second field is missing on
> the last line.
> Then, I created a simple test using the "cluster fixture" framework:
> {code}
> @Test
> public void readerTest() throws Exception {
> FixtureBuilder builder = ClusterFixture.builder()
> .maxParallelization(1);
> try (ClusterFixture cluster = builder.build();
> ClientFixture client = cluster.clientFixture()) {
> TextFormatConfig csvFormat = new TextFormatConfig();
> csvFormat.fieldDelimiter = ',';
> csvFormat.skipFirstLine = false;
> csvFormat.extractHeader = true;
> cluster.defineWorkspace("dfs", "data", "/tmp/data", "csv", csvFormat);
> String sql = "SELECT * FROM `dfs.data`.`csv/test4.csv` LIMIT 10";
> client.queryBuilder().sql(sql).printCsv();
> }
> }
> {code}
> The results show we've got a problem:
> {code}
> Exception (no rows returned):
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> IllegalArgumentException: length: -3 (expected: >= 0)
> {code}
> If the last line were:
> {code}
> efg,
> {code}
> Then the offset vector should look like this:
> {code}
> [0, 3, 3]
> {code}
> Very likely we have an offset vector that looks like this instead:
> {code}
> [0, 3, 0]
> {code}
> When we compute the second column of the second row, we should compute:
> {code}
> length = offset[2] - offset[1] = 3 - 3 = 0
> {code}
> Instead we get:
> {code}
> length = offset[2] - offset[1] = 0 - 3 = -3
> {code}
> The summary is that a premature EOF appears to cause the "missing" columns to
> be skipped; they are not filled with a blank value to "bump" the offset
> vectors to fill in the last row. Instead, they are left at 0, causing havoc
> downstream in the query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5777) Oracle JDBC Error while access synonym
[ https://issues.apache.org/jira/browse/DRILL-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5777: Component/s: Storage - JDBC > Oracle JDBC Error while access synonym > -- > > Key: DRILL-5777 > URL: https://issues.apache.org/jira/browse/DRILL-5777 > Project: Apache Drill > Issue Type: Bug > Components: Client - Java, Storage - JDBC >Affects Versions: 1.10.0 >Reporter: Sudhir Kumar >Priority: Major > > Error while accessing individual column in oracle table accessing via synonym. > Query : select from > .. > Error: > 1 > 2017-09-08 10:13:46,451 [264d3035-1605-9f5b-084f-09a1b525ef75:foreman] INFO > o.a.d.exec.planner.sql.SqlConverter - User Error Occurred: From line 1, > column 8 to line 1, column 17: Column not found in any table > (From line 1, column 8 to line 1, column 17: Column not found > in any table) > org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: From line > 1, column 8 to line 1, column 17: Column not found in any table > SQL Query null > [Error Id: 2b7c7c2d-664e-4c90-ba20-67509de90f09 ] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544) > ~[drill-common-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:178) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:622) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:192) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79) > [drill-java-exec-1.10.0.jar:1.10.0] > at > org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1050) > [drill-java-exec-1.10.0.jar:1.10.0] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) > [drill-java-exec-1.10.0.jar:1.10.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_92] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_92] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92] > Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, > column 8 to line 1, column 17: Column 'TABLE_NAME' not found in any table > at sun.reflect.GeneratedConstructorAccessor67.newInstance(Unknown > Source) ~[na:na] > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > ~[na:1.8.0_92] > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > ~[na:1.8.0_92] > at > org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:405) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:765) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:753) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:3974) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.EmptyScope.findQualifyingTableName(EmptyScope.java:108) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.DelegatingScope.findQualifyingTableName(DelegatingScope.java:112) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.ListScope.findQualifyingTableName(ListScope.java:150) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.DelegatingScope.fullyQualify(DelegatingScope.java:154) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:4460) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visit(SqlValidatorImpl.java:4440) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:274) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.
[jira] [Updated] (DRILL-3970) Oracle JDBC configuration filters necessary Calcite parameter
[
https://issues.apache.org/jira/browse/DRILL-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-3970:
Component/s: Storage - JDBC
> Oracle JDBC configuration filters necessary Calcite parameter
> -
>
> Key: DRILL-3970
> URL: https://issues.apache.org/jira/browse/DRILL-3970
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.2.0
> Environment: Mac OS X 10.10
>Reporter: Kyle Marek-Spartz
>Priority: Major
>
> After getting around DRILL-3958, I created a JDBC storage connection to an
> Oracle database. However, when I execute queries, I get errors of this sort:
> {code}
> 0: jdbc:drill:zk=local> show tables from oracle;
> Error: SYSTEM ERROR: IllegalArgumentException: duplicate key: WWV_FLOW_DUAL100
> Fragment 0:0
> [Error Id: f34cc14b-42ea-4caf-a849-205eb5570ba2 on .:31010]
> (state=,code=0)
> {code}
> This seems similar to CALCITE-880, which was closed as a configuration issue,
> suggesting adding {{"jdbcSchema": "ORACLE_SCHEMA_NAME"}} to the JSON
> configuration. Adding this to the Drill storage configuration option gives a
> {{Invalid JSON mapping}} error, which likely is due to a lack of
> {{jdbcSchema}} in {{JdbcStorageConfig.java}}:
> https://github.com/apache/drill/blob/d7eebec41a1636055be1b2c79b693d76c52d8932/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStorageConfig.java#L38-L41
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-4200) drill-jdbc-storage: applies timezone to java.sql.Date field and fails
[
https://issues.apache.org/jira/browse/DRILL-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-4200:
Component/s: Storage - JDBC
> drill-jdbc-storage: applies timezone to java.sql.Date field and fails
> -
>
> Key: DRILL-4200
> URL: https://issues.apache.org/jira/browse/DRILL-4200
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC, Storage - Other
>Affects Versions: 1.3.0
> Environment: drill-jdbc-storage plugin configured (based on
> https://drill.apache.org/docs/rdbms-storage-plugin) with
> org.relique.jdbc.csv.CsvDriver to access dbf (dbase) files.
>Reporter: Karol Potocki
>Priority: Major
>
> When using org.relique.jdbc.csv.CsvDriver to query files with date fields
> (i.e. 2012-05-01) causes:
> {code}
> UnsupportedOperationException: Method not supported: ResultSet.getDate(int,
> Calendar)
> {code}
> In JdbcRecordReader.java:406 there is getDate which tries to apply timezone
> to java.sql.Date which probably is not timezone related and this brings the
> error.
> Quick fix is to use ResultSet.getDate(int) instead.
> Details:
> {code}
> Caused by: java.lang.UnsupportedOperationException: Method not supported:
> Result
> Set.getDate(int, Calendar)
> at org.relique.jdbc.csv.CsvResultSet.getDate(Unknown Source)
> ~[csvjdbc-1
> .0-28.jar:na]
> at
> org.apache.commons.dbcp.DelegatingResultSet.getDate(DelegatingResultS
> et.java:574) ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.DelegatingResultSet.getDate(DelegatingResultS
> et.java:574) ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.drill.exec.store.jdbc.JdbcRecordReader$DateCopier.copy(Jdb
> cRecordReader.java:406)
> ~[drill-jdbc-storage-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.next(JdbcRecordRead
> er.java:242) ~[drill-jdbc-storage-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-4211) Column aliases not pushed down to JDBC stores in some cases when Drill expects aliased columns to be returned.
[
https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-4211:
Component/s: Storage - JDBC
> Column aliases not pushed down to JDBC stores in some cases when Drill
> expects aliased columns to be returned.
> --
>
> Key: DRILL-4211
> URL: https://issues.apache.org/jira/browse/DRILL-4211
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators, Storage - JDBC
>Affects Versions: 1.3.0, 1.11.0
> Environment: Postgres db storage
>Reporter: Robert Hamilton-Smith
>Assignee: Timothy Farkas
>Priority: Major
> Labels: newbie
>
> When making an sql statement that incorporates a join to a table and then a
> self join to that table to get a parent value , Drill brings back
> inconsistent results.
> Here is the sql in postgres with correct output:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from transactions trx
> join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join categories w1 on (cat.categoryparentguid = w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL;
> {code}
> Output:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|food&Dining|
> |id1|restaurants|food&Dining|
> |id2|Coffee Shops|food&Dining|
> |id2|Coffee Shops|food&Dining|
> When run in Drill with correct storage prefix:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from db.schema.transactions trx
> join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join db.schema.wpfm_categories w1 on (cat.categoryparentguid =
> w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL
> {code}
> Results are:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|null|
> |id1|restaurants|null|
> |id2|Coffee Shops|null|
> |id2|Coffee Shops|null|
> Physical plan is:
> {code:sql}
> 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50)
> categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost =
> {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293
> 00-01 Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) :
> rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname,
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows,
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292
> 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47])
> : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname,
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows,
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291
> 00-03 Jdbc(sql=[SELECT *
> FROM "public"."transactions"
> INNER JOIN (SELECT *
> FROM "public"."categories"
> WHERE "categoryparentguid" IS NOT NULL) AS "t" ON
> "transactions"."categoryguid" = "t"."categoryguid"
> INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid"
> = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255)
> transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255)
> transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50)
> transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid,
> VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate,
> VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3)
> transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3)
> transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512)
> transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20)
> transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode,
> VARCHAR(50) transactionorigpartyguid, VARCHAR(255)
> transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255)
> transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1,
> 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0)
> transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6)
> transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype, VARCHAR(50)
> transactionmatrulerefobjguid, VARCHAR(50) transactionmatrulerefobjvalue,
> VARCHAR(50) transactionuserruleguid, DECIMAL(2, 0) transactionsplitorder,
> TIMESTAMP(6) transactionprocessedat, TIMESTAMP(6)
> transactioncategoryassignat, VARCHAR(50) transactionsystemcategoryguid,
> VARCHAR(50) transactionorigmandateid, VARCHAR(100) fingerprint, VARCHAR(50)
> categoryguid0, VARCHAR(50) categoryparentguid, DECIMAL(3, 0) categorytype,
> VARCHAR(50) categoryname, VARCHAR(50) categorydescription, VARCHAR
[jira] [Updated] (DRILL-3977) unable to configure postgreSQL JDBC driver using apache drill 1.2
[
https://issues.apache.org/jira/browse/DRILL-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-3977:
Component/s: Storage - JDBC
> unable to configure postgreSQL JDBC driver using apache drill 1.2
> -
>
> Key: DRILL-3977
> URL: https://issues.apache.org/jira/browse/DRILL-3977
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Storage - JDBC
>Reporter: Xadi Del
>Priority: Major
>
> Hello,
> I'd like to use the new RDBMS capability working with apache drill 1.2.0.
> I followed this: https://drill.apache.org/docs/jdbc-storage-plugin/
> I put the mysql drivier jar file in this folder
> 'apache-drill-1.2.0\jars\3rdparty'
> And now, I'm trying to create a storage configuration. I am doing it using
> the web console (http://127.0.0.1:8047). When I try to add:
> {
> "type": "jdbc",
> "driver": "com.mysql.jdbc.Driver",
> "url": "jdbc:mysql://localhost:3306",
> "username": "root",
> "password": "mypassword",
> "enabled": true
> }
> It is showing error: error(Unable to create / update storage)
> Even with just the following it's not working (same error):
> {
> "type": "jdbc"
> }
> To link apache drill classpath and the jar I added
> drill.exec.sys.store.provider.local.path =
> "mysql-connector-java-5.1.37-bin.jar"
> to the drill-override.conf -> result is:
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
> drill.exec.sys.store.provider.local.path =
> "mysql-connector-java-5.1.37-bin.jar"
> }
> Any idea? Thanks,
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-3916) Assembly for JDBC storage plugin missing
[ https://issues.apache.org/jira/browse/DRILL-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-3916: Component/s: Storage - JDBC > Assembly for JDBC storage plugin missing > > > Key: DRILL-3916 > URL: https://issues.apache.org/jira/browse/DRILL-3916 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JDBC, Storage - Other >Affects Versions: 1.2.0 >Reporter: Andrew >Assignee: Andrew >Priority: Major > > The JDBC storage plugin is missing from the assembly instructions, which > means that the plugin fails to be loaded by the drill bit on start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5607) NullPointer Exception on jdbc source using limit 0
[
https://issues.apache.org/jira/browse/DRILL-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-5607:
Component/s: Storage - JDBC
> NullPointer Exception on jdbc source using limit 0
> --
>
> Key: DRILL-5607
> URL: https://issues.apache.org/jira/browse/DRILL-5607
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata, Storage - JDBC
>Affects Versions: 1.10.0, 1.11.0, 1.12.0
> Environment: mariadb 10.0.29
> OpenJDK build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11
>Reporter: Holger Kiel
>Priority: Major
>
> Querying a jdbc datasource in a 2 node cluster when 1 node is down throws an
> exception when 'limit 0' is used to discover the schema:
> {code}
> 0: jdbc:drill:zk=localhost:2181> !verbose
> verbose: on
> 0: jdbc:drill:zk=localhost:2181> select * from mysql.drill_mysql_test.person
> limit 1;
> +---++---+-+--+---+-+--+--++---+---+--+-+++-+++++-+---+-++---++
> | person_id | first_name | last_name | address | city | state | zip | json |
> bigint_field | smallint_field | numeric_field | boolean_field | double_field
> | float_field | real_field | time_field | timestamp_field | date_field |
> datetime_field | year_field | text_field | tiny_text_field |
> medium_text_field | long_text_field | blob_field | bit_field | enum_field |
> +---++---+-+--+---+-+--+--++---+---+--+-+++-+++++-+---+-++---++
> | 1 | first_name_1 | last_name_1 | 1401 John F Kennedy Blvd | Philadelphia |
> PA | 19107 | { a : 5, b : 6 } | 123456789 | 1 | 10.01 | false | 1.0 | 1.1 |
> 1.2 | 12:00:01 | 2012-02-29 12:00:01.0 | 2012-02-29 | 2012-02-29 12:00:01.0 |
> 2015-01-01 | It is a long established fact that a reader will be distracted
> by the readable content of a page when looking at its layout | xxx | a medium
> piece of text | a longer piece of text this is going on. | [B@79c5636f |
> true | XXX |
> +---++---+-+--+---+-+--+--++---+---+--+-+++-+++++-+---+-++---++
> 1 row selected (0,257 seconds)
> 0: jdbc:drill:zk=localhost:2181> select * from mysql.drill_mysql_test.person
> limit 0;
> Error: SYSTEM ERROR: NullPointerException
> [Error Id: 977858d8-ed05-4a94-a28b-931ee870826f on localhost:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> [Error Id: 977858d8-ed05-4a94-a28b-931ee870826f on localhost:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:180)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at
> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:121)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:101)
> at sqlline.Commands.execute(Commands.java:841)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:746)
> at sqlline.SqlLine.begin(SqlLine.java:621)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: NullPointerException
> [Error Id: 977858d8-ed05-4a94-a28b-931ee870826f on localhost:3101
[jira] [Updated] (DRILL-5713) Doing joins on tables that share column names in a JDBC store returns incorrect results
[ https://issues.apache.org/jira/browse/DRILL-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5713: Component/s: Storage - JDBC > Doing joins on tables that share column names in a JDBC store returns > incorrect results > --- > > Key: DRILL-5713 > URL: https://issues.apache.org/jira/browse/DRILL-5713 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators, Storage - JDBC >Affects Versions: 1.11.0 > Environment: My mac running the latest drill in embedded mode. >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: newbie > > If there are two tables in Postgres that share column names, incorrect > results are returned when a join is done between the two tables. > For example if we have two tables: categories and categories2 with the > following contents: +---+-+---+ > | categoryguid | categoryparentguid | categoryname | > +---+-+---+ > | id1 | null| restaurants | > | null | id1 | food&Dining | > | id2 | null| Coffee Shops | > | null | id2 | food&Dining | > +---+-+---+ > Then the following join query returns incorrectly names columns and incorrect > null values: select cat.categoryname, cat2.categoryname from > postgres.public.categories cat join postgres.public.categories2 cat2 on > (cat.categoryguid = cat2.categoryguid) where cat.categoryguid IS NOT NULL; > +---++ > | categoryname | categoryname0 | > +---++ > | restaurants | null | > | Coffee Shops | null | > +---++ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4302) when running select count(*) from table;alert:Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL query
[ https://issues.apache.org/jira/browse/DRILL-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-4302: Component/s: Storage - JDBC > when running select count(*) from table;alert:Error: DATA_READ ERROR: The > JDBC storage plugin failed while trying setup the SQL query > - > > Key: DRILL-4302 > URL: https://issues.apache.org/jira/browse/DRILL-4302 > Project: Apache Drill > Issue Type: Improvement > Components: Client - JDBC, Storage - JDBC >Affects Versions: Future >Reporter: david_hudavy >Priority: Major > > 0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from > mysql256.datamart.tab1; > +--+ > | EXPR$0 | > +--+ > | 1225578 | > +--+ > 1 row selected (1.819 seconds) > 0: jdbc:drill:zk=SC-1:6181,SC-2:6181,PL-3:618> select count(1) from > mysql256.datamart.tab2; > Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the > SQL query. > sql SELECT COUNT(*) AS `EXPR$0` > FROM (SELECT 1 AS `$f0` > FROM `datamart`.`tab2`) AS `t` > plugin mysql256 > Fragment 0:0 > [Error Id: 9e919a48-bae5-4835-a9cd-64066bf560b0 on SC-2:31010] (state=,code=0) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5622) DISTINCT predicate generates invalid SQL for a JDBC data source such as ORACLE
[ https://issues.apache.org/jira/browse/DRILL-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5622: Component/s: Storage - JDBC > DISTINCT predicate generates invalid SQL for a JDBC data source such as ORACLE > -- > > Key: DRILL-5622 > URL: https://issues.apache.org/jira/browse/DRILL-5622 > Project: Apache Drill > Issue Type: Bug > Components: Server, Storage - JDBC >Affects Versions: 1.10.0 > Environment: ORACLE 12 > Apache Drill 1.10 >Reporter: N Campbell >Priority: Major > > The SQL generated by DRILL to a database such as ORACLE is inavlid > SELECT RNUM, TINT.CINT FROM certora.DBCERT.TINT WHERE TINT.CINT IS DISTINCT > FROM 1 > DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL > query. > SELECT * > FROM "DBCERT"."TINT" > WHERE CAST(CASE WHEN "CINT" IS NULL THEN 1 IS NOT NULL WHEN 1 IS NULL THEN > "CINT" IS NOT NULL ELSE "CINT" <> 1 END AS BOOLEAN) > plugin certora > Fragment 0:0 > CREATE TABLE "DBCERT"."TINT" >( "RNUM" NUMBER, > "CINT" NUMBER(9,0) >) ; > Insert into DBCERT.TINT (RNUM,CINT) values (0,null); > Insert into DBCERT.TINT (RNUM,CINT) values (1,-1); > Insert into DBCERT.TINT (RNUM,CINT) values (2,0); > Insert into DBCERT.TINT (RNUM,CINT) values (3,1); > Insert into DBCERT.TINT (RNUM,CINT) values (4,10); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-4088) XML Type not supported by JDBC plugin
[ https://issues.apache.org/jira/browse/DRILL-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-4088: Component/s: Storage - JDBC > XML Type not supported by JDBC plugin > - > > Key: DRILL-4088 > URL: https://issues.apache.org/jira/browse/DRILL-4088 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JDBC, Storage - Other >Affects Versions: 1.3.0 >Reporter: Andrew >Priority: Minor > > The JDBC storage plugin does not support column types of 'XML'. Any query > that references such a column, including a '*' query, will fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5932) HiveServer2 queries throw error with jdbc connection
[
https://issues.apache.org/jira/browse/DRILL-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-5932:
Component/s: Storage - JDBC
> HiveServer2 queries throw error with jdbc connection
>
>
> Key: DRILL-5932
> URL: https://issues.apache.org/jira/browse/DRILL-5932
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC, Storage - Hive, Storage - JDBC
>Affects Versions: 1.11.0
> Environment: linux
> 2.3 hive version
>Reporter: tooptoop4
>Priority: Blocker
>
> Basic hive queries all throw error!
> {code:sql}
> copied
> https://repo1.maven.org/maven2/org/apache/hive/hive-jdbc/2.3.0/hive-jdbc-2.3.0-standalone.jar
> to /usr/lib/apache-drill-1.11.0/jars/3rdparty/hive-jdbc-2.3.0-standalone.jar
> added this storage plugin:
> {
> "type": "jdbc",
> "driver": "org.apache.hive.jdbc.HiveDriver",
> "url": "jdbc:hive2://host:1/default",
> "username": "hive",
> "password": "hive1234",
> "enabled": true
> }
> [ec2-user@host ~]$ cd /usr/lib/apache-drill-1.11.0
> [ec2-user@host apache-drill-1.11.0]$ ./bin/drill-embedded
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
> was removed in 8.0
> Nov 01, 2017 7:53:53 AM org.glassfish.jersey.server.ApplicationHandler
> initialize
> INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29
> 01:25:26...
> apache drill 1.11.0
> "this isn't your grandfather's sql"
> 0: jdbc:drill:zk=local> SELECT count(*) FROM hive2.`contact`;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the
> SQL query.
> sql SELECT COUNT(*) AS EXPR$0
> FROM (SELECT 0 AS $f0
> FROM.default.contact) AS t
> plugin hive2
> Fragment 0:0
> [Error Id: 4b293e97-7547-49c5-91da-b9ee2f2184fc on
> ip-myip.mydomain.orghere.com:31010] (state=,code=0)
> 0: jdbc:drill:zk=local> ALTER SESSION SET `exec.errors.verbose` = true;
> +---+---+
> | ok |summary|
> +---+---+
> | true | exec.errors.verbose updated. |
> +---+---+
> 1 row selected (0.351 seconds)
> 0: jdbc:drill:zk=local> select * from hive2.contact;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the
> SQL query.
> sql SELECT *
> FROM.default.contact
> plugin hive2
> Fragment 0:0
> [Error Id: fe36b026-e8ff-4354-af6c-6073130680c9 on ip-ip.domain.org.com:31010]
> (org.apache.hive.service.cli.HiveSQLException) Error while compiling
> statement: FAILED: ParseException line 2:4 cannot recognize input near '.'
> 'default' '.' in join source
> org.apache.hive.jdbc.Utils.verifySuccess():267
> org.apache.hive.jdbc.Utils.verifySuccessWithInfo():253
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer():313
> org.apache.hive.jdbc.HiveStatement.execute():253
> org.apache.hive.jdbc.HiveStatement.executeQuery():476
> org.apache.commons.dbcp.DelegatingStatement.executeQuery():208
> org.apache.commons.dbcp.DelegatingStatement.executeQuery():208
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup():177
> org.apache.drill.exec.physical.impl.ScanBatch.():104
> org.apache.drill.exec.physical.impl.ScanBatch.():126
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch():40
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch():33
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():156
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():179
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():179
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():109
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():87
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():207
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
> Caused By (org.apache.hive.service.cli.HiveSQLException) Error while
> compiling statement: FAILED: ParseException line 2:4 cannot recognize input
> near '.' 'default' '.' in join source
> org.apache.hive.service.cli.operation.Operation.toSQLException():380
> org.apache.hive.service.cli.operation.SQLOperation.prepare():206
> org.apache.hive.service.cli.operation.SQLOperation.runInternal():290
> org.apache.hive.service.cli.operation.Operation.run():320
>
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal():530
>
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync():517
> sun.reflect.GeneratedMethodAccessor86.i
[jira] [Updated] (DRILL-4312) JDBC PlugIN - MySQL Causes errors in Drill INFORMATION_SCHEMA
[
https://issues.apache.org/jira/browse/DRILL-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-4312:
Component/s: Storage - JDBC
> JDBC PlugIN - MySQL Causes errors in Drill INFORMATION_SCHEMA
> -
>
> Key: DRILL-4312
> URL: https://issues.apache.org/jira/browse/DRILL-4312
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC, Storage - Other
>Affects Versions: 1.4.0
>Reporter: Andries Engelbrecht
>Priority: Major
>
> When connecting MySQL with the JDBC PlugIn queries to INFORMATION_SCHEMA
> fails. Specifically for COLUMNS and on mysql.performance_schema.
> {query}
> SELECT DISTINCT TABLE_SCHEMA as NAME_SPACE, TABLE_NAME as TAB_NAME FROM
> INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA <>'INFORMATION_SCHEMA' and
> TABLE_SCHEMA <> 'sys';
> {/query}
> {result}
> Error: SYSTEM ERROR: MySQLSyntaxErrorException: Unknown table engine
> 'PERFORMANCE_SCHEMA'
> Fragment 0:0
> {/result}
> {query}
> 0: jdbc:drill:> select * from INFORMATION_SCHEMA.`COLUMNS` where TABLE_SCHEMA
> = 'mysql.performance_schema';
> {/query}
> {result}
> Error: SYSTEM ERROR: MySQLSyntaxErrorException: Unknown table engine
> 'PERFORMANCE_SCHEMA'
> Fragment 0:0
> {/result}
> {drillbit.log}
> [Error Id: 45d23eb8-0bcf-41e2-84e2-4626e7fb0d33 on drilldemo:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0.jar:1.4.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> Caused by: java.lang.RuntimeException: Exception while reading definition of
> table 'cond_instances'
> at
> org.apache.calcite.adapter.jdbc.JdbcTable.getRowType(JdbcTable.java:103)
> ~[calcite-core-1.4.0-drill-1.4.0-mapr-r1.jar:1.4.0-drill-1.4.0-mapr-r1]
> at
> org.apache.drill.exec.store.ischema.RecordGenerator.scanSchema(RecordGenerator.java:140)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.store.ischema.RecordGenerator.scanSchema(RecordGenerator.java:120)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.store.ischema.RecordGenerator.scanSchema(RecordGenerator.java:120)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.store.ischema.RecordGenerator.scanSchema(RecordGenerator.java:108)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.store.ischema.SelectedTable.getRecordReader(SelectedTable.java:57)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:36)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:147)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:170)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:127)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:170)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:101)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79)
> ~[drill-java-exec-1.4.0.jar:1.4.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230)
> [drill-java-exec-1.4.0.jar:1.4.0]
> ... 4 common frames omitted
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown
> table engine 'PERFORMANCE_SCHEMA'
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.8.0_51]
>
[jira] [Updated] (DRILL-6200) ERROR Quering hive through HiverServer2 via JDBC!
[ https://issues.apache.org/jira/browse/DRILL-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-6200: Component/s: Storage - JDBC > ERROR Quering hive through HiverServer2 via JDBC! > - > > Key: DRILL-6200 > URL: https://issues.apache.org/jira/browse/DRILL-6200 > Project: Apache Drill > Issue Type: Bug > Components: Client - JDBC, Storage - Hive, Storage - JDBC >Affects Versions: 1.12.0 > Environment: CentOS Linux release 7.4.1708 (Core) (It's not a cluster > node - thats why i'm using jdbc on hiveserver2). > Hive 2.1.1 Version >Reporter: Willian Mattos Ribeiro >Priority: Major > > ERROR [HY000] [MapR][Drill] (1040) Drill failed to execute the query: select > * from dim_parameter > [30034]Query execution error. Details:[ > DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL > query. > sql SELECT * > FROM.dw.dim_parameter > plugin hive > Fragment 0:0 > [Error Id: e522f220-b857-4273-af0a-2a2d05d992f2 on 172.28.32.7:31010] > (org.apache.hive.service.cli.HiveSQLException) Error while compiling > statement: FAILED: ParseException line 2:4 cannot recognize input near '.' > 'dw' '.' in join source > org.apache.hive.jdbc.Utils.verifySuccess():267 > org.apache.hive.jdbc.Utils.verifySuccessWithInfo():253 > org.apache.hive.jdbc.HiveStatement.runAsyncOnServer():309 > org.apache.hive.jdbc.HiveStatement.execute():250 > org.apache.hive.jdbc.HiveStatement.executeQuery():434 > org.apache.commons.dbcp.DelegatingStatement.executeQuery():208 > org.apache.commons.dbcp.DelegatingStatement.executeQuery():208 > org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup():177 > org.apache.drill.exec.p... > em System.Data.Odbc.OdbcConnection.HandleError(OdbcHandle hrHandle, RetCode > retcode) > em System.Data.Odbc.OdbcCommand.ExecuteReaderObject(CommandBehavior > behavior, String method, Boolean needReader, Object[] methodArguments, > SQL_API odbcApiMethod) > em System.Data.Odbc.OdbcCommand.ExecuteReaderObject(CommandBehavior > behavior, String method, Boolean needReader) > em System.Data.Odbc.OdbcCommand.ExecuteReader(CommandBehavior behavior) > em DrillExplorer.DROdbcProvider.GetStatmentColumns(String in_query) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-105) JDBC Storage engine
[ https://issues.apache.org/jira/browse/DRILL-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-105: --- Component/s: Storage - JDBC > JDBC Storage engine > --- > > Key: DRILL-105 > URL: https://issues.apache.org/jira/browse/DRILL-105 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - JDBC >Reporter: Ted Dunning >Priority: Major > Fix For: Future > > > We should allow query pushdown into JDBC. Note that this is very different > from JDBC support that allows a Java program to send a query to Drill. > Instead, it is a way for Drill to push down a query into a JDBC driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6450) Visualized plans for profiles querying JDBC sources is broken
[
https://issues.apache.org/jira/browse/DRILL-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-6450:
Component/s: Storage - JDBC
> Visualized plans for profiles querying JDBC sources is broken
> -
>
> Key: DRILL-6450
> URL: https://issues.apache.org/jira/browse/DRILL-6450
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization, Storage - JDBC
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> When viewing a profile for a query against a JDBC source, the visualized plan
> is not rendered. This is because the generated SQL pushed down to the JDBC
> source has a line break injected just before the {{FROM}} clause.
> The workaround is to strip away any injected newlines at least for the SQL
> defined in the text plan, so that the backend Javascript can render it
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-5578) Drill fails on date functions in 'where clause' when queried on a JDBC Storage plugin
[ https://issues.apache.org/jira/browse/DRILL-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5578: Component/s: Storage - JDBC > Drill fails on date functions in 'where clause' when queried on a JDBC > Storage plugin > - > > Key: DRILL-5578 > URL: https://issues.apache.org/jira/browse/DRILL-5578 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JDBC >Affects Versions: 1.9.0, 1.10.0 >Reporter: Rahul Raj >Priority: Major > > Drill 1.9/1.10 fails on any date manipulation in the where part while > querying on a table from JDBC Storage plugin. Issue happens on postgres and > Oracle. > The following two queries error out: > select * from config_1.APP.EXECUTIONSTEP where DATE_ADD(CAST(STARTED_AT as > DATE),interval '1' second) < CAST(CURRENT_DATE as DATE) > select `id` from (select * from config_1.public.project_release) where > CAST(DATE_ADD(`start_date`,interval '19800' second(5)) AS DATE) = DATE > '2011-11-11' > However date manipulation succeeds when date function are applied on the > selected fields > > select DATE_ADD(CAST(STARTED_AT as DATE),interval '1' second) < > CAST(CURRENT_DATE as DATE) from config_1.APP.EXECUTIONSTEP > select `id` from (select * from config_1.public.project_release) where > CAST(`start_date` AS DATE) = DATE '2011-11-11' > Error: > [Error Id: 048fe4b9-ecab-40fe-aca9-d57eb2df9b0c on vpc12.o3c.in:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > AssertionError: 1000: INTERVAL_DAY_TIME > [Error Id: 048fe4b9-ecab-40fe-aca9-d57eb2df9b0c on vpc12.o3c.in:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.9.0.jar:1.9.0] > at > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:825) > [drill-java-exec-1.9.0.jar:1.9.0] > at > org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:935) > [drill-java-exec-1.9.0.jar:1.9.0] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) > [drill-java-exec-1.9.0.jar:1.9.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_60] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected > exception during fragment initialization: 1000: INTERVAL_DAY_TIME > ... 4 common frames omitted > Caused by: java.lang.AssertionError: 1000: INTERVAL_DAY_TIME > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:179) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:268) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:212) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:268) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:212) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:268) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor$Context.toSql(JdbcImplementor.java:212) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.implement(JdbcRules.java:658) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.adapter.jdbc.JdbcImplementor.visitChild(JdbcImplementor.java:118) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at org.apache.drill.exec.store.jdbc.JdbcPrel.(JdbcPrel.java:62) > ~[drill-jdbc-storage-1.9.0.jar:1.9.0] > at > org.apache.drill.exec.store.jdbc.JdbcIntermediatePrel.finalizeRel(JdbcIntermediatePrel.java:66) > ~[drill-jdbc-storage-1.9.0.jar:1.9.0] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler$PrelFinalizer.visit(DefaultSqlHandler.java:306) > ~[drill-java-exec-1.9.0.jar:1.9.0] > at > org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:256) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:53) > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19] > at > org.a
[jira] [Commented] (DRILL-6032) Use RecordBatchSizer to estimate size of columns in HashAgg
[ https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497241#comment-16497241 ] ASF GitHub Bot commented on DRILL-6032: --- ilooner commented on issue #1101: DRILL-6032: Made the batch sizing for HashAgg more accurate. URL: https://github.com/apache/drill/pull/1101#issuecomment-393694572 While making this change I learned that it is extremely difficult to make changes to the HashAgg operator, specifically the memory calculations. After a month of work I was not able to bring even a simple change like this to a state where it would not OOM. Others experience similar frustrations when they make changes that either impact the HashAgg operator directly, or indirectly. Considering the fact that a comprehensive and stable memory calculator was created for HashJoin in less than a month, I feel that there is an easier way forward. As a result of this experience I was motivated to create https://issues.apache.org/jira/browse/DRILL-6253 which focuses on refactoring and unit testing the memory calculations for HashAgg . DRILL-6253 will fulfill the original intention of this PR, which was to make the memory calculations more accurate, and also improve the general stability and testability of HashAgg. As a result of this, I am going to abandon this PR and all future work will be done on the PR for DRILL-6253 once it is opened. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use RecordBatchSizer to estimate size of columns in HashAgg > --- > > Key: DRILL-6032 > URL: https://issues.apache.org/jira/browse/DRILL-6032 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.15.0 > > > We need to use the RecordBatchSize to estimate the size of columns in the > Partition batches created by HashAgg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6032) Use RecordBatchSizer to estimate size of columns in HashAgg
[
https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497242#comment-16497242
]
ASF GitHub Bot commented on DRILL-6032:
---
ilooner closed pull request #1101: DRILL-6032: Made the batch sizing for
HashAgg more accurate.
URL: https://github.com/apache/drill/pull/1101
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
index 47f1017b34..374dcdd3a4 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java
@@ -25,7 +25,6 @@
import org.apache.drill.common.exceptions.UserException;
import org.apache.drill.common.expression.ErrorCollector;
import org.apache.drill.common.expression.ErrorCollectorImpl;
-import org.apache.drill.common.expression.FunctionHolderExpression;
import org.apache.drill.common.expression.IfExpression;
import org.apache.drill.common.expression.LogicalExpression;
import org.apache.drill.common.logical.data.NamedExpression;
@@ -65,7 +64,7 @@
private HashAggregator aggregator;
private RecordBatch incoming;
- private LogicalExpression[] aggrExprs;
+ private ValueVectorWriteExpression[] aggrExprs;
private TypedFieldId[] groupByOutFieldIds;
private TypedFieldId[] aggrOutFieldIds; // field ids for the outgoing
batch
private final List comparators;
@@ -231,7 +230,7 @@ private HashAggregator createAggregatorInternal() throws
SchemaChangeException,
int numGroupByExprs = (popConfig.getGroupByExprs() != null) ?
popConfig.getGroupByExprs().size() : 0;
int numAggrExprs = (popConfig.getAggrExprs() != null) ?
popConfig.getAggrExprs().size() : 0;
-aggrExprs = new LogicalExpression[numAggrExprs];
+aggrExprs = new ValueVectorWriteExpression[numAggrExprs];
groupByOutFieldIds = new TypedFieldId[numGroupByExprs];
aggrOutFieldIds = new TypedFieldId[numAggrExprs];
@@ -255,7 +254,6 @@ private HashAggregator createAggregatorInternal() throws
SchemaChangeException,
groupByOutFieldIds[i] = container.add(vv);
}
-int extraNonNullColumns = 0; // each of SUM, MAX and MIN gets an extra
bigint column
for (i = 0; i < numAggrExprs; i++) {
NamedExpression ne = popConfig.getAggrExprs().get(i);
final LogicalExpression expr =
@@ -273,15 +271,10 @@ private HashAggregator createAggregatorInternal() throws
SchemaChangeException,
continue;
}
- if ( expr instanceof FunctionHolderExpression ) {
- String funcName = ((FunctionHolderExpression) expr).getName();
- if ( funcName.equals("sum") || funcName.equals("max") ||
funcName.equals("min") ) {extraNonNullColumns++;}
- }
final MaterializedField outputField =
MaterializedField.create(ne.getRef().getAsNamePart().getName(),
expr.getMajorType());
@SuppressWarnings("resource")
ValueVector vv = TypeHelper.getNewVector(outputField,
oContext.getAllocator());
aggrOutFieldIds[i] = container.add(vv);
-
aggrExprs[i] = new ValueVectorWriteExpression(aggrOutFieldIds[i], expr,
true);
}
@@ -301,7 +294,7 @@ private HashAggregator createAggregatorInternal() throws
SchemaChangeException,
aggrExprs,
cgInner.getWorkspaceTypes(),
groupByOutFieldIds,
-this.container, extraNonNullColumns * 8 /* sizeof(BigInt) */);
+container);
return agg;
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
index 4c54080169..1a33577120 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
@@ -17,20 +17,16 @@
*/
package org.apache.drill.exec.physical.impl.aggregate;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Iterator;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-
-import javax.inject.Named;
-
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
import org.apache.drill.common.exceptions.RetryAfterSpillException;
import org.apache.drill.common.exceptions.UserException;
import org.apache.drill.common.expression.ExpressionPosition;
import org.apache.drill.common.expression.FieldReferenc
[jira] [Commented] (DRILL-6456) Planner shouldn't create any exchanges on the right side of Lateral Join.
[
https://issues.apache.org/jira/browse/DRILL-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497270#comment-16497270
]
ASF GitHub Bot commented on DRILL-6456:
---
amansinha100 commented on a change in pull request #1299: DRILL-6456: Planner
shouldn't create any exchanges on the right side …
URL: https://github.com/apache/drill/pull/1299#discussion_r192256195
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/lateraljoin/TestLateralPlans.java
##
@@ -275,4 +284,153 @@ public void testUnnestColumnAlias() throws Exception {
assert(ex.getMessage().contains("Alias table and column name are
required for UNNEST"));
}
}
+
+
/***
+ Following test cases are introduced to make sure no exchanges are present
on right side of
+ Lateral join.
+
**/
+
+ @Test
+ public void testNoExchangeWithAggWithoutGrpBy() throws Exception {
+String Sql = "select d1.totalprice from dfs.`lateraljoin/multipleFiles`
t," +
+" lateral ( select sum(t2.ord.o_totalprice) as totalprice from
unnest(t.c_orders) t2(ord)) d1";
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher)
+.setOptionDefault(ExecConstants.ENABLE_UNNEST_LATERAL_KEY, true)
+.setOptionDefault(ExecConstants.SLICE_TARGET, 1);
+
+try (ClusterFixture cluster = builder.build();
+ ClientFixture client = cluster.clientFixture()) {
+ String explain = client.queryBuilder().sql(Sql).explainText();
+ String rightChild = getRightChildOfLateral(explain);
+ assert(!rightChild.contains("Exchange"));
Review comment:
Can you use the JUnit's Assert ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Planner shouldn't create any exchanges on the right side of Lateral Join.
> -
>
> Key: DRILL-6456
> URL: https://issues.apache.org/jira/browse/DRILL-6456
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, there is no restriction placed on right side of the LateralJoin.
> This is causing planner to generate an Exchange when there are operators like
> (Agg, Limit, Sort etc).
> Due to this unnest operator cannot retrieve the row from lateral's left side
> to process the pipeline further. Enhance the planner to not generate
> exchanges on the right side of the LateralJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5281) JdbcSchema throws exception when detecting nullable for columns
[ https://issues.apache.org/jira/browse/DRILL-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497284#comment-16497284 ] Pritesh Maker commented on DRILL-5281: -- With Drill 1.13, we moved to using Apache Calcite 1.15 (and moving to 1.16 with Drill 1.14). I am marking the issue as resolved [~usmanshahid92] please verify with the newer version of Apache Drill (1.13+) and reopen if it is not yet addressed. > JdbcSchema throws exception when detecting nullable for columns > --- > > Key: DRILL-5281 > URL: https://issues.apache.org/jira/browse/DRILL-5281 > Project: Apache Drill > Issue Type: Bug >Reporter: Usman Shahid >Priority: Major > Fix For: 1.14.0 > > > Technically, it's a calcite bug i.e. > [CALCITE-1572|https://issues.apache.org/jira/browse/CALCITE-1572] which has > been recently fixed in the original Calcite source. Drill, however uses a > specific port of [Calcite by MapR|https://github.com/mapr/incubator-calcite] > and does not include most of the fixes from the original calcite source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-5281) JdbcSchema throws exception when detecting nullable for columns
[ https://issues.apache.org/jira/browse/DRILL-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker resolved DRILL-5281. -- Resolution: Resolved Resolved with upgrading the dependency to Apache Calcite 1.15 > JdbcSchema throws exception when detecting nullable for columns > --- > > Key: DRILL-5281 > URL: https://issues.apache.org/jira/browse/DRILL-5281 > Project: Apache Drill > Issue Type: Bug >Reporter: Usman Shahid >Priority: Major > Fix For: 1.14.0 > > > Technically, it's a calcite bug i.e. > [CALCITE-1572|https://issues.apache.org/jira/browse/CALCITE-1572] which has > been recently fixed in the original Calcite source. Drill, however uses a > specific port of [Calcite by MapR|https://github.com/mapr/incubator-calcite] > and does not include most of the fixes from the original calcite source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-5796) Filter pruning for multi rowgroup parquet file
[ https://issues.apache.org/jira/browse/DRILL-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker reassigned DRILL-5796: Assignee: Damien Profeta Reviewer: Arina Ielchiieva > Filter pruning for multi rowgroup parquet file > -- > > Key: DRILL-5796 > URL: https://issues.apache.org/jira/browse/DRILL-5796 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Reporter: Damien Profeta >Assignee: Damien Profeta >Priority: Major > Fix For: 1.14.0 > > > Today, filter pruning use the file name as the partitioning key. This means > you can remove a partition only if the whole file is for the same partition. > With parquet, you can prune the filter if the rowgroup make a partition of > your dataset as the unit of work if the rowgroup not the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5796) Filter pruning for multi rowgroup parquet file
[ https://issues.apache.org/jira/browse/DRILL-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-5796: - Fix Version/s: 1.14.0 > Filter pruning for multi rowgroup parquet file > -- > > Key: DRILL-5796 > URL: https://issues.apache.org/jira/browse/DRILL-5796 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Parquet >Reporter: Damien Profeta >Assignee: Damien Profeta >Priority: Major > Fix For: 1.14.0 > > > Today, filter pruning use the file name as the partitioning key. This means > you can remove a partition only if the whole file is for the same partition. > With parquet, you can prune the filter if the rowgroup make a partition of > your dataset as the unit of work if the rowgroup not the file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5365) FileNotFoundException when reading a parquet file
[
https://issues.apache.org/jira/browse/DRILL-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497294#comment-16497294
]
ASF GitHub Bot commented on DRILL-5365:
---
ilooner closed pull request #796: DRILL-5365: DrillFileSystem setConf in
constructor. DrillFileSystem c…
URL: https://github.com/apache/drill/pull/796
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/DrillFileSystem.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/DrillFileSystem.java
index e03cf3..9ce7c6d68d 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/DrillFileSystem.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/DrillFileSystem.java
@@ -89,22 +89,36 @@ public DrillFileSystem(Configuration fsConf) throws
IOException {
}
public DrillFileSystem(Configuration fsConf, OperatorStats operatorStats)
throws IOException {
-this.underlyingFs = FileSystem.get(fsConf);
+this(fsConf, URI.create(fsConf.getRaw(FS_DEFAULT_NAME_KEY)),
operatorStats);
+ }
+
+ public DrillFileSystem(Configuration fsConf, URI Uri, OperatorStats
operatorStats) throws IOException {
+this.underlyingFs = FileSystem.get(Uri, fsConf);
+logger.trace("Configuration for the DrillFileSystem " +
fsConf.getRaw(FS_DEFAULT_NAME_KEY) +
+", underlyingFs: " + this.underlyingFs.getUri());
this.codecFactory = new CompressionCodecFactory(fsConf);
this.operatorStats = operatorStats;
+setConf(fsConf);
}
@Override
public void setConf(Configuration conf) {
// Guard against setConf(null) call that is called as part of superclass
constructor (Configured) of the
// DrillFileSystem, at which point underlyingFs is null.
-if (conf != null && underlyingFs != null) {
- underlyingFs.setConf(conf);
+if(conf != null) {
+ super.setConf(conf);
+ if (underlyingFs != null && underlyingFs.getConf() == null) {
+underlyingFs.setConf(conf);
+ }
+
}
}
@Override
public Configuration getConf() {
+if (super.getConf() != null) {
+ return super.getConf();
+}
return underlyingFs.getConf();
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
index 7f2a9c16a7..b5f087117b 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
@@ -74,6 +74,7 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
fsConf.set(s, config.config.get(s));
}
}
+ fsConf.set("fs.default.name", config.connection);
fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, config.connection);
fsConf.set("fs.classpath.impl", ClassPathFileSystem.class.getName());
fsConf.set("fs.drill-local.impl",
LocalSyncableFileSystem.class.getName());
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> FileNotFoundException when reading a parquet file
> -
>
> Key: DRILL-5365
> URL: https://issues.apache.org/jira/browse/DRILL-5365
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.10.0
>Reporter: Chun Chang
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> The parquet file is generated through the following CTAS.
> To reproduce the issue: 1) two or more nodes cluster; 2) enable
> impersonation; 3) set "fs.default.name": "file:///" in hive storage plugin;
> 4) restart drillbits; 5) as a regular user, on node A, drop the table/file;
> 6) ctas from a large enough hive table as source to recreate the table/file;
> 7) query the table from node A should work; 8) query from node B as same user
> should reproduce the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3313) Eliminate redundant #load methods and unit-test loading & exporting of vectors
[
https://issues.apache.org/jira/browse/DRILL-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497297#comment-16497297
]
ASF GitHub Bot commented on DRILL-3313:
---
ilooner closed pull request #81: DRILL-3313: Eliminate redundant #load methods
and unit-test loading & exporting of vectors
URL: https://github.com/apache/drill/pull/81
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/exec/java-exec/src/main/codegen/templates/FixedValueVectors.java
b/exec/java-exec/src/main/codegen/templates/FixedValueVectors.java
index cadcfd92ba..fa14fce059 100644
--- a/exec/java-exec/src/main/codegen/templates/FixedValueVectors.java
+++ b/exec/java-exec/src/main/codegen/templates/FixedValueVectors.java
@@ -156,21 +156,17 @@ public void zeroVector() {
}
@Override
- public int load(int valueCount, DrillBuf buf){
+ public void load(SerializedField metadata, DrillBuf buffer) {
+final int actualLength = metadata.getBufferLength();
+final int valueCount = metadata.getValueCount();
+final int expectedLength = valueCount * ${type.width};
+assert actualLength == expectedLength : String.format("Expected to load %d
bytes but actually loaded %d bytes", expectedLength, actualLength);
+
clear();
-int len = valueCount * ${type.width};
-data = buf.slice(0, len);
+data = buffer.slice(0, actualLength);
+data.writerIndex(actualLength);
data.retain();
-data.writerIndex(len);
-return len;
- }
-
- @Override
- public void load(SerializedField metadata, DrillBuf buffer) {
-assert this.field.matches(metadata) : String.format("The field %s doesn't
match the provided metadata %s.", this.field, metadata);
-int loaded = load(metadata.getValueCount(), buffer);
-assert metadata.getBufferLength() == loaded : String.format("Expected to
load %d bytes but actually loaded %d bytes", metadata.getBufferLength(),
loaded);
- }
+}
public TransferPair getTransferPair(){
return new TransferImpl(getField());
diff --git
a/exec/java-exec/src/main/codegen/templates/NullableValueVectors.java
b/exec/java-exec/src/main/codegen/templates/NullableValueVectors.java
index 7fa0d551fe..7b8d3eb160 100644
--- a/exec/java-exec/src/main/codegen/templates/NullableValueVectors.java
+++ b/exec/java-exec/src/main/codegen/templates/NullableValueVectors.java
@@ -50,8 +50,10 @@
private final FieldReader reader = new
Nullable${minor.class}ReaderImpl(Nullable${minor.class}Vector.this);
- private final UInt1Vector bits = new
UInt1Vector(MaterializedField.create(field + "_bits",
Types.required(MinorType.UINT1)), allocator);
+ private final MaterializedField bitsField =
MaterializedField.create("$bits$", Types.required(MinorType.UINT1));
+ private final UInt1Vector bits = new UInt1Vector(bitsField, allocator);
private final ${valuesName} values = new ${minor.class}Vector(field,
allocator);
+
private final Mutator mutator = new Mutator();
private final Accessor accessor = new Accessor();
@@ -106,16 +108,14 @@ public void setInitialCapacity(int numRecords) {
values.setInitialCapacity(numRecords);
}
- <#if type.major == "VarLen">
@Override
- public SerializedField getMetadata() {
-return getMetadataBuilder()
- .setValueCount(getAccessor().getValueCount())
- .setVarByteLength(values.getVarByteLength())
- .setBufferLength(getBufferSize())
- .build();
+ public SerializedField.Builder getMetadataBuilder() {
+return super.getMetadataBuilder()
+ .addChild(bits.getMetadata())
+ .addChild(values.getMetadata());
}
+ @Override
public void allocateNew() {
if(!allocateNewSafe()){
throw new OutOfMemoryRuntimeException("Failure while allocating
buffer.");
@@ -143,6 +143,7 @@ public boolean allocateNewSafe() {
return success;
}
+ <#if type.major == "VarLen">
@Override
public void allocateNew(int totalBytes, int valueCount) {
try {
@@ -157,26 +158,6 @@ public void allocateNew(int totalBytes, int valueCount) {
accessor.reset();
}
- @Override
- public int load(int dataBytes, int valueCount, DrillBuf buf){
-clear();
-int loaded = bits.load(valueCount, buf);
-
-// remove bits part of buffer.
-buf = buf.slice(loaded, buf.capacity() - loaded);
-dataBytes -= loaded;
-loaded += values.load(dataBytes, valueCount, buf);
-this.mutator.lastSet = valueCount;
-return loaded;
- }
-
- @Override
- public void load(SerializedField metadata, DrillBuf buffer) {
-assert this.field.matches(metadata) : String.format("The field %s doesn't
match the provided metadata %s.", this.field, metadata);
-int loaded = lo
[jira] [Commented] (DRILL-3313) Eliminate redundant #load methods and unit-test loading & exporting of vectors
[ https://issues.apache.org/jira/browse/DRILL-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497298#comment-16497298 ] ASF GitHub Bot commented on DRILL-3313: --- ilooner commented on issue #81: DRILL-3313: Eliminate redundant #load methods and unit-test loading & exporting of vectors URL: https://github.com/apache/drill/pull/81#issuecomment-393709996 This commit was merged as part of this PR https://github.com/apache/drill/pull/134. So closing this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Eliminate redundant #load methods and unit-test loading & exporting of vectors > -- > > Key: DRILL-3313 > URL: https://issues.apache.org/jira/browse/DRILL-3313 > Project: Apache Drill > Issue Type: Sub-task > Components: Execution - Data Types >Affects Versions: 1.0.0 >Reporter: Hanifi Gunes >Assignee: Hanifi Gunes >Priority: Major > Fix For: 1.3.0 > > > Vectors have multiple #load methods that are used to populate data from raw > buffers. It is relatively tough to reason, maintain and unit-test loading and > exporting of data since there is many redundant code around load methods. > This issue proposes to have single #load method conforming to VV#load(def, > buffer) signature eliminating all other #load overrides. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5030) Drill SSL Docs have Bad Link to Oracle Website
[ https://issues.apache.org/jira/browse/DRILL-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-5030: - Labels: documentation (was: doc-impacting) > Drill SSL Docs have Bad Link to Oracle Website > -- > > Key: DRILL-5030 > URL: https://issues.apache.org/jira/browse/DRILL-5030 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.8.0 >Reporter: John Omernik >Assignee: Bridget Bevens >Priority: Major > Labels: documentation > Fix For: 1.14.0 > > > When going to setup custom SSL certs on Drill, I found that the link to the > oracle website was broken on this page: > https://drill.apache.org/docs/configuring-web-console-and-rest-api-security/ > at: > As cluster administrator, you can set the following SSL configuration > parameters in the conf/drill-override.conf file, as described in the Java > product documentation: > Obviously fixing the link is one option, another would be to provide > instructions for SSL certs directly in the drill docs so we are not reliant > on Oracle's website. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5030) Drill SSL Docs have Bad Link to Oracle Website
[ https://issues.apache.org/jira/browse/DRILL-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-5030: - Labels: doc-impacting (was: ) > Drill SSL Docs have Bad Link to Oracle Website > -- > > Key: DRILL-5030 > URL: https://issues.apache.org/jira/browse/DRILL-5030 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.8.0 >Reporter: John Omernik >Assignee: Bridget Bevens >Priority: Major > Labels: documentation > Fix For: 1.14.0 > > > When going to setup custom SSL certs on Drill, I found that the link to the > oracle website was broken on this page: > https://drill.apache.org/docs/configuring-web-console-and-rest-api-security/ > at: > As cluster administrator, you can set the following SSL configuration > parameters in the conf/drill-override.conf file, as described in the Java > product documentation: > Obviously fixing the link is one option, another would be to provide > instructions for SSL certs directly in the drill docs so we are not reliant > on Oracle's website. > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6393) Radians should take an argument (x)
[
https://issues.apache.org/jira/browse/DRILL-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6393:
-
Labels: documentaion (was: )
> Radians should take an argument (x)
> ---
>
> Key: DRILL-6393
> URL: https://issues.apache.org/jira/browse/DRILL-6393
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.13.0
>Reporter: Robert Hou
>Assignee: Bridget Bevens
>Priority: Major
> Labels: documentaion
> Fix For: 1.14.0
>
>
> The radians function is missing an argument on this webpage:
>https://drill.apache.org/docs/math-and-trig/
> The table has this information:
> {noformat}
> RADIANS FLOAT8 Converts x degress to radians.
> {noformat}
> It should be:
> {noformat}
> RADIANS(x)FLOAT8 Converts x degrees to radians.
> {noformat}
> Also, degress is mis-spelled. It should be degrees.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6431) Unnest operator requires table and a single column alias to be specified.
[ https://issues.apache.org/jira/browse/DRILL-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6431: - Reviewer: Aman Sinha > Unnest operator requires table and a single column alias to be specified. > - > > Key: DRILL-6431 > URL: https://issues.apache.org/jira/browse/DRILL-6431 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, SQL Parser >Reporter: Hanumath Rao Maduri >Assignee: Hanumath Rao Maduri >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Currently, unnest operator is not required to specify alias neither for table > name nor column name. This has some implications on what name the unnest > operator output column should use. One can use a common name like "unnest" as > the output name. It means, customers need to be educated on what to expect > from unnest operator. This might confuse some customers and also prone to > introduce errors in the query. > The design decision for DRILL is that unnest always produces either a scalar > column or a map (depending upon the input schema for it), but it is always a > single column. > Given this scenario, it is better to enforce the requirement that unnest > operator requires a table alias and a column alias(single column). This can > help to disambiguate the column and further can easily be referenced in the > query. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-5201) Query bug: null values in result of a conditioned query
[ https://issues.apache.org/jira/browse/DRILL-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker resolved DRILL-5201. -- Resolution: Cannot Reproduce Marked as not reproducible based on above comments > Query bug: null values in result of a conditioned query > - > > Key: DRILL-5201 > URL: https://issues.apache.org/jira/browse/DRILL-5201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.9.0 >Reporter: mehran >Assignee: Jinfeng Ni >Priority: Critical > Fix For: 1.14.0 > > Attachments: 20.tar.gz > > > I have a table with 300 columns. > When I select a query , some null columns are also returned. > This is the query that I used. > "select a,b from dfs.root.`/2016/1/17` where a=1" > For example in this query some columns are returned that "a" value is null. > I think null values are problems. Because when I add a condition to it like > "select a,b from dfs.root.`/2016/1/17` where a=1 and a IS NOT NULL" > there is not any change and still NULL values are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4973) Sqlline history
[ https://issues.apache.org/jira/browse/DRILL-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497310#comment-16497310 ] Andries Engelbrecht commented on DRILL-4973: [~kkhatua] in old versions of sqlline the history feature stopped working after 500 commands. Newer versions (not sure when this was resolved) no longer seem to have this issue, tested Drill 1.13 which keeps a rolling history of the last 500 command which is sufficient for most use cases. I think this can be marked as fixed now. > Sqlline history > --- > > Key: DRILL-4973 > URL: https://issues.apache.org/jira/browse/DRILL-4973 > Project: Apache Drill > Issue Type: Improvement > Components: Client - CLI >Reporter: Andries Engelbrecht >Priority: Minor > > Currently the history on sqlline stops working after 500 queries have been > logged in the users .sqlline/history file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3300) Parameter never used in HBaseGroupScan
[
https://issues.apache.org/jira/browse/DRILL-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497316#comment-16497316
]
ASF GitHub Bot commented on DRILL-3300:
---
ilooner closed pull request #183: [DRILL-3300] Bug fix: Unused method parameter
URL: https://github.com/apache/drill/pull/183
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Parameter never used in HBaseGroupScan
> --
>
> Key: DRILL-3300
> URL: https://issues.apache.org/jira/browse/DRILL-3300
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: 1.0.0
>Reporter: Andrew
>Assignee: Andrew
>Priority: Trivial
> Fix For: Future
>
>
> The method HBaseGroupScan.setFilterPushedDown(boolean) does not read it's
> parameter value:
> public void setFilterPushedDown(boolean b) {
> this.filterPushedDown = true;
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3300) Parameter never used in HBaseGroupScan
[
https://issues.apache.org/jira/browse/DRILL-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497315#comment-16497315
]
ASF GitHub Bot commented on DRILL-3300:
---
ilooner commented on issue #183: [DRILL-3300] Bug fix: Unused method parameter
URL: https://github.com/apache/drill/pull/183#issuecomment-393712404
This change has already been made in the latest master.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Parameter never used in HBaseGroupScan
> --
>
> Key: DRILL-3300
> URL: https://issues.apache.org/jira/browse/DRILL-3300
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: 1.0.0
>Reporter: Andrew
>Assignee: Andrew
>Priority: Trivial
> Fix For: Future
>
>
> The method HBaseGroupScan.setFilterPushedDown(boolean) does not read it's
> parameter value:
> public void setFilterPushedDown(boolean b) {
> this.filterPushedDown = true;
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3723) RemoteServiceSet.getServiceSetWithFullCache() ignores arguments
[ https://issues.apache.org/jira/browse/DRILL-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497322#comment-16497322 ] ASF GitHub Bot commented on DRILL-3723: --- ilooner closed pull request #184: [DRILL-3723] Remove unused RemoteServiceSet method URL: https://github.com/apache/drill/pull/184 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > RemoteServiceSet.getServiceSetWithFullCache() ignores arguments > --- > > Key: DRILL-3723 > URL: https://issues.apache.org/jira/browse/DRILL-3723 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Affects Versions: 1.1.0 >Reporter: Andrew >Assignee: Andrew >Priority: Minor > Fix For: Future > > > RemoteServiceSet.getServiceSetWithFullCache() ignores both of its arguments > and is therefore functionally equivalent to getLocalServiceSet(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3723) RemoteServiceSet.getServiceSetWithFullCache() ignores arguments
[ https://issues.apache.org/jira/browse/DRILL-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497321#comment-16497321 ] ASF GitHub Bot commented on DRILL-3723: --- ilooner commented on issue #184: [DRILL-3723] Remove unused RemoteServiceSet method URL: https://github.com/apache/drill/pull/184#issuecomment-393712834 These changes are already in master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > RemoteServiceSet.getServiceSetWithFullCache() ignores arguments > --- > > Key: DRILL-3723 > URL: https://issues.apache.org/jira/browse/DRILL-3723 > Project: Apache Drill > Issue Type: Bug > Components: Execution - RPC >Affects Versions: 1.1.0 >Reporter: Andrew >Assignee: Andrew >Priority: Minor > Fix For: Future > > > RemoteServiceSet.getServiceSetWithFullCache() ignores both of its arguments > and is therefore functionally equivalent to getLocalServiceSet(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-4973) Sqlline history
[ https://issues.apache.org/jira/browse/DRILL-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua resolved DRILL-4973. - Resolution: Fixed > Sqlline history > --- > > Key: DRILL-4973 > URL: https://issues.apache.org/jira/browse/DRILL-4973 > Project: Apache Drill > Issue Type: Improvement > Components: Client - CLI >Reporter: Andries Engelbrecht >Priority: Minor > > Currently the history on sqlline stops working after 500 queries have been > logged in the users .sqlline/history file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4070) Files written with versions of Drill before v1.3 record metadata that is indistinguishable from bad metadata from other Parquet creators
[ https://issues.apache.org/jira/browse/DRILL-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497328#comment-16497328 ] ASF GitHub Bot commented on DRILL-4070: --- ilooner commented on issue #278: DRILL-4070: Add note about parquet file migration in 1.3 URL: https://github.com/apache/drill/pull/278#issuecomment-393714327 @bbevens Could you take a look at this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Files written with versions of Drill before v1.3 record metadata that is > indistinguishable from bad metadata from other Parquet creators > > > Key: DRILL-4070 > URL: https://issues.apache.org/jira/browse/DRILL-4070 > Project: Apache Drill > Issue Type: Bug > Components: Metadata >Affects Versions: 1.3.0 >Reporter: Rahul Challapalli >Assignee: Parth Chandra >Priority: Blocker > Fix For: 1.3.0 > > Attachments: cache.txt, fewtypes_varcharpartition.tar.tgz > > > Drill uses the parquet-mr library to write Parquet files. The metadata > signature that Drill produced in 1.2 and earlier versions of Drill is > indistinguishable from older footers written by other tools (such as Pig and > Hive). There was a known bug when those tools wrote metadata that caused the > statistics to be incorrect. To correct this, the parquet-mr library adopted a > behavior of ignoring statistics from the old form of the Parquet footer. > With 1.3, Drill upgraded to the latest version of parquet-mr and has now > started ignoring these statistics as well. This ensures correct result but > produces performance regressions (compared to Drill v1 and v2) when querying > against partitioned Parquet files generated in Drill 1.1 and 1.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6027) Implement spill to disk for the Hash Join
[
https://issues.apache.org/jira/browse/DRILL-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497330#comment-16497330
]
ASF GitHub Bot commented on DRILL-6027:
---
Ben-Zvi commented on a change in pull request #1248: DRILL-6027: Implement
Spilling for the Hash-Join
URL: https://github.com/apache/drill/pull/1248#discussion_r192265913
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableConfig.java
##
@@ -29,27 +29,58 @@
public class HashTableConfig {
private final int initialCapacity;
+ private final boolean initialSizeIsFinal;
private final float loadFactor;
private final List keyExprsBuild;
private final List keyExprsProbe;
private final List comparators;
@JsonCreator
- public HashTableConfig(@JsonProperty("initialCapacity") int initialCapacity,
@JsonProperty("loadFactor") float loadFactor,
+ public HashTableConfig(@JsonProperty("initialCapacity") int initialCapacity,
+ @JsonProperty("loadFactor") float loadFactor,
@JsonProperty("keyExprsBuild") List
keyExprsBuild,
@JsonProperty("keyExprsProbe") List
keyExprsProbe,
@JsonProperty("comparators") List
comparators) {
this.initialCapacity = initialCapacity;
+this.initialSizeIsFinal = false;
this.loadFactor = loadFactor;
this.keyExprsBuild = keyExprsBuild;
this.keyExprsProbe = keyExprsProbe;
this.comparators = comparators;
}
+ @JsonCreator
+ public HashTableConfig(@JsonProperty("initialCapacity") int initialCapacity,
+ @JsonProperty("initialCapacity") boolean
initialSizeIsFinal,
Review comment:
Yes, that one looks like an error. And yes, we should reuse the "more
comprehensive" constructor. Does this work call for a special Jira, or could it
be lumped together with other expected changes ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement spill to disk for the Hash Join
> -
>
> Key: DRILL-6027
> URL: https://issues.apache.org/jira/browse/DRILL-6027
> Project: Apache Drill
> Issue Type: New Feature
> Components: Execution - Relational Operators
>Affects Versions: 1.11.0
>Reporter: Boaz Ben-Zvi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> Implement the spill memory to disk (as needed) feature for the Hash Join
> operator (similar to the prior work on the Hash Aggregate).
> A design draft document has been published:
> [https://docs.google.com/document/d/1-c_oGQY4E5d58qJYv_zc7ka834hSaB3wDQwqKcMoSAI/edit?usp=sharing]
> Functional spec is available:
> [https://docs.google.com/document/d/1bPAddVCRxKHxi2RjqUvISIWXNqLdbRzZT9CWmanh4h0/edit]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4196) some TPCDS queries return wrong result when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497339#comment-16497339
]
ASF GitHub Bot commented on DRILL-4196:
---
ilooner closed pull request #338: DRILL-4196 Fix to stop returning no more data
when output batch is fu…
URL: https://github.com/apache/drill/pull/338
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinStatus.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinStatus.java
index 9e31763398..8e48515150 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinStatus.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinStatus.java
@@ -42,6 +42,7 @@
private boolean allowMarking;
public boolean ok = true;
+ public boolean hasMoreData = false;
public JoinStatus(RecordIterator left, RecordIterator right, MergeJoinBatch
output) {
this.left = left;
@@ -120,6 +121,14 @@ public boolean shouldMark() {
return allowMarking;
}
+ public boolean isHasMoreData() {
+return hasMoreData;
+ }
+
+ public void setHasMoreData(boolean hasMoreData) {
+this.hasMoreData = hasMoreData;
+ }
+
/**
* Return state of join based on status of left and right iterator.
* @return
@@ -132,6 +141,9 @@ public JoinOutcome getOutcome() {
if (!ok) {
return JoinOutcome.FAILURE;
}
+if (hasMoreData) {
+ return JoinOutcome.BATCH_RETURNED;
+}
if (bothMatches(IterOutcome.NONE) ||
(joinType == JoinRelType.INNER && eitherMatches(IterOutcome.NONE)) ||
(joinType == JoinRelType.LEFT && getLeftStatus() == IterOutcome.NONE) ||
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java
index 43cbf71d70..37cf0ed559 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java
@@ -42,12 +42,13 @@ public void setupJoin(FragmentContext context, JoinStatus
status, VectorContaine
*/
public final boolean doJoin(final JoinStatus status) {
final boolean isLeftJoin =
(((MergeJoinPOP)status.outputBatch.getPopConfig()).getJoinType() ==
JoinRelType.LEFT);
-
+status.setHasMoreData(false);
while (!status.isOutgoingBatchFull()) {
if (status.right.finished()) {
if (isLeftJoin) {
while (!status.left.finished()) {
if (status.isOutgoingBatchFull()) {
+ status.setHasMoreData(true);
return true;
}
doCopyLeft(status.left.getCurrentPosition(),
status.getOutPosition());
@@ -86,6 +87,7 @@ public final boolean doJoin(final JoinStatus status) {
if (status.isOutgoingBatchFull()) {
// Leave iterators at their current positions and markers.
// Don't mark on all subsequent doJoin iterations.
+status.setHasMoreData(true);
status.disableMarking();
return true;
}
@@ -97,6 +99,7 @@ public final boolean doJoin(final JoinStatus status) {
doCopyRight(status.right.getCurrentPosition(),
status.getOutPosition());
status.incOutputPos();
if (status.isOutgoingBatchFull()) {
+status.setHasMoreData(true);
status.disableMarking();
return true;
}
diff --git
a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoinAdvanced.java
b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoinAdvanced.java
index ac6ac89369..3e0deb206a 100644
---
a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoinAdvanced.java
+++
b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/join/TestMergeJoinAdvanced.java
@@ -213,4 +213,38 @@ public void testDrill4165() throws Exception {
.baselineValues(202452l)
.go();
}
+
+ @Test
+ public void testDrill4196() throws Exception {
+final String leftSide = BaseTestQuery.getTempDir("merge-join-left.json");
+final String rightSide = BaseTestQuery.getTempDir("merge-join-right.json");
+final BufferedWriter leftWriter = new BufferedWriter(new FileWriter(new
File(leftSide)));
+final BufferedWriter rightWriter = new BufferedWriter(new FileWriter(new
File(rightSide)));
+
+// output batch is 32k, create 60k left batch
+leftWriter
[jira] [Commented] (DRILL-4196) some TPCDS queries return wrong result when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497338#comment-16497338
]
ASF GitHub Bot commented on DRILL-4196:
---
ilooner commented on issue #338: DRILL-4196 Fix to stop returning no more data
when output batch is fu…
URL: https://github.com/apache/drill/pull/338#issuecomment-393715672
This change is in master.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> some TPCDS queries return wrong result when hash join is disabled
> -
>
> Key: DRILL-4196
> URL: https://issues.apache.org/jira/browse/DRILL-4196
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: 1.5.0-amit-branch_tpcds_sf1.txt, query40.tar, query52.tar
>
>
> With hash join disabled query52.sql and query40.sql returned incorrect result
> with 1.4.0 :
> {noformat}
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.13 seconds)
> {noformat}
> Setup and options are the same as in DRILL-4190
> See attached queries (.sql), expected result (.e_tsv) and actual output (.out)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
