[jira] [Updated] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Description:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error:
{code:java}
Caused by: KrbException: Identifier doesn't match expected value{code}
due to the mismatch between the explicit sqlline kerberos principal and
zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
was:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error due to the mismatch between the explicit sqlline kerberos
principal and zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
> Sqlline - infer Kerberos principal dynamically to be able to use individual
> keytabs across Drill nodes and still use ZooKeeper connection string for High
> Availability
> --
>
> Key: DRILL-6457
> URL: https://issues.apache.org/jira/browse/DRILL-6457
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - CLI, Client - JDBC, Security, Tools, Build
> & Test
>Affects Versions: 1.13.0
> Environment: MapR 6
>Reporter: Hari Sekhon
>Priority: Major
>
> Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
> connection string, eg:
> {code:java}
> zk=;auth=kerberos;principal=mapr/@REALM{code}
> When Drill nodes are configured with individual keytabs containing the node's
> fqdn and configured like so:
> {code:java}
> security: { auth.principal: mapr/_HOST@REALM }{code}
> then the ZooKeeper connection string from sqlline does not work and results
> in GSS Kerberos error:
> {code:java}
> Caused by: KrbException: Identifier doesn't match expected value{code}
> due to the mismatch between the explicit sqlline kerberos principal and
> zookeeper's random drillbit's principal.
> For the connection to work in this case requires something more like:
> {code:java}
> drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname
> -f)@REALM{code}
> but this lacks the high availability of using the ZooKeeper connection string
> to connect to any available node
> Hence it would be good if there was a way for sqlline arguments to be able to
> either infer the correct kerberos principal to match the host that zookeeper
> tells it to connect to or else accept a more generic parameter such as:
> {code:java}
> zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
> I've tested the above but it doesn't work showing that sqlline is not using a
> dynamic kerberos principal to match the host it is connecting to.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Av
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Comment: was deleted
(was: For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
org.apache.drill.exec.rpc.AbstractHandshakeHandler.decode(AbstractHandshake
[jira] [Commented] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496490#comment-16496490
]
Hari Sekhon commented on DRILL-6457:
For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
org.apache.drill.exec.rpc.AbstractHandshakeHandle
[jira] [Comment Edited] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availabil
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496484#comment-16496484
]
Hari Sekhon edited comment on DRILL-6457 at 5/31/18 12:45 PM:
--
For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
[jira] [Commented] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496484#comment-16496484
]
Hari Sekhon commented on DRILL-6457:
For reference:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,host3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:143)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.set(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.BasicClient$ClientHandshakeHandler.consumeHandshake(BasicClient.java:318)
at
org.apache.drill.exec.rpc.AbstractHandshakeHandle
[jira] [Updated] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Description:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error due to the mismatch between the explicit sqlline kerberos
principal and zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
was:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,hsot3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed
[jira] [Updated] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
[
https://issues.apache.org/jira/browse/DRILL-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6457:
---
Description:
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos error:
{code:java}
sqlline -u
"jdbc:drill:zk=host1:5181,host2:5181,hsot3:5181;auth=kerberos;principal=mapr/_HOST@$REALM"
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))] (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill:
org.apache.drill.exec.rpc.RpcException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7))]
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
database (7))]
at
org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:296)
at
org.apache.drill.exec.rpc.user.UserClient$1.mapException(UserClient.java:286)
at
com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
at
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:202)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:458)
at
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:402)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:170)
... 18 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7))]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:241)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener$1.run(AuthenticationOutcomeListener.java:238)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.evaluateChallenge(AuthenticationOutcomeListener.java:238)
at
org.apache.drill.exec.rpc.security.AuthenticationOutcomeListener.initiate(AuthenticationOutcomeListener.java:91)
at
org.apache.drill.exec.rpc.BasicClient.startSaslHandshake(BasicClient.java:263)
at
org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:463)
at
org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiList
[jira] [Created] (DRILL-6457) Sqlline - infer Kerberos principal dynamically to be able to use individual keytabs across Drill nodes and still use ZooKeeper connection string for High Availability
Hari Sekhon created DRILL-6457:
--
Summary: Sqlline - infer Kerberos principal dynamically to be able
to use individual keytabs across Drill nodes and still use ZooKeeper connection
string for High Availability
Key: DRILL-6457
URL: https://issues.apache.org/jira/browse/DRILL-6457
Project: Apache Drill
Issue Type: Improvement
Components: Client - CLI, Client - JDBC, Security, Tools, Build &
Test
Affects Versions: 1.13.0
Environment: MapR 6
Reporter: Hari Sekhon
Sqlline requires explicit kerberos 'principal=' parameter in its JDBC
connection string, eg:
{code:java}
zk=;auth=kerberos;principal=mapr/@REALM{code}
When Drill nodes are configured with individual keytabs containing the node's
fqdn and configured like so:
{code:java}
security: { auth.principal: mapr/_HOST@REALM }{code}
then the ZooKeeper connection string from sqlline does not work and results in
GSS Kerberos errors due to the mismatch between the explicit sqlline kerberos
principal and zookeeper's random drillbit's principal.
For the connection to work in this case requires something more like:
{code:java}
drillbits=$(hostname -f);auth=kerberos;principal=mapr/$(hostname -f)@REALM{code}
but this lacks the high availability of using the ZooKeeper connection string
to connect to any available node
Hence it would be good if there was a way for sqlline arguments to be able to
either infer the correct kerberos principal to match the host that zookeeper
tells it to connect to or else accept a more generic parameter such as:
{code:java}
zk=;auth=kerberos;principal=mapr/_HOST@REALM{code}
I've tested the above but it doesn't work showing that sqlline is not using a
dynamic kerberos principal to match the host it is connecting to.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6332) DrillbitStartupException: Failure while initializing values in Drillbit
[
https://issues.apache.org/jira/browse/DRILL-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489327#comment-16489327
]
Hari Sekhon edited comment on DRILL-6332 at 5/24/18 4:28 PM:
-
Yes we have found you need to leave Plain auth enabled for username + password
web form login to the UI. Otherwise you would have to use Kerberos SpNego which
only recently became available in Drill.
was (Author: harisekhon):
Yes we have found you need to leave Plain auth enabled for username + password
form login. Otherwise you would have to use Kerberos SpNego which only recently
became available in Drill.
> DrillbitStartupException: Failure while initializing values in Drillbit
> ---
>
> Key: DRILL-6332
> URL: https://issues.apache.org/jira/browse/DRILL-6332
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
>Affects Versions: 1.10.0
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement request to make this error more specific so we can tell what is
> causing it:
> {code:java}
> ==> /opt/mapr/drill/drill-1.10.0/logs/drillbit.out <==
> Exception in thread "main"
> org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> initializing values in Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> Caused by: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
> at
> org.apache.drill.common.KerberosUtil.splitPrincipalIntoParts(KerberosUtil.java:59)
> at
> org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:130)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:77)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6332) DrillbitStartupException: Failure while initializing values in Drillbit
[
https://issues.apache.org/jira/browse/DRILL-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489327#comment-16489327
]
Hari Sekhon edited comment on DRILL-6332 at 5/24/18 4:28 PM:
-
Yes we have found you need to leave Plain auth enabled for username + password
form login. Otherwise you would have to use Kerberos SpNego which only recently
became available in Drill.
was (Author: harisekhon):
Yes we have found you need to leave Plain auth enabled for username + password
form login. Otherwise you would have to use Kerberos SpNego.
> DrillbitStartupException: Failure while initializing values in Drillbit
> ---
>
> Key: DRILL-6332
> URL: https://issues.apache.org/jira/browse/DRILL-6332
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
>Affects Versions: 1.10.0
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement request to make this error more specific so we can tell what is
> causing it:
> {code:java}
> ==> /opt/mapr/drill/drill-1.10.0/logs/drillbit.out <==
> Exception in thread "main"
> org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> initializing values in Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> Caused by: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
> at
> org.apache.drill.common.KerberosUtil.splitPrincipalIntoParts(KerberosUtil.java:59)
> at
> org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:130)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:77)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6332) DrillbitStartupException: Failure while initializing values in Drillbit
[
https://issues.apache.org/jira/browse/DRILL-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489327#comment-16489327
]
Hari Sekhon commented on DRILL-6332:
Yes we have found you need to leave Plain auth enabled for username + password
form login. Otherwise you would have to use Kerberos SpNego.
> DrillbitStartupException: Failure while initializing values in Drillbit
> ---
>
> Key: DRILL-6332
> URL: https://issues.apache.org/jira/browse/DRILL-6332
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
>Affects Versions: 1.10.0
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement request to make this error more specific so we can tell what is
> causing it:
> {code:java}
> ==> /opt/mapr/drill/drill-1.10.0/logs/drillbit.out <==
> Exception in thread "main"
> org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> initializing values in Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> Caused by: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
> at
> org.apache.drill.common.KerberosUtil.splitPrincipalIntoParts(KerberosUtil.java:59)
> at
> org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:130)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:77)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6332) DrillbitStartupException: Failure while initializing values in Drillbit
[
https://issues.apache.org/jira/browse/DRILL-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488822#comment-16488822
]
Hari Sekhon commented on DRILL-6332:
In my case this was caused by auth.principal not having a hostname component eg:
{code:java}
mapr/@REALM{code}
rather than
{code:java}
mapr@REALM{code}
> DrillbitStartupException: Failure while initializing values in Drillbit
> ---
>
> Key: DRILL-6332
> URL: https://issues.apache.org/jira/browse/DRILL-6332
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
>Affects Versions: 1.10.0
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement request to make this error more specific so we can tell what is
> causing it:
> {code:java}
> ==> /opt/mapr/drill/drill-1.10.0/logs/drillbit.out <==
> Exception in thread "main"
> org.apache.drill.exec.exception.DrillbitStartupException: Failure while
> initializing values in Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> Caused by: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
> at
> org.apache.drill.common.KerberosUtil.splitPrincipalIntoParts(KerberosUtil.java:59)
> at
> org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:130)
> at
> org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:77)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6408) Document Drill API /cluster.json states
[
https://issues.apache.org/jira/browse/DRILL-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472081#comment-16472081
]
Hari Sekhon commented on DRILL-6408:
Looks like the states can be found here:
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/DrillbitStateManager.java#L25
> Document Drill API /cluster.json states
> ---
>
> Key: DRILL-6408
> URL: https://issues.apache.org/jira/browse/DRILL-6408
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server, Client - HTTP, Documentation, Execution -
> Monitoring, Tools, Build & Test
>Affects Versions: 1.13.0
> Environment: MapR 6.0 / Apache Drill in Docker
>Reporter: Hari Sekhon
>Priority: Major
>
> Request to document the possible states returned by Drill API's newer
> /cluster.json endpoint:
> [https://drill.apache.org/docs/rest-api-introduction/#get-/cluster.json]
> Real world output shows an interesting 'state' field which is currently
> undocumented in the link above, but you can see here:
> {code:java}
> {
> "drillbits" : [ {
> "address" : "1939919665a9",
> "userPort" : "31010",
> "controlPort" : "31011",
> "dataPort" : "31012",
> "version" : "1.13.0",
> "current" : true,
> "versionMatch" : true,
> "state" : "ONLINE"
> } ],
> "currentVersion" : "1.13.0",
> "mismatchedVersions" : [ ],
> "userEncryptionEnabled" : false,
> "bitEncryptionEnabled" : false,
> "authEnabled" : false
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6408) Document Drill API /cluster.json states
[
https://issues.apache.org/jira/browse/DRILL-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6408:
---
Environment: MapR 6.0 / Apache Drill in Docker (was: Request to document
the possible states returned by Drill API's newer /cluster.json endpoint:
[https://drill.apache.org/docs/rest-api-introduction/#get-/cluster.json]
Real world output shows an interesting 'state' field which is currently
undocumented in the link above, but you can see here:
{code:java}
{
"drillbits" : [ {
"address" : "1939919665a9",
"userPort" : "31010",
"controlPort" : "31011",
"dataPort" : "31012",
"version" : "1.13.0",
"current" : true,
"versionMatch" : true,
"state" : "ONLINE"
} ],
"currentVersion" : "1.13.0",
"mismatchedVersions" : [ ],
"userEncryptionEnabled" : false,
"bitEncryptionEnabled" : false,
"authEnabled" : false
}
{code})
Description:
Request to document the possible states returned by Drill API's newer
/cluster.json endpoint:
[https://drill.apache.org/docs/rest-api-introduction/#get-/cluster.json]
Real world output shows an interesting 'state' field which is currently
undocumented in the link above, but you can see here:
{code:java}
{
"drillbits" : [ {
"address" : "1939919665a9",
"userPort" : "31010",
"controlPort" : "31011",
"dataPort" : "31012",
"version" : "1.13.0",
"current" : true,
"versionMatch" : true,
"state" : "ONLINE"
} ],
"currentVersion" : "1.13.0",
"mismatchedVersions" : [ ],
"userEncryptionEnabled" : false,
"bitEncryptionEnabled" : false,
"authEnabled" : false
}{code}
was:MapR 6.0 / Apache Drill in Docker
> Document Drill API /cluster.json states
> ---
>
> Key: DRILL-6408
> URL: https://issues.apache.org/jira/browse/DRILL-6408
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server, Client - HTTP, Documentation, Execution -
> Monitoring, Tools, Build & Test
>Affects Versions: 1.13.0
> Environment: MapR 6.0 / Apache Drill in Docker
>Reporter: Hari Sekhon
>Priority: Major
>
> Request to document the possible states returned by Drill API's newer
> /cluster.json endpoint:
> [https://drill.apache.org/docs/rest-api-introduction/#get-/cluster.json]
> Real world output shows an interesting 'state' field which is currently
> undocumented in the link above, but you can see here:
> {code:java}
> {
> "drillbits" : [ {
> "address" : "1939919665a9",
> "userPort" : "31010",
> "controlPort" : "31011",
> "dataPort" : "31012",
> "version" : "1.13.0",
> "current" : true,
> "versionMatch" : true,
> "state" : "ONLINE"
> } ],
> "currentVersion" : "1.13.0",
> "mismatchedVersions" : [ ],
> "userEncryptionEnabled" : false,
> "bitEncryptionEnabled" : false,
> "authEnabled" : false
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6408) Document Drill API /cluster.json states
Hari Sekhon created DRILL-6408:
--
Summary: Document Drill API /cluster.json states
Key: DRILL-6408
URL: https://issues.apache.org/jira/browse/DRILL-6408
Project: Apache Drill
Issue Type: Improvement
Components: Server, Client - HTTP, Documentation, Execution -
Monitoring, Tools, Build & Test
Affects Versions: 1.13.0
Environment: Request to document the possible states returned by Drill
API's newer /cluster.json endpoint:
[https://drill.apache.org/docs/rest-api-introduction/#get-/cluster.json]
Real world output shows an interesting 'state' field which is currently
undocumented in the link above, but you can see here:
{code:java}
{
"drillbits" : [ {
"address" : "1939919665a9",
"userPort" : "31010",
"controlPort" : "31011",
"dataPort" : "31012",
"version" : "1.13.0",
"current" : true,
"versionMatch" : true,
"state" : "ONLINE"
} ],
"currentVersion" : "1.13.0",
"mismatchedVersions" : [ ],
"userEncryptionEnabled" : false,
"bitEncryptionEnabled" : false,
"authEnabled" : false
}
{code}
Reporter: Hari Sekhon
MapR 6.0 / Apache Drill in Docker
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6407) Document API /status responses, currently only "Running" is shown in docs
[ https://issues.apache.org/jira/browse/DRILL-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472034#comment-16472034 ] Hari Sekhon edited comment on DRILL-6407 at 5/11/18 2:52 PM: - Looking at [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StatusResources.java#L75|http://example.com/] I'm not sure there are other statuses but I'm not familiar with the internals of the code base and don't want to spend too much time on this. Suffice it to say, there should be different statuses reflecting the true condition of Drill, see DRILL-6406 and DRILL-5990, and they should be clearly documented. Right now if "Running" is the only thing that the /status API will ever output then under what conditions would say /status be blank or return 404 or 500? was (Author: harisekhon): Looking at [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StatusResources.java#L75|http://example.com/] I'm not sure there are other statuses but I'm not familiar with the internals of the code base and don't want to spend too much time on this. Suffice it to say, there should be different statuses reflecting the true condition of Drill, see DRILL-6406 and DRILL-5990, and they should be clearly documented. Right now if Running is the only thing that the /status API will ever output then under what conditions would say /status be blank or return 404 or 500? > Document API /status responses, currently only "Running" is shown in docs > - > > Key: DRILL-6407 > URL: https://issues.apache.org/jira/browse/DRILL-6407 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Documentation >Affects Versions: 1.10.0, 1.11.0, 1.13.0 > Environment: MapR Enterprise 5.2 / 6.0 >Reporter: Hari Sekhon >Priority: Major > > Request to document the results of the API /status endpoint. Currently only > the "Running" result is shown in the docs here: > https://drill.apache.org/docs/rest-api-introduction/#get-/status -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6407) Document API /status responses, currently only "Running" is shown in docs
[ https://issues.apache.org/jira/browse/DRILL-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472034#comment-16472034 ] Hari Sekhon edited comment on DRILL-6407 at 5/11/18 2:52 PM: - Looking at [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StatusResources.java#L75|http://example.com/] I'm not sure there are other statuses but I'm not familiar with the internals of the code base and don't want to spend too much time on this. Suffice it to say, there should be different statuses reflecting the true condition of Drill, see DRILL-6406 and DRILL-5990, and they should be clearly documented. Right now if Running is the only thing that the /status API will ever output then under what conditions would say /status be blank or return 404 or 500? was (Author: harisekhon): Looking at [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StatusResources.java#L75|http://example.com] I'm not sure there are other statuses but I'm not familiar with the internals of the code base and don't want to spend too much time on this. Suffice to say there should be different statuses reflecting the true condition of Drill, see DRILL-6406 and DRILL-5990 > Document API /status responses, currently only "Running" is shown in docs > - > > Key: DRILL-6407 > URL: https://issues.apache.org/jira/browse/DRILL-6407 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Documentation >Affects Versions: 1.10.0, 1.11.0, 1.13.0 > Environment: MapR Enterprise 5.2 / 6.0 >Reporter: Hari Sekhon >Priority: Major > > Request to document the results of the API /status endpoint. Currently only > the "Running" result is shown in the docs here: > https://drill.apache.org/docs/rest-api-introduction/#get-/status -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6407) Document API /status responses, currently only "Running" is shown in docs
[ https://issues.apache.org/jira/browse/DRILL-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472034#comment-16472034 ] Hari Sekhon commented on DRILL-6407: Looking at [https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/StatusResources.java#L75|http://example.com] I'm not sure there are other statuses but I'm not familiar with the internals of the code base and don't want to spend too much time on this. Suffice to say there should be different statuses reflecting the true condition of Drill, see DRILL-6406 and DRILL-5990 > Document API /status responses, currently only "Running" is shown in docs > - > > Key: DRILL-6407 > URL: https://issues.apache.org/jira/browse/DRILL-6407 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Documentation >Affects Versions: 1.10.0, 1.11.0, 1.13.0 > Environment: MapR Enterprise 5.2 / 6.0 >Reporter: Hari Sekhon >Priority: Major > > Request to document the results of the API /status endpoint. Currently only > the "Running" result is shown in the docs here: > https://drill.apache.org/docs/rest-api-introduction/#get-/status -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6407) Document API /status responses, currently only "Running" is shown in docs
[ https://issues.apache.org/jira/browse/DRILL-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6407: --- Summary: Document API /status responses, currently only "Running" is shown in docs (was: Document API /status possible responses other then "Running") > Document API /status responses, currently only "Running" is shown in docs > - > > Key: DRILL-6407 > URL: https://issues.apache.org/jira/browse/DRILL-6407 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Documentation >Affects Versions: 1.10.0, 1.11.0, 1.13.0 > Environment: MapR Enterprise 5.2 / 6.0 >Reporter: Hari Sekhon >Priority: Major > > Request to document the results of the API /status endpoint. Currently only > the "Running" result is shown in the docs here: > https://drill.apache.org/docs/rest-api-introduction/#get-/status -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6407) Document API /status possible responses other then "Running"
Hari Sekhon created DRILL-6407: -- Summary: Document API /status possible responses other then "Running" Key: DRILL-6407 URL: https://issues.apache.org/jira/browse/DRILL-6407 Project: Apache Drill Issue Type: Improvement Components: Client - HTTP, Documentation Affects Versions: 1.13.0, 1.11.0, 1.10.0 Environment: MapR Enterprise 5.2 / 6.0 Reporter: Hari Sekhon Request to document the results of the API /status endpoint. Currently only the "Running" result is shown in the docs here: https://drill.apache.org/docs/rest-api-introduction/#get-/status -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6406) Apache Drill API /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.j
[ https://issues.apache.org/jira/browse/DRILL-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6406: --- Summary: Apache Drill API /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute any query (was: Apache Drill /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute any query) > Apache Drill API /status unreliable - add Proper Health Check to API - > /status returns "Running" even though Drill isn't usable eg. java classpath > errors for flexjson-*.jar when trying to execute any query > - > > Key: DRILL-6406 > URL: https://issues.apache.org/jira/browse/DRILL-6406 > Project: Apache Drill > Issue Type: Bug > Components: Server, Client - HTTP, Execution - Monitoring, Tools, > Build & Test, Web Server >Affects Versions: 1.10.0, 1.13.0 > Environment: MapR Enterprise 5.2 / 6.0 >Reporter: Hari Sekhon >Priority: Major > > I've come across situations on MapR 5.2 / 6.0 where Drill 1.10 / 1.13 is up > and running but there are flexjson jar classpath errors meaning that Drill is > not usable so queries fail even though the API gives the impression that > Drill is available. This is bad for load balancers. > This API needs to be much more robust and terms of checking if Drill is fully > working. I suggest internally doing a health check real query and if that > executes successfully and returns the expected result, only then updating the > status to OK / Running. > This internal health check query should be periodically re-executed (eg. > every 30 secs) and the status set to ERROR or something if if fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6406) Apache Drill /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar w
[ https://issues.apache.org/jira/browse/DRILL-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6406: --- Environment: MapR Enterprise 5.2 / 6.0 > Apache Drill /status unreliable - add Proper Health Check to API - /status > returns "Running" even though Drill isn't usable eg. java classpath errors > for flexjson-*.jar when trying to execute any query > - > > Key: DRILL-6406 > URL: https://issues.apache.org/jira/browse/DRILL-6406 > Project: Apache Drill > Issue Type: Bug > Components: Server, Client - HTTP, Execution - Monitoring, Tools, > Build & Test, Web Server >Affects Versions: 1.10.0, 1.13.0 > Environment: MapR Enterprise 5.2 / 6.0 >Reporter: Hari Sekhon >Priority: Major > > I've come across situations on MapR 5.2 / 6.0 where Drill 1.10 / 1.13 is up > and running but there are flexjson jar classpath errors meaning that Drill is > not usable so queries fail even though the API gives the impression that > Drill is available. This is bad for load balancers. > This API needs to be much more robust and terms of checking if Drill is fully > working. I suggest internally doing a health check real query and if that > executes successfully and returns the expected result, only then updating the > status to OK / Running. > This internal health check query should be periodically re-executed (eg. > every 30 secs) and the status set to ERROR or something if if fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6406) Apache Drill /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar w
[ https://issues.apache.org/jira/browse/DRILL-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6406: --- Summary: Apache Drill /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute any query (was: Apache Drill Add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute any query) > Apache Drill /status unreliable - add Proper Health Check to API - /status > returns "Running" even though Drill isn't usable eg. java classpath errors > for flexjson-*.jar when trying to execute any query > - > > Key: DRILL-6406 > URL: https://issues.apache.org/jira/browse/DRILL-6406 > Project: Apache Drill > Issue Type: Bug > Components: Server, Client - HTTP, Execution - Monitoring, Tools, > Build & Test, Web Server >Affects Versions: 1.10.0, 1.13.0 >Reporter: Hari Sekhon >Priority: Major > > I've come across situations on MapR 5.2 / 6.0 where Drill 1.10 / 1.13 is up > and running but there are flexjson jar classpath errors meaning that Drill is > not usable so queries fail even though the API gives the impression that > Drill is available. This is bad for load balancers. > This API needs to be much more robust and terms of checking if Drill is fully > working. I suggest internally doing a health check real query and if that > executes successfully and returns the expected result, only then updating the > status to OK / Running. > This internal health check query should be periodically re-executed (eg. > every 30 secs) and the status set to ERROR or something if if fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5990) Apache Drill /status API returns OK ('Running') even with JRE while queries will not work - make status API reflect the fact that Drill is broken on JRE or stop Drill sta
[
https://issues.apache.org/jira/browse/DRILL-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-5990:
---
Component/s: Query Planning & Optimization
> Apache Drill /status API returns OK ('Running') even with JRE while queries
> will not work - make status API reflect the fact that Drill is broken on JRE
> or stop Drill starting up with JRE
> ---
>
> Key: DRILL-5990
> URL: https://issues.apache.org/jira/browse/DRILL-5990
> Project: Apache Drill
> Issue Type: Bug
> Components: Server, Client - HTTP, Execution - Monitoring, Query
> Planning & Optimization
>Affects Versions: 1.10.0, 1.11.0
> Environment: Docker
>Reporter: Hari Sekhon
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
>
> If running Apache Drill on versions 1.10 / 1.11 on JRE it appears that
> queries no longer run without JDK, but the /status monitoring API endpoint
> does not reflect the fact the Apache Drill will not work and still returns
> 'Running'.
> While 'Running' is technically true the process is up, it's effectively
> broken and Apache Drill should either reflect this in /status that is is
> broken or refuse to start up on JRE altogether.
> See this ticket for more information:
> https://github.com/HariSekhon/Dockerfiles/pull/15
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6406) Apache Drill Add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute
[ https://issues.apache.org/jira/browse/DRILL-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6406: --- Issue Type: Bug (was: Improvement) > Apache Drill Add Proper Health Check to API - /status returns "Running" even > though Drill isn't usable eg. java classpath errors for flexjson-*.jar when > trying to execute any query > > > Key: DRILL-6406 > URL: https://issues.apache.org/jira/browse/DRILL-6406 > Project: Apache Drill > Issue Type: Bug > Components: Server, Client - HTTP, Execution - Monitoring, Tools, > Build & Test, Web Server >Affects Versions: 1.10.0, 1.13.0 >Reporter: Hari Sekhon >Priority: Major > > I've come across situations on MapR 5.2 / 6.0 where Drill 1.10 / 1.13 is up > and running but there are flexjson jar classpath errors meaning that Drill is > not usable so queries fail even though the API gives the impression that > Drill is available. This is bad for load balancers. > This API needs to be much more robust and terms of checking if Drill is fully > working. I suggest internally doing a health check real query and if that > executes successfully and returns the expected result, only then updating the > status to OK / Running. > This internal health check query should be periodically re-executed (eg. > every 30 secs) and the status set to ERROR or something if if fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6406) Apache Drill Add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute
Hari Sekhon created DRILL-6406: -- Summary: Apache Drill Add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute any query Key: DRILL-6406 URL: https://issues.apache.org/jira/browse/DRILL-6406 Project: Apache Drill Issue Type: Improvement Components: Server, Client - HTTP, Execution - Monitoring, Tools, Build & Test, Web Server Affects Versions: 1.13.0, 1.10.0 Reporter: Hari Sekhon I've come across situations on MapR 5.2 / 6.0 where Drill 1.10 / 1.13 is up and running but there are flexjson jar classpath errors meaning that Drill is not usable so queries fail even though the API gives the impression that Drill is available. This is bad for load balancers. This API needs to be much more robust and terms of checking if Drill is fully working. I suggest internally doing a health check real query and if that executes successfully and returns the expected result, only then updating the status to OK / Running. This internal health check query should be periodically re-executed (eg. every 30 secs) and the status set to ERROR or something if if fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[
https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6095:
---
Priority: Minor (was: Trivial)
> Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per
> standard practice
> ---
>
> Key: DRILL-6095
> URL: https://issues.apache.org/jira/browse/DRILL-6095
> Project: Apache Drill
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.12.0
> Environment: MapR 5.2
>
>Reporter: Hari Sekhon
>Assignee: Bridget Bevens
>Priority: Minor
>
> Drill's documentation on PStore configuration to HBase shows sample config
> with 4 ZooKeeper ip addresses in the {{hbase.zookeeper.quorum field}}, this
> should be fixed to 3 or 5 as per ZooKeeper best practices.
> [https://drill.apache.org/docs/persistent-configuration-storage/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6394) Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object '/tmp/file.dat' not found within 'dfs'
Hari Sekhon created DRILL-6394:
--
Summary: Error: VALIDATION ERROR: From line 1, column 15 to line
1, column 17: Object '/tmp/file.dat' not found within 'dfs'
Key: DRILL-6394
URL: https://issues.apache.org/jira/browse/DRILL-6394
Project: Apache Drill
Issue Type: Improvement
Components: Server, Execution - Codegen, Metadata, Query Planning
& Optimization, SQL Parser, Storage - Text & CSV
Affects Versions: 1.13.0
Environment: MapR 6.0
Reporter: Hari Sekhon
Improvement request for the following error to be made more specific to mention
that this is caused by the file extension (.dat) not being one of the expected
ones, even though it was a CSV file (renaming it to .csv worked):
{code:java}
0: jdbc:drill:drillbit=> select * from dfs.`/tmp/file.dat`;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 17: Object
'/tmp/file.dat' not found within 'dfs'
SQL Query null
[Error Id: e7c2863e-0feb-4b80-82b7-008056f0fcef on :31010] (state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6332) DrillbitStartupException: Failure while initializing values in Drillbit
Hari Sekhon created DRILL-6332:
--
Summary: DrillbitStartupException: Failure while initializing
values in Drillbit
Key: DRILL-6332
URL: https://issues.apache.org/jira/browse/DRILL-6332
Project: Apache Drill
Issue Type: Improvement
Components: Server
Affects Versions: 1.10.0
Reporter: Hari Sekhon
Improvement request to make this error more specific so we can tell what is
causing it:
{code:java}
==> /opt/mapr/drill/drill-1.10.0/logs/drillbit.out <==
Exception in thread "main"
org.apache.drill.exec.exception.DrillbitStartupException: Failure while
initializing values in Drillbit.
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
at
org.apache.drill.common.KerberosUtil.splitPrincipalIntoParts(KerberosUtil.java:59)
at
org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:130)
at
org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:77)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
... 2 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6309) Show currently logged in user inside SQL session eg. SELECT CURRENT_USER();
Hari Sekhon created DRILL-6309:
--
Summary: Show currently logged in user inside SQL session eg.
SELECT CURRENT_USER();
Key: DRILL-6309
URL: https://issues.apache.org/jira/browse/DRILL-6309
Project: Apache Drill
Issue Type: New Feature
Components: Execution - Monitoring, Functions - Drill, Metadata
Affects Versions: 1.10.0
Environment: MapR 5.2
Reporter: Hari Sekhon
Feature Request to add ability to show the currently logged in user in the
current SQL session, something analogous to MySQL's:
{code:java}
SELECT CURRENT_USER();{code}
https://dev.mysql.com/doc/refman/5.7/en/information-functions.html#function_current-user
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417646#comment-16417646 ] Hari Sekhon commented on DRILL-6245: If I use a fresh browser such as Firefox or go Chrome Incognito to avoid using the existing cookie, log in again on the http login page (and I double checked it's http and not https, firefox even warned about it), then I am able to get to the query page normally. What's weird if I don't think this node has ever been https so I don't understand how it could have gotten a secure cookie set. > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.14.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417646#comment-16417646 ] Hari Sekhon edited comment on DRILL-6245 at 3/28/18 4:07 PM: - If I use a fresh browser such as Firefox or go Chrome Incognito to avoid using the existing cookie, log in again on the http login page (and I double checked it's http and not https, firefox even warned about it), then I am able to get to the query page normally. What's weird is I don't think this node has ever been https so I don't understand how it could have gotten a secure cookie set that could now cause this issue. was (Author: harisekhon): If I use a fresh browser such as Firefox or go Chrome Incognito to avoid using the existing cookie, log in again on the http login page (and I double checked it's http and not https, firefox even warned about it), then I am able to get to the query page normally. What's weird is I don't think this node has ever been https so I don't understand how it could have gotten a secure cookie set. > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.14.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417646#comment-16417646 ] Hari Sekhon edited comment on DRILL-6245 at 3/28/18 4:07 PM: - If I use a fresh browser such as Firefox or go Chrome Incognito to avoid using the existing cookie, log in again on the http login page (and I double checked it's http and not https, firefox even warned about it), then I am able to get to the query page normally. What's weird is I don't think this node has ever been https so I don't understand how it could have gotten a secure cookie set. was (Author: harisekhon): If I use a fresh browser such as Firefox or go Chrome Incognito to avoid using the existing cookie, log in again on the http login page (and I double checked it's http and not https, firefox even warned about it), then I am able to get to the query page normally. What's weird if I don't think this node has ever been https so I don't understand how it could have gotten a secure cookie set. > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.14.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415305#comment-16415305 ] Hari Sekhon commented on DRILL-6245: [~vdonapati] - What I don't understand is the login page is non-SSL on this dev cluster (I just double checked) and so it's not a case of going from https to http, it's http the whole time, both on the login page and the query page. Is this a bug with Apache Drill trying to set a secure cookie when it should be setting a normal one? Perhaps Drill is assuming SSL and always sets a secure cookie regardless without checking, thereby breaking authenticated non-SSL? > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.14.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414109#comment-16414109 ] Hari Sekhon commented on DRILL-6245: I've got this same problem on MapR 5.2 with Drill 1.10 without SSL using Chrome 65 (which according to the docs prevents setting secure cookies). This works fine in IE 11. Some URL requests seemed to append jsessionid to the path, so trying the same explicitly on the /query path does allow that page to be returned: http://host.domain.com:8047/query;jsessionid= Of course submitting a query still kicks out to the Login page again but it does imply that the cookie isn't being sent back and forth so might be the above issue [~vdonapati] mentioned. > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Assignee: Venkata Jyothsna Donapati >Priority: Minor > Fix For: 1.14.0 > > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6135) New Feature: SHOW CREATE TABLE / VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353638#comment-16353638
]
Hari Sekhon edited comment on DRILL-6135 at 2/6/18 9:36 AM:
[~julianhyde]
I think SHOW CREATE is better as it stays in line with other SQL
systems. This should also be applied to tables etc. Examples include MySQL and
Presto / AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
was (Author: harisekhon):
[~julianhyde]
I think SHOW CREATE ... is better as it stays in line with other SQL systems.
This should also be applied to tables etc. Examples include MySQL and Presto /
AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
> New Feature: SHOW CREATE TABLE / VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6135) New Feature: SHOW CREATE TABLE / VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353638#comment-16353638
]
Hari Sekhon edited comment on DRILL-6135 at 2/6/18 9:36 AM:
[~julianhyde]
I think SHOW CREATE is better as it stays in line with other SQL
systems. This should also be applied to tables etc. Examples include MySQL and
Presto / AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
was (Author: harisekhon):
[~julianhyde]
I think SHOW CREATE is better as it stays in line with other SQL
systems. This should also be applied to tables etc. Examples include MySQL and
Presto / AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
> New Feature: SHOW CREATE TABLE / VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6135) New Feature: SHOW CREATE TABLE / VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353638#comment-16353638
]
Hari Sekhon edited comment on DRILL-6135 at 2/6/18 9:37 AM:
[~julianhyde]
I think SHOW CREATE is better as it stays in line with other SQL
systems. This should also be applied to both tables and views. Examples include
MySQL and Presto / AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
was (Author: harisekhon):
[~julianhyde]
I think SHOW CREATE is better as it stays in line with other SQL
systems. This should also be applied to tables etc. Examples include MySQL and
Presto / AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
> New Feature: SHOW CREATE TABLE / VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6135) New Feature: SHOW CREATE VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353638#comment-16353638
]
Hari Sekhon edited comment on DRILL-6135 at 2/6/18 9:35 AM:
[~julianhyde]
I think SHOW CREATE ... is better as it stays in line with other SQL systems.
This should also be applied to tables etc. Examples include MySQL and Presto /
AWS Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
was (Author: harisekhon):
[~julianhyde]
SHOW CREATE ... is better as it stays in line with other SQL systems. This
should also be applied to tables etc. Examples include MySQL and Presto / AWS
Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
> New Feature: SHOW CREATE VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6135) New Feature: SHOW CREATE TABLE / VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6135:
---
Summary: New Feature: SHOW CREATE TABLE / VIEW command (was: New Feature:
SHOW CREATE VIEW command)
> New Feature: SHOW CREATE TABLE / VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6135) New Feature: SHOW CREATE VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353638#comment-16353638
]
Hari Sekhon edited comment on DRILL-6135 at 2/6/18 9:35 AM:
[~julianhyde]
SHOW CREATE ... is better as it stays in line with other SQL systems. This
should also be applied to tables etc. Examples include MySQL and Presto / AWS
Athena:
[https://dev.mysql.com/doc/refman/5.7/en/show-create-table.html]
[https://docs.aws.amazon.com/athena/latest/ug/show-create-table.html]
was (Author: harisekhon):
[~julianhyde]
SHOW CREATE ... is better as it stays in line with other SQL systems. This
should also be applied to tables etc.
> New Feature: SHOW CREATE VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6135) New Feature: SHOW CREATE VIEW command
[
https://issues.apache.org/jira/browse/DRILL-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353638#comment-16353638
]
Hari Sekhon commented on DRILL-6135:
[~julianhyde]
SHOW CREATE ... is better as it stays in line with other SQL systems. This
should also be applied to tables etc.
> New Feature: SHOW CREATE VIEW command
> -
>
> Key: DRILL-6135
> URL: https://issues.apache.org/jira/browse/DRILL-6135
> Project: Apache Drill
> Issue Type: New Feature
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.10.0
> Environment: MapR 5.2 + Kerberos
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to implement
> {code:java}
> SHOW CREATE VIEW ;{code}
> A colleague and I just had to cat the view file which is non-pretty json and
> hard to read a large view creation statement that could have been presented
> in drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6135) New Feature: SHOW CREATE VIEW command
Hari Sekhon created DRILL-6135:
--
Summary: New Feature: SHOW CREATE VIEW command
Key: DRILL-6135
URL: https://issues.apache.org/jira/browse/DRILL-6135
Project: Apache Drill
Issue Type: New Feature
Components: Metadata, Storage - Information Schema
Affects Versions: 1.10.0
Environment: MapR 5.2 + Kerberos
Reporter: Hari Sekhon
Feature Request to implement
{code:java}
SHOW CREATE VIEW ;{code}
A colleague and I just had to cat the view file which is non-pretty json and
hard to read a large view creation statement that could have been presented in
drill shell and formatted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-1709) desc => describe command
[
https://issues.apache.org/jira/browse/DRILL-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352626#comment-16352626
]
Hari Sekhon commented on DRILL-1709:
A colleague just tried the commonly used SQL {code}desc{code} command which led
us back to this ticket (I forgot I raised this it was so many years ago).
Any update on implementing this?
> desc => describe command
>
>
> Key: DRILL-1709
> URL: https://issues.apache.org/jira/browse/DRILL-1709
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 0.6.0
> Environment: MapR 4.0.1
>Reporter: Hari Sekhon
>Priority: Minor
> Fix For: Future
>
>
> There is no desc command, can you please add that shortcut to describe.
> Regards,
> Hari Sekhon
> http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient as other projects seem to use a Security component type to be allocated to whoever is working on the security component(s) / layer. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient as other projects seem to use a Security component type to be allocated to whoever is working on the security component(s)/layer. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > Labels: security > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security component issues, not sure if labels are sufficient > as other projects seem to use a Security component type to be allocated to > whoever is working on the security component(s) / layer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient as other projects seem to use a Security component type to be allocated to whoever is working on the security component(s)/layer. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > Labels: security > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security component issues, not sure if labels are sufficient > as other projects seem to use a Security component type to be allocated to > whoever is working on the security component(s)/layer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient as other more mature projects like Hive seem to use a Security component type to be allocated to whoever is working on the security component(s) / layer. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient as other projects seem to use a Security component type to be allocated to whoever is working on the security component(s) / layer. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > Labels: security > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security component issues, not sure if labels are sufficient > as other more mature projects like Hive seem to use a Security component type > to be allocated to whoever is working on the security component(s) / layer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security issues. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > Labels: security > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security component issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues, not sure if labels are sufficient. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security component issues. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > Labels: security > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security component issues, not sure if labels are sufficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6108) Drill Ranger integration
[ https://issues.apache.org/jira/browse/DRILL-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6108: --- Labels: security (was: ) > Drill Ranger integration > > > Key: DRILL-6108 > URL: https://issues.apache.org/jira/browse/DRILL-6108 > Project: Apache Drill > Issue Type: New Feature > Components: Documentation, Metadata, Query Planning & > Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Major > Labels: security > > Feature Request for integration with Apache Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6109) Drill Sentry integration
[ https://issues.apache.org/jira/browse/DRILL-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6109: --- Labels: security (was: ) > Drill Sentry integration > > > Key: DRILL-6109 > URL: https://issues.apache.org/jira/browse/DRILL-6109 > Project: Apache Drill > Issue Type: New Feature > Components: Metadata, Query Planning & Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Minor > Labels: security > > Feature Request for integration with Apache Sentry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Labels: security (was: ) > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > Labels: security > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security issues. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no SQL Grants or Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security issues. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no Ranger or > Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able > to be tagged as security issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
[ https://issues.apache.org/jira/browse/DRILL-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6110: --- Description: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no SQL Grants or Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security issues. was: Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no SQL Grants or Ranger or Drill integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security issues. > Add Jira Security Component Type for tracking security related Jiras > > > Key: DRILL-6110 > URL: https://issues.apache.org/jira/browse/DRILL-6110 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Trivial > > Please add a Security component type. > It appears Security is a very undeveloped aspect of Drill with no SQL Grants > or Ranger or Sentry integrations and Jiras such as DRILL-6108 and DRILL-6109 > aren't able to be tagged as security issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6110) Add Jira Security Component Type for tracking security related Jiras
Hari Sekhon created DRILL-6110: -- Summary: Add Jira Security Component Type for tracking security related Jiras Key: DRILL-6110 URL: https://issues.apache.org/jira/browse/DRILL-6110 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.12.0 Reporter: Hari Sekhon Please add a Security component type. It appears Security is a very undeveloped aspect of Drill with no SQL Grants or Ranger or Drill integrations and Jiras such as DRILL-6108 and DRILL-6109 aren't able to be tagged as security issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6108) Drill Ranger integration
[ https://issues.apache.org/jira/browse/DRILL-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339512#comment-16339512 ] Hari Sekhon edited comment on DRILL-6108 at 1/25/18 5:08 PM: - Ranger integration would be my first choice over Sentry as I see Ranger in far more in deployments, especially newer deployments in the last few years. was (Author: harisekhon): Ranger integration would be my first choice as I see it far more in deployments, especially newer deployments in the last few years. > Drill Ranger integration > > > Key: DRILL-6108 > URL: https://issues.apache.org/jira/browse/DRILL-6108 > Project: Apache Drill > Issue Type: New Feature > Components: Documentation, Metadata, Query Planning & > Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Major > > Feature Request for integration with Apache Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6109) Drill Sentry integration
[ https://issues.apache.org/jira/browse/DRILL-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339513#comment-16339513 ] Hari Sekhon commented on DRILL-6109: Ranger integration would be my first choice over Sentry as I see Ranger in far more in deployments, especially newer deployments in the last few years. > Drill Sentry integration > > > Key: DRILL-6109 > URL: https://issues.apache.org/jira/browse/DRILL-6109 > Project: Apache Drill > Issue Type: New Feature > Components: Metadata, Query Planning & Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Minor > > Feature Request for integration with Apache Sentry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6108) Drill Ranger integration
[ https://issues.apache.org/jira/browse/DRILL-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339512#comment-16339512 ] Hari Sekhon commented on DRILL-6108: Ranger integration would be my first choice as I see it far more in deployments, especially newer deployments in the last few years. > Drill Ranger integration > > > Key: DRILL-6108 > URL: https://issues.apache.org/jira/browse/DRILL-6108 > Project: Apache Drill > Issue Type: New Feature > Components: Documentation, Metadata, Query Planning & > Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Major > > Feature Request for integration with Apache Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6109) Drill Sentry integration
Hari Sekhon created DRILL-6109: -- Summary: Drill Sentry integration Key: DRILL-6109 URL: https://issues.apache.org/jira/browse/DRILL-6109 Project: Apache Drill Issue Type: New Feature Components: Metadata, Query Planning & Optimization, SQL Parser Affects Versions: 1.12.0 Reporter: Hari Sekhon Feature Request for integration with Apache Sentry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6108) Drill Ranger integration
[ https://issues.apache.org/jira/browse/DRILL-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6108: --- Priority: Critical (was: Major) > Drill Ranger integration > > > Key: DRILL-6108 > URL: https://issues.apache.org/jira/browse/DRILL-6108 > Project: Apache Drill > Issue Type: New Feature > Components: Documentation, Metadata, Query Planning & > Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Critical > > Feature Request for integration with Apache Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6108) Drill Ranger integration
[ https://issues.apache.org/jira/browse/DRILL-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6108: --- Priority: Major (was: Critical) > Drill Ranger integration > > > Key: DRILL-6108 > URL: https://issues.apache.org/jira/browse/DRILL-6108 > Project: Apache Drill > Issue Type: New Feature > Components: Documentation, Metadata, Query Planning & > Optimization, SQL Parser >Affects Versions: 1.12.0 >Reporter: Hari Sekhon >Priority: Major > > Feature Request for integration with Apache Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6108) Drill Ranger integration
Hari Sekhon created DRILL-6108: -- Summary: Drill Ranger integration Key: DRILL-6108 URL: https://issues.apache.org/jira/browse/DRILL-6108 Project: Apache Drill Issue Type: New Feature Components: Documentation, Metadata, Query Planning & Optimization, SQL Parser Affects Versions: 1.12.0 Reporter: Hari Sekhon Feature Request for integration with Apache Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[
https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated DRILL-6095:
---
Description:
Drill's documentation on PStore configuration to HBase shows sample config with
4 ZooKeeper ip addresses in the {{hbase.zookeeper.quorum field}}, this should
be fixed to 3 or 5 as per ZooKeeper best practices.
[https://drill.apache.org/docs/persistent-configuration-storage/]
was:
Drill's documentation on PStore configuration to HBase shows sample config with
4 ZooKeeper servers in a quorum, this should be fixed to 3 or 5 as per
ZooKeeper best practices.
[https://drill.apache.org/docs/persistent-configuration-storage/]
> Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per
> standard practice
> ---
>
> Key: DRILL-6095
> URL: https://issues.apache.org/jira/browse/DRILL-6095
> Project: Apache Drill
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.12.0
> Environment: MapR 5.2
>
>Reporter: Hari Sekhon
>Priority: Trivial
>
> Drill's documentation on PStore configuration to HBase shows sample config
> with 4 ZooKeeper ip addresses in the {{hbase.zookeeper.quorum field}}, this
> should be fixed to 3 or 5 as per ZooKeeper best practices.
> [https://drill.apache.org/docs/persistent-configuration-storage/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[ https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6095: --- Description: Drill's documentation on PStore configuration to HBase shows sample config with 4 ZooKeeper servers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. [https://drill.apache.org/docs/persistent-configuration-storage/] was: Drill's documentation on PStore configuration to HBase shows sample config with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. [https://drill.apache.org/docs/persistent-configuration-storage/] > Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per > standard practice > --- > > Key: DRILL-6095 > URL: https://issues.apache.org/jira/browse/DRILL-6095 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.12.0 > Environment: MapR 5.2 > >Reporter: Hari Sekhon >Priority: Trivial > > Drill's documentation on PStore configuration to HBase shows sample config > with 4 ZooKeeper servers in a quorum, this should be fixed to 3 or 5 as per > ZooKeeper best practices. > [https://drill.apache.org/docs/persistent-configuration-storage/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[ https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6095: --- Description: Drill's documentation on PStore configuration to HBase shows sample config with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. [https://drill.apache.org/docs/persistent-configuration-storage/] was: Drill's PStore doc for HBase shows config with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. [https://drill.apache.org/docs/persistent-configuration-storage/] > Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per > standard practice > --- > > Key: DRILL-6095 > URL: https://issues.apache.org/jira/browse/DRILL-6095 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.12.0 > Environment: MapR 5.2 > >Reporter: Hari Sekhon >Priority: Trivial > > Drill's documentation on PStore configuration to HBase shows sample config > with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per > ZooKeeper best practices. > [https://drill.apache.org/docs/persistent-configuration-storage/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[ https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6095: --- Description: Drill's PStore doc for HBase shows config with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. [https://drill.apache.org/docs/persistent-configuration-storage/] > Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per > standard practice > --- > > Key: DRILL-6095 > URL: https://issues.apache.org/jira/browse/DRILL-6095 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.12.0 > Environment: MapR 5.2 > >Reporter: Hari Sekhon >Priority: Trivial > > Drill's PStore doc for HBase shows config with 4 zookeepers in a quorum, this > should be fixed to 3 or 5 as per ZooKeeper best practices. > [https://drill.apache.org/docs/persistent-configuration-storage/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
Hari Sekhon created DRILL-6095: -- Summary: Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice Key: DRILL-6095 URL: https://issues.apache.org/jira/browse/DRILL-6095 Project: Apache Drill Issue Type: Improvement Components: Documentation Affects Versions: 1.12.0 Environment: Drill's PStore doc for HBase shows config with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. https://drill.apache.org/docs/persistent-configuration-storage/ Reporter: Hari Sekhon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6095) Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per standard practice
[ https://issues.apache.org/jira/browse/DRILL-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6095: --- Environment: MapR 5.2 was: Drill's PStore doc for HBase shows config with 4 zookeepers in a quorum, this should be fixed to 3 or 5 as per ZooKeeper best practices. https://drill.apache.org/docs/persistent-configuration-storage/ > Drill PStore HBase doc shows 4 zookeepers in quorum, fix to 3 or 5 as per > standard practice > --- > > Key: DRILL-6095 > URL: https://issues.apache.org/jira/browse/DRILL-6095 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.12.0 > Environment: MapR 5.2 > >Reporter: Hari Sekhon >Priority: Trivial > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Doc Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Description: Documentation Request to improve doc around Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes (eg. [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy|https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy]) then the query list will be spread across lots of different nodes with no global timeline of queries. was: Documentation Request to improve doc around Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes (eg. [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy|https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy)]) then the query list will be spread across lots of different nodes with no global timeline of queries. > Doc Request: Global Query List showing queries from all Drill foreman nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation, Metadata, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Documentation Request to improve doc around Global Query List to show all > queries executed across all Drill nodes in a cluster for better management > and auditing. > It wasn't obvious to be able to see all queries across all nodes in a Drill > cluster. The Web UI on any given Drill node only shows the queries > coordinated by that local node if acting as the foreman for the query, so if > using ZooKeeper or a Load Balancer to distribute queries via different Drill > nodes (eg. > [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy|https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy]) > then the query list will be spread across lots of different nodes with no > global timeline of queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Doc Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Description: Documentation Request to improve doc around Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes (eg. [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy|https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy)]) then the query list will be spread across lots of different nodes with no global timeline of queries. was: Documentation Request to improve doc around Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes (eg. [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy)] then the query list will be spread across lots of different nodes with no global timeline of queries. > Doc Request: Global Query List showing queries from all Drill foreman nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation, Metadata, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Documentation Request to improve doc around Global Query List to show all > queries executed across all Drill nodes in a cluster for better management > and auditing. > It wasn't obvious to be able to see all queries across all nodes in a Drill > cluster. The Web UI on any given Drill node only shows the queries > coordinated by that local node if acting as the foreman for the query, so if > using ZooKeeper or a Load Balancer to distribute queries via different Drill > nodes (eg. > [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy|https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy)]) > then the query list will be spread across lots of different nodes with no > global timeline of queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Doc Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Description: Documentation Request to improve doc around Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes (eg. [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy)] then the query list will be spread across lots of different nodes with no global timeline of queries. was: Documentation Request to add a Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes then the query list will be spread across lots of different nodes with no global timeline of queries. > Doc Request: Global Query List showing queries from all Drill foreman nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation, Metadata, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Documentation Request to improve doc around Global Query List to show all > queries executed across all Drill nodes in a cluster for better management > and auditing. > It wasn't obvious to be able to see all queries across all nodes in a Drill > cluster. The Web UI on any given Drill node only shows the queries > coordinated by that local node if acting as the foreman for the query, so if > using ZooKeeper or a Load Balancer to distribute queries via different Drill > nodes (eg. > [https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy)] then the > query list will be spread across lots of different nodes with no global > timeline of queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Doc Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Description: Documentation Request to add a Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. It wasn't obvious to be able to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes then the query list will be spread across lots of different nodes with no global timeline of queries. was: Feature Request to add a Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. Right now there doesn't appear to be a way to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes then the query list will be spread across lots of different nodes with no global timeline of queries. This seems to leave a bit of a gap in auditing functionality, with the only other option that I can think of being immediately available is to limit all query submissions via a single foreman node so the query list is complete on that node - although that doesn't seem like a great idea in terms of load distribution of query planning, coordination and final aggregation steps. I've made load balancing configurations for Apache Drill and similar technologies that could be used for that purpose with failover support to maintain high availability at https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would still prefer if Drill was designed to store the global list of queries submitted in a centralized place. > Doc Request: Global Query List showing queries from all Drill foreman nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation, Metadata, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Documentation Request to add a Global Query List to show all queries executed > across all Drill nodes in a cluster for better management and auditing. > It wasn't obvious to be able to see all queries across all nodes in a Drill > cluster. The Web UI on any given Drill node only shows the queries > coordinated by that local node if acting as the foreman for the query, so if > using ZooKeeper or a Load Balancer to distribute queries via different Drill > nodes then the query list will be spread across lots of different nodes with > no global timeline of queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Doc Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Summary: Doc Request: Global Query List showing queries from all Drill foreman nodes (was: Feature Request: Global Query List showing queries from all Drill foreman nodes) > Doc Request: Global Query List showing queries from all Drill foreman nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation, Metadata, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add a Global Query List to show all queries executed > across all Drill nodes in a cluster for better management and auditing. > Right now there doesn't appear to be a way to see all queries across all > nodes in a Drill cluster. The Web UI on any given Drill node only shows the > queries coordinated by that local node if acting as the foreman for the > query, so if using ZooKeeper or a Load Balancer to distribute queries via > different Drill nodes then the query list will be spread across lots of > different nodes with no global timeline of queries. > This seems to leave a bit of a gap in auditing functionality, with the only > other option that I can think of being immediately available is to limit all > query submissions via a single foreman node so the query list is complete on > that node - although that doesn't seem like a great idea in terms of load > distribution of query planning, coordination and final aggregation steps. > I've made load balancing configurations for Apache Drill and similar > technologies that could be used for that purpose with failover support to > maintain high availability at > https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would > still prefer if Drill was designed to store the global list of queries > submitted in a centralized place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Component/s: (was: Tools, Build & Test) (was: Query Planning & Optimization) (was: Server) > Feature Request: Global Query List showing queries from all Drill foreman > nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation, Metadata, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add a Global Query List to show all queries executed > across all Drill nodes in a cluster for better management and auditing. > Right now there doesn't appear to be a way to see all queries across all > nodes in a Drill cluster. The Web UI on any given Drill node only shows the > queries coordinated by that local node if acting as the foreman for the > query, so if using ZooKeeper or a Load Balancer to distribute queries via > different Drill nodes then the query list will be spread across lots of > different nodes with no global timeline of queries. > This seems to leave a bit of a gap in auditing functionality, with the only > other option that I can think of being immediately available is to limit all > query submissions via a single foreman node so the query list is complete on > that node - although that doesn't seem like a great idea in terms of load > distribution of query planning, coordination and final aggregation steps. > I've made load balancing configurations for Apache Drill and similar > technologies that could be used for that purpose with failover support to > maintain high availability at > https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would > still prefer if Drill was designed to store the global list of queries > submitted in a centralized place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[ https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-6061: --- Issue Type: Improvement (was: New Feature) > Feature Request: Global Query List showing queries from all Drill foreman > nodes > --- > > Key: DRILL-6061 > URL: https://issues.apache.org/jira/browse/DRILL-6061 > Project: Apache Drill > Issue Type: Improvement > Components: Server, Documentation, Metadata, Query Planning & > Optimization, Tools, Build & Test, Web Server >Affects Versions: 1.11.0 > Environment: MapR 5.2 >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add a Global Query List to show all queries executed > across all Drill nodes in a cluster for better management and auditing. > Right now there doesn't appear to be a way to see all queries across all > nodes in a Drill cluster. The Web UI on any given Drill node only shows the > queries coordinated by that local node if acting as the foreman for the > query, so if using ZooKeeper or a Load Balancer to distribute queries via > different Drill nodes then the query list will be spread across lots of > different nodes with no global timeline of queries. > This seems to leave a bit of a gap in auditing functionality, with the only > other option that I can think of being immediately available is to limit all > query submissions via a single foreman node so the query list is complete on > that node - although that doesn't seem like a great idea in terms of load > distribution of query planning, coordination and final aggregation steps. > I've made load balancing configurations for Apache Drill and similar > technologies that could be used for that purpose with failover support to > maintain high availability at > https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would > still prefer if Drill was designed to store the global list of queries > submitted in a centralized place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[
https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327012#comment-16327012
]
Hari Sekhon edited comment on DRILL-6061 at 1/16/18 11:10 AM:
--
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community thread on this as well:
[https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles]
I recommend changing the Apache Drill documentation at:
{{[https://drill.apache.org/docs/persistent-configuration-storage/]}}
{{}} to a standardized best practice location
of {{/apps/drill/pstore}} to fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
was (Author: harisekhon):
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection in response to this as well:
[https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles]
I recommend changing the Apache Drill documentation at:
{{[https://drill.apache.org/docs/persistent-configuration-storage/]}}
{{}} to a standardized best practice location
of {{/apps/drill/pstore}} to fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
> Feature Request: Global Query List showing queries from all Drill foreman
> nodes
> ---
>
> Key: DRILL-6061
> URL: https://issues.apache.org/jira/browse/DRILL-6061
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server, Documentation, Metadata, Query Planning &
> Optimization, Tools, Build & Test, Web Server
>Affects Versions: 1.11.0
> Environment: MapR 5.2
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to add a Global Query List to show all queries executed
> across all Drill nodes in a cluster for better management and auditing.
> Right now there doesn't appear to be a way to see all queries across all
> nodes in a Drill cluster. The Web UI on any given Drill node only shows the
> queries coordinated by that local node if acting as the foreman for the
> query, so if using ZooKeeper or a Load Balancer to distribute queries via
> different Drill nodes then the query list will be spread across lots of
> different nodes with no global timeline of queries.
> This seems to leave a bit of a gap in auditing functionality, with the only
> other option that I can think of being immediately available is to limit all
> query submissions via a single foreman node so the query list is complete on
> that node - although that doesn't seem like a great idea in terms of load
> distribution of query planning, coordination and final aggregation steps.
> I've made load balancing configurations for Apache Drill and similar
> technologies that could be used for that purpose with failover support to
> maintain high availability at
> https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would
> still prefer if Drill was designed to store the global list of queries
> submitted in a centralized place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[
https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327012#comment-16327012
]
Hari Sekhon edited comment on DRILL-6061 at 1/16/18 11:09 AM:
--
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection in response to this as well:
[https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles]
I recommend changing the Apache Drill documentation at:
{{[https://drill.apache.org/docs/persistent-configuration-storage/]}}
{{}} to a standardized best practice location
of {{/apps/drill/pstore}} to fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
was (Author: harisekhon):
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection to response to this as well:
[https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles]
I recommend changing the Apache Drill documentation at:
{{[https://drill.apache.org/docs/persistent-configuration-storage/]}}
{{}} to a standardized best practice location
of {{/apps/drill/pstore}} to fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
> Feature Request: Global Query List showing queries from all Drill foreman
> nodes
> ---
>
> Key: DRILL-6061
> URL: https://issues.apache.org/jira/browse/DRILL-6061
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server, Documentation, Metadata, Query Planning &
> Optimization, Tools, Build & Test, Web Server
>Affects Versions: 1.11.0
> Environment: MapR 5.2
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to add a Global Query List to show all queries executed
> across all Drill nodes in a cluster for better management and auditing.
> Right now there doesn't appear to be a way to see all queries across all
> nodes in a Drill cluster. The Web UI on any given Drill node only shows the
> queries coordinated by that local node if acting as the foreman for the
> query, so if using ZooKeeper or a Load Balancer to distribute queries via
> different Drill nodes then the query list will be spread across lots of
> different nodes with no global timeline of queries.
> This seems to leave a bit of a gap in auditing functionality, with the only
> other option that I can think of being immediately available is to limit all
> query submissions via a single foreman node so the query list is complete on
> that node - although that doesn't seem like a great idea in terms of load
> distribution of query planning, coordination and final aggregation steps.
> I've made load balancing configurations for Apache Drill and similar
> technologies that could be used for that purpose with failover support to
> maintain high availability at
> https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would
> still prefer if Drill was designed to store the global list of queries
> submitted in a centralized place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[
https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327012#comment-16327012
]
Hari Sekhon edited comment on DRILL-6061 at 1/16/18 11:08 AM:
--
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection to response to this as well:
[https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles]
I recommend changing the Apache Drill documentation at:
{{[https://drill.apache.org/docs/persistent-configuration-storage/]}}
{{}} to a standardized best practice location
of {{/apps/drill/pstore}} to fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
was (Author: harisekhon):
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection to response to this as well:
https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles
I recommend changing the Apache Drill documentation {{}} with a single best practice path of {{/apps/drill/pstore}} to
standardize this and fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
> Feature Request: Global Query List showing queries from all Drill foreman
> nodes
> ---
>
> Key: DRILL-6061
> URL: https://issues.apache.org/jira/browse/DRILL-6061
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server, Documentation, Metadata, Query Planning &
> Optimization, Tools, Build & Test, Web Server
>Affects Versions: 1.11.0
> Environment: MapR 5.2
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to add a Global Query List to show all queries executed
> across all Drill nodes in a cluster for better management and auditing.
> Right now there doesn't appear to be a way to see all queries across all
> nodes in a Drill cluster. The Web UI on any given Drill node only shows the
> queries coordinated by that local node if acting as the foreman for the
> query, so if using ZooKeeper or a Load Balancer to distribute queries via
> different Drill nodes then the query list will be spread across lots of
> different nodes with no global timeline of queries.
> This seems to leave a bit of a gap in auditing functionality, with the only
> other option that I can think of being immediately available is to limit all
> query submissions via a single foreman node so the query list is complete on
> that node - although that doesn't seem like a great idea in terms of load
> distribution of query planning, coordination and final aggregation steps.
> I've made load balancing configurations for Apache Drill and similar
> technologies that could be used for that purpose with failover support to
> maintain high availability at
> https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would
> still prefer if Drill was designed to store the global list of queries
> submitted in a centralized place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[
https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327012#comment-16327012
]
Hari Sekhon commented on DRILL-6061:
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection to response to this as well:
. This then makes sense to
I recommend also replacing {{}} with a single
best practice path of {{/apps/drill/pstore}} to standardize this and fall in
line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
> Feature Request: Global Query List showing queries from all Drill foreman
> nodes
> ---
>
> Key: DRILL-6061
> URL: https://issues.apache.org/jira/browse/DRILL-6061
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server, Documentation, Metadata, Query Planning &
> Optimization, Tools, Build & Test, Web Server
>Affects Versions: 1.11.0
> Environment: MapR 5.2
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to add a Global Query List to show all queries executed
> across all Drill nodes in a cluster for better management and auditing.
> Right now there doesn't appear to be a way to see all queries across all
> nodes in a Drill cluster. The Web UI on any given Drill node only shows the
> queries coordinated by that local node if acting as the foreman for the
> query, so if using ZooKeeper or a Load Balancer to distribute queries via
> different Drill nodes then the query list will be spread across lots of
> different nodes with no global timeline of queries.
> This seems to leave a bit of a gap in auditing functionality, with the only
> other option that I can think of being immediately available is to limit all
> query submissions via a single foreman node so the query list is complete on
> that node - although that doesn't seem like a great idea in terms of load
> distribution of query planning, coordination and final aggregation steps.
> I've made load balancing configurations for Apache Drill and similar
> technologies that could be used for that purpose with failover support to
> maintain high availability at
> https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would
> still prefer if Drill was designed to store the global list of queries
> submitted in a centralized place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
[
https://issues.apache.org/jira/browse/DRILL-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327012#comment-16327012
]
Hari Sekhon edited comment on DRILL-6061 at 1/16/18 11:08 AM:
--
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection to response to this as well:
https://community.mapr.com/thread/21498-what-are-best-practices-for-managing-drill-query-profiles
I recommend changing the Apache Drill documentation {{}} with a single best practice path of {{/apps/drill/pstore}} to
standardize this and fall in line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
was (Author: harisekhon):
Yes this is what we found after raising this, we will point to MapR-FS
/apps/drill/pstore to follow Hadoop layout best practices convention and test.
I think this should be documented a bit better / easier to find, perhaps in FAQ
or a section stating something like "Global Query List - how to see the queries
on the cluster from any Drill node" in the Apache Drill documentation. There is
a MapR community connection to response to this as well:
. This then makes sense to
I recommend also replacing {{}} with a single
best practice path of {{/apps/drill/pstore}} to standardize this and fall in
line with other apps on Hadoop clusters.
It's also worth documenting the load balancing algorithm used for load
balancing across Drill nodes when acquiring a drillbit via zookeeper quorum
referral (random, round robin, least connection etc).
> Feature Request: Global Query List showing queries from all Drill foreman
> nodes
> ---
>
> Key: DRILL-6061
> URL: https://issues.apache.org/jira/browse/DRILL-6061
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server, Documentation, Metadata, Query Planning &
> Optimization, Tools, Build & Test, Web Server
>Affects Versions: 1.11.0
> Environment: MapR 5.2
>Reporter: Hari Sekhon
>Priority: Major
>
> Feature Request to add a Global Query List to show all queries executed
> across all Drill nodes in a cluster for better management and auditing.
> Right now there doesn't appear to be a way to see all queries across all
> nodes in a Drill cluster. The Web UI on any given Drill node only shows the
> queries coordinated by that local node if acting as the foreman for the
> query, so if using ZooKeeper or a Load Balancer to distribute queries via
> different Drill nodes then the query list will be spread across lots of
> different nodes with no global timeline of queries.
> This seems to leave a bit of a gap in auditing functionality, with the only
> other option that I can think of being immediately available is to limit all
> query submissions via a single foreman node so the query list is complete on
> that node - although that doesn't seem like a great idea in terms of load
> distribution of query planning, coordination and final aggregation steps.
> I've made load balancing configurations for Apache Drill and similar
> technologies that could be used for that purpose with failover support to
> maintain high availability at
> https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would
> still prefer if Drill was designed to store the global list of queries
> submitted in a centralized place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6061) Feature Request: Global Query List showing queries from all Drill foreman nodes
Hari Sekhon created DRILL-6061: -- Summary: Feature Request: Global Query List showing queries from all Drill foreman nodes Key: DRILL-6061 URL: https://issues.apache.org/jira/browse/DRILL-6061 Project: Apache Drill Issue Type: New Feature Components: Server, Documentation, Metadata, Query Planning & Optimization, Tools, Build & Test, Web Server Affects Versions: 1.11.0 Environment: MapR 5.2 Reporter: Hari Sekhon Feature Request to add a Global Query List to show all queries executed across all Drill nodes in a cluster for better management and auditing. Right now there doesn't appear to be a way to see all queries across all nodes in a Drill cluster. The Web UI on any given Drill node only shows the queries coordinated by that local node if acting as the foreman for the query, so if using ZooKeeper or a Load Balancer to distribute queries via different Drill nodes then the query list will be spread across lots of different nodes with no global timeline of queries. This seems to leave a bit of a gap in auditing functionality, with the only other option that I can think of being immediately available is to limit all query submissions via a single foreman node so the query list is complete on that node - although that doesn't seem like a great idea in terms of load distribution of query planning, coordination and final aggregation steps. I've made load balancing configurations for Apache Drill and similar technologies that could be used for that purpose with failover support to maintain high availability at https://github.com/HariSekhon/nagios-plugins/tree/master/haproxy) but would still prefer if Drill was designed to store the global list of queries submitted in a centralized place. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5990) Apache Drill /status API returns OK ('Running') even with JRE while queries will not work - make status API reflect the fact that Drill is broken on JRE or stop Drill sta
Hari Sekhon created DRILL-5990:
--
Summary: Apache Drill /status API returns OK ('Running') even with
JRE while queries will not work - make status API reflect the fact that Drill
is broken on JRE or stop Drill starting up with JRE
Key: DRILL-5990
URL: https://issues.apache.org/jira/browse/DRILL-5990
Project: Apache Drill
Issue Type: Bug
Components: Server, Client - HTTP, Execution - Monitoring
Affects Versions: 1.11.0, 1.10.0
Environment: Docker
Reporter: Hari Sekhon
If running Apache Drill on versions 1.10 / 1.11 on JRE it appears that queries
no longer run without JDK, but the /status monitoring API endpoint does not
reflect the fact the Apache Drill will not work and still returns 'Running'.
While 'Running' is technically true the process is up, it's effectively broken
and Apache Drill should either reflect this in /status that is is broken or
refuse to start up on JRE altogether.
See this ticket for more information:
https://github.com/HariSekhon/Dockerfiles/pull/15
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (DRILL-3584) Drill Kerberos HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066323#comment-15066323 ] Hari Sekhon edited comment on DRILL-3584 at 12/21/15 11:06 AM: --- Last time I pressed this issue Ted Dunning suggested that I implemented the HDFS Kerberos integration myself. If only If I was still on a vendor payroll and getting steady salary to spend time on this kind of thing... I'm not holding my breath for this to happen soon... we ended up dropping Drill here - as I mentioned in my original post it's Dead-On-Arrival without the HDFS Kerberos integration for most governed enterprise environments. was (Author: harisekhon): Last time I pressed this issue Ted Dunning suggested that I implemented the HDFS Kerberos integration myself. If only If I was still on a vendor payroll and getting steady salary to spend time on this kind of thing... > Drill Kerberos HDFS Support / Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066323#comment-15066323 ] Hari Sekhon commented on DRILL-3584: Last time I pressed this issue Ted Dunning suggested that I implemented the HDFS Kerberos integration myself. If only If I was still on a vendor payroll and getting steady salary to spend time on this kind of thing... > Drill Kerberos HDFS Support / Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066302#comment-15066302 ] Hari Sekhon commented on DRILL-3584: MapR preferred not to run Kerberos on their clusters last I used them (about a year ago). While they have some kerberos integration they usually recommended their own ticketing system, MapR "native security", to secure the base filesystem... it's not clear how much of the Apache ecosystem actually integrates with this alternative mechanism though. I agree this is a disadvantage to the Drill project and the limited adoption will prevent the implicit additional QA of wider usage on the HDFS based platforms which require the standard Kerberos security. > Drill Kerberos HDFS Support / Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3524) Drill proper DESCRIBE support for MongoDB
[
https://issues.apache.org/jira/browse/DRILL-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14996526#comment-14996526
]
Hari Sekhon commented on DRILL-3524:
Why would the query have to fail? It doesn't fail now so that behaviour
shouldn't change.
I'm not talking about validating queries or enforcing schema, just purely a
convenience function for the describe command to return something useful for a
user, with default best effort or optional full scan.
> Drill proper DESCRIBE support for MongoDB
> -
>
> Key: DRILL-3524
> URL: https://issues.apache.org/jira/browse/DRILL-3524
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata, Storage - MongoDB
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
> Fix For: Future
>
>
> Request to add full DESCRIBE support for MongoDB collections.
> I understand this may be difficult / sub-optimal due to the flexible schema
> nature of Mongo docs but if you can tabulate results when reading directly
> from MongoDB for which you have read the field names, then it's also possible
> to extract all field names to present for the describe command, albeit an
> inefficient scan to do so.
> Currently describe returns a pseudo / inaccurate / unhelpful metadata:
> {code}+--++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +--++--+
> | *| ANY| YES |
> +--++--+{code}
> Perhaps you could extend DESCRIBE to scan the first few dozen docs by default
> to create a merged schema as well as adding an optional argument to the
> describe command to allow for scanning a user-specified number of docs from
> which to describe the schema, or an ALL argument keyword to describe to scan
> all docs in a collection to get the complete global schema for the collection?
> In case of schema evolution it might be an interesting option to additionally
> read the newest and oldest records, maybe the first and last records by ID
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3524) Drill proper DESCRIBE support for MongoDB
[
https://issues.apache.org/jira/browse/DRILL-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14995665#comment-14995665
]
Hari Sekhon commented on DRILL-3524:
I already addressed the schema issue in the original post.
I've also suggested workarounds such as scanning the first few docs and return
whatever fields are found.
Right now I have to run a sample query which is much harder for the human eye
to parse than having a describe function provide a condensed unique list of
discovered fields. This function also provides Drill engineers to determine the
best performance / schema completeness trade-off via testing as well as
providing the user a more comprehensive schema description via keyword ALL
(full scan).
Considering the current 'describe' command returns completely useless output,
any of the suggestions in my original post would be better than the current
implementation.
> Drill proper DESCRIBE support for MongoDB
> -
>
> Key: DRILL-3524
> URL: https://issues.apache.org/jira/browse/DRILL-3524
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata, Storage - MongoDB
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
> Fix For: Future
>
>
> Request to add full DESCRIBE support for MongoDB collections.
> I understand this may be difficult / sub-optimal due to the flexible schema
> nature of Mongo docs but if you can tabulate results when reading directly
> from MongoDB for which you have read the field names, then it's also possible
> to extract all field names to present for the describe command, albeit an
> inefficient scan to do so.
> Currently describe returns a pseudo / inaccurate / unhelpful metadata:
> {code}+--++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +--++--+
> | *| ANY| YES |
> +--++--+{code}
> Perhaps you could extend DESCRIBE to scan the first few dozen docs by default
> to create a merged schema as well as adding an optional argument to the
> describe command to allow for scanning a user-specified number of docs from
> which to describe the schema, or an ALL argument keyword to describe to scan
> all docs in a collection to get the complete global schema for the collection?
> In case of schema evolution it might be an interesting option to additionally
> read the newest and oldest records, maybe the first and last records by ID
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3646) Show tables in DFS workspace
[
https://issues.apache.org/jira/browse/DRILL-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725193#comment-14725193
]
Hari Sekhon commented on DRILL-3646:
Show files does work, but then you have to know which files are data sources
and which directories are tables. One could easily set up a specific workspace
where everything inside is a data source to work around this, but it still
seems a little counterintuitive if coming from RDBMS or even Hive to be able to
select from a "phantom" table that doesn't appear in "show tables".
I believe the more intuitive behaviour would be for Drill to show all files and
directories with relevant data formats such as CSV, JSON, *.log, parquet files
etc. I believe this would only slow things down if there are a lot of
subdirectories to do listings on to determine if they're tables?
> Show tables in DFS workspace
>
>
> Key: DRILL-3646
> URL: https://issues.apache.org/jira/browse/DRILL-3646
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata, Storage - Information Schema
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Assignee: Steven Phillips
>
> Drill does not show tables in a DFS workspace, even when I just created a
> parquet table there using CTAS through Drill itself.
> The output results are blank, a zero row table with blank column header.
> {code}
> 0: jdbc:drill:zk=local> show tables in dfs.hari;
> +--+
> | |
> +--+
> +--+
> No rows selected (0.137 seconds)
> {code}
> although I can still query the table I just created as long as I know it's
> there and query it blindly:
> {code}
> 0: jdbc:drill:zk=local> select count(*) from dfs.hari.auditlogs_parquet_drill;
> +-+
> | EXPR$0 |
> +-+
> | 2579|
> +-+
> 1 row selected (0.129 seconds)
> {code}
> I can't describe the table so I really do have to query it blindly too (I
> previously raised a different jira for parquet describe support DRILL-3525
> and other formats DRILL-3524 to DRILL-3529):
> {code}
> 0: jdbc:drill:zk=local> select count(*) from dfs.hari.auditlogs_parquet_drill;
> +-+
> | EXPR$0 |
> +-+
> | 2579|
> +-+
> 1 row selected (0.129 seconds)
> {code}
> This jira is specifically to address the inability to list tables (or perhaps
> files/dirs) in a DFS workspace though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3713) Drill-embedded doesn't work with VPN due to use of local IP, should use 127.0.0.1 instead
Hari Sekhon created DRILL-3713: -- Summary: Drill-embedded doesn't work with VPN due to use of local IP, should use 127.0.0.1 instead Key: DRILL-3713 URL: https://issues.apache.org/jira/browse/DRILL-3713 Project: Apache Drill Issue Type: Bug Components: Client - JDBC Affects Versions: 1.1.0 Environment: Drill embedded Reporter: Hari Sekhon Assignee: Daniel Barclay (Drill) Priority: Critical Drill appears to be using my local IP address (192.168.x.x that is assigned by DHCP on wifi) but when connected to my company VPN this breaks drill's access due to routing changes done by the VPN client which I have no control over (this is a locked down company laptop without admin rights, not my personal laptop). I've noticed that 127.0.0.1 remains accessible however. I believe it would be better for drill-embedded to do everything on 127.0.0.1 to avoid these kinds of issues. Currently I can only use drill-embedded when not on the VPN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3523) Drill client CLI does not reconnect to Drillbit after long period of inactivity
[
https://issues.apache.org/jira/browse/DRILL-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703247#comment-14703247
]
Hari Sekhon commented on DRILL-3523:
More like 30+ mins. I don't know the exact time, as I was working on other
things and only noticed this when returning to the drill prompt, after this
happened a few times I thought I'd raise it. Since this is drill-embedded there
are no intermediate firewalls that could be resetting the connection either.
The error that occurs when trying to enter another SQL statement was shown in
the original post above.
Good point about session alteration. It might require the drill client to
remember the session alteration statements that were sent to Drill so it could
re-run them on re-connect. This would be a significant usability improvement
rather than making the user reconnect and deal with it every time multiple
times a day as I was having to do.
> Drill client CLI does not reconnect to Drillbit after long period of
> inactivity
> ---
>
> Key: DRILL-3523
> URL: https://issues.apache.org/jira/browse/DRILL-3523
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - CLI
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Assignee: Daniel Barclay (Drill)
> Fix For: 1.2.0
>
>
> After a period of inactivity in the Drill CLI client before issuing another
> statement, it appears to time out the session/connection and the client
> returns an error instead of reconnecting to Drill.
> It should instead just implicitly "!reconnect" itself since I can't think of
> a good reason why I have to do this by hand every time.
> Unlike in DRILL-3514 the drillbit in this case was never restarted, although
> the same fix could probably solve both issues.
> The error received when trying to issue a new query after some absence is:
> {code}Error: CONNECTION ERROR: Connection /x.x.x.x:54367 <--> /x.x.x.x:31010
> (user client) closed unexpectedly.
> [Error Id: 68714cc2-65f5-4cac-b062-44331f8f4c31 ] (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3646) Show tables in DFS workspace
Hari Sekhon created DRILL-3646:
--
Summary: Show tables in DFS workspace
Key: DRILL-3646
URL: https://issues.apache.org/jira/browse/DRILL-3646
Project: Apache Drill
Issue Type: Bug
Components: Metadata, Storage - Information Schema
Affects Versions: 1.1.0
Reporter: Hari Sekhon
Assignee: Steven Phillips
Drill does not show tables in a DFS workspace, even when I just created a
parquet table there using CTAS through Drill itself.
The output results are blank, a zero row table with blank column header.
{code}
0: jdbc:drill:zk=local> show tables in dfs.hari;
+--+
| |
+--+
+--+
No rows selected (0.137 seconds)
{code}
although I can still query the table I just created as long as I know it's
there and query it blindly:
{code}
0: jdbc:drill:zk=local> select count(*) from dfs.hari.auditlogs_parquet_drill;
+-+
| EXPR$0 |
+-+
| 2579|
+-+
1 row selected (0.129 seconds)
{code}
I can't describe the table so I really do have to query it blindly too (I
previously raised a different jira for parquet describe support DRILL-3525 and
other formats DRILL-3524 to DRILL-3529):
{code}
0: jdbc:drill:zk=local> select count(*) from dfs.hari.auditlogs_parquet_drill;
+-+
| EXPR$0 |
+-+
| 2579|
+-+
1 row selected (0.129 seconds)
{code}
This jira is specifically to address the inability to list tables (or perhaps
files/dirs) in a DFS workspace though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3625) Dynamic Format Detection in DFS backend for unmapped file extensions / files without extensions
[
https://issues.apache.org/jira/browse/DRILL-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14682071#comment-14682071
]
Hari Sekhon commented on DRILL-3625:
That would create the same problem as mapping the file extension in the DFS
configuration that I mentioned which is that it's not generic. What if one log
file is json and another is csv?
The Dynamic Format Detection is still needed when traversing filesystems like
this since in the real-world there will almost certainly be different formats
found and marking everything as json under the dfs workspace is not flexible.
> Dynamic Format Detection in DFS backend for unmapped file extensions / files
> without extensions
> ---
>
> Key: DRILL-3625
> URL: https://issues.apache.org/jira/browse/DRILL-3625
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - JSON, Storage - Other, Storage - Parquet,
> Storage - Text & CSV
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Assignee: Steven Phillips
>
> When querying a json file that doesn't have a ".json" extension such as
> ".log" I get this exception:
> {code}0: jdbc:drill:zk=local> select * from dfs.down.`auditOut.log` limit 1;
> Aug 11, 2015 4:01:38 PM org.apache.calcite.sql.validate.SqlValidatorException
>
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
> 'dfs.down.auditOut.log' not found
> Aug 11, 2015 4:01:38 PM org.apache.calcite.runtime.CalciteException
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
> column 15 to line 1, column 17: Table 'dfs.down.auditOut.log' not found
> Error: PARSE ERROR: From line 1, column 15 to line 1, column 17: Table
> 'dfs.down.auditOut.log' not found
> [Error Id: 5610210b-3eb2-497f-9443-c725b29733b6 on :31010]
> (state=,code=0)
> {code}
> However when renaming the file to have a .json extension then the query
> succeeds.
> Now while I could reconfigure the DFS plugin to associate all files with
> *.log extension to be mapped to json, this doesn't seem like the right thing
> to do. I could rename the file to have a .json extension of course which is
> the better thing to do but this highlights another question, why doesn't this
> just work as-is?
> Hence I'd like to raise this as a feature request that when an unmapped
> extension or file without any extension is encountered Drill should do a few
> quick checks on the file type and then use the appropriate storage backend
> for the file.
> Adding this "Dynamic Format Detection" as I have dubbed it would tie in
> nicely with Drill's style and existing features like the dynamic schema
> detection already used for json.
> This may also come in handy for dealing with outputs from MapReduce jobs
> where the files may be named part-m-N or part-r-N without any
> extension and for example if those files were text then the text storage
> backend could be immediately invoked upon them in Drill.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3625) Dynamic Format Detection in DFS backend for unmapped file extensions / files without extensions
Hari Sekhon created DRILL-3625:
--
Summary: Dynamic Format Detection in DFS backend for unmapped file
extensions / files without extensions
Key: DRILL-3625
URL: https://issues.apache.org/jira/browse/DRILL-3625
Project: Apache Drill
Issue Type: New Feature
Components: Storage - JSON, Storage - Other, Storage - Parquet,
Storage - Text & CSV
Affects Versions: 1.1.0
Reporter: Hari Sekhon
Assignee: Steven Phillips
When querying a json file that doesn't have a ".json" extension such as ".log"
I get this exception:
{code}0: jdbc:drill:zk=local> select * from dfs.down.`auditOut.log` limit 1;
Aug 11, 2015 4:01:38 PM org.apache.calcite.sql.validate.SqlValidatorException
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
'dfs.down.auditOut.log' not found
Aug 11, 2015 4:01:38 PM org.apache.calcite.runtime.CalciteException
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1, column
15 to line 1, column 17: Table 'dfs.down.auditOut.log' not found
Error: PARSE ERROR: From line 1, column 15 to line 1, column 17: Table
'dfs.down.auditOut.log' not found
[Error Id: 5610210b-3eb2-497f-9443-c725b29733b6 on :31010] (state=,code=0)
{code}
However when renaming the file to have a .json extension then the query
succeeds.
Now while I could reconfigure the DFS plugin to associate all files with *.log
extension to be mapped to json, this doesn't seem like the right thing to do. I
could rename the file to have a .json extension of course which is the better
thing to do but this highlights another question, why doesn't this just work
as-is?
Hence I'd like to raise this as a feature request that when an unmapped
extension or file without any extension is encountered Drill should do a few
quick checks on the file type and then use the appropriate storage backend for
the file.
Adding this "Dynamic Format Detection" as I have dubbed it would tie in nicely
with Drill's style and existing features like the dynamic schema detection
already used for json.
This may also come in handy for dealing with outputs from MapReduce jobs where
the files may be named part-m-N or part-r-N without any extension and
for example if those files were text then the text storage backend could be
immediately invoked upon them in Drill.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3584) Drill Kerberos HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-3584: --- Summary: Drill Kerberos HDFS Support / Documentation (was: Drill Kerberos + HDFS Support / Documentation) > Drill Kerberos HDFS Support / Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3584) Drill Kerberos HDFS Support + Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-3584: --- Description: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure kerberized Hive/HCatalog etc.) was: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure Hive/HCatalog etc.) > Drill Kerberos HDFS Support + Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3584) Drill Kerberos + HDFS Support / Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-3584: --- Summary: Drill Kerberos + HDFS Support / Documentation (was: Drill Kerberos HDFS Support + Documentation) > Drill Kerberos + HDFS Support / Documentation > - > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > kerberized Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3584) Drill Kerberos HDFS Support + Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-3584: --- Description: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure Hive/HCatalog etc.) was: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos (plus obviouslyu. > Drill Kerberos HDFS Support + Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure > Hive/HCatalog etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3584) Drill Kerberos HDFS Support + Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-3584: --- Description: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos (plus obviouslyu. was: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos. > Drill Kerberos HDFS Support + Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos (plus obviouslyu. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3584) Drill Kerberos HDFS Support + Documentation
[ https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated DRILL-3584: --- Description: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it doesn't appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos. was: I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it does appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos. > Drill Kerberos HDFS Support + Documentation > --- > > Key: DRILL-3584 > URL: https://issues.apache.org/jira/browse/DRILL-3584 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.1.0 >Reporter: Hari Sekhon >Priority: Blocker > > I'm trying to find Drill docs for Kerberos support for secure HDFS clusters > and it doesn't appear to be well tested / supported / documented yet. > This product is Dead-on-Arrival if it doesn't integrate well with secure > Hadoop clusters, specifically HDFS + Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3584) Drill Kerberos HDFS Support + Documentation
Hari Sekhon created DRILL-3584: -- Summary: Drill Kerberos HDFS Support + Documentation Key: DRILL-3584 URL: https://issues.apache.org/jira/browse/DRILL-3584 Project: Apache Drill Issue Type: New Feature Affects Versions: 1.1.0 Reporter: Hari Sekhon Priority: Blocker I'm trying to find Drill docs for Kerberos support for secure HDFS clusters and it does appear to be well tested / supported / documented yet. This product is Dead-on-Arrival if it doesn't integrate well with secure Hadoop clusters, specifically HDFS + Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
