[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389759#comment-17389759 ] Oleksiy Sayankin commented on HIVE-6679: [~kgyrtkirk] Could you please review the PR? > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15, pull-request-available > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386528#comment-17386528 ] Oleksiy Sayankin commented on HIVE-6679: Guys, please review the PR. > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15, pull-request-available > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378010#comment-17378010 ] Oleksiy Sayankin commented on HIVE-6679: Created PR with fix https://github.com/apache/hive/pull/2461 Please review > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15, pull-request-available > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-6679: -- Assignee: Oleksiy Sayankin (was: Navis Ryu) > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0 >Reporter: Prasad Suresh Mujumdar >Assignee: Oleksiy Sayankin >Priority: Major > Labels: TODOC1.0, TODOC15 > Fix For: 1.3.0 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch, HIVE-6679.6.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21075: Status: Patch Available (was: In Progress) > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21075.2.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21075: Attachment: HIVE-21075.2.patch > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21075.2.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21075 started by Oleksiy Sayankin. --- > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351720#comment-17351720 ] Oleksiy Sayankin edited comment on HIVE-21075 at 5/26/21, 11:23 AM: Created [PR|https://github.com/apache/hive/pull/2323]. Solution is simple: {code} if (isPostgres()){ // use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1 } else { // use select count(1) from org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd == inCD) } {code} was (Author: osayankin): Created [PR|https://github.com/apache/hive/pull/2323]. Solution is simple: {code} if (isPostgres()){ // use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1 } else { // use select count(1) from org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd == inCD) } {code} > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351720#comment-17351720 ] Oleksiy Sayankin edited comment on HIVE-21075 at 5/26/21, 11:22 AM: Created [PR|https://github.com/apache/hive/pull/2323]. Solution is simple: {code} if (isPostgres()){ // use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1 } else { // use select count(1) from org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd == inCD) } {code} was (Author: osayankin): Created [PR|https://github.com/apache/hive/pull/2323]. Solution is simple: {code} if (isPostgres()){ // use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1 } else { // select count(1) from org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd == inCD) } {code} > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351720#comment-17351720 ] Oleksiy Sayankin commented on HIVE-21075: - Created [PR|https://github.com/apache/hive/pull/2323]. Solution is simple: {code} if (isPostgres()){ // use SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1 } else { // select count(1) from org.apache.hadoop.hive.metastore.model.MStorageDescriptor where (this.cd == inCD) } {code} > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-21075: --- Assignee: Oleksiy Sayankin > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[ https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304055#comment-17304055 ] Oleksiy Sayankin edited comment on HIVE-24904 at 3/18/21, 11:23 AM: The latest supported release of the lib is 1.9.13 ([https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl]) for updating the lib to version with fix we have 3 options: 1. [https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl/1.9.14.jdk17-redhat-1] update to lib that was bundled by RedHat 2. Build our own lib from the master: [https://github.com/FasterXML/jackson-1] 3. Move to new artifact {panel} com.fasterxml.jackson.core » jackson-databind {panel} FYI: [~kgyrtkirk], [~jcamachorodriguez], [~pvary] was (Author: osayankin): The latest supported release of the lib is 1.9.13 ([https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl]) for updating the lib to version with fix we have 3 options: 1. [https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl/1.9.14.jdk17-redhat-1] update to lib that was bundled by RedHat 2. Build our own lib from the master: [https://github.com/FasterXML/jackson-1] 3. Move to new artifact {panel} com.fasterxml.jackson.core » jackson-databind{panel} > CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar > -- > > Key: HIVE-24904 > URL: https://issues.apache.org/jira/browse/HIVE-24904 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Priority: Critical > > CVE list: CVE-2019-10172,CVE-2019-10202 > CVSS score: High > {code} > ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[ https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-24904: --- Assignee: Zoltan Haindrich > CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar > -- > > Key: HIVE-24904 > URL: https://issues.apache.org/jira/browse/HIVE-24904 > Project: Hive > Issue Type: Bug > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Zoltan Haindrich >Priority: Critical > Labels: CVE > > CVE list: CVE-2019-10172,CVE-2019-10202 > CVSS score: High > {code} > ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[ https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-24904: Labels: CVE (was: ) > CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar > -- > > Key: HIVE-24904 > URL: https://issues.apache.org/jira/browse/HIVE-24904 > Project: Hive > Issue Type: Bug > Components: Security >Reporter: Oleksiy Sayankin >Priority: Critical > Labels: CVE > > CVE list: CVE-2019-10172,CVE-2019-10202 > CVSS score: High > {code} > ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[ https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-24904: Component/s: Security > CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar > -- > > Key: HIVE-24904 > URL: https://issues.apache.org/jira/browse/HIVE-24904 > Project: Hive > Issue Type: Bug > Components: Security >Reporter: Oleksiy Sayankin >Priority: Critical > > CVE list: CVE-2019-10172,CVE-2019-10202 > CVSS score: High > {code} > ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24904) CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar
[ https://issues.apache.org/jira/browse/HIVE-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304055#comment-17304055 ] Oleksiy Sayankin commented on HIVE-24904: - The latest supported release of the lib is 1.9.13 ([https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl]) for updating the lib to version with fix we have 3 options: 1. [https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-mapper-asl/1.9.14.jdk17-redhat-1] update to lib that was bundled by RedHat 2. Build our own lib from the master: [https://github.com/FasterXML/jackson-1] 3. Move to new artifact {panel} com.fasterxml.jackson.core » jackson-databind{panel} > CVE-2019-10172,CVE-2019-10202 vulnerabilities in jackson-mapper-asl-1.9.13.jar > -- > > Key: HIVE-24904 > URL: https://issues.apache.org/jira/browse/HIVE-24904 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Priority: Critical > > CVE list: CVE-2019-10172,CVE-2019-10202 > CVSS score: High > {code} > ./packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/lib/jackson-mapper-asl-1.9.13.jar > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-24740: Description: {code} CREATE TABLE t1 (column1 STRING); {code} {code} select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) order by column1; {code} {code} org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table alias or column reference 'column1': (possible column names are: _c0, _c1, .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg (tok_table_or_col column1))) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJU
[jira] [Updated] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-24740: Description: {code} CREATE TABLE t1 (column1 STRING); {code} {code} select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) order by column1; {code} {code} org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table alias or column reference 'column1': (possible column names are: _c0, _c1, .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg (tok_table_or_col column1))) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJU
[jira] [Commented] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279025#comment-17279025 ] Oleksiy Sayankin commented on HIVE-24740: - [~pxiong] could you please take a look here? > Invalid table alias or column reference: Can't order by an unselected column > > > Key: HIVE-24740 > URL: https://issues.apache.org/jira/browse/HIVE-24740 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Pengcheng Xiong >Priority: Blocker > > {code} > CREATE TABLE t1 (column1 STRING); > {code} > {code} > select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) > order by column1; > {code} > {code} > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table > alias or column reference 'column1': (possible column names are: _c0, _c1, > .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg > (tok_table_or_col column1))) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) > at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) > at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) > at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) > at > org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) >
[jira] [Assigned] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-24740: --- Assignee: Pengcheng Xiong (was: Peng Cheng) > Invalid table alias or column reference: Can't order by an unselected column > > > Key: HIVE-24740 > URL: https://issues.apache.org/jira/browse/HIVE-24740 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Pengcheng Xiong >Priority: Blocker > > {code} > CREATE TABLE t1 (column1 STRING); > {code} > {code} > select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) > order by column1; > {code} > {code} > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table > alias or column reference 'column1': (possible column names are: _c0, _c1, > .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg > (tok_table_or_col column1))) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) > at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) > at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) > at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) > at > org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMet
[jira] [Assigned] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-24740: --- Assignee: Peng Cheng > Invalid table alias or column reference: Can't order by an unselected column > > > Key: HIVE-24740 > URL: https://issues.apache.org/jira/browse/HIVE-24740 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Peng Cheng >Priority: Blocker > > {code} > CREATE TABLE t1 (column1 STRING); > {code} > {code} > select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) > order by column1; > {code} > {code} > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table > alias or column reference 'column1': (possible column names are: _c0, _c1, > .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg > (tok_table_or_col column1))) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) > at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) > at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) > at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) > at > org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(Framewo
[jira] [Updated] (HIVE-24740) Invalid table alias or column reference: Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-24740: Summary: Invalid table alias or column reference: Can't order by an unselected column (was: Can't order by an unselected column) > Invalid table alias or column reference: Can't order by an unselected column > > > Key: HIVE-24740 > URL: https://issues.apache.org/jira/browse/HIVE-24740 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Priority: Blocker > > {code} > CREATE TABLE t1 (column1 STRING); > {code} > {code} > select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) > order by column1; > {code} > {code} > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table > alias or column reference 'column1': (possible column names are: _c0, _c1, > .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg > (tok_table_or_col column1))) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) > at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) > at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) > at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) > at > org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at
[jira] [Updated] (HIVE-24740) Can't order by an unselected column
[ https://issues.apache.org/jira/browse/HIVE-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-24740: Priority: Blocker (was: Major) > Can't order by an unselected column > --- > > Key: HIVE-24740 > URL: https://issues.apache.org/jira/browse/HIVE-24740 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Priority: Blocker > > {code} > CREATE TABLE t1 (column1 STRING); > {code} > {code} > select substr(column1,1,4), avg(column1) from t1 group by substr(column1,1,4) > order by column1; > {code} > {code} > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:87 Invalid table > alias or column reference 'column1': (possible column names are: _c0, _c1, > .(tok_function substr (tok_table_or_col column1) 1 4), .(tok_function avg > (tok_table_or_col column1))) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5645) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genAllRexNode(CalcitePlanner.java:5576) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.getOrderByExpression(CalcitePlanner.java:4326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.beginGenOBLogicalPlan(CalcitePlanner.java:4230) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genOBLogicalPlan(CalcitePlanner.java:4136) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5326) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1864) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1810) > at > org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:130) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:915) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:179) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:125) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1571) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:562) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12538) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:456) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:315) > at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) > at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:492) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:445) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:403) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:125) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) > at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:355) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:744) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:714) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:170) > at > org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:62) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-15444: Status: Patch Available (was: In Progress) > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 2.1.1 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch, HIVE-15444.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > {code} > {code} > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > {code} > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254765#comment-17254765 ] Oleksiy Sayankin commented on HIVE-15444: - These are results of verification of [HIVE-14397|https://issues.apache.org/jira/browse/HIVE-14397] with disabled queue name removal via {{hive.server2.tez.unset.tez.queue.name = false}} {code} conf.unset(TezConfiguration.TEZ_QUEUE_NAME) {code} statement. *1.* Configure Hive on Tez. *2.* Configure in {{hive-site.xml}} {code} hive.server2.tez.unset.tez.queue.name false hive.server2.tez.default.queues q1,q2 {code} Here we define two queues for Tez jobs: q1 and q2. We also disable removing Tez queue name after query execution. *3.* Configure in {{tez-site.xml}} {code} tez.session.am.dag.submit.timeout.secs 60 {code} This property allows to set Tez session timeout to 60 seconds. *4.* In console 1: run Beeline, connect to HS2 and set {code} 0: jdbc:hive2://node1.cluster.com:1/defau> set tez.queue.name=q1; {code} *5.* In console 2: run Beeline, connect to HS2 and set {code} 0: jdbc:hive2://node1.cluster.com:1/defau> set tez.queue.name=q2; {code} *6.* In console 1: run any simple query that triggers Tez job {code} 0: jdbc:hive2://node1.cluster.com:1/defau> select sum(id) from test; {code} *7.* In console 2: run any simple query that triggers Tez job {code} 0: jdbc:hive2://node1.cluster.com:1/defau> select sum(id) from test; {code} *8.* See two YARN applications: each for a queue: q1 and q2. {code} [myuser@node1 ~]$ yarn application -list {code} {code} Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):2 Application-Id Application-NameApplication-Type User Queue State Final-State ProgressTracking-URL application_1608718387253_0011 HIVE-e5b114cc-5b31-465e-9204-3b49daf71038 TEZ myuser root.q2 RUNNING UNDEFINED 0% http://node1.cluster.com:46621/ui/ application_1608718387253_0012 HIVE-b307590b-1fe6-445c-bf6b-57e72d88d92d TEZ myuser root.q1 RUNNING UNDEFINED 0% http://node1.cluster.com:42410/ui/ {code} *9.* Wait for 60 seconds ans see that Tez sessions are expired {code} [myuser@node1 ~]$ yarn application -list {code} {code} Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):0 Application-Id Application-NameApplication-Type User Queue State Final-State ProgressTracking-URL [myuser@node1 ~]$ {code} *10.* Run queries from console 1 and 2 and check that there is no 4 YARN applications after that: {code} [myuser@node1 ~]$ yarn application -list {code} {code} Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):2 Application-Id Application-NameApplication-Type User Queue State Final-State ProgressTracking-URL application_1608718387253_0011 HIVE-e5b114cc-5b31-465e-9204-3b49daf71038 TEZ myuser root.q2 RUNNING UNDEFINED 0% http://node1.cluster.com:46621/ui/ application_1608718387253_0012 HIVE-b307590b-1fe6-445c-bf6b-57e72d88d92d TEZ myuser root.q1 RUNNING UNDEFINED 0% http://node1.cluster.com:42410/ui/ {code} *RESOLUTION* We did not observe [HIVE-14397|https://issues.apache.org/jira/browse/HIVE-14397] after applying the fix. > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch, HIVE-15444.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > {code} > {code} > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-15444: Attachment: HIVE-15444.2.patch > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch, HIVE-15444.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > {code} > {code} > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > {code} > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-15444: Description: {code} hive> set tez.queue.name; tez.queue.name is undefined hive> set tez.queue.name=HQ_OLPS; hive> set tez.queue.name; tez.queue.name=HQ_OLPS {code} {code} hive> insert into abc values(2,2); Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1481877998678_0011) -- VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -- Map 1 .. container SUCCEEDED 1 100 0 0 -- VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s -- Loading data to table default.abc OK Time taken: 19.983 seconds {code} {code} hive> set tez.queue.name; tez.queue.name is undefined hive> set hive.execution.engine; hive.execution.engine=tez {code} was: hive> set tez.queue.name; tez.queue.name is undefined hive> set tez.queue.name=HQ_OLPS; hive> set tez.queue.name; tez.queue.name=HQ_OLPS hive> insert into abc values(2,2); Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1481877998678_0011) -- VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -- Map 1 .. container SUCCEEDED 1 100 0 0 -- VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s -- Loading data to table default.abc OK Time taken: 19.983 seconds hive> set tez.queue.name; tez.queue.name is undefined hive> set hive.execution.engine; hive.execution.engine=tez > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > {code} > {code} > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > {code} > {code} > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254762#comment-17254762 ] Oleksiy Sayankin commented on HIVE-15444: - Created https://github.com/apache/hive/pull/1815 > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-15444: --- Assignee: Oleksiy Sayankin (was: Hui Fei) > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-15444: Status: In Progress (was: Patch Available) > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0, 2.1.1 >Reporter: Hui Fei >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-15444.1.patch > > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21489) EXPLAIN command throws ClassCastException in Hive
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247291#comment-17247291 ] Oleksiy Sayankin commented on HIVE-21489: - I've tested the patched and it works for Storage Base Authorization as well. > EXPLAIN command throws ClassCastException in Hive > - > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1)hive.execution.engine is the default value mr > 2)hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3)start hivecli to run command:explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true,the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21489) EXPLAIN command throw ClassCastException in Hive
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21489: Summary: EXPLAIN command throw ClassCastException in Hive (was: explain command throw ClassCastException in 2.3.4) > EXPLAIN command throw ClassCastException in Hive > > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1)hive.execution.engine is the default value mr > 2)hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3)start hivecli to run command:explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true,the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21489) EXPLAIN command throws ClassCastException in Hive
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-21489: Summary: EXPLAIN command throws ClassCastException in Hive (was: EXPLAIN command throw ClassCastException in Hive) > EXPLAIN command throws ClassCastException in Hive > - > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1)hive.execution.engine is the default value mr > 2)hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3)start hivecli to run command:explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true,the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21489) explain command throw ClassCastException in 2.3.4
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247289#comment-17247289 ] Oleksiy Sayankin edited comment on HIVE-21489 at 12/10/20, 2:53 PM: [~Ping Lu] [~daijy] The same error happens when Hive is configured for Storage Base Authorization. {code} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} {code} FAILED: ClassCastException org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer {code} So the patch is valid. Please commit to the master. was (Author: osayankin): The same error happens when Hive is configured for Storage Base Authorization. {code} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} {code} FAILED: ClassCastException org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer {code} So the patch is valid. Please commit to the master. > explain command throw ClassCastException in 2.3.4 > - > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1)hive.execution.engine is the default value mr > 2)hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3)start hivecli to run command:explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true,the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21489) explain command throw ClassCastException in 2.3.4
[ https://issues.apache.org/jira/browse/HIVE-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247289#comment-17247289 ] Oleksiy Sayankin commented on HIVE-21489: - The same error happens when Hive is configured for Storage Base Authorization. {code} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} {code} FAILED: ClassCastException org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer {code} So the patch is valid. Please commit to the master. > explain command throw ClassCastException in 2.3.4 > - > > Key: HIVE-21489 > URL: https://issues.apache.org/jira/browse/HIVE-21489 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Ping Lu >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21489.1.patch, HIVE-21489.2.patch > > > I'm trying to run commands like explain select * from src in hive-2.3.4,but > it falls with the ClassCastException: > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer cannot be cast to > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer > Steps to reproduce: > 1)hive.execution.engine is the default value mr > 2)hive.security.authorization.enabled is set to true, and > hive.security.authorization.manager is set to > org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider > 3)start hivecli to run command:explain select * from src > I debug the code and find the issue HIVE-18778 causing the above > ClassCastException.If I set hive.in.test to true,the explain command can be > successfully executed。 > Now,I have one question,due to hive.in.test cann't be modified at runtime.how > to run explain command with using default authorization in hive-2.3.4, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21843) UNION query with regular expressions for column name does not work
[ https://issues.apache.org/jira/browse/HIVE-21843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241432#comment-17241432 ] Oleksiy Sayankin commented on HIVE-21843: - Thanks > UNION query with regular expressions for column name does not work > -- > > Key: HIVE-21843 > URL: https://issues.apache.org/jira/browse/HIVE-21843 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-21843.1.patch, HIVE-21843.10.patch, > HIVE-21843.4.patch, HIVE-21843.6.patch, HIVE-21843.7.patch, > HIVE-21843.8.patch, HIVE-21843.9.patch > > Time Spent: 40m > Remaining Estimate: 0h > > *STEPS TO REPRODUCE:* > 1. Create a table: > {code:java}CREATE TABLE t (a1 INT, a2 INT); > INSERT INTO TABLE t VALUES (1,1),(1,2),(2,1),(2,2);{code} > 2. SET hive.support.quoted.identifiers to "none": > {code:java}SET hive.support.quoted.identifiers=none;{code} > 3. Run the query: > {code:java}SELECT `(a1)?+.+` FROM t > UNION > SELECT `(a2)?+.+` FROM t;{code} > *ACTUAL RESULT:* > The query fails with an exception: > {code:java}2019-05-23T01:36:53,604 ERROR > [9aa457a9-1c74-466e-abef-ec2f007117f3 main] ql.Driver: FAILED: > SemanticException Line 0:-1 Invalid column reference '`(a1)?+.+`' > org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid column > reference '`(a1)?+.+`' > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11734) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11674) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11642) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11620) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5225) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6330) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9659) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10579) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10457) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:11202) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:481) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11215) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:774) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:697) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:692) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code} > FYI: [~sershe] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[ https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237685#comment-17237685 ] Oleksiy Sayankin commented on HIVE-18728: - [~hunterl] wrote: {quote} Unfortunately there's no way for me to change the PR on github to your authorship, which is why I suggested you open your own PR and I close mine. If you do chose to do this just leave a comment here or on the github link I posted above and I'll be sure to close mine out. {quote} You can easily change the ownership if existing commit is on top of the PR. {code} git commit --amend --author="Name " git push origin +master --force {code} > Secure webHCat with SSL > --- > > Key: HIVE-18728 > URL: https://issues.apache.org/jira/browse/HIVE-18728 > Project: Hive > Issue Type: New Feature > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch, > HIVE-18728.3.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Doc for the issue: > *Configure WebHCat server to use SSL encryption* > You can configure WebHCat REST-API to use SSL (Secure Sockets Layer) > encryption. The following WebHCat properties are added to enable SSL. > {{templeton.use.ssl}} > Default value: {{false}} > Description: Set this to true for using SSL encryption for WebHCat server > {{templeton.keystore.path}} > Default value: {{}} > Description: SSL certificate keystore location for WebHCat server > {{templeton.keystore.password}} > Default value: {{}} > Description: SSL certificate keystore password for WebHCat server > {{templeton.ssl.protocol.blacklist}} > Default value: {{SSLv2,SSLv3}} > Description: SSL Versions to disable for WebHCat server > {{templeton.host}} > Default value: {{0.0.0.0}} > Description: The host address the WebHCat server will listen on. > *Modifying the {{webhcat-site.xml}} file* > Configure the following properties in the {{webhcat-site.xml}} file to enable > SSL encryption on each node where WebHCat is installed: > {code} > > > templeton.use.ssl > true > > > templeton.keystore.path > /path/to/ssl_keystore > > > templeton.keystore.password > password > > {code} > *Example:* To check status of WebHCat server configured for SSL encryption > use following command > {code} > curl -k 'https://:@:50111/templeton/v1/status' > {code} > replace {{}} and {{}} with valid user/password. Replace > {{}} with your host name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[ https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237684#comment-17237684 ] Oleksiy Sayankin commented on HIVE-18728: - Created https://github.com/apache/hive/pull/1699 > Secure webHCat with SSL > --- > > Key: HIVE-18728 > URL: https://issues.apache.org/jira/browse/HIVE-18728 > Project: Hive > Issue Type: New Feature > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch, > HIVE-18728.3.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Doc for the issue: > *Configure WebHCat server to use SSL encryption* > You can configure WebHCat REST-API to use SSL (Secure Sockets Layer) > encryption. The following WebHCat properties are added to enable SSL. > {{templeton.use.ssl}} > Default value: {{false}} > Description: Set this to true for using SSL encryption for WebHCat server > {{templeton.keystore.path}} > Default value: {{}} > Description: SSL certificate keystore location for WebHCat server > {{templeton.keystore.password}} > Default value: {{}} > Description: SSL certificate keystore password for WebHCat server > {{templeton.ssl.protocol.blacklist}} > Default value: {{SSLv2,SSLv3}} > Description: SSL Versions to disable for WebHCat server > {{templeton.host}} > Default value: {{0.0.0.0}} > Description: The host address the WebHCat server will listen on. > *Modifying the {{webhcat-site.xml}} file* > Configure the following properties in the {{webhcat-site.xml}} file to enable > SSL encryption on each node where WebHCat is installed: > {code} > > > templeton.use.ssl > true > > > templeton.keystore.path > /path/to/ssl_keystore > > > templeton.keystore.password > password > > {code} > *Example:* To check status of WebHCat server configured for SSL encryption > use following command > {code} > curl -k 'https://:@:50111/templeton/v1/status' > {code} > replace {{}} and {{}} with valid user/password. Replace > {{}} with your host name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[ https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237553#comment-17237553 ] Oleksiy Sayankin commented on HIVE-18728: - I have no enough permissions to create a PR in Hive so that's why I added the files with patches. If you are able to set me as an author in the PR, then the rest is ok from my side. > Secure webHCat with SSL > --- > > Key: HIVE-18728 > URL: https://issues.apache.org/jira/browse/HIVE-18728 > Project: Hive > Issue Type: New Feature > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Hunter Logan >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch, > HIVE-18728.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Doc for the issue: > *Configure WebHCat server to use SSL encryption* > You can configure WebHCat REST-API to use SSL (Secure Sockets Layer) > encryption. The following WebHCat properties are added to enable SSL. > {{templeton.use.ssl}} > Default value: {{false}} > Description: Set this to true for using SSL encryption for WebHCat server > {{templeton.keystore.path}} > Default value: {{}} > Description: SSL certificate keystore location for WebHCat server > {{templeton.keystore.password}} > Default value: {{}} > Description: SSL certificate keystore password for WebHCat server > {{templeton.ssl.protocol.blacklist}} > Default value: {{SSLv2,SSLv3}} > Description: SSL Versions to disable for WebHCat server > {{templeton.host}} > Default value: {{0.0.0.0}} > Description: The host address the WebHCat server will listen on. > *Modifying the {{webhcat-site.xml}} file* > Configure the following properties in the {{webhcat-site.xml}} file to enable > SSL encryption on each node where WebHCat is installed: > {code} > > > templeton.use.ssl > true > > > templeton.keystore.path > /path/to/ssl_keystore > > > templeton.keystore.password > password > > {code} > *Example:* To check status of WebHCat server configured for SSL encryption > use following command > {code} > curl -k 'https://:@:50111/templeton/v1/status' > {code} > replace {{}} and {{}} with valid user/password. Replace > {{}} with your host name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18728) Secure webHCat with SSL
[ https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237539#comment-17237539 ] Oleksiy Sayankin commented on HIVE-18728: - For some reason my solution was ignored by the community. I am still here and I wish to commit this to Hive > Secure webHCat with SSL > --- > > Key: HIVE-18728 > URL: https://issues.apache.org/jira/browse/HIVE-18728 > Project: Hive > Issue Type: New Feature > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Hunter Logan >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch, > HIVE-18728.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Doc for the issue: > *Configure WebHCat server to use SSL encryption* > You can configure WebHCat REST-API to use SSL (Secure Sockets Layer) > encryption. The following WebHCat properties are added to enable SSL. > {{templeton.use.ssl}} > Default value: {{false}} > Description: Set this to true for using SSL encryption for WebHCat server > {{templeton.keystore.path}} > Default value: {{}} > Description: SSL certificate keystore location for WebHCat server > {{templeton.keystore.password}} > Default value: {{}} > Description: SSL certificate keystore password for WebHCat server > {{templeton.ssl.protocol.blacklist}} > Default value: {{SSLv2,SSLv3}} > Description: SSL Versions to disable for WebHCat server > {{templeton.host}} > Default value: {{0.0.0.0}} > Description: The host address the WebHCat server will listen on. > *Modifying the {{webhcat-site.xml}} file* > Configure the following properties in the {{webhcat-site.xml}} file to enable > SSL encryption on each node where WebHCat is installed: > {code} > > > templeton.use.ssl > true > > > templeton.keystore.path > /path/to/ssl_keystore > > > templeton.keystore.password > password > > {code} > *Example:* To check status of WebHCat server configured for SSL encryption > use following command > {code} > curl -k 'https://:@:50111/templeton/v1/status' > {code} > replace {{}} and {{}} with valid user/password. Replace > {{}} with your host name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Attachment: HIVE-23075.2.patch > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch, HIVE-23075.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Status: Patch Available (was: In Progress) > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch, HIVE-23075.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Status: In Progress (was: Patch Available) > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1700#comment-1700 ] Oleksiy Sayankin commented on HIVE-23075: - *FIXED* *SOLUTION* *1.* Add new property {{hive.ssl.protocol.version}} with default value {{TLSv1.2}}. This is SSL protocol versions for all Hive Servers. This property is set in {{hive-site.xml}} and requires Hive services to be restarted to make the change take effect. *2.* Add logging with SSL version. See example in HiveServer2: {code} 2020-03-23T14:24:57,907 INFO [main] http.HttpServer: Current SSL protocol version is TLSv1.2 2020-03-23T14:24:58,008 INFO [Thread-8] auth.HiveAuthUtils: SSL Server Socket Enabled Protocols: [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] 2020-03-23T14:24:58,008 INFO [Thread-8] thrift.ThriftCLIService: Current SSL protocol version is TLSv1.2 2020-03-23T14:24:58,119 INFO [main] server.AbstractConnector: Started ServerConnector@71978f46{SSL,[ssl, http/1.1]}{0.0.0.0:10002} {code} In webhcat: {code} INFO | 23 Mar 2020 14:25:03,363 | org.apache.hive.hcatalog.templeton.Main | Using SSL for templeton. INFO | 23 Mar 2020 14:25:03,641 | org.apache.hive.hcatalog.templeton.Main | Current SSL protocol version is TLSv1.2 {code} *EFFECTS* 1. JDBC SSL connection 2. WebHCat SSL conection > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Attachment: HIVE-23075.1.patch > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-23075: Status: Patch Available (was: In Progress) > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-23075.1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23075 started by Oleksiy Sayankin. --- > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23075) Add property for manual configuration of SSL version
[ https://issues.apache.org/jira/browse/HIVE-23075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-23075: --- > Add property for manual configuration of SSL version > > > Key: HIVE-23075 > URL: https://issues.apache.org/jira/browse/HIVE-23075 > Project: Hive > Issue Type: Improvement > Components: Security >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > Add property for manual configuration of SSL version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Status: Patch Available (was: In Progress) > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch, > HIVE-22980.3.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Status: In Progress (was: Patch Available) > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch, > HIVE-22980.3.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Attachment: HIVE-22980.3.patch > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch, > HIVE-22980.3.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Attachment: HIVE-22980.2.patch > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Status: In Progress (was: Patch Available) > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Status: Patch Available (was: In Progress) > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052071#comment-17052071 ] Oleksiy Sayankin commented on HIVE-22980: - *FIXED* *SOLUTION* Add processing of {{mapreduce.input.pathFilter.class}} property in {{AcidUtils}} for ORC tables. E.g. to enable custom filter: \\ \\ 1. Implement {{CustomPathFilter}}: {code} class CustomPathFilter implements PathFilter{ @Override public boolean accept(Path path) { String name = path.getName(); return name.startsWith("a"); } } {code} 2. Add {{CustomPathFilter}} to configuration. {code} PathFilter customPathFilter = new CustomPathFilter(); Configuration conf = new Configuration(); conf.setClass("mapreduce.input.pathFilter.class", customPathFilter.getClass(), PathFilter.class); {code} 3. Pass {{Configuration}} to Hive: {code} AcidUtils.Directory dir = AcidUtils.getAcidState(fs, new MockPath(fs, "/tbl/part1"), conf, new ValidReaderWriteIdList("tbl:100:" + Long.MAX_VALUE + ":"), null, false, null, false); {code} *EFFECTS* ORC processing. > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Status: Patch Available (was: In Progress) > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22980: Attachment: HIVE-22980.1.patch > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22980.1.patch > > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22980 started by Oleksiy Sayankin. --- > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22980) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-22980: --- > Support custom path filter for ORC tables > - > > Key: HIVE-22980 > URL: https://issues.apache.org/jira/browse/HIVE-22980 > Project: Hive > Issue Type: New Feature > Components: ORC >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > > The customer is looking for an option to specify custom path filter for ORC > tables. Please find the details below from customer requirement. > Problem Statement/Approach in customer words : > {quote} > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > On local testing, mapreduce.input.pathfilter.class seems to be working for > Text tables but not for ORC tables. > {quote} > Our analysis: > {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for > {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is > only respected by {{FileInputFormat}}, but not by any other implementations > of {{InputFormat}}. The customer wants to have the ability to filter out rows > based on path/filenames, current ORC features like bloomfilters and indexes > are not good enough for them to minimize number of disk read operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: In Progress (was: Patch Available) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch, HIVE-22919.6.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Attachment: HIVE-22919.6.patch > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch, HIVE-22919.6.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: Patch Available (was: In Progress) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch, HIVE-22919.6.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: In Progress (was: Patch Available) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: Patch Available (was: In Progress) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Attachment: HIVE-22919.5.patch > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch, HIVE-22919.5.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Attachment: HIVE-22919.4.patch > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: Patch Available (was: In Progress) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch, HIVE-22919.4.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: In Progress (was: Patch Available) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041814#comment-17041814 ] Oleksiy Sayankin edited comment on HIVE-22919 at 2/25/20 10:39 AM: --- *FIXED* *ROOT-CAUSE* The root-cause of the issue does not relate to Storage Base Authorization, it is about correct update of Hive variable {{hive.metastore.warehouse.dir}}. As one can see from the exception: {code} FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db {code} Hive wants to create new database at {{hdfs:/tmp/m2}} despite the operator {{SET}} that was executed before {code} SET hive.metastore.warehouse.dir=/tmp/m3; {code} This happens because {{StorageBasedAuthorizationProvider}} has an instance of {{Warehouse}} object that has value for {{hive.metastore.warehouse.dir}}. When a user updates the {{hive.metastore.warehouse.dir}} in {{Configuration}} instance, this action does not force {{Warehouse}} object to refresh the value of {{hive.metastore.warehouse.dir}} and hence it has the old one. *SOLUTION* Add {{isWarehouseChanged()}} method to check whether {{hive.metastore.warehouse.dir}} has been changed and recreate {{Warehouse}} in {{StorageBasedAuthorizationProvider}} if yes. *EFFECTS* {{StorageBasedAuthorizationProvider}} initialization. was (Author: osayankin): *FIXED* *ROOT-CAUSE* The root-cause of the issue does not relate to Storage Base Authorization, it is about correct update of Hive variable {{hive.metastore.warehouse.dir}}. As one can see from the exception: {code} FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db {code} Hive wants to create new database at {{hdfs:/tmp/m2}} despite the operator {{SET}} that was executed before {code} SET hive.metastore.warehouse.dir=/tmp/m3; {code} This happens because {{StorageBasedAuthorizationProvider}} has an instance of {{Warehouse}} object that has value for {{hive.metastore.warehouse.dir}}. When a user updates the {{hive.metastore.warehouse.dir}} in {{HiveConf}} instance, this action does not force {{Warehouse}} object to refresh the value of {{hive.metastore.warehouse.dir}} and hence it has the old one. *SOLUTION* Add {{isWarehouseChanged()}} method to check whether {{hive.metastore.warehouse.dir}} has been changed and recreate {{Warehouse}} in {{StorageBasedAuthorizationProvider}} if yes. *EFFECTS* {{StorageBasedAuthorizationProvider}} initialization. > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: Patch Available (was: In Progress) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Attachment: HIVE-22919.3.patch > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch, > HIVE-22919.3.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: In Progress (was: Patch Available) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043449#comment-17043449 ] Oleksiy Sayankin commented on HIVE-22919: - Hi [~kgyrtkirk]. I understand your idea. This line {code} hive [...] --hiveconf hive.metastore.warehouse.dir=/tmp/m2 {code} is only an example. Customer wants to do following in CLI: {code} SET hive.metastore.warehouse.dir=/tmp/path1; SET hive.metastore.warehouse.dir=/tmp/path2; SET hive.metastore.warehouse.dir=/tmp/path3; ... -- some other value for hive.metastore.warehouse.dir {code} and so on in one CLI session. And this is real use-case from real customer and they ask me to fix it. > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: In Progress (was: Patch Available) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: Patch Available (was: In Progress) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Attachment: HIVE-22919.2.patch > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch, HIVE-22919.2.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041814#comment-17041814 ] Oleksiy Sayankin commented on HIVE-22919: - *FIXED* *ROOT-CAUSE* The root-cause of the issue does not relate to Storage Base Authorization, it is about correct update of Hive variable {{hive.metastore.warehouse.dir}}. As one can see from the exception: {code} FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db {code} Hive wants to create new database at {{hdfs:/tmp/m2}} despite the operator {{SET}} that was executed before {code} SET hive.metastore.warehouse.dir=/tmp/m3; {code} This happens because {{StorageBasedAuthorizationProvider}} has an instance of {{Warehouse}} object that has value for {{hive.metastore.warehouse.dir}}. When a user updates the {{hive.metastore.warehouse.dir}} in {{HiveConf}} instance, this action does not force {{Warehouse}} object to refresh the value of {{hive.metastore.warehouse.dir}} and hence it has the old one. *SOLUTION* Add {{isWarehouseChanged()}} method to check whether {{hive.metastore.warehouse.dir}} has been changed and recreate {{Warehouse}} in {{StorageBasedAuthorizationProvider}} if yes. *EFFECTS* {{StorageBasedAuthorizationProvider}} initialization. > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Status: Patch Available (was: In Progress) > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Attachment: HIVE-22919.1.patch > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-22919.1.patch > > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]$ > {code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java} > sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1 > {code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3; > {code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java} > hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive> > {code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Description: *ENVIRONMENT:* Hive-2.3 *STEPS TO REPRODUCE:* 1. Configure Storage Based Authorization: {code:xml} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} 2. Create a few directories, change owners and permissions to it: {code:java}hadoop fs -mkdir /tmp/m1 hadoop fs -mkdir /tmp/m2 hadoop fs -mkdir /tmp/m3 hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] hadoop fs -chmod 700 /tmp/m[1-3]{code} 3. Check permissions: {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 [test@node2 ~]$ {code} 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: {code:java} sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf hive.metastore.warehouse.dir=/tmp/m1 {code} 5. Perform the next steps: {code:sql }-- 1. Check "hive.metastore.warehouse.dir" value: SET hive.metastore.warehouse.dir; -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user does not have an access: SET hive.metastore.warehouse.dir=/tmp/m2; -- 3. Try to create a database: CREATE DATABASE m2; -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user has an access: SET hive.metastore.warehouse.dir=/tmp/m3; -- 5. Try to create a database: CREATE DATABASE m3; {code} *ACTUAL RESULT:* Query 5 fails with an exception below. It does not handle "hive.metastore.warehouse.dir" proprty: {code:java} hive> -- 5. Try to create a database: hive> CREATE DATABASE m3; FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db hive> {code} *EXPECTED RESULT:* Query 5 creates a database; was: *ENVIRONMENT:* Hive-2.3 *STEPS TO REPRODUCE:* 1. Configure Storage Based Authorization: {code:xml} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} 2. Create a few directories, change owners and permissions to it: {code:java}hadoop fs -mkdir /tmp/m1 hadoop fs -mkdir /tmp/m2 hadoop fs -mkdir /tmp/m3 hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] hadoop fs -chmod 700 /tmp/m[1-3]{code} 3. Check permissions: {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 [test@node2 ~]${code} 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: {code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf hive.metastore.warehouse.dir=/tmp/m1{code} 5. Perform the next steps: {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: SET hive.metastore.warehouse.dir; -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user does not have an access: SET hive.metastore.warehouse.dir=/tmp/m2; -- 3. Try to create a database: CREATE DATABASE m2; -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user has an access: SET hive.metastore.warehouse.dir=/tmp/m3; -- 5. Try to create a database: CREATE DATABASE m3;{code} *ACTUAL RESULT:* Query 5 fails with an exception below. It does not handle "hive.metastore.warehouse.dir" proprty: {code:java}hive> -- 5. Try to create a database: hive> CREATE DATABASE m3; FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db hive>{code} *EXPECTED RE
[jira] [Updated] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22919: Description: *ENVIRONMENT:* Hive-2.3 *STEPS TO REPRODUCE:* 1. Configure Storage Based Authorization: {code:xml} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} 2. Create a few directories, change owners and permissions to it: {code:java}hadoop fs -mkdir /tmp/m1 hadoop fs -mkdir /tmp/m2 hadoop fs -mkdir /tmp/m3 hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] hadoop fs -chmod 700 /tmp/m[1-3]{code} 3. Check permissions: {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 [test@node2 ~]$ {code} 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: {code:java} sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf hive.metastore.warehouse.dir=/tmp/m1 {code} 5. Perform the next steps: {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: SET hive.metastore.warehouse.dir; -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user does not have an access: SET hive.metastore.warehouse.dir=/tmp/m2; -- 3. Try to create a database: CREATE DATABASE m2; -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user has an access: SET hive.metastore.warehouse.dir=/tmp/m3; -- 5. Try to create a database: CREATE DATABASE m3; {code} *ACTUAL RESULT:* Query 5 fails with an exception below. It does not handle "hive.metastore.warehouse.dir" proprty: {code:java} hive> -- 5. Try to create a database: hive> CREATE DATABASE m3; FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db hive> {code} *EXPECTED RESULT:* Query 5 creates a database; was: *ENVIRONMENT:* Hive-2.3 *STEPS TO REPRODUCE:* 1. Configure Storage Based Authorization: {code:xml} hive.security.authorization.enabled true hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator hive.metastore.pre.event.listeners org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener {code} 2. Create a few directories, change owners and permissions to it: {code:java}hadoop fs -mkdir /tmp/m1 hadoop fs -mkdir /tmp/m2 hadoop fs -mkdir /tmp/m3 hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] hadoop fs -chmod 700 /tmp/m[1-3]{code} 3. Check permissions: {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 [test@node2 ~]$ {code} 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: {code:java} sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf hive.metastore.warehouse.dir=/tmp/m1 {code} 5. Perform the next steps: {code:sql }-- 1. Check "hive.metastore.warehouse.dir" value: SET hive.metastore.warehouse.dir; -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user does not have an access: SET hive.metastore.warehouse.dir=/tmp/m2; -- 3. Try to create a database: CREATE DATABASE m2; -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user has an access: SET hive.metastore.warehouse.dir=/tmp/m3; -- 5. Try to create a database: CREATE DATABASE m3; {code} *ACTUAL RESULT:* Query 5 fails with an exception below. It does not handle "hive.metastore.warehouse.dir" proprty: {code:java} hive> -- 5. Try to create a database: hive> CREATE DATABASE m3; FAILED: HiveException org.apache.hadoop.security.AccessControlException: User testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db hive> {code} *EXPEC
[jira] [Work started] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22919 started by Oleksiy Sayankin. --- > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]${code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1{code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3;{code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java}hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive>{code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/HIVE-22919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-22919: --- > StorageBasedAuthorizationProvider does not allow create databases after > changing hive.metastore.warehouse.dir > - > > Key: HIVE-22919 > URL: https://issues.apache.org/jira/browse/HIVE-22919 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > > *ENVIRONMENT:* > Hive-2.3 > *STEPS TO REPRODUCE:* > 1. Configure Storage Based Authorization: > {code:xml} > hive.security.authorization.enabled > true > > > hive.security.metastore.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.authorization.manager > > org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider > > > hive.security.metastore.authenticator.manager > > org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator > > > hive.metastore.pre.event.listeners > > org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener > {code} > 2. Create a few directories, change owners and permissions to it: > {code:java}hadoop fs -mkdir /tmp/m1 > hadoop fs -mkdir /tmp/m2 > hadoop fs -mkdir /tmp/m3 > hadoop fs -chown testuser1:testuser1 /tmp/m[1,3] > hadoop fs -chmod 700 /tmp/m[1-3]{code} > 3. Check permissions: > {code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3] > drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1 > drwx-- - test test 0 2020-02-11 10:25 /tmp/m2 > drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3 > [test@node2 ~]${code} > 4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user, > with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*: > {code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf > hive.metastore.warehouse.dir=/tmp/m1{code} > 5. Perform the next steps: > {code:sql}-- 1. Check "hive.metastore.warehouse.dir" value: > SET hive.metastore.warehouse.dir; > -- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user does not have an access: > SET hive.metastore.warehouse.dir=/tmp/m2; > -- 3. Try to create a database: > CREATE DATABASE m2; > -- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" > user has an access: > SET hive.metastore.warehouse.dir=/tmp/m3; > -- 5. Try to create a database: > CREATE DATABASE m3;{code} > *ACTUAL RESULT:* > Query 5 fails with an exception below. It does not handle > "hive.metastore.warehouse.dir" proprty: > {code:java}hive> -- 5. Try to create a database: > hive> CREATE DATABASE m3; > FAILED: HiveException org.apache.hadoop.security.AccessControlException: User > testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db > hive>{code} > *EXPECTED RESULT:* > Query 5 creates a database; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Description: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | db1 | table3 | 9315 | … {code} For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive session is waiting for 22328 read locks. This is because this table “db1. table1” is a huge partition table, and it has more than 200k child partitions. I am guessing each of Hive session was trying to do a full table scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list: {code} mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as dt,count(*) as cnt from HIVE_LOCKS -> group by 1 order by 1; +++ | dt | cnt| +++ | 1969-12-31 | 2 | | 2019-05-20 | 10 | | 2019-05-21 | 3 | | 2019-05-23 | 5 | | 2019-05-24 | 2 | | 2019-05-25 | 1 | | 2019-05-29 | 7 | | 2019-05-30 | 2 | | 2019-06-11 | 13 | | 2019-06-28 | 3 | | 2019-07-02 | 2 | | 2019-07-04 | 5 | | 2019-07-09 | 1 | | 2019-07-15 | 2 | | 2019-07-16 | 1 | | 2019-07-18 | 2 | | 2019-07-20 | 3 | | 2019-07-29 | 5 | | 2019-07-30 | 9 | | 2019-07-31 | 7 | | 2019-08-02 | 2 | | 2019-08-06 | 5 | | 2019-08-07 | 17 | | 2019-08-08 | 8 | | 2019-08-09 | 5 | | 2019-08-21 | 1 | | 2019-08-22 | 20 | | 2019-08-23 | 1 | | 2019-08-26 | 5 | | 2019-08-27 | 98 | | 2019-08-28 | 3 | | 2019-08-29 | 1 | | 2019-09-02 | 3 | | 2019-09-04 | 3 | | 2019-09-05 |105 | | 2019-09-06 | 3 | | 2019-09-07 | 2 | | 2019-09-09 | 6 | | 2019-09-12 | 9 | | 2019-09-13 | 1 | | 2019-09-17 | 1 | | 2019-09-24 | 3 | | 2019-09-26 | 6 | | 2019-09-27 | 4 | | 2019-09-30 | 1 | | 2019-10-01 | 2 | | 2019-10-03 | 9 | | 2019-10-04 | 2 | | 2019-10-06 | 1 | | 2019-10-08 | 1 | | 2019-10-09 | 1 | | 2019-10-10 | 6 | | 2019-10-11 | 1 | | 2019-10-16 | 13 | | 2019-10-17 | 1 | | 2019-10-18 | 2 | | 2019-10-19 | 2 | | 2019-10-21 | 10 | | 2019-10-22 | 6 | | 2019-10-28 | 2 | | 2019-10-29 | 4 | | 2019-10-30 | 2 | | 2019-10-31 | 2 | | 2019-11-05 | 2 | | 2019-11-06 | 2 | | 2019-11-11 | 1 | | 2019-11-13 | 1 | | 2019-11-14 | 1 | | 2019-11-21 | 4 | | 2019-11-26 | 1 | | 2019-11-27 | 1 | | 2019-12-05 | 4 | | 2019-12-06 | 2 | | 2019-12-12 | 1 | | 2019-12-14 | 1 | | 2019-12-15 | 3 | | 2019-12-16 | 1 | | 2019-12-17 | 1 | | 2019-12-18 | 1 | | 2019-12-19 | 2 | | 2019-12-20 | 2 | | 2019-12-23 | 1 | | 2019-12-27 | 1 | | 2020-01-07 | 1 | | 2020-01-08 | 14 | | 2020-01-09 | 2 | | 2020-01-12 |372 | | 2020-01-14 | 2 | | 2020-01-15 | 1 | | 2020-01-20 | 11 | | 2020-01-21 | 119253 | | 2020-01-23 |113 | | 2020-01-24 | 4 | | 2020-01-25 |536 | | 2020-01-26 | 2132 | | 2020-01-27 |396 | | 2020-01-28 | 1 | | 2020-01-29 | 3 | | 2020-01-30 | 11 | | 2020-01-31 | 11 | | 2020-02-03 | 2 | | 2020-02-04 | 4 | | 2020-02-05 | 5 | | 2020-02-06 | 8 | | 2020-02-10 | 32 | | 2020-02-11 | 15 | | 2020-02-12 | 14 | | 2020-02-13 | 1 | | 2020-02-14 | 92 | +++ 109 rows in set (0.16 sec) {code} However most of the entries have {{HL_ACQUIRED_AT=NULL}} in {{HIVE_LOCKS}}: {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null; +--+ | COUNT(*) | +--+ | 123437 | +--+ 1 row in set (0.04 sec) {code} {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null; +--+ | COUNT(*) | +--+ | 97 | +--+ 1 row in set (0.04 sec) {code} Hot-fix is to remove orphan/lost locks from {{HIVE_LOCKS}}, but this does not solve the problem in the future. was: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Description: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | db1 | table3 | 9315 | … {code} For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive session is waiting for 22328 read locks. This is because this table “db1. table1” is a huge partition table, and it has more than 200k child partitions. I am guessing each of Hive session was trying to do a full table scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list: {code} mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as dt,count(*) as cnt from HIVE_LOCKS -> group by 1 order by 1; +++ | dt | cnt| +++ | 1969-12-31 | 2 | | 2019-05-20 | 10 | | 2019-05-21 | 3 | | 2019-05-23 | 5 | | 2019-05-24 | 2 | | 2019-05-25 | 1 | | 2019-05-29 | 7 | | 2019-05-30 | 2 | | 2019-06-11 | 13 | | 2019-06-28 | 3 | | 2019-07-02 | 2 | | 2019-07-04 | 5 | | 2019-07-09 | 1 | | 2019-07-15 | 2 | | 2019-07-16 | 1 | | 2019-07-18 | 2 | | 2019-07-20 | 3 | | 2019-07-29 | 5 | | 2019-07-30 | 9 | | 2019-07-31 | 7 | | 2019-08-02 | 2 | | 2019-08-06 | 5 | | 2019-08-07 | 17 | | 2019-08-08 | 8 | | 2019-08-09 | 5 | | 2019-08-21 | 1 | | 2019-08-22 | 20 | | 2019-08-23 | 1 | | 2019-08-26 | 5 | | 2019-08-27 | 98 | | 2019-08-28 | 3 | | 2019-08-29 | 1 | | 2019-09-02 | 3 | | 2019-09-04 | 3 | | 2019-09-05 |105 | | 2019-09-06 | 3 | | 2019-09-07 | 2 | | 2019-09-09 | 6 | | 2019-09-12 | 9 | | 2019-09-13 | 1 | | 2019-09-17 | 1 | | 2019-09-24 | 3 | | 2019-09-26 | 6 | | 2019-09-27 | 4 | | 2019-09-30 | 1 | | 2019-10-01 | 2 | | 2019-10-03 | 9 | | 2019-10-04 | 2 | | 2019-10-06 | 1 | | 2019-10-08 | 1 | | 2019-10-09 | 1 | | 2019-10-10 | 6 | | 2019-10-11 | 1 | | 2019-10-16 | 13 | | 2019-10-17 | 1 | | 2019-10-18 | 2 | | 2019-10-19 | 2 | | 2019-10-21 | 10 | | 2019-10-22 | 6 | | 2019-10-28 | 2 | | 2019-10-29 | 4 | | 2019-10-30 | 2 | | 2019-10-31 | 2 | | 2019-11-05 | 2 | | 2019-11-06 | 2 | | 2019-11-11 | 1 | | 2019-11-13 | 1 | | 2019-11-14 | 1 | | 2019-11-21 | 4 | | 2019-11-26 | 1 | | 2019-11-27 | 1 | | 2019-12-05 | 4 | | 2019-12-06 | 2 | | 2019-12-12 | 1 | | 2019-12-14 | 1 | | 2019-12-15 | 3 | | 2019-12-16 | 1 | | 2019-12-17 | 1 | | 2019-12-18 | 1 | | 2019-12-19 | 2 | | 2019-12-20 | 2 | | 2019-12-23 | 1 | | 2019-12-27 | 1 | | 2020-01-07 | 1 | | 2020-01-08 | 14 | | 2020-01-09 | 2 | | 2020-01-12 |372 | | 2020-01-14 | 2 | | 2020-01-15 | 1 | | 2020-01-20 | 11 | | 2020-01-21 | 119253 | | 2020-01-23 |113 | | 2020-01-24 | 4 | | 2020-01-25 |536 | | 2020-01-26 | 2132 | | 2020-01-27 |396 | | 2020-01-28 | 1 | | 2020-01-29 | 3 | | 2020-01-30 | 11 | | 2020-01-31 | 11 | | 2020-02-03 | 2 | | 2020-02-04 | 4 | | 2020-02-05 | 5 | | 2020-02-06 | 8 | | 2020-02-10 | 32 | | 2020-02-11 | 15 | | 2020-02-12 | 14 | | 2020-02-13 | 1 | | 2020-02-14 | 92 | +++ 109 rows in set (0.16 sec) {code} However most of the entries are {{HL_ACQUIRED_AT=NULL}} in {{HIVE_LOCKS}}: {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null; +--+ | COUNT(*) | +--+ | 123437 | +--+ 1 row in set (0.04 sec) {code} {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null; +--+ | COUNT(*) | +--+ | 97 | +--+ 1 row in set (0.04 sec) {code} was: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | d
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Description: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | db1 | table3 | 9315 | … {code} For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive session is waiting for 22328 read locks. This is because this table “db1. table1” is a huge partition table, and it has more than 200k child partitions. I am guessing each of Hive session was trying to do a full table scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list: {code} mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as dt,count(*) as cnt from HIVE_LOCKS -> group by 1 order by 1; +++ | dt | cnt| +++ | 1969-12-31 | 2 | | 2019-05-20 | 10 | | 2019-05-21 | 3 | | 2019-05-23 | 5 | | 2019-05-24 | 2 | | 2019-05-25 | 1 | | 2019-05-29 | 7 | | 2019-05-30 | 2 | | 2019-06-11 | 13 | | 2019-06-28 | 3 | | 2019-07-02 | 2 | | 2019-07-04 | 5 | | 2019-07-09 | 1 | | 2019-07-15 | 2 | | 2019-07-16 | 1 | | 2019-07-18 | 2 | | 2019-07-20 | 3 | | 2019-07-29 | 5 | | 2019-07-30 | 9 | | 2019-07-31 | 7 | | 2019-08-02 | 2 | | 2019-08-06 | 5 | | 2019-08-07 | 17 | | 2019-08-08 | 8 | | 2019-08-09 | 5 | | 2019-08-21 | 1 | | 2019-08-22 | 20 | | 2019-08-23 | 1 | | 2019-08-26 | 5 | | 2019-08-27 | 98 | | 2019-08-28 | 3 | | 2019-08-29 | 1 | | 2019-09-02 | 3 | | 2019-09-04 | 3 | | 2019-09-05 |105 | | 2019-09-06 | 3 | | 2019-09-07 | 2 | | 2019-09-09 | 6 | | 2019-09-12 | 9 | | 2019-09-13 | 1 | | 2019-09-17 | 1 | | 2019-09-24 | 3 | | 2019-09-26 | 6 | | 2019-09-27 | 4 | | 2019-09-30 | 1 | | 2019-10-01 | 2 | | 2019-10-03 | 9 | | 2019-10-04 | 2 | | 2019-10-06 | 1 | | 2019-10-08 | 1 | | 2019-10-09 | 1 | | 2019-10-10 | 6 | | 2019-10-11 | 1 | | 2019-10-16 | 13 | | 2019-10-17 | 1 | | 2019-10-18 | 2 | | 2019-10-19 | 2 | | 2019-10-21 | 10 | | 2019-10-22 | 6 | | 2019-10-28 | 2 | | 2019-10-29 | 4 | | 2019-10-30 | 2 | | 2019-10-31 | 2 | | 2019-11-05 | 2 | | 2019-11-06 | 2 | | 2019-11-11 | 1 | | 2019-11-13 | 1 | | 2019-11-14 | 1 | | 2019-11-21 | 4 | | 2019-11-26 | 1 | | 2019-11-27 | 1 | | 2019-12-05 | 4 | | 2019-12-06 | 2 | | 2019-12-12 | 1 | | 2019-12-14 | 1 | | 2019-12-15 | 3 | | 2019-12-16 | 1 | | 2019-12-17 | 1 | | 2019-12-18 | 1 | | 2019-12-19 | 2 | | 2019-12-20 | 2 | | 2019-12-23 | 1 | | 2019-12-27 | 1 | | 2020-01-07 | 1 | | 2020-01-08 | 14 | | 2020-01-09 | 2 | | 2020-01-12 |372 | | 2020-01-14 | 2 | | 2020-01-15 | 1 | | 2020-01-20 | 11 | | 2020-01-21 | 119253 | | 2020-01-23 |113 | | 2020-01-24 | 4 | | 2020-01-25 |536 | | 2020-01-26 | 2132 | | 2020-01-27 |396 | | 2020-01-28 | 1 | | 2020-01-29 | 3 | | 2020-01-30 | 11 | | 2020-01-31 | 11 | | 2020-02-03 | 2 | | 2020-02-04 | 4 | | 2020-02-05 | 5 | | 2020-02-06 | 8 | | 2020-02-10 | 32 | | 2020-02-11 | 15 | | 2020-02-12 | 14 | | 2020-02-13 | 1 | | 2020-02-14 | 92 | +++ 109 rows in set (0.16 sec) {code} The delete sql on {{HIVE_LOCKS}} is based on column {{HL_ACQUIRED_AT}}, however most of the entries are {{NULL}}: {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null; +--+ | COUNT(*) | +--+ | 123437 | +--+ 1 row in set (0.04 sec) {code} {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null; +--+ | COUNT(*) | +--+ | 97 | +--+ 1 row in set (0.04 sec) {code} So after deleting, was: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+-
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Description: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | db1 | table3 | 9315 | … {code} For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive session is waiting for 22328 read locks. This is because this table “db1. table1” is a huge partition table, and it has more than 200k child partitions. I am guessing each of Hive session was trying to do a full table scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list: {code} mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as dt,count(*) as cnt from HIVE_LOCKS -> group by 1 order by 1; +++ | dt | cnt| +++ | 1969-12-31 | 2 | | 2019-05-20 | 10 | | 2019-05-21 | 3 | | 2019-05-23 | 5 | | 2019-05-24 | 2 | | 2019-05-25 | 1 | | 2019-05-29 | 7 | | 2019-05-30 | 2 | | 2019-06-11 | 13 | | 2019-06-28 | 3 | | 2019-07-02 | 2 | | 2019-07-04 | 5 | | 2019-07-09 | 1 | | 2019-07-15 | 2 | | 2019-07-16 | 1 | | 2019-07-18 | 2 | | 2019-07-20 | 3 | | 2019-07-29 | 5 | | 2019-07-30 | 9 | | 2019-07-31 | 7 | | 2019-08-02 | 2 | | 2019-08-06 | 5 | | 2019-08-07 | 17 | | 2019-08-08 | 8 | | 2019-08-09 | 5 | | 2019-08-21 | 1 | | 2019-08-22 | 20 | | 2019-08-23 | 1 | | 2019-08-26 | 5 | | 2019-08-27 | 98 | | 2019-08-28 | 3 | | 2019-08-29 | 1 | | 2019-09-02 | 3 | | 2019-09-04 | 3 | | 2019-09-05 |105 | | 2019-09-06 | 3 | | 2019-09-07 | 2 | | 2019-09-09 | 6 | | 2019-09-12 | 9 | | 2019-09-13 | 1 | | 2019-09-17 | 1 | | 2019-09-24 | 3 | | 2019-09-26 | 6 | | 2019-09-27 | 4 | | 2019-09-30 | 1 | | 2019-10-01 | 2 | | 2019-10-03 | 9 | | 2019-10-04 | 2 | | 2019-10-06 | 1 | | 2019-10-08 | 1 | | 2019-10-09 | 1 | | 2019-10-10 | 6 | | 2019-10-11 | 1 | | 2019-10-16 | 13 | | 2019-10-17 | 1 | | 2019-10-18 | 2 | | 2019-10-19 | 2 | | 2019-10-21 | 10 | | 2019-10-22 | 6 | | 2019-10-28 | 2 | | 2019-10-29 | 4 | | 2019-10-30 | 2 | | 2019-10-31 | 2 | | 2019-11-05 | 2 | | 2019-11-06 | 2 | | 2019-11-11 | 1 | | 2019-11-13 | 1 | | 2019-11-14 | 1 | | 2019-11-21 | 4 | | 2019-11-26 | 1 | | 2019-11-27 | 1 | | 2019-12-05 | 4 | | 2019-12-06 | 2 | | 2019-12-12 | 1 | | 2019-12-14 | 1 | | 2019-12-15 | 3 | | 2019-12-16 | 1 | | 2019-12-17 | 1 | | 2019-12-18 | 1 | | 2019-12-19 | 2 | | 2019-12-20 | 2 | | 2019-12-23 | 1 | | 2019-12-27 | 1 | | 2020-01-07 | 1 | | 2020-01-08 | 14 | | 2020-01-09 | 2 | | 2020-01-12 |372 | | 2020-01-14 | 2 | | 2020-01-15 | 1 | | 2020-01-20 | 11 | | 2020-01-21 | 119253 | | 2020-01-23 |113 | | 2020-01-24 | 4 | | 2020-01-25 |536 | | 2020-01-26 | 2132 | | 2020-01-27 |396 | | 2020-01-28 | 1 | | 2020-01-29 | 3 | | 2020-01-30 | 11 | | 2020-01-31 | 11 | | 2020-02-03 | 2 | | 2020-02-04 | 4 | | 2020-02-05 | 5 | | 2020-02-06 | 8 | | 2020-02-10 | 32 | | 2020-02-11 | 15 | | 2020-02-12 | 14 | | 2020-02-13 | 1 | | 2020-02-14 | 92 | +++ 109 rows in set (0.16 sec) {code} The delete sql on {{HIVE_LOCKS}} is based on column {{HL_ACQUIRED_AT}}, however most of the entries are {{NULL}}: {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is null; +--+ | COUNT(*) | +--+ | 123437 | +--+ 1 row in set (0.04 sec) {code} {code} mysql> SELECT COUNT(*) FROM HIVE_LOCKS WHERE HL_ACQUIRED_AT is not null; +--+ | COUNT(*) | +--+ | 97 | +--+ 1 row in set (0.04 sec) {code} was: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Environment: Hive-2.3. > Locks entries are left over inside HIVE_LOCKS when using DbTxnManager > - > > Key: HIVE-22911 > URL: https://issues.apache.org/jira/browse/HIVE-22911 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions > Environment: Hive-2.3. >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Critical > > We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. > There are more than 120k locks in HIVE_LOCKS of MySQL database. We also > checked the top 3 tables which are related to the existing locks: > > {code} > mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by > 3 desc limit 10; > +---+--+--+ > | HL_DB | HL_TABLE | count(*) | > +---+--+--+ > | db1 | table1 | 66984 | > | db1 | table2 | 33208 | > | db1 | table3 | 9315 | > … > {code} > For table “db1. table1”, here are 3 Hive sessions related, and each of the > Hive session is waiting for 22328 read locks. This is because this table > “db1. table1” is a huge partition table, and it has more than 200k child > partitions. I am guessing each of Hive session was trying to do a full table > scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is > the list: > > {code} > mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as > dt,count(*) as cnt from HIVE_LOCKS > -> group by 1 order by 1; > +++ > | dt | cnt| > +++ > | 1969-12-31 | 2 | > | 2019-05-20 | 10 | > | 2019-05-21 | 3 | > | 2019-05-23 | 5 | > | 2019-05-24 | 2 | > | 2019-05-25 | 1 | > | 2019-05-29 | 7 | > | 2019-05-30 | 2 | > | 2019-06-11 | 13 | > | 2019-06-28 | 3 | > | 2019-07-02 | 2 | > | 2019-07-04 | 5 | > | 2019-07-09 | 1 | > | 2019-07-15 | 2 | > | 2019-07-16 | 1 | > | 2019-07-18 | 2 | > | 2019-07-20 | 3 | > | 2019-07-29 | 5 | > | 2019-07-30 | 9 | > | 2019-07-31 | 7 | > | 2019-08-02 | 2 | > | 2019-08-06 | 5 | > | 2019-08-07 | 17 | > | 2019-08-08 | 8 | > | 2019-08-09 | 5 | > | 2019-08-21 | 1 | > | 2019-08-22 | 20 | > | 2019-08-23 | 1 | > | 2019-08-26 | 5 | > | 2019-08-27 | 98 | > | 2019-08-28 | 3 | > | 2019-08-29 | 1 | > | 2019-09-02 | 3 | > | 2019-09-04 | 3 | > | 2019-09-05 |105 | > | 2019-09-06 | 3 | > | 2019-09-07 | 2 | > | 2019-09-09 | 6 | > | 2019-09-12 | 9 | > | 2019-09-13 | 1 | > | 2019-09-17 | 1 | > | 2019-09-24 | 3 | > | 2019-09-26 | 6 | > | 2019-09-27 | 4 | > | 2019-09-30 | 1 | > | 2019-10-01 | 2 | > | 2019-10-03 | 9 | > | 2019-10-04 | 2 | > | 2019-10-06 | 1 | > | 2019-10-08 | 1 | > | 2019-10-09 | 1 | > | 2019-10-10 | 6 | > | 2019-10-11 | 1 | > | 2019-10-16 | 13 | > | 2019-10-17 | 1 | > | 2019-10-18 | 2 | > | 2019-10-19 | 2 | > | 2019-10-21 | 10 | > | 2019-10-22 | 6 | > | 2019-10-28 | 2 | > | 2019-10-29 | 4 | > | 2019-10-30 | 2 | > | 2019-10-31 | 2 | > | 2019-11-05 | 2 | > | 2019-11-06 | 2 | > | 2019-11-11 | 1 | > | 2019-11-13 | 1 | > | 2019-11-14 | 1 | > | 2019-11-21 | 4 | > | 2019-11-26 | 1 | > | 2019-11-27 | 1 | > | 2019-12-05 | 4 | > | 2019-12-06 | 2 | > | 2019-12-12 | 1 | > | 2019-12-14 | 1 | > | 2019-12-15 | 3 | > | 2019-12-16 | 1 | > | 2019-12-17 | 1 | > | 2019-12-18 | 1 | > | 2019-12-19 | 2 | > | 2019-12-20 | 2 | > | 2019-12-23 | 1 | > | 2019-12-27 | 1 | > | 2020-01-07 | 1 | > | 2020-01-08 | 14 | > | 2020-01-09 | 2 | > | 2020-01-12 |372 | > | 2020-01-14 | 2 | > | 2020-01-15 | 1 | > | 2020-01-20 | 11 | > | 2020-01-21 | 119253 | > | 2020-01-23 |113 | > | 2020-01-24 | 4 | > | 2020-01-25 |536 | > | 2020-01-26 | 2132 | > | 2020-01-27 |396 | > | 2020-01-28 | 1 | > | 2020-01-29 | 3 | > | 2020-01-30 | 11 | > | 2020-01-31 | 11 | > | 2020-02-03 | 2 | > | 2020-02-04 | 4 | > | 2020-02-05 | 5 | > | 2020-02-06 | 8 | > | 2020-02-10 | 32 | > | 2020-02-11 | 15 | > | 2020-02-12 | 14 | > | 2020-02-13 | 1 | > | 2020-02-14 | 92 | > +++ > 109 rows in set (0.16 sec) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Component/s: Transactions > Locks entries are left over inside HIVE_LOCKS when using DbTxnManager > - > > Key: HIVE-22911 > URL: https://issues.apache.org/jira/browse/HIVE-22911 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Critical > > We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. > There are more than 120k locks in HIVE_LOCKS of MySQL database. We also > checked the top 3 tables which are related to the existing locks: > > {code} > mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by > 3 desc limit 10; > +---+--+--+ > | HL_DB | HL_TABLE | count(*) | > +---+--+--+ > | db1 | table1 | 66984 | > | db1 | table2 | 33208 | > | db1 | table3 | 9315 | > … > {code} > For table “db1. table1”, here are 3 Hive sessions related, and each of the > Hive session is waiting for 22328 read locks. This is because this table > “db1. table1” is a huge partition table, and it has more than 200k child > partitions. I am guessing each of Hive session was trying to do a full table > scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is > the list: > > {code} > mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as > dt,count(*) as cnt from HIVE_LOCKS > -> group by 1 order by 1; > +++ > | dt | cnt| > +++ > | 1969-12-31 | 2 | > | 2019-05-20 | 10 | > | 2019-05-21 | 3 | > | 2019-05-23 | 5 | > | 2019-05-24 | 2 | > | 2019-05-25 | 1 | > | 2019-05-29 | 7 | > | 2019-05-30 | 2 | > | 2019-06-11 | 13 | > | 2019-06-28 | 3 | > | 2019-07-02 | 2 | > | 2019-07-04 | 5 | > | 2019-07-09 | 1 | > | 2019-07-15 | 2 | > | 2019-07-16 | 1 | > | 2019-07-18 | 2 | > | 2019-07-20 | 3 | > | 2019-07-29 | 5 | > | 2019-07-30 | 9 | > | 2019-07-31 | 7 | > | 2019-08-02 | 2 | > | 2019-08-06 | 5 | > | 2019-08-07 | 17 | > | 2019-08-08 | 8 | > | 2019-08-09 | 5 | > | 2019-08-21 | 1 | > | 2019-08-22 | 20 | > | 2019-08-23 | 1 | > | 2019-08-26 | 5 | > | 2019-08-27 | 98 | > | 2019-08-28 | 3 | > | 2019-08-29 | 1 | > | 2019-09-02 | 3 | > | 2019-09-04 | 3 | > | 2019-09-05 |105 | > | 2019-09-06 | 3 | > | 2019-09-07 | 2 | > | 2019-09-09 | 6 | > | 2019-09-12 | 9 | > | 2019-09-13 | 1 | > | 2019-09-17 | 1 | > | 2019-09-24 | 3 | > | 2019-09-26 | 6 | > | 2019-09-27 | 4 | > | 2019-09-30 | 1 | > | 2019-10-01 | 2 | > | 2019-10-03 | 9 | > | 2019-10-04 | 2 | > | 2019-10-06 | 1 | > | 2019-10-08 | 1 | > | 2019-10-09 | 1 | > | 2019-10-10 | 6 | > | 2019-10-11 | 1 | > | 2019-10-16 | 13 | > | 2019-10-17 | 1 | > | 2019-10-18 | 2 | > | 2019-10-19 | 2 | > | 2019-10-21 | 10 | > | 2019-10-22 | 6 | > | 2019-10-28 | 2 | > | 2019-10-29 | 4 | > | 2019-10-30 | 2 | > | 2019-10-31 | 2 | > | 2019-11-05 | 2 | > | 2019-11-06 | 2 | > | 2019-11-11 | 1 | > | 2019-11-13 | 1 | > | 2019-11-14 | 1 | > | 2019-11-21 | 4 | > | 2019-11-26 | 1 | > | 2019-11-27 | 1 | > | 2019-12-05 | 4 | > | 2019-12-06 | 2 | > | 2019-12-12 | 1 | > | 2019-12-14 | 1 | > | 2019-12-15 | 3 | > | 2019-12-16 | 1 | > | 2019-12-17 | 1 | > | 2019-12-18 | 1 | > | 2019-12-19 | 2 | > | 2019-12-20 | 2 | > | 2019-12-23 | 1 | > | 2019-12-27 | 1 | > | 2020-01-07 | 1 | > | 2020-01-08 | 14 | > | 2020-01-09 | 2 | > | 2020-01-12 |372 | > | 2020-01-14 | 2 | > | 2020-01-15 | 1 | > | 2020-01-20 | 11 | > | 2020-01-21 | 119253 | > | 2020-01-23 |113 | > | 2020-01-24 | 4 | > | 2020-01-25 |536 | > | 2020-01-26 | 2132 | > | 2020-01-27 |396 | > | 2020-01-28 | 1 | > | 2020-01-29 | 3 | > | 2020-01-30 | 11 | > | 2020-01-31 | 11 | > | 2020-02-03 | 2 | > | 2020-02-04 | 4 | > | 2020-02-05 | 5 | > | 2020-02-06 | 8 | > | 2020-02-10 | 32 | > | 2020-02-11 | 15 | > | 2020-02-12 | 14 | > | 2020-02-13 | 1 | > | 2020-02-14 | 92 | > +++ > 109 rows in set (0.16 sec) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Component/s: Locking > Locks entries are left over inside HIVE_LOCKS when using DbTxnManager > - > > Key: HIVE-22911 > URL: https://issues.apache.org/jira/browse/HIVE-22911 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Critical > > We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. > There are more than 120k locks in HIVE_LOCKS of MySQL database. We also > checked the top 3 tables which are related to the existing locks: > > {code} > mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by > 3 desc limit 10; > +---+--+--+ > | HL_DB | HL_TABLE | count(*) | > +---+--+--+ > | db1 | table1 | 66984 | > | db1 | table2 | 33208 | > | db1 | table3 | 9315 | > … > {code} > For table “db1. table1”, here are 3 Hive sessions related, and each of the > Hive session is waiting for 22328 read locks. This is because this table > “db1. table1” is a huge partition table, and it has more than 200k child > partitions. I am guessing each of Hive session was trying to do a full table > scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is > the list: > > {code} > mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as > dt,count(*) as cnt from HIVE_LOCKS > -> group by 1 order by 1; > +++ > | dt | cnt| > +++ > | 1969-12-31 | 2 | > | 2019-05-20 | 10 | > | 2019-05-21 | 3 | > | 2019-05-23 | 5 | > | 2019-05-24 | 2 | > | 2019-05-25 | 1 | > | 2019-05-29 | 7 | > | 2019-05-30 | 2 | > | 2019-06-11 | 13 | > | 2019-06-28 | 3 | > | 2019-07-02 | 2 | > | 2019-07-04 | 5 | > | 2019-07-09 | 1 | > | 2019-07-15 | 2 | > | 2019-07-16 | 1 | > | 2019-07-18 | 2 | > | 2019-07-20 | 3 | > | 2019-07-29 | 5 | > | 2019-07-30 | 9 | > | 2019-07-31 | 7 | > | 2019-08-02 | 2 | > | 2019-08-06 | 5 | > | 2019-08-07 | 17 | > | 2019-08-08 | 8 | > | 2019-08-09 | 5 | > | 2019-08-21 | 1 | > | 2019-08-22 | 20 | > | 2019-08-23 | 1 | > | 2019-08-26 | 5 | > | 2019-08-27 | 98 | > | 2019-08-28 | 3 | > | 2019-08-29 | 1 | > | 2019-09-02 | 3 | > | 2019-09-04 | 3 | > | 2019-09-05 |105 | > | 2019-09-06 | 3 | > | 2019-09-07 | 2 | > | 2019-09-09 | 6 | > | 2019-09-12 | 9 | > | 2019-09-13 | 1 | > | 2019-09-17 | 1 | > | 2019-09-24 | 3 | > | 2019-09-26 | 6 | > | 2019-09-27 | 4 | > | 2019-09-30 | 1 | > | 2019-10-01 | 2 | > | 2019-10-03 | 9 | > | 2019-10-04 | 2 | > | 2019-10-06 | 1 | > | 2019-10-08 | 1 | > | 2019-10-09 | 1 | > | 2019-10-10 | 6 | > | 2019-10-11 | 1 | > | 2019-10-16 | 13 | > | 2019-10-17 | 1 | > | 2019-10-18 | 2 | > | 2019-10-19 | 2 | > | 2019-10-21 | 10 | > | 2019-10-22 | 6 | > | 2019-10-28 | 2 | > | 2019-10-29 | 4 | > | 2019-10-30 | 2 | > | 2019-10-31 | 2 | > | 2019-11-05 | 2 | > | 2019-11-06 | 2 | > | 2019-11-11 | 1 | > | 2019-11-13 | 1 | > | 2019-11-14 | 1 | > | 2019-11-21 | 4 | > | 2019-11-26 | 1 | > | 2019-11-27 | 1 | > | 2019-12-05 | 4 | > | 2019-12-06 | 2 | > | 2019-12-12 | 1 | > | 2019-12-14 | 1 | > | 2019-12-15 | 3 | > | 2019-12-16 | 1 | > | 2019-12-17 | 1 | > | 2019-12-18 | 1 | > | 2019-12-19 | 2 | > | 2019-12-20 | 2 | > | 2019-12-23 | 1 | > | 2019-12-27 | 1 | > | 2020-01-07 | 1 | > | 2020-01-08 | 14 | > | 2020-01-09 | 2 | > | 2020-01-12 |372 | > | 2020-01-14 | 2 | > | 2020-01-15 | 1 | > | 2020-01-20 | 11 | > | 2020-01-21 | 119253 | > | 2020-01-23 |113 | > | 2020-01-24 | 4 | > | 2020-01-25 |536 | > | 2020-01-26 | 2132 | > | 2020-01-27 |396 | > | 2020-01-28 | 1 | > | 2020-01-29 | 3 | > | 2020-01-30 | 11 | > | 2020-01-31 | 11 | > | 2020-02-03 | 2 | > | 2020-02-04 | 4 | > | 2020-02-05 | 5 | > | 2020-02-06 | 8 | > | 2020-02-10 | 32 | > | 2020-02-11 | 15 | > | 2020-02-12 | 14 | > | 2020-02-13 | 1 | > | 2020-02-14 | 92 | > +++ > 109 rows in set (0.16 sec) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22911 started by Oleksiy Sayankin. --- > Locks entries are left over inside HIVE_LOCKS when using DbTxnManager > - > > Key: HIVE-22911 > URL: https://issues.apache.org/jira/browse/HIVE-22911 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Critical > > We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. > There are more than 120k locks in HIVE_LOCKS of MySQL database. We also > checked the top 3 tables which are related to the existing locks: > > {code} > mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by > 3 desc limit 10; > +---+--+--+ > | HL_DB | HL_TABLE | count(*) | > +---+--+--+ > | db1 | table1 | 66984 | > | db1 | table2 | 33208 | > | db1 | table3 | 9315 | > … > {code} > For table “db1. table1”, here are 3 Hive sessions related, and each of the > Hive session is waiting for 22328 read locks. This is because this table > “db1. table1” is a huge partition table, and it has more than 200k child > partitions. I am guessing each of Hive session was trying to do a full table > scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is > the list: > > {code} > MariaDB [customer]> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as > date) as dt,count(*) as cnt from HIVE_LOCKS > -> group by 1 order by 1; > +++ > | dt | cnt| > +++ > | 1969-12-31 | 2 | > | 2019-05-20 | 10 | > | 2019-05-21 | 3 | > | 2019-05-23 | 5 | > | 2019-05-24 | 2 | > | 2019-05-25 | 1 | > | 2019-05-29 | 7 | > | 2019-05-30 | 2 | > | 2019-06-11 | 13 | > | 2019-06-28 | 3 | > | 2019-07-02 | 2 | > | 2019-07-04 | 5 | > | 2019-07-09 | 1 | > | 2019-07-15 | 2 | > | 2019-07-16 | 1 | > | 2019-07-18 | 2 | > | 2019-07-20 | 3 | > | 2019-07-29 | 5 | > | 2019-07-30 | 9 | > | 2019-07-31 | 7 | > | 2019-08-02 | 2 | > | 2019-08-06 | 5 | > | 2019-08-07 | 17 | > | 2019-08-08 | 8 | > | 2019-08-09 | 5 | > | 2019-08-21 | 1 | > | 2019-08-22 | 20 | > | 2019-08-23 | 1 | > | 2019-08-26 | 5 | > | 2019-08-27 | 98 | > | 2019-08-28 | 3 | > | 2019-08-29 | 1 | > | 2019-09-02 | 3 | > | 2019-09-04 | 3 | > | 2019-09-05 |105 | > | 2019-09-06 | 3 | > | 2019-09-07 | 2 | > | 2019-09-09 | 6 | > | 2019-09-12 | 9 | > | 2019-09-13 | 1 | > | 2019-09-17 | 1 | > | 2019-09-24 | 3 | > | 2019-09-26 | 6 | > | 2019-09-27 | 4 | > | 2019-09-30 | 1 | > | 2019-10-01 | 2 | > | 2019-10-03 | 9 | > | 2019-10-04 | 2 | > | 2019-10-06 | 1 | > | 2019-10-08 | 1 | > | 2019-10-09 | 1 | > | 2019-10-10 | 6 | > | 2019-10-11 | 1 | > | 2019-10-16 | 13 | > | 2019-10-17 | 1 | > | 2019-10-18 | 2 | > | 2019-10-19 | 2 | > | 2019-10-21 | 10 | > | 2019-10-22 | 6 | > | 2019-10-28 | 2 | > | 2019-10-29 | 4 | > | 2019-10-30 | 2 | > | 2019-10-31 | 2 | > | 2019-11-05 | 2 | > | 2019-11-06 | 2 | > | 2019-11-11 | 1 | > | 2019-11-13 | 1 | > | 2019-11-14 | 1 | > | 2019-11-21 | 4 | > | 2019-11-26 | 1 | > | 2019-11-27 | 1 | > | 2019-12-05 | 4 | > | 2019-12-06 | 2 | > | 2019-12-12 | 1 | > | 2019-12-14 | 1 | > | 2019-12-15 | 3 | > | 2019-12-16 | 1 | > | 2019-12-17 | 1 | > | 2019-12-18 | 1 | > | 2019-12-19 | 2 | > | 2019-12-20 | 2 | > | 2019-12-23 | 1 | > | 2019-12-27 | 1 | > | 2020-01-07 | 1 | > | 2020-01-08 | 14 | > | 2020-01-09 | 2 | > | 2020-01-12 |372 | > | 2020-01-14 | 2 | > | 2020-01-15 | 1 | > | 2020-01-20 | 11 | > | 2020-01-21 | 119253 | > | 2020-01-23 |113 | > | 2020-01-24 | 4 | > | 2020-01-25 |536 | > | 2020-01-26 | 2132 | > | 2020-01-27 |396 | > | 2020-01-28 | 1 | > | 2020-01-29 | 3 | > | 2020-01-30 | 11 | > | 2020-01-31 | 11 | > | 2020-02-03 | 2 | > | 2020-02-04 | 4 | > | 2020-02-05 | 5 | > | 2020-02-06 | 8 | > | 2020-02-10 | 32 | > | 2020-02-11 | 15 | > | 2020-02-12 | 14 | > | 2020-02-13 | 1 | > | 2020-02-14 | 92 | > +++ > 109 rows in set (0.16 sec) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-22911: --- > Locks entries are left over inside HIVE_LOCKS when using DbTxnManager > - > > Key: HIVE-22911 > URL: https://issues.apache.org/jira/browse/HIVE-22911 > Project: Hive > Issue Type: Bug >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Critical > > We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. > There are more than 120k locks in HIVE_LOCKS of MySQL database. We also > checked the top 3 tables which are related to the existing locks: > > {code} > mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by > 3 desc limit 10; > +---+--+--+ > | HL_DB | HL_TABLE | count(*) | > +---+--+--+ > | db1 | table1 | 66984 | > | db1 | table2 | 33208 | > | db1 | table3 | 9315 | > … > {code} > For table “db1. table1”, here are 3 Hive sessions related, and each of the > Hive session is waiting for 22328 read locks. This is because this table > “db1. table1” is a huge partition table, and it has more than 200k child > partitions. I am guessing each of Hive session was trying to do a full table > scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is > the list: > > {code} > MariaDB [customer]> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as > date) as dt,count(*) as cnt from HIVE_LOCKS > -> group by 1 order by 1; > +++ > | dt | cnt| > +++ > | 1969-12-31 | 2 | > | 2019-05-20 | 10 | > | 2019-05-21 | 3 | > | 2019-05-23 | 5 | > | 2019-05-24 | 2 | > | 2019-05-25 | 1 | > | 2019-05-29 | 7 | > | 2019-05-30 | 2 | > | 2019-06-11 | 13 | > | 2019-06-28 | 3 | > | 2019-07-02 | 2 | > | 2019-07-04 | 5 | > | 2019-07-09 | 1 | > | 2019-07-15 | 2 | > | 2019-07-16 | 1 | > | 2019-07-18 | 2 | > | 2019-07-20 | 3 | > | 2019-07-29 | 5 | > | 2019-07-30 | 9 | > | 2019-07-31 | 7 | > | 2019-08-02 | 2 | > | 2019-08-06 | 5 | > | 2019-08-07 | 17 | > | 2019-08-08 | 8 | > | 2019-08-09 | 5 | > | 2019-08-21 | 1 | > | 2019-08-22 | 20 | > | 2019-08-23 | 1 | > | 2019-08-26 | 5 | > | 2019-08-27 | 98 | > | 2019-08-28 | 3 | > | 2019-08-29 | 1 | > | 2019-09-02 | 3 | > | 2019-09-04 | 3 | > | 2019-09-05 |105 | > | 2019-09-06 | 3 | > | 2019-09-07 | 2 | > | 2019-09-09 | 6 | > | 2019-09-12 | 9 | > | 2019-09-13 | 1 | > | 2019-09-17 | 1 | > | 2019-09-24 | 3 | > | 2019-09-26 | 6 | > | 2019-09-27 | 4 | > | 2019-09-30 | 1 | > | 2019-10-01 | 2 | > | 2019-10-03 | 9 | > | 2019-10-04 | 2 | > | 2019-10-06 | 1 | > | 2019-10-08 | 1 | > | 2019-10-09 | 1 | > | 2019-10-10 | 6 | > | 2019-10-11 | 1 | > | 2019-10-16 | 13 | > | 2019-10-17 | 1 | > | 2019-10-18 | 2 | > | 2019-10-19 | 2 | > | 2019-10-21 | 10 | > | 2019-10-22 | 6 | > | 2019-10-28 | 2 | > | 2019-10-29 | 4 | > | 2019-10-30 | 2 | > | 2019-10-31 | 2 | > | 2019-11-05 | 2 | > | 2019-11-06 | 2 | > | 2019-11-11 | 1 | > | 2019-11-13 | 1 | > | 2019-11-14 | 1 | > | 2019-11-21 | 4 | > | 2019-11-26 | 1 | > | 2019-11-27 | 1 | > | 2019-12-05 | 4 | > | 2019-12-06 | 2 | > | 2019-12-12 | 1 | > | 2019-12-14 | 1 | > | 2019-12-15 | 3 | > | 2019-12-16 | 1 | > | 2019-12-17 | 1 | > | 2019-12-18 | 1 | > | 2019-12-19 | 2 | > | 2019-12-20 | 2 | > | 2019-12-23 | 1 | > | 2019-12-27 | 1 | > | 2020-01-07 | 1 | > | 2020-01-08 | 14 | > | 2020-01-09 | 2 | > | 2020-01-12 |372 | > | 2020-01-14 | 2 | > | 2020-01-15 | 1 | > | 2020-01-20 | 11 | > | 2020-01-21 | 119253 | > | 2020-01-23 |113 | > | 2020-01-24 | 4 | > | 2020-01-25 |536 | > | 2020-01-26 | 2132 | > | 2020-01-27 |396 | > | 2020-01-28 | 1 | > | 2020-01-29 | 3 | > | 2020-01-30 | 11 | > | 2020-01-31 | 11 | > | 2020-02-03 | 2 | > | 2020-02-04 | 4 | > | 2020-02-05 | 5 | > | 2020-02-06 | 8 | > | 2020-02-10 | 32 | > | 2020-02-11 | 15 | > | 2020-02-12 | 14 | > | 2020-02-13 | 1 | > | 2020-02-14 | 92 | > +++ > 109 rows in set (0.16 sec) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22911) Locks entries are left over inside HIVE_LOCKS when using DbTxnManager
[ https://issues.apache.org/jira/browse/HIVE-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-22911: Description: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | db1 | table3 | 9315 | … {code} For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive session is waiting for 22328 read locks. This is because this table “db1. table1” is a huge partition table, and it has more than 200k child partitions. I am guessing each of Hive session was trying to do a full table scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list: {code} mysql> select cast(FROM_UNIXTIME(HL_LAST_HEARTBEAT/1000) as date) as dt,count(*) as cnt from HIVE_LOCKS -> group by 1 order by 1; +++ | dt | cnt| +++ | 1969-12-31 | 2 | | 2019-05-20 | 10 | | 2019-05-21 | 3 | | 2019-05-23 | 5 | | 2019-05-24 | 2 | | 2019-05-25 | 1 | | 2019-05-29 | 7 | | 2019-05-30 | 2 | | 2019-06-11 | 13 | | 2019-06-28 | 3 | | 2019-07-02 | 2 | | 2019-07-04 | 5 | | 2019-07-09 | 1 | | 2019-07-15 | 2 | | 2019-07-16 | 1 | | 2019-07-18 | 2 | | 2019-07-20 | 3 | | 2019-07-29 | 5 | | 2019-07-30 | 9 | | 2019-07-31 | 7 | | 2019-08-02 | 2 | | 2019-08-06 | 5 | | 2019-08-07 | 17 | | 2019-08-08 | 8 | | 2019-08-09 | 5 | | 2019-08-21 | 1 | | 2019-08-22 | 20 | | 2019-08-23 | 1 | | 2019-08-26 | 5 | | 2019-08-27 | 98 | | 2019-08-28 | 3 | | 2019-08-29 | 1 | | 2019-09-02 | 3 | | 2019-09-04 | 3 | | 2019-09-05 |105 | | 2019-09-06 | 3 | | 2019-09-07 | 2 | | 2019-09-09 | 6 | | 2019-09-12 | 9 | | 2019-09-13 | 1 | | 2019-09-17 | 1 | | 2019-09-24 | 3 | | 2019-09-26 | 6 | | 2019-09-27 | 4 | | 2019-09-30 | 1 | | 2019-10-01 | 2 | | 2019-10-03 | 9 | | 2019-10-04 | 2 | | 2019-10-06 | 1 | | 2019-10-08 | 1 | | 2019-10-09 | 1 | | 2019-10-10 | 6 | | 2019-10-11 | 1 | | 2019-10-16 | 13 | | 2019-10-17 | 1 | | 2019-10-18 | 2 | | 2019-10-19 | 2 | | 2019-10-21 | 10 | | 2019-10-22 | 6 | | 2019-10-28 | 2 | | 2019-10-29 | 4 | | 2019-10-30 | 2 | | 2019-10-31 | 2 | | 2019-11-05 | 2 | | 2019-11-06 | 2 | | 2019-11-11 | 1 | | 2019-11-13 | 1 | | 2019-11-14 | 1 | | 2019-11-21 | 4 | | 2019-11-26 | 1 | | 2019-11-27 | 1 | | 2019-12-05 | 4 | | 2019-12-06 | 2 | | 2019-12-12 | 1 | | 2019-12-14 | 1 | | 2019-12-15 | 3 | | 2019-12-16 | 1 | | 2019-12-17 | 1 | | 2019-12-18 | 1 | | 2019-12-19 | 2 | | 2019-12-20 | 2 | | 2019-12-23 | 1 | | 2019-12-27 | 1 | | 2020-01-07 | 1 | | 2020-01-08 | 14 | | 2020-01-09 | 2 | | 2020-01-12 |372 | | 2020-01-14 | 2 | | 2020-01-15 | 1 | | 2020-01-20 | 11 | | 2020-01-21 | 119253 | | 2020-01-23 |113 | | 2020-01-24 | 4 | | 2020-01-25 |536 | | 2020-01-26 | 2132 | | 2020-01-27 |396 | | 2020-01-28 | 1 | | 2020-01-29 | 3 | | 2020-01-30 | 11 | | 2020-01-31 | 11 | | 2020-02-03 | 2 | | 2020-02-04 | 4 | | 2020-02-05 | 5 | | 2020-02-06 | 8 | | 2020-02-10 | 32 | | 2020-02-11 | 15 | | 2020-02-12 | 14 | | 2020-02-13 | 1 | | 2020-02-14 | 92 | +++ 109 rows in set (0.16 sec) {code} was: We found lots of orphan/old/leftover lock entries inside {{HIVE_LOCKS}}. There are more than 120k locks in HIVE_LOCKS of MySQL database. We also checked the top 3 tables which are related to the existing locks: {code} mysql> select HL_DB,HL_TABLE, count(*) from HIVE_LOCKS group by 1,2 order by 3 desc limit 10; +---+--+--+ | HL_DB | HL_TABLE | count(*) | +---+--+--+ | db1 | table1 | 66984 | | db1 | table2 | 33208 | | db1 | table3 | 9315 | … {code} For table “db1. table1”, here are 3 Hive sessions related, and each of the Hive session is waiting for 22328 read locks. This is because this table “db1. table1” is a huge partition table, and it has more than 200k child partitions. I am guessing each of Hive session was trying to do a full table scan on it. I group-by based on column {{HL_LAST_HEARTBEAT}} instead, here is the list:
[jira] [Assigned] (HIVE-7089) StorageBasedAuthorizationProvider fails to allow non-admin users to create databases in writable directories
[ https://issues.apache.org/jira/browse/HIVE-7089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-7089: -- Assignee: Oleksiy Sayankin > StorageBasedAuthorizationProvider fails to allow non-admin users to create > databases in writable directories > > > Key: HIVE-7089 > URL: https://issues.apache.org/jira/browse/HIVE-7089 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Craig Condit >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-7089.patch > > > When attempting to create a database with a custom location and using > hive.security.authorizationmanager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider, > an AccessControlException is generated for the default warehouse location, > not the location which was given in the create database command. > {noformat} > hive> create database test LOCATION '/user/ccondit/test'; > > Authorization failed:java.security.AccessControlException: action WRITE not > permitted on path hdfs://example.com:8020/apps/hive/warehouse for user > ccondit. Use SHOW GRANT to get more details. > 14/05/19 09:50:59 ERROR ql.Driver: Authorization > failed:java.security.AccessControlException: action WRITE not permitted on > path hdfs://example.com:8020/apps/hive/warehouse for user ccondit. Use SHOW > GRANT to get more details. > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-7089) StorageBasedAuthorizationProvider fails to allow non-admin users to create databases in writable directories
[ https://issues.apache.org/jira/browse/HIVE-7089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-7089: --- Status: In Progress (was: Patch Available) > StorageBasedAuthorizationProvider fails to allow non-admin users to create > databases in writable directories > > > Key: HIVE-7089 > URL: https://issues.apache.org/jira/browse/HIVE-7089 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Craig Condit >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-7089.patch > > > When attempting to create a database with a custom location and using > hive.security.authorizationmanager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider, > an AccessControlException is generated for the default warehouse location, > not the location which was given in the create database command. > {noformat} > hive> create database test LOCATION '/user/ccondit/test'; > > Authorization failed:java.security.AccessControlException: action WRITE not > permitted on path hdfs://example.com:8020/apps/hive/warehouse for user > ccondit. Use SHOW GRANT to get more details. > 14/05/19 09:50:59 ERROR ql.Driver: Authorization > failed:java.security.AccessControlException: action WRITE not permitted on > path hdfs://example.com:8020/apps/hive/warehouse for user ccondit. Use SHOW > GRANT to get more details. > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900162#comment-16900162 ] Oleksiy Sayankin commented on HIVE-4605: Added not null verification for {{finalPaths[idx]}}. > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > {code} > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > {code} > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > {code} > java.lang.RuntimeException: Hive Runtime Error while closing operators: > Unable to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-4605: --- Attachment: HIVE-4605.3.patch > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > {code} > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > {code} > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > {code} > java.lang.RuntimeException: Hive Runtime Error while closing operators: > Unable to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-4605: --- Status: Patch Available (was: In Progress) > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.3.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > {code} > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > {code} > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > {code} > java.lang.RuntimeException: Hive Runtime Error while closing operators: > Unable to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-4605: --- Status: In Progress (was: Patch Available) > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > {code} > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > {code} > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > {code} > java.lang.RuntimeException: Hive Runtime Error while closing operators: > Unable to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897390#comment-16897390 ] Oleksiy Sayankin commented on HIVE-4605: I've re-testes failed tests: {code} mvn test -Dtest=TestCliDriver -Dqfile=orc_llap_counters1.q,orc_llap_counters.q,orc_ppd_basic.q,orc_ppd_schema_evol_3a.q,materialized_view_rewrite_4.q {code} The fix does not change their behaviour: they fail even without any changes in master branch. > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > {code} > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > {code} > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > {code} > java.lang.RuntimeException: Hive Runtime Error while closing operators: > Unable to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016)