[jira] [Assigned] (HIVE-24169) HiveServer2 UDF cache
[ https://issues.apache.org/jira/browse/HIVE-24169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-24169: - > HiveServer2 UDF cache > - > > Key: HIVE-24169 > URL: https://issues.apache.org/jira/browse/HIVE-24169 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > UDF is cache per session. This optional feature can help speed up UDF access > in S3 scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24097) correct NPE exception in HiveMetastoreAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-24097: - > correct NPE exception in HiveMetastoreAuthorizer > > > Key: HIVE-24097 > URL: https://issues.apache.org/jira/browse/HIVE-24097 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > In some testing, we found it's possible to have NPE if the preEventType does > not fall within the several the HMS currently checks. This makes the > AuthzContext a null pointer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24022) Optimise HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-24022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-24022: - Assignee: Sam An > Optimise HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer > -- > > Key: HIVE-24022 > URL: https://issues.apache.org/jira/browse/HIVE-24022 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Sam An >Priority: Minor > Labels: performance, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > For a table with 3000+ partitions, analyze table takes a lot longer time as > HiveMetaStoreAuthorizer tries to create HiveConf for every partition request. > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L319] > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L447] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24022) Optimise HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-24022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186143#comment-17186143 ] Sam An commented on HIVE-24022: --- have a upstream patch. [https://github.com/apache/hive/pull/1385] > Optimise HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer > -- > > Key: HIVE-24022 > URL: https://issues.apache.org/jira/browse/HIVE-24022 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Sam An >Priority: Minor > Labels: performance, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > For a table with 3000+ partitions, analyze table takes a lot longer time as > HiveMetaStoreAuthorizer tries to create HiveConf for every partition request. > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L319] > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L447] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24022) Optimise HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer
[ https://issues.apache.org/jira/browse/HIVE-24022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176513#comment-17176513 ] Sam An commented on HIVE-24022: --- I am trying to use ThreadLocal to skip creating HiveConf each and every time createHiveMetastoreAuthorizer gets called. Currently there are 2 Unit Test failures. The root cause is in the TestHiveMetastoreAuthorizer unit test set up, the Config overlay was not visible inside the ThreadLocal hiveconf, so it is not using the DummyHiveAuthorizerFactory as Authorization manager, but instead look for SqlStdAuthorizationFactoryForTest, and it fails to find the class, causing failure. [https://github.com/apache/hive/blob/2b3c689baff857c18164a9610f2854583105734a/ql/src/test/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/TestHiveMetaStoreAuthorizer.java#L83] So I see 2 problem, 1. DummyHiveAuthorizerFactory is not passed in to ThreadLocal<> for unit test. 2. classloader not able to find SQLStdHiveAuthorizerFactoryForTest class in classpath. Without my change, it was using the DummyHiveAuthorizerFactory for unit test. So I should make the overlay happen first, then see if it can find the class DummyHiveAuthorizerFactory. > Optimise HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer > -- > > Key: HIVE-24022 > URL: https://issues.apache.org/jira/browse/HIVE-24022 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Labels: performance, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > For a table with 3000+ partitions, analyze table takes a lot longer time as > HiveMetaStoreAuthorizer tries to create HiveConf for every partition request. > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L319] > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L447] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24004) Improve performance for filter hook for superuser path
[ https://issues.apache.org/jira/browse/HIVE-24004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-24004: - > Improve performance for filter hook for superuser path > -- > > Key: HIVE-24004 > URL: https://issues.apache.org/jira/browse/HIVE-24004 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > In HiveMetastoreAuthoriver, the sequence of creating the authorizer can be > optimized so that for super user when we can skip authorization, we can skip > creating authorizer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23967) Give HiveMetastore thread pool a more descriptive name
[ https://issues.apache.org/jira/browse/HIVE-23967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23967: - > Give HiveMetastore thread pool a more descriptive name > -- > > Key: HIVE-23967 > URL: https://issues.apache.org/jira/browse/HIVE-23967 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently HiveMetastore threadpool name is the default generic format > pool-id-thread-id. We should use a more descriptive one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23943) Improve HiveMetastoreAuthorizer filter performance
[ https://issues.apache.org/jira/browse/HIVE-23943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23943: - > Improve HiveMetastoreAuthorizer filter performance > -- > > Key: HIVE-23943 > URL: https://issues.apache.org/jira/browse/HIVE-23943 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > The current HiveMetastoreAuthorizer is per thread, and it can have problem in > some heavy use scenario. Let's see if we can improve the performance by > changing the class to a singleton or some other more performant option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23786) HMS Server side filter
[ https://issues.apache.org/jira/browse/HIVE-23786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23786: - > HMS Server side filter > -- > > Key: HIVE-23786 > URL: https://issues.apache.org/jira/browse/HIVE-23786 > Project: Hive > Issue Type: Improvement >Reporter: Sam An >Assignee: Sam An >Priority: Major > > HMS server side filter of results based on authorization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23467) Add a skip.trash config for HMS to skip trash when deleting external table data
[ https://issues.apache.org/jira/browse/HIVE-23467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23467: - Assignee: Yu-Wen Lai (was: Sam An) > Add a skip.trash config for HMS to skip trash when deleting external table > data > --- > > Key: HIVE-23467 > URL: https://issues.apache.org/jira/browse/HIVE-23467 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sam An >Assignee: Yu-Wen Lai >Priority: Trivial > > We have an auto.purge flag, which means skip trash. It can be confusing as we > have 'external.table.purge'='true' to indicate delete table data when this > tblproperties is set. > We should make the meaning clearer by introducing a skip trash alias/option. > Additionally, we shall add an alias for external.table.purge, and name it > external.table.autodelete, and document it more prominently, so as to > maintain backward compatibility, and make the meaning of auto deletion of > data more obvious. > The net effect of these 2 changes will be. If the user sets > 'external.table.autodelete'='true' > the table data will be removed when table is dropped. and if > 'skip.trash'='true' > is set, HMS will not move the table data to trash folder when removing the > files. This will result in faster removal, especially when underlying FS is > S3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23467) Add a skip.trash config for HMS to skip trash when deleting external table data
[ https://issues.apache.org/jira/browse/HIVE-23467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23467: -- Description: We have an auto.purge flag, which means skip trash. It can be confusing as we have 'external.table.purge'='true' to indicate delete table data when this tblproperties is set. We should make the meaning clearer by introducing a skip trash alias/option. Additionally, we shall add an alias for external.table.purge, and name it external.table.autodelete, and document it more prominently, so as to maintain backward compatibility, and make the meaning of auto deletion of data more obvious. The net effect of these 2 changes will be. If the user sets 'external.table.autodelete'='true' the table data will be removed when table is dropped. and if 'skip.trash'='true' is set, HMS will not move the table data to trash folder when removing the files. This will result in faster removal, especially when underlying FS is S3. was: We have an auto.purge flag, which means skip trash. It can be confusing as we have 'external.table.purge'='true' to indicate delete table data when this tblproperties is set. We should make the meaning clearer by introducing a skip trash alias/option. > Add a skip.trash config for HMS to skip trash when deleting external table > data > --- > > Key: HIVE-23467 > URL: https://issues.apache.org/jira/browse/HIVE-23467 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Trivial > > We have an auto.purge flag, which means skip trash. It can be confusing as we > have 'external.table.purge'='true' to indicate delete table data when this > tblproperties is set. > We should make the meaning clearer by introducing a skip trash alias/option. > Additionally, we shall add an alias for external.table.purge, and name it > external.table.autodelete, and document it more prominently, so as to > maintain backward compatibility, and make the meaning of auto deletion of > data more obvious. > The net effect of these 2 changes will be. If the user sets > 'external.table.autodelete'='true' > the table data will be removed when table is dropped. and if > 'skip.trash'='true' > is set, HMS will not move the table data to trash folder when removing the > files. This will result in faster removal, especially when underlying FS is > S3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23467) Add a skip.trash config for HMS to skip trash when deleting external table data
[ https://issues.apache.org/jira/browse/HIVE-23467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23467: - > Add a skip.trash config for HMS to skip trash when deleting external table > data > --- > > Key: HIVE-23467 > URL: https://issues.apache.org/jira/browse/HIVE-23467 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Trivial > > We have an auto.purge flag, which means skip trash. It can be confusing as we > have 'external.table.purge'='true' to indicate delete table data when this > tblproperties is set. > > We should make the meaning clearer by introducing a skip trash alias/option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23425) HS2 disconnect should not affect DDL operation status
[
https://issues.apache.org/jira/browse/HIVE-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam An updated HIVE-23425:
--
Description:
I observe in a cluster that HS2 connection to HMS was timed out.
{code:java}
// code placeholder
{code}
2020-05-07 19:28:46,718 WARN
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: MetaStoreClient lost connection.
Attempting to reconnect (1 of 1) after 1s. dropTable
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714] at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
~[hive-exec-3.1.3000.7.1.0.0-714.jar:3.1.3000.7.1.0.0-714]
2020-05-07 19:28:47,718 INFO
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: RetryingMetaStoreClient trying
reconnect as
hive/ascend-px-ha2.ascend-p.a8du-8mgv.cloudera.s...@ascend-p.a8du-8mgv.cloudera.site
(auth:KERBEROS)
2020-05-07 19:28:47,719 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Closed a connection to metastore,
current connections: 29
2020-05-07 19:28:47,719 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Trying to connect to metastore
with URI thrift://ascend-px-master4.ascend-p.a8du-8mgv.cloudera.site:9083
2020-05-07 19:28:47,719 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: HMSC::open(): Could not find
delegation token. Creating KERBEROS-based thrift connection.
2020-05-07 19:28:47,724 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Opened a connection to metastore,
current connections: 30
2020-05-07 19:28:47,725 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient:
[HiveServer2-Background-Pool: Thread-21790]: Connected to metastore.
2020-05-07 19:28:47,730 INFO
org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook:
[HiveServer2-Background-Pool: Thread-21790]: Received post-hook notification
for: hive_20200507192346_a1a0bad2-ca4f-4cd0-a0e7-22415c7050f0
2020-05-07 19:28:47,731 INFO
org.apache.hive.service.cli.operation.OperationManager:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
getHandleIdentifier()=db40b2c8-d291-4f34-9809-24b6300ee45c]
2020-05-07 19:28:47,731 INFO org.apache.hadoop.hive.common.LogUtils:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: Thread context registration is done.
2020-05-07 19:28:47,731 INFO org.apache.hive.service.cli.operation.Operation:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: [opType=EXECUTE_STATEMENT,
queryId=hive_20200507192847_08dbb746-b38f-405c-a92c-f6d2dd02b6f8,
startTime=1588879727730, sessionId=d900945d-9eb5-4b63-8ce9-c1166bc2474a,
createTime=1588879461427, userName=hive, ipAddress=10.28.140.56]
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.Driver:
[d900945d-9eb5-4b63-8ce9-c1166bc2474a HiveServer2-HttpHandler-Pool:
Thread-19285]: Compiling
command(queryId=hive_20200507192847_08dbb746-b38f-405c-a92c-f6d2dd02b6f8): repl
dump `*` from 1501982 with ('hive.repl.dump.metadata.only'='true',
'hive.repl.dump.include.acid.tables'='true')
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.Driver:
[HiveServer2-Background-Pool: Thread-21790]: Completed executing
command(queryId=hive_20200507192346_a1a0bad2-ca4f-4cd0-a0e7-22415c7050f0); Time
taken: 301.177 seconds
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.Driver:
[HiveServer2-Background-Pool: Thread-21790]: OK
2020-05-07 19:28:47,732 INFO org.apache.hadoop.hive.ql.lockmgr.DbTxnManager:
[HiveServer2-Background-Pool: Thread-21790]: Stopped heartbeat for query:
hive_20200507192346_a1a0bad2-ca4f-4cd0-a0e7-22415c7050f0
2020-05-07 19:28:47,738 INFO org.apache.hadoop.hive.common.LogUtils:
[HiveServer2-Background-Pool: Thread-21790]: Unregistered logging context.
However, on HMS, the drop table operation has not been completed till 5 minutes
later.
[jira] [Assigned] (HIVE-23425) HS2 disconnect should not affect DDL operation status
[ https://issues.apache.org/jira/browse/HIVE-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23425: - > HS2 disconnect should not affect DDL operation status > - > > Key: HIVE-23425 > URL: https://issues.apache.org/jira/browse/HIVE-23425 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > I observe in a cluster that HS2 connection to HMS was timed out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23260) Add support for unmodified_metadata capability
[ https://issues.apache.org/jira/browse/HIVE-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093948#comment-17093948 ] Sam An commented on HIVE-23260: --- +1 > Add support for unmodified_metadata capability > -- > > Key: HIVE-23260 > URL: https://issues.apache.org/jira/browse/HIVE-23260 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-23260.patch > > > Currently, the translator removes bucketing info for tables for clients that > do not possess the HIVEBUCKET2 capability. While this is desirable, some > clients that have write access to these tables can turn around overwrite the > metadata thus corrupting original bucketing info. > So adding support for a capability for client that are capable of > interpreting the original metadata would prevent such corruption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23308) Skip authorization for create db statement if it's default location.
[ https://issues.apache.org/jira/browse/HIVE-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23308: - > Skip authorization for create db statement if it's default location. > - > > Key: HIVE-23308 > URL: https://issues.apache.org/jira/browse/HIVE-23308 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > > When user create database and specify a db location that's the same as > default db location, we can skip authorization. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23192) "default" database locationUri should be external warehouse root.
[ https://issues.apache.org/jira/browse/HIVE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093900#comment-17093900 ] Sam An commented on HIVE-23192: --- +1 > "default" database locationUri should be external warehouse root. > - > > Key: HIVE-23192 > URL: https://issues.apache.org/jira/browse/HIVE-23192 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-23192.patch, HIVE-23192.patch, HIVE-23192.patch > > > When creating the default database, the database locationUri should be set to > external warehouse. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23299) Ranger authorization of managed location
[ https://issues.apache.org/jira/browse/HIVE-23299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23299: -- Attachment: HIVE-23299.1.patch Status: Patch Available (was: Open) > Ranger authorization of managed location > > > Key: HIVE-23299 > URL: https://issues.apache.org/jira/browse/HIVE-23299 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-23299.1.patch > > > With this feature in HIVE-22995, we have introduced a new location for > Databases. This location is meant to be the designated location for all > managed tables in this database. Ranger should also check if the user has > access to this location before the database can be created. > This location can be retrieved via Database.getManagedLocationUri() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23299) Ranger authorization of managed location
[ https://issues.apache.org/jira/browse/HIVE-23299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23299: - > Ranger authorization of managed location > > > Key: HIVE-23299 > URL: https://issues.apache.org/jira/browse/HIVE-23299 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > With this feature in HIVE-22995, we have introduced a new location for > Databases. This location is meant to be the designated location for all > managed tables in this database. Ranger should also check if the user has > access to this location before the database can be created. > This location can be retrieved via Database.getManagedLocationUri() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077464#comment-17077464 ] Sam An commented on HIVE-23111: --- Thanks a lot, [~ngangam] > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Fix For: 4.0.0 > > Attachments: Hive-23111.1.patch, Hive-23111.2.patch, > Hive-23111.3.patch, Hive-23111.4.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23111: -- Attachment: Hive-23111.4.patch > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: Hive-23111.1.patch, Hive-23111.2.patch, > Hive-23111.3.patch, Hive-23111.4.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17075986#comment-17075986 ] Sam An edited comment on HIVE-23111 at 4/6/20, 11:59 PM: - I posted a new PR, incorporating Prasanth's review comments. [https://github.com/apache/hive/pull/968] Just addressed additional comments for Timestamp and exact string match, and pushed. was (Author: samuelan): I posted a new PR, incorporating Prasanth's review comments. [https://github.com/apache/hive/pull/968] > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: Hive-23111.1.patch, Hive-23111.2.patch, > Hive-23111.3.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23111: -- Attachment: Hive-23111.3.patch > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: Hive-23111.1.patch, Hive-23111.2.patch, > Hive-23111.3.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23111: -- Attachment: Hive-23111.2.patch > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: Hive-23111.1.patch, Hive-23111.2.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17075986#comment-17075986 ] Sam An commented on HIVE-23111: --- I posted a new PR, incorporating Prasanth's review comments. [https://github.com/apache/hive/pull/968] > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: Hive-23111.1.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23111: -- Attachment: Hive-23111.1.patch Status: Patch Available (was: In Progress) > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: Hive-23111.1.patch > > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23111 started by Sam An. - > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23111) MsckPartitionExpressionProxy should filter partitions
[ https://issues.apache.org/jira/browse/HIVE-23111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23111: - > MsckPartitionExpressionProxy should filter partitions > - > > Key: HIVE-23111 > URL: https://issues.apache.org/jira/browse/HIVE-23111 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently MsckPartitionExpressionProxy does not filter partition names, this > causes problem for partition auto discovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067919#comment-17067919
]
Sam An edited comment on HIVE-22098 at 3/26/20, 9:58 PM:
-
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you be able to work on this further at this time?
was (Author: samuelan):
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time?
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067919#comment-17067919
]
Sam An edited comment on HIVE-22098 at 3/26/20, 9:57 PM:
-
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time?
was (Author: samuelan):
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch?
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067920#comment-17067920
]
Sam An edited comment on HIVE-18983 at 3/26/20, 9:57 PM:
-
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time? I
can see you have done quite a bit of coding and testing.
was (Author: samuelan):
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch? I can see you
have done quite a bit of coding and testing.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067920#comment-17067920
]
Sam An edited comment on HIVE-18983 at 3/26/20, 9:57 PM:
-
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you be able to work on this further at this time?
I can see you have done quite a bit of coding and testing.
was (Author: samuelan):
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you be able work on this further at this time? I
can see you have done quite a bit of coding and testing.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067920#comment-17067920
]
Sam An commented on HIVE-18983:
---
[~ikryvenko] hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch? I can see you
have done quite a bit of coding and testing.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22098) Data loss occurs when multiple tables are join with different bucket_version
[
https://issues.apache.org/jira/browse/HIVE-22098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067919#comment-17067919
]
Sam An commented on HIVE-22098:
---
[~luguangming], hope the message finds you well. There are some production use
case for your solution. Would you like to complete this patch?
> Data loss occurs when multiple tables are join with different bucket_version
>
>
> Key: HIVE-22098
> URL: https://issues.apache.org/jira/browse/HIVE-22098
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.1.0, 3.1.2
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Blocker
> Labels: data-loss, wrongresults
> Attachments: HIVE-22098.1.patch, image-2019-08-12-18-45-15-771.png,
> join_test.sql, table_a_data.orc, table_b_data.orc, table_c_data.orc
>
>
> When different bucketVersion of tables do join and no of reducers is greater
> than 2, the result is incorrect (*data loss*).
> *Scenario 1*: Three tables join. The temporary result data of table_a in the
> first table and table_b in the second table joins result is recorded as
> tmp_a_b, When it joins with the third table, the bucket_version=2 of the
> table created by default after hive-3.0.0, temporary data tmp_a_b initialized
> the bucketVerison=-1, and then ReduceSinkOperator Verketison=-1 is joined. In
> the init method, the hash algorithm of selecting join column is selected
> according to bucketVersion. If bucketVersion = 2 and is not an acid
> operation, it will acquired the new algorithm of hash. Otherwise, the old
> algorithm of hash is acquired. Because of the inconsistency of the algorithm
> of hash, the partition of data allocation caused are different. At stage of
> Reducer, Data with the same key can not be paired resulting in data loss.
> *Scenario 2*: create two test tables, create table
> table_bucketversion_1(col_1 string, col_2 string) TBLPROPERTIES
> ('bucketing_version'='1'); table_bucketversion_2(col_1 string, col_2 string)
> TBLPROPERTIES ('bucketing_version'='2');
> when use table_bucketversion_1 to join table_bucketversion_2, partial result
> data will be loss due to bucketVerison is different.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064459#comment-17064459 ] Sam An commented on HIVE-23062: --- Thanks, guys. I made the changes accordingly and tests just ran clean. Will merge tomorrow shortly. Appreciate your time for review. > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch, HIVE-23062.2.patch, > HIVE-23062.3.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23062: -- Attachment: HIVE-23062.3.patch > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch, HIVE-23062.2.patch, > HIVE-23062.3.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23062: -- Attachment: HIVE-23062.2.patch > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch, HIVE-23062.2.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23062: -- Attachment: HIVE-23062.1.patch Status: Patch Available (was: In Progress) > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23062 started by Sam An. - > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23062: - > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22961) Drop function in Hive should not send request for drop database to Ranger plugin.
[
https://issues.apache.org/jira/browse/HIVE-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam An reassigned HIVE-22961:
-
> Drop function in Hive should not send request for drop database to Ranger
> plugin.
> --
>
> Key: HIVE-22961
> URL: https://issues.apache.org/jira/browse/HIVE-22961
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
>
> Issue here is how HIVE sends the "HivePrivilegeObjects" to Ranger when DROP
> fUNTION is done. This is different from how DROP TABLE is done.
> DROP TABLE the following is the request:
> {code:java}
> 'checkPrivileges':{'hiveOpType':DROPTABLE,
> 'inputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW, 'dbName':testdemo,
> 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'outputHObjs':['HivePrivilegeObject':{'type':TABLE_OR_VIEW,
> 'dbName':testdemo, 'objectType':TABLE_OR_VIEW, 'objectName':t1, 'columns':[],
> 'partKeys':[], 'commandParams':[], 'actionType':OTHER, 'owner':systest}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop table t1,
> 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':58f89a16-2df5-4124-af0e-913aabbefe06}, 'user':systest,
> 'groups':[systest, wheel]}{code}
> Where as in DROP FUNCTION:
> {code:java}
> {'hiveOpType':DROPFUNCTION,
> 'inputHObjs':['HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'outputHObjs':['HivePrivilegeObject':{'type':DATABASE, 'dbName':udfdemo,
> 'objectType':DATABASE, 'objectName':null, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER,
> 'owner':systest},'HivePrivilegeObject':{'type':FUNCTION, 'dbName':udfdemo,
> 'objectType':FUNCTION, 'objectName':aes1, 'columns':[], 'partKeys':[],
> 'commandParams':[], 'actionType':OTHER, 'owner':null}],
> 'context':{'clientType':HIVESERVER2, 'commandString':drop function
> udfdemo.aes1, 'ipAddress':10.65.42.125, 'forwardedAddresses':null,
> 'sessionString':442ca4d3-f34a-470c-878a-18542b99016c}, 'user':systest,
> 'groups':[systest, wheel]}
> {code}
> in DROP function in outputHObjs, there is this addition, DATABASE object
> which should not there and this causes the Ranger requested to be generated
> differently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012289#comment-17012289 ] Sam An commented on HIVE-22573: --- Yes, I think so. It already went in cdpd I recall. > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, > HIVE-22573.4.patch, HIVE-22573.5.patch, HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22498) Schema tool enhancements to merge catalogs
[ https://issues.apache.org/jira/browse/HIVE-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991792#comment-16991792 ] Sam An commented on HIVE-22498: --- I clicked the ship it on review board. +1 here, too > Schema tool enhancements to merge catalogs > -- > > Key: HIVE-22498 > URL: https://issues.apache.org/jira/browse/HIVE-22498 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22498.2.patch, HIVE-22498.patch > > > Schema tool currently supports relocation of database from one catalog to > another, one at a time. While having to do this one at a time is painful, it > also lacks support for converting them to external tables during migration, > in lieu of the changes to the translation layer where a MANAGED table is > strictly ACID-only table. > Hence we also need to convert them to external tables during relocation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-22573: - Assignee: Ramesh Mani (was: Sam An) > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22573: -- Attachment: HIVE-22573.patch Status: Patch Available (was: In Progress) ramesh's HMS ranger plugin code > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22573 started by Sam An. - > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-22573: - > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22497) Remove default value for Capabilities from HiveConf
[ https://issues.apache.org/jira/browse/HIVE-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976759#comment-16976759 ] Sam An edited comment on HIVE-22497 at 11/18/19 6:15 PM: - +1. I am ok with the change, except we shall use the defined constant instead string literal for the "hive.metastore.client.capabilities" was (Author: samuelan): +1. I am ok with the change. > Remove default value for Capabilities from HiveConf > --- > > Key: HIVE-22497 > URL: https://issues.apache.org/jira/browse/HIVE-22497 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22497.patch > > > This class is used and bundled in other jars that 3rd party connectors like > teradata etc. So it would be good to remove this default value from HiveConf > but rely on it being set in hive-site.xml instead. The HiveServer2 should > still set this as part of HS2 initialization or via hiveserver2-site.xml -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22497) Remove default value for Capabilities from HiveConf
[ https://issues.apache.org/jira/browse/HIVE-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976759#comment-16976759 ] Sam An commented on HIVE-22497: --- +1. I am ok with the change. > Remove default value for Capabilities from HiveConf > --- > > Key: HIVE-22497 > URL: https://issues.apache.org/jira/browse/HIVE-22497 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22497.patch > > > This class is used and bundled in other jars that 3rd party connectors like > teradata etc. So it would be good to remove this default value from HiveConf > but rely on it being set in hive-site.xml instead. The HiveServer2 should > still set this as part of HS2 initialization or via hiveserver2-site.xml -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22410) CachedStore Prewarm Failure NPE
[
https://issues.apache.org/jira/browse/HIVE-22410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970494#comment-16970494
]
Sam An commented on HIVE-22410:
---
Finally got a change to review the patch. Here are my comments.
* We need unit test
* The logic will add the missing estimator. However, it might not be
addressing the real issue here. The estimator will be null only if the if
statement above it
if (lastClass != elementClass) {
...
}
is false, which means lastClass == elementClass. When that is the case,
lastEstimator should not have been null. But in this case, it is null, and that
in my opinion, points to some assumption in the code is wrong, and it is not
just in array estimation, but also in estimating Collection, Map sizes. This
highlights the importance of having the reproduction scenario and unit test,
otherwise, we don't know what we are actually fixing here.
> CachedStore Prewarm Failure NPE
> ---
>
> Key: HIVE-22410
> URL: https://issues.apache.org/jira/browse/HIVE-22410
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.2, 2.3.7
> Environment: JDK 1.7.0_40
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: cachedstore
> Fix For: 2.3.7, 3.1.3
>
> Attachments: HIVE-22410.1.patch, image-2019-11-01-11-14-42-114.png
>
>
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimateArrayElements(IncrementalObjectSizeEstimator.java:501)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:443)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:433)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:369)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.cached.SharedCache.populateTableInCache(SharedCache.java:1028)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.cached.CachedStore.prewarm(CachedStore.java:314)
> ~[hive-metastore-2.3.5.jar:2.3.5]
> at
> org.apache.hadoop.hive.metastore.cached.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:492)
> [hive-metastore-2.3.5.jar:2.3.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [?:1.7.0_40]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [?:1.7.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [?:1.7.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [?:1.7.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [?:1.7.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [?:1.7.0_40]
> at java.lang.Thread.run(Thread.java:724) [?:1.7.0_40]
> *reason:*
> Methods estimateArrayElements throw NPE
> I found that the last class does not appear in the parent (HashMap,
> Object Estimator> ), and the parent does not contain the information of the
> ConcurrentHashMap Class.
> The solution is to initialize the class information before using it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22410) CachedStore Prewarm Failure NPE
[ https://issues.apache.org/jira/browse/HIVE-22410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964536#comment-16964536 ] Sam An commented on HIVE-22410: --- What I was looking for was the query that resulted in the backtrace you originally listed. If you can just paste the SQL query that crashed in problem description, it'd be great. > CachedStore Prewarm Failure NPE > --- > > Key: HIVE-22410 > URL: https://issues.apache.org/jira/browse/HIVE-22410 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.2, 2.3.7 > Environment: JDK 1.7.0_40 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: cachedstore > Fix For: 2.3.7, 3.1.3 > > Attachments: HIVE-22410.1.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimateArrayElements(IncrementalObjectSizeEstimator.java:501) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:443) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:433) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:369) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.SharedCache.populateTableInCache(SharedCache.java:1028) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.CachedStore.prewarm(CachedStore.java:314) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:492) > [hive-metastore-2.3.5.jar:2.3.5] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [?:1.7.0_40] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [?:1.7.0_40] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [?:1.7.0_40] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.7.0_40] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [?:1.7.0_40] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [?:1.7.0_40] > at java.lang.Thread.run(Thread.java:724) [?:1.7.0_40] > *reason:* > Methods estimateArrayElements throw NPE > I found that the last class does not appear in the parent (HashMap, > Object Estimator> ), and the parent does not contain the information of the > ConcurrentHashMap Class. > The solution is to initialize the class information before using it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22384) Hive MetaStore CachedStore Funcion Cache Dev
[ https://issues.apache.org/jira/browse/HIVE-22384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964389#comment-16964389 ] Sam An commented on HIVE-22384: --- [~slfan1989], thanks for working on this. Could you please add a feature description? I know your codes are about adding/removing functions from cached store. Brief description would help other developers to understand the context. > Hive MetaStore CachedStore Funcion Cache Dev > > > Key: HIVE-22384 > URL: https://issues.apache.org/jira/browse/HIVE-22384 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 2.3.7, 3.1.3 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: cachedstore > Fix For: 2.3.7, 3.1.3 > > Attachments: HIVE-22384.1.patch, HIVE-22384.2.patch, > HIVE-22384.3.patch, HIVE-22384.4.patch, HIVE-22384.5.patch, > HIVE-22384.6.patch, HIVE-22384.7.patch, HIVE-22384.8.patch, HIVE-22384.9.patch > > > Hive MetaStore CachedStore Funcion Cache Dev -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22410) CachedStore Prewarm Failure NPE
[ https://issues.apache.org/jira/browse/HIVE-22410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964388#comment-16964388 ] Sam An commented on HIVE-22410: --- [~slfan1989], thanks for working on it. Can you provide the query that caused the NPE in problem description? > CachedStore Prewarm Failure NPE > --- > > Key: HIVE-22410 > URL: https://issues.apache.org/jira/browse/HIVE-22410 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.2, 2.3.7 > Environment: JDK 1.7.0_40 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: cachedstore > Fix For: 2.3.7, 3.1.3 > > Attachments: HIVE-22410.1.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimateArrayElements(IncrementalObjectSizeEstimator.java:501) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:443) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:433) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:369) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.SharedCache.populateTableInCache(SharedCache.java:1028) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.CachedStore.prewarm(CachedStore.java:314) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:492) > [hive-metastore-2.3.5.jar:2.3.5] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [?:1.7.0_40] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [?:1.7.0_40] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [?:1.7.0_40] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.7.0_40] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [?:1.7.0_40] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [?:1.7.0_40] > at java.lang.Thread.run(Thread.java:724) [?:1.7.0_40] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22406) TRUNCATE TABLE fails due MySQL limitations on limit value
[
https://issues.apache.org/jira/browse/HIVE-22406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962232#comment-16962232
]
Sam An commented on HIVE-22406:
---
+1
> TRUNCATE TABLE fails due MySQL limitations on limit value
> -
>
> Key: HIVE-22406
> URL: https://issues.apache.org/jira/browse/HIVE-22406
> Project: Hive
> Issue Type: Bug
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22406.patch, HIVE-22406.patch
>
>
> HMS currently has some APIs that accepts an integer limit value. Prior to the
> change in HIVE-21734, HMS was silently converting this int to short and thus
> we havent seen this issue. But semantically, its incorrect to do so quietly.
> {noformat}
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> Caused by: java.sql.SQLException: setMaxRows() out of range. 2147483647 >
> 5000.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.StatementImpl.setMaxRows(StatementImpl.java:2525) ~[mysql-
> connector-java.jar:5.1.33]
> at
> com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setMaxRows(HikariProxyPreparedS
> tatement.java) ~[HikariCP-2.6.1.jar:?]
> {noformat}
> We cannot change the RawStore api to accept shorts instead of ints.
> So we have to fix the caller to use a lower limit instead of
> Integer.MAX_VALUE.
> {noformat}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Exception
> thrown when executing query : SELECT DISTINCT
> 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`WRITE_ID`,`A0`.`PART_ID`,`A0`.`PART_NAME`
> AS `NUCORDER0` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON
> `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` =
> `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND
> `C0`.`CTLG_NAME` = ? ORDER BY `NUCORDER0` LIMIT 0,2147483647
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result$truncate_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result$truncate_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result.read(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_truncate_table_req(ThriftHiveMetastore.java:1999)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.truncate_table_req(ThriftHiveMetastore.java:1986)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTableInternal(HiveMetaStoreClient.java:1450)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTable(HiveMetaStoreClient.java:1427)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.truncateTable(SessionHiveMetaStoreClient.java:171)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at com.sun.proxy.$Proxy59.truncateTable(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
[jira] [Commented] (HIVE-22342) HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables
[ https://issues.apache.org/jira/browse/HIVE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955214#comment-16955214 ] Sam An commented on HIVE-22342: --- +1. Except To have consistent style, we can change "true" to "TRUE" in MetastoreDefaultTransformer.java > HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables > > > Key: HIVE-22342 > URL: https://issues.apache.org/jira/browse/HIVE-22342 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22342.2.patch, HIVE-22342.3.patch, HIVE-22342.patch > > > HIVE-22189 restricts EXTERNAL tables being created to be restricted to the > EXTERNAL_WAREHOUSE_DIR. This might be too strict as any other location should > be allowed as long as the location is outside the MANAGED warehouse directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943045#comment-16943045 ] Sam An commented on HIVE-5: --- Thanks, Naveen. As communicated offline. I believe we should be OK in not automatically filling in the ownership for null owner DBs. The reasons are First, it might not be always possible to figure who owns the DB. Second, if ownership is needed, the user can ask admin to assign a new owner. Once 5 goes in, most likely null ownership won’t happen much. > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch, HIVE-5.2.patch, > HIVE-5.4.patch, Hive-5.3.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22266) Addendum fix to have HS2 pom add explicit curator dependency
[ https://issues.apache.org/jira/browse/HIVE-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941153#comment-16941153 ] Sam An commented on HIVE-22266: --- +1 > Addendum fix to have HS2 pom add explicit curator dependency > > > Key: HIVE-22266 > URL: https://issues.apache.org/jira/browse/HIVE-22266 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22266.patch > > > It might be better to add an explicit dependency on apache-curator in the > service/pom.xml. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-5: -- Attachment: HIVE-5.4.patch > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch, HIVE-5.2.patch, > HIVE-5.4.patch, Hive-5.3.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938128#comment-16938128 ] Sam An commented on HIVE-22205: --- +1 > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, > HIVE-22205.4.patch, HIVE-22205.6.patch, HIVE-22205.7.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937935#comment-16937935 ] Sam An commented on HIVE-5: --- Test failures for .2 patch do not seem to be related to my changes. Recheck... > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch, HIVE-5.2.patch, > Hive-5.3.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-5: -- Attachment: Hive-5.3.patch > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch, HIVE-5.2.patch, > Hive-5.3.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-5: -- Attachment: HIVE-5.2.patch > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch, HIVE-5.2.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-5: -- Attachment: HIVE-5.1.patch Status: Patch Available (was: Open) > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-5: - > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22138) Upgrade script for Hive version
[ https://issues.apache.org/jira/browse/HIVE-22138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-22138: - > Upgrade script for Hive version > --- > > Key: HIVE-22138 > URL: https://issues.apache.org/jira/browse/HIVE-22138 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Hive metastore and Hive schema version will be bumped to 3.1.2000, and we > need to have corresponding sql script to handle that. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.8.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch, HIVE-22063.4.patch, HIVE-22063.5.patch, > HIVE-22063.6.patch, HIVE-22063.7.patch, HIVE-22063.8.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.7.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch, HIVE-22063.4.patch, HIVE-22063.5.patch, > HIVE-22063.6.patch, HIVE-22063.7.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.6.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch, HIVE-22063.4.patch, HIVE-22063.5.patch, HIVE-22063.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22101) table RUNTIME_STATS Schema Script for Postgres Metastore is wrong
[ https://issues.apache.org/jira/browse/HIVE-22101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22101: -- Attachment: HIVE-22101.1.patch Status: Patch Available (was: Open) > table RUNTIME_STATS Schema Script for Postgres Metastore is wrong > - > > Key: HIVE-22101 > URL: https://issues.apache.org/jira/browse/HIVE-22101 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-22101.1.patch > > > In some test clusters, we found the following backtrace for Postgres backed > metastore. > 2019-07-19 14:15:43,847 WARN DataNucleus.Query: [pool-5-thread-2]: Query for > candidates of org.apache.hadoop.hive.metastore.model.MRuntimeStat and > subclasses resulted in no possible candidates > org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table > missing : ""RUNTIME_STATS"" in Catalog "" Schema "". DataNucleus requires > this table to perform its persistence operations. Either your MetaData is > incorrect, or you need to enable "datanucleus.schema.autoCreateTables" > at > org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:606) > ~[datanucleus-rdbms-4.1.19.jar:?] > at > org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3385) > ~[datanucleus-rdbms-4.1.19.jar:?] > at > org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2896) > ~[datanucleus-rdbms-4.1.19.jar:?] > at > org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119) > ~[datanucleus-rdbms-4.1.19.jar:?] > The root cause of this is RUNTIME_STATS table is created without quotes, > resulting in lower case table. When it's queried, the table cannot be found. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22101) table RUNTIME_STATS Schema Script for Postgres Metastore is wrong
[ https://issues.apache.org/jira/browse/HIVE-22101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-22101: - > table RUNTIME_STATS Schema Script for Postgres Metastore is wrong > - > > Key: HIVE-22101 > URL: https://issues.apache.org/jira/browse/HIVE-22101 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > In some test clusters, we found the following backtrace for Postgres backed > metastore. > 2019-07-19 14:15:43,847 WARN DataNucleus.Query: [pool-5-thread-2]: Query for > candidates of org.apache.hadoop.hive.metastore.model.MRuntimeStat and > subclasses resulted in no possible candidates > org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table > missing : ""RUNTIME_STATS"" in Catalog "" Schema "". DataNucleus requires > this table to perform its persistence operations. Either your MetaData is > incorrect, or you need to enable "datanucleus.schema.autoCreateTables" > at > org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:606) > ~[datanucleus-rdbms-4.1.19.jar:?] > at > org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3385) > ~[datanucleus-rdbms-4.1.19.jar:?] > at > org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2896) > ~[datanucleus-rdbms-4.1.19.jar:?] > at > org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119) > ~[datanucleus-rdbms-4.1.19.jar:?] > The root cause of this is RUNTIME_STATS table is created without quotes, > resulting in lower case table. When it's queried, the table cannot be found. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.5.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch, HIVE-22063.4.patch, HIVE-22063.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898261#comment-16898261 ] Sam An commented on HIVE-22063: --- Local test of the 2 failures for .1 and .2 patches are clean. Resubmit to check again. > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch, HIVE-22063.4.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.4.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch, HIVE-22063.4.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897738#comment-16897738 ] Sam An commented on HIVE-22063: --- 22063.3.patch is a dummy that contains only latest master commit. Just test to see if PreCommit test is flaky. > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.3.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.2.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.1.patch Status: Patch Available (was: In Progress) > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22063 started by Sam An. - > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-22063: - > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21787) Metastore table cache LRU eviction
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.14.patch > Metastore table cache LRU eviction > -- > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, > HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.14.patch, > HIVE-21787.2.patch, HIVE-21787.3.patch, HIVE-21787.4.patch, > HIVE-21787.5.patch, HIVE-21787.6.patch, HIVE-21787.7.patch, > HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache LRU eviction
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.13.patch > Metastore table cache LRU eviction > -- > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, > HIVE-21787.11.patch, HIVE-21787.13.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache LRU eviction
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: (was: HIVE-21787.12.patch) > Metastore table cache LRU eviction > -- > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, > HIVE-21787.11.patch, HIVE-21787.2.patch, HIVE-21787.3.patch, > HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch, > HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache LRU eviction
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.12.patch > Metastore table cache LRU eviction > -- > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, > HIVE-21787.11.patch, HIVE-21787.12.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache LRU eviction
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.11.patch > Metastore table cache LRU eviction > -- > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, > HIVE-21787.11.patch, HIVE-21787.2.patch, HIVE-21787.3.patch, > HIVE-21787.4.patch, HIVE-21787.5.patch, HIVE-21787.6.patch, > HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.10.patch > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.10.patch, > HIVE-21787.2.patch, HIVE-21787.3.patch, HIVE-21787.4.patch, > HIVE-21787.5.patch, HIVE-21787.6.patch, HIVE-21787.7.patch, > HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866061#comment-16866061 ] Sam An commented on HIVE-21833: --- Thanks a lot, Dan. I will cherry pick to Gerrit soon as well. > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch, HIVE-21833.8.patch, HIVE-21833.9.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866056#comment-16866056 ] Sam An commented on HIVE-21833: --- We choose not to add ownername and ownertype in the toString() method, because in many NegativeCliDriver tests, when authorization fails, tests print out the HivePrivilegeObject content. By adding these 2, there will be mismatches each time a new user submit the test. That's why we don't add it in toString, but only in compareTo method. > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch, HIVE-21833.8.patch, HIVE-21833.9.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Attachment: (was: HIVE-21833.10.patch) > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch, HIVE-21833.8.patch, HIVE-21833.9.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Attachment: HIVE-21833.10.patch > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.10.patch, > HIVE-21833.2.patch, HIVE-21833.6.patch, HIVE-21833.7.patch, > HIVE-21833.8.patch, HIVE-21833.9.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.9.patch > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21787.9.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: (was: HIVE-21833.8.patch) > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21787.8.patch Status: Patch Available (was: Open) > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21787.8.patch, HIVE-21833.8.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Status: Open (was: Patch Available) wrong patch > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21833.8.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Status: Open (was: Patch Available) > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch, HIVE-21833.8.patch, HIVE-21833.9.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Attachment: HIVE-21833.9.patch Status: Patch Available (was: Open) > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch, HIVE-21833.8.patch, HIVE-21833.9.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Attachment: HIVE-21833.8.patch Status: Patch Available (was: Open) > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch, HIVE-21833.8.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21787) Metastore table cache enhancement
[ https://issues.apache.org/jira/browse/HIVE-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21787: -- Status: Open (was: Patch Available) > Metastore table cache enhancement > - > > Key: HIVE-21787 > URL: https://issues.apache.org/jira/browse/HIVE-21787 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21787.1.patch, HIVE-21787.2.patch, > HIVE-21787.3.patch, HIVE-21787.4.patch, HIVE-21787.5.patch, > HIVE-21787.6.patch, HIVE-21787.7.patch > > > Metastore currently uses black/white list to specify patterns of tables to > load into the cache. Cache is loaded in one shot "prewarm", and updated by a > background thread. This is not a very efficient design. > In this feature, we try to enhance the cache for Tables with LRU to improve > cache utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Attachment: HIVE-21833.8.patch Status: Patch Available (was: Open) > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch, HIVE-21833.8.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Attachment: (was: HIVE-21833.3.patch) > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21833) Ranger Authorization in Hive based on object ownership
[ https://issues.apache.org/jira/browse/HIVE-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-21833: -- Status: Open (was: Patch Available) > Ranger Authorization in Hive based on object ownership > -- > > Key: HIVE-21833 > URL: https://issues.apache.org/jira/browse/HIVE-21833 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-21833.1.patch, HIVE-21833.2.patch, > HIVE-21833.6.patch, HIVE-21833.7.patch > > > Background: Currently Hive Authorizer for Ranger does not provide owner > information for Hive objects as part of AuthZ calls. This has resulted in > gaps with respect to Sentry AuthZ and customers/partners cannot leverage > privileges for owners in their authorization model. > > User Story: As an enterprise security admin, I need to be able to set > privileges based on Hive object ownership for setting up access controls in > Ranger so that I can provide appropriate protections and permissions for my > enterprise users. > > Acceptance criteria: > 1) Owner information is available in Hive -Ranger AuthZ calls > 2) Ranger admin users can use owner information to set policies based on > object ownership in Ranger UI and APIs > 3) OWNER Macro based policies continue to work for Hive objects -- This message was sent by Atlassian JIRA (v7.6.3#76005)
