[jira] [Updated] (HIVE-27114) Provide a configurable filter for removing useless properties in Partition objects from listPartitions HMS Calls

2023-09-20 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-27114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-27114:
--
Labels: pull-request-available  (was: )

> Provide a configurable filter for removing useless properties in Partition 
> objects from listPartitions HMS Calls
> 
>
> Key: HIVE-27114
> URL: https://issues.apache.org/jira/browse/HIVE-27114
> Project: Hive
>  Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
>
> HMS API calls are throwing following exception because of thrift upgrade
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
>         at 
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>  
>         at 
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>  
>         at 
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) 
>         at 
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>  
>         at 
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) 
>         at 
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>  
>         at 
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
>  
>         at 
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) 
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
>  
>         at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) 
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
>  
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1782)
>  
>         at 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitions(SessionHiveMetaStoreClient.java:1134)
>  
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1775)
>  
>         at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) 
> ~[?:?]
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_311]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311]
>         at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
>  
>         at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?]
>         at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) 
> ~[?:?]
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_311]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311]
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3550)
>  
>         at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?]
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:3793) 
>         at 
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getAllPartitions(PartitionPruner.java:485)
>    {code}
> Large size partition metadata is causing this issue
> eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys = 
> (impala_intermediate_stats_chunk{*}{*}), these PARTITION_PARAM_KEYS are not 
> required for Hive. These params should be skipped while preparing partition 
> object from HMS to HS2.
> Similar to HIVE-25501, any user defined regex param_keys should be skipped in 
> listPartitions HMS API call response.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (HIVE-27114) Provide a configurable filter for removing useless properties in Partition objects from listPartitions HMS Calls

2023-02-28 Thread Naresh P R (Jira)


 [ 
https://issues.apache.org/jira/browse/HIVE-27114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naresh P R updated HIVE-27114:
--
Summary: Provide a configurable filter for removing useless properties in 
Partition objects from listPartitions HMS Calls  (was: Provide a configurable 
filter for removing useless properties in Partition objects from getPartitions 
HMS Calls)

> Provide a configurable filter for removing useless properties in Partition 
> objects from listPartitions HMS Calls
> 
>
> Key: HIVE-27114
> URL: https://issues.apache.org/jira/browse/HIVE-27114
> Project: Hive
>  Issue Type: Bug
>Reporter: Naresh P R
>Priority: Major
>
> HMS API calls are throwing following exception because of thrift upgrade
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
>         at 
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>  
>         at 
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>  
>         at 
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390) 
>         at 
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>  
>         at 
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) 
>         at 
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>  
>         at 
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:417)
>  
>         at 
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1286)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062) 
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java)
>  
>         at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) 
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:3290)
>  
>         at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:3275)
>  
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1782)
>  
>         at 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitions(SessionHiveMetaStoreClient.java:1134)
>  
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1775)
>  
>         at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) 
> ~[?:?]
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_311]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311]
>         at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
>  
>         at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?]
>         at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source) 
> ~[?:?]
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_311]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311]
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3550)
>  
>         at com.sun.proxy.$Proxy52.listPartitions(Unknown Source) ~[?:?]
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.getAllPartitionsOf(Hive.java:3793) 
>         at 
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.getAllPartitions(PartitionPruner.java:485)
>    {code}
> Large size partition metadata is causing this issue
> eg., impala stores huge stats chunk in partitionMetadata with {*}param_keys = 
> (impala_intermediate_stats_chunk{*}{*}), these PARTITION_PARAM_KEYS are not 
> required for Hive. These params should be skipped while preparing partition 
> object from HMS to HS2.
> Similar to HIVE-25501, any user defined regex param_keys should be skipped in 
> listPartitions HMS