[jira] [Updated] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4322: --- Fix Version/s: (was: 0.11.0) 0.12.0 SkewedInfo in Metastore Thrift API cannot be deserialized in Python --- Key: HIVE-4322 URL: https://issues.apache.org/jira/browse/HIVE-4322 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch The Thrift-generated Python code that deserializes Thrift objects fails whenever a complex type is used as a map key, because by default mutable Python objects such as lists do not have a hash function. See https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. The SkewedInfo struct contains a map which uses a list as a key, breaking the Python Thrift interface. It is not possible to specify the mapping from Thrift types to Python types, or otherwise we could map Thrift lists to Python tuples. Instead, the proposed workaround wraps the list inside a new struct. This alone does not accomplish anything, but allows Python clients to define a hash function for the struct class, e.g.: def f(object): return hash(tuple(object.skewedValueList)) SkewedValueList.__hash__ = f In practice a more efficient hash might be defined that does not involve copying the list. The advantage of wrapping the list inside a struct is that the client does not have to define the hash on the list itself, which would change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Tim Liu updated HIVE-4322: --- Resolution: Fixed Fix Version/s: 0.11.0 Status: Resolved (was: Patch Available) SkewedInfo in Metastore Thrift API cannot be deserialized in Python --- Key: HIVE-4322 URL: https://issues.apache.org/jira/browse/HIVE-4322 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Minor Fix For: 0.11.0 Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch The Thrift-generated Python code that deserializes Thrift objects fails whenever a complex type is used as a map key, because by default mutable Python objects such as lists do not have a hash function. See https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. The SkewedInfo struct contains a map which uses a list as a key, breaking the Python Thrift interface. It is not possible to specify the mapping from Thrift types to Python types, or otherwise we could map Thrift lists to Python tuples. Instead, the proposed workaround wraps the list inside a new struct. This alone does not accomplish anything, but allows Python clients to define a hash function for the struct class, e.g.: def f(object): return hash(tuple(object.skewedValueList)) SkewedValueList.__hash__ = f In practice a more efficient hash might be defined that does not involve copying the list. The advantage of wrapping the list inside a struct is that the client does not have to define the hash on the list itself, which would change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4322: -- Attachment: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch sxyuan requested code review of HIVE-4322 [jira] SkewedInfo in Metastore Thrift API cannot be deserialized in Python. Reviewers: gangtimliu See JIRA for description. TEST PLAN Ran both hadoop20 and hadoop23 tests. REVISION DETAIL https://reviews.facebook.net/D10203 AFFECTED FILES service/src/gen/thrift/gen-php/hive_service service/src/gen/thrift/gen-php/hive_service/ThriftHive.php service/src/gen/thrift/gen-php/hive_service/hive_service_types.php metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb metastore/src/gen/thrift/gen-php/hive_metastore_constants.php metastore/src/gen/thrift/gen-php/metastore/Types.php metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php metastore/src/gen/thrift/gen-php/hive_metastore_types.php metastore/src/gen/thrift/gen-php/hive_metastore metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java metastore/if/hive_metastore.thrift ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPruner.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java ql/src/gen/thrift/gen-php/queryplan ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/24381/ To: gangtimliu, sxyuan Cc: kevinwilfong, JIRA SkewedInfo in Metastore Thrift API cannot be deserialized in Python --- Key: HIVE-4322 URL: https://issues.apache.org/jira/browse/HIVE-4322 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Minor Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch The Thrift-generated Python code that deserializes Thrift objects fails whenever a complex type is used as a map key, because by default mutable Python objects such as lists do not have a hash function. See https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. The SkewedInfo struct contains a map which uses a list as a key, breaking the Python Thrift interface. It is not possible to specify the mapping from Thrift types to Python types, or otherwise we could map Thrift lists to Python tuples. Instead, the proposed workaround wraps the list inside a new struct. This alone does not accomplish anything, but allows Python clients to define a hash function for the struct class, e.g.: def f(object):
[jira] [Updated] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samuel Yuan updated HIVE-4322: -- Status: Patch Available (was: Open) SkewedInfo in Metastore Thrift API cannot be deserialized in Python --- Key: HIVE-4322 URL: https://issues.apache.org/jira/browse/HIVE-4322 Project: Hive Issue Type: Bug Components: Metastore, Thrift API Affects Versions: 0.11.0 Reporter: Samuel Yuan Assignee: Samuel Yuan Priority: Minor Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch The Thrift-generated Python code that deserializes Thrift objects fails whenever a complex type is used as a map key, because by default mutable Python objects such as lists do not have a hash function. See https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. The SkewedInfo struct contains a map which uses a list as a key, breaking the Python Thrift interface. It is not possible to specify the mapping from Thrift types to Python types, or otherwise we could map Thrift lists to Python tuples. Instead, the proposed workaround wraps the list inside a new struct. This alone does not accomplish anything, but allows Python clients to define a hash function for the struct class, e.g.: def f(object): return hash(tuple(object.skewedValueList)) SkewedValueList.__hash__ = f In practice a more efficient hash might be defined that does not involve copying the list. The advantage of wrapping the list inside a struct is that the client does not have to define the hash on the list itself, which would change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
