[
https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-4322:
------------------------------
Attachment: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch
sxyuan requested code review of "HIVE-4322 [jira] SkewedInfo in Metastore
Thrift API cannot be deserialized in Python".
Reviewers: gangtimliu
See JIRA for description.
TEST PLAN
Ran both hadoop20 and hadoop23 tests.
REVISION DETAIL
https://reviews.facebook.net/D10203
AFFECTED FILES
service/src/gen/thrift/gen-php/hive_service
service/src/gen/thrift/gen-php/hive_service/ThriftHive.php
service/src/gen/thrift/gen-php/hive_service/hive_service_types.php
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
metastore/src/gen/thrift/gen-php/hive_metastore_constants.php
metastore/src/gen/thrift/gen-php/metastore/Types.php
metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php
metastore/src/gen/thrift/gen-php/hive_metastore_types.php
metastore/src/gen/thrift/gen-php/hive_metastore
metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php
metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
metastore/if/hive_metastore.thrift
ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPruner.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
ql/src/gen/thrift/gen-php/queryplan
ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
MANAGE HERALD RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/24381/
To: gangtimliu, sxyuan
Cc: kevinwilfong, JIRA
> SkewedInfo in Metastore Thrift API cannot be deserialized in Python
> -------------------------------------------------------------------
>
> Key: HIVE-4322
> URL: https://issues.apache.org/jira/browse/HIVE-4322
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Thrift API
> Affects Versions: 0.11.0
> Reporter: Samuel Yuan
> Assignee: Samuel Yuan
> Priority: Minor
> Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch
>
>
> The Thrift-generated Python code that deserializes Thrift objects fails
> whenever a complex type is used as a map key, because by default mutable
> Python objects such as lists do not have a hash function. See
> https://issues.apache.org/jira/browse/THRIFT-162 for related discussion.
> The SkewedInfo struct contains a map which uses a list as a key, breaking the
> Python Thrift interface. It is not possible to specify the mapping from
> Thrift types to Python types, or otherwise we could map Thrift lists to
> Python tuples. Instead, the proposed workaround wraps the list inside a new
> struct. This alone does not accomplish anything, but allows Python clients to
> define a hash function for the struct class, e.g.:
> def f(object):
> return hash(tuple(object.skewedValueList))
> SkewedValueList.__hash__ = f
> In practice a more efficient hash might be defined that does not involve
> copying the list. The advantage of wrapping the list inside a struct is that
> the client does not have to define the hash on the list itself, which would
> change the behaviour of lists everywhere else in the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
