[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631241#comment-13631241 ] Hudson commented on HIVE-4322: -- Integrated in Hive-trunk-hadoop2 #157 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/157/]) HIVE-4322: SkewedInfo in Metastore Thrift API cannot be deserialized in Python (Samuel Yuan via Gang Tim Liu) (Revision 1467491) Result = FAILURE gangtimliu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467491 Files : * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPruner.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Fix For: 0.11.0 > > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631186#comment-13631186 ] Hudson commented on HIVE-4322: -- Integrated in Hive-trunk-h0.21 #2062 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2062/]) HIVE-4322: SkewedInfo in Metastore Thrift API cannot be deserialized in Python (Samuel Yuan via Gang Tim Liu) (Revision 1467491) Result = FAILURE gangtimliu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467491 Files : * /hive/trunk/metastore/if/hive_metastore.thrift * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp * /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java * /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java * /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php * /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py * /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPruner.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Fix For: 0.11.0 > > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more i
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630689#comment-13630689 ] Gang Tim Liu commented on HIVE-4322: Committed. thank Samuel Yuan. > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630605#comment-13630605 ] Phabricator commented on HIVE-4322: --- gangtimliu has accepted the revision "HIVE-4322 [jira] SkewedInfo in Metastore Thrift API cannot be deserialized in Python". thanks REVISION DETAIL https://reviews.facebook.net/D10203 BRANCH svn ARCANIST PROJECT hive To: gangtimliu, sxyuan Cc: kevinwilfong, JIRA > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630553#comment-13630553 ] Phabricator commented on HIVE-4322: --- sxyuan has commented on the revision "HIVE-4322 [jira] SkewedInfo in Metastore Thrift API cannot be deserialized in Python". INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1127 Just fixed the name of the function. REVISION DETAIL https://reviews.facebook.net/D10203 To: gangtimliu, sxyuan Cc: kevinwilfong, JIRA > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630549#comment-13630549 ] Phabricator commented on HIVE-4322: --- gangtimliu has commented on the revision "HIVE-4322 [jira] SkewedInfo in Metastore Thrift API cannot be deserialized in Python". code lgtm. looking at hadoop 23 tests. INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1127 what's difference in this line? thanks REVISION DETAIL https://reviews.facebook.net/D10203 To: gangtimliu, sxyuan Cc: kevinwilfong, JIRA > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630546#comment-13630546 ] Gang Tim Liu commented on HIVE-4322: +1 after test passes > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch > > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
[ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627275#comment-13627275 ] Gang Tim Liu commented on HIVE-4322: [~sxyuan] Good write up. thank you for working on it. > SkewedInfo in Metastore Thrift API cannot be deserialized in Python > --- > > Key: HIVE-4322 > URL: https://issues.apache.org/jira/browse/HIVE-4322 > Project: Hive > Issue Type: Bug > Components: Metastore, Thrift API >Affects Versions: 0.11.0 >Reporter: Samuel Yuan >Assignee: Samuel Yuan >Priority: Minor > > The Thrift-generated Python code that deserializes Thrift objects fails > whenever a complex type is used as a map key, because by default mutable > Python objects such as lists do not have a hash function. See > https://issues.apache.org/jira/browse/THRIFT-162 for related discussion. > The SkewedInfo struct contains a map which uses a list as a key, breaking the > Python Thrift interface. It is not possible to specify the mapping from > Thrift types to Python types, or otherwise we could map Thrift lists to > Python tuples. Instead, the proposed workaround wraps the list inside a new > struct. This alone does not accomplish anything, but allows Python clients to > define a hash function for the struct class, e.g.: > def f(object): > return hash(tuple(object.skewedValueList)) > SkewedValueList.__hash__ = f > In practice a more efficient hash might be defined that does not involve > copying the list. The advantage of wrapping the list inside a struct is that > the client does not have to define the hash on the list itself, which would > change the behaviour of lists everywhere else in the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
