[
https://issues.apache.org/jira/browse/PIG-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959895#comment-15959895
]
liyunzhang_intel edited comment on PIG-5197 at 4/6/17 10:28 PM:
----------------------------------------------------------------
[~rohini]: we can not replace IndexedKey with PigNullableWritable.I replaced
IndexedKey with PigNullableWriable in PIG-5197.patch. Just run
TestSparkSecondarySort to verify.
TestSecondarySortSpark#testNestedSortMultiQueryEndToEnd3 fails and throws
exception like
{code}
had a not serializable result: org.apache.hadoop.io.Text$
{code}
It is because {code} HDataType.getWritableComparableTypes ->
org.apache.pig.impl.io.NullableText#NullableText(java.lang.String)->org.apache.hadoop.io.Text{code}
For the case we use chararray as type, this exception will be thrown out as
org.apache.hadoop.io.Text is not serializable. Can you provide suggestion to
solve it or remain IndexedKey in spark package?
was (Author: kellyzly):
[~rohini]: we can not replace IndexedKey with PigNullableWritable.I replaced
IndexedKey with PigNullableWriable in PIG-5197.patch. Just run
TestSparkSecondarySort to verify.
TestSecondarySortSpark#testNestedSortMultiQueryEndToEnd3 fails and throws
exception like
{code}
had a not serializable result: org.apache.hadoop.io.Text$
{code}
It is because {code} HDataType.getWritableComparableTypes ->
org.apache.pig.impl.io.NullableText#NullableText(java.lang.String)->org.apache.hadoop.io.Text{code}
For the case we use chararray as type, this exception will be thrown out. Can
you provide suggestion to solve it or remain IndexedKey in spark package?
> Replace IndexedKey with PigNullableWritable in spark branch
> -----------------------------------------------------------
>
> Key: PIG-5197
> URL: https://issues.apache.org/jira/browse/PIG-5197
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-5197.patch
>
>
> The function of IndexedKey and PigNullableWritable is similar.
> The difference between these two is IndexedKey contains Index,key while
> PigNullableWritable contains index,key,value.
> Besides,the comparators for PigNullableWritable have lot of conditions for
> the different data types taken care of and IndexedKey can miss some of that.
> We can try to replace IndexedKey with PigNullableWritable.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
