[
https://issues.apache.org/jira/browse/PIG-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-2999:
------------------------------
Attachment: pig-2999-v2.txt
In WritableComparator, it converts to 'int' and then compare.
{noformat}
123 public static int compareBytes(byte[] b1, int s1, int l1,
124 byte[] b2, int s2, int l2) {
125 int end1 = s1 + l1;
126 int end2 = s2 + l2;
127 for (int i = s1, j = s2; i < end1 && j < end2; i++, j++) {
128 int a = (b1[i] & 0xff);
129 int b = (b2[j] & 0xff);
130 if (a != b) {
131 return a - b;
132 }
133 }
134 return l1 - l2;
135 }
{noformat}
whereas, DataByteArray compares directly as byte. To make it consistent with
other shortcurts we added in PIG-2975, making the DataByteArray.compare simply
call WritableComparator.compareBytes. This fixes the
TestPigTupleRawComparator.testRandomTuples failure.
> Regression after PIG-2975: BinInterSedesTupleRawComparator secondary sort
> failing
> ---------------------------------------------------------------------------------
>
> Key: PIG-2999
> URL: https://issues.apache.org/jira/browse/PIG-2999
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11, 0.12
> Reporter: Koji Noguchi
> Assignee: Jonathan Coveney
> Attachments: pig-2999-v1.txt, pig-2999-v2.txt
>
>
> I think I broke the build from PIG-2975. I see couple of tests failing at
> BinInterSedesTupleRawComparator.
> {noformat}
> 12/10/22 22:26:15 WARN mapred.LocalJobRunner: job_local_0022
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:478)
> at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:387)
> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:829)
> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:732)
> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
> at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
> at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:139)
> at
> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
> at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
> at org.apache.hadoop.mapred.ReduceTask$4.next(ReduceTask.java:625)
> at
> org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:117)
> at
> org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
