[
https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044886#comment-15044886
]
Colonel.Hou commented on PIG-282:
---------------------------------
Daniel Dai, thank you for your answer, all exception is below:
Backend error message
---------------------
Error: java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
at
com.compcc.ps.pig.TotalSortPartitioner.getPartition(TotalSortPartitioner.java:21)
at
com.compcc.ps.pig.TotalSortPartitioner.getPartition(TotalSortPartitioner.java:12)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:135
)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
My write partition is :
public class TotalSortPartitioner extends Partitioner<PigNullableWritable,
Writable>
{
public static final Log log = LogFactory.getLog(TotalSortPartitioner.class);
public static final int ONE_HOUR_MINUTE = 60;
public int getPartition(PigNullableWritable key, Writable value, int
numPartitions)
{
int minute = -1;
try {
minute = Integer.parseInt(key.getValueAsPigType().toString());
} catch (RuntimeException e) {
log.error("Convert String minute to Integer error. " + e.getMessage());
return minute;
}
...
return xxx;
}
}
> Custom Partitioner
> ------------------
>
> Key: PIG-282
> URL: https://issues.apache.org/jira/browse/PIG-282
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.7.0
> Reporter: Amir Youssefi
> Assignee: Aniket Mokashi
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: CustomPartitioner.patch, CustomPartitionerFinale.patch,
> CustomPartitionerTest.patch
>
>
> By adding custom partitioner we can give control over which output partition
> a key (/value) goes to. We can add keywords to language e.g.
> PARTITION BY UDF(...)
> or a similar syntax. UDF returns a number between 0 and n-1 where n is number
> of output partitions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
