[
https://issues.apache.org/jira/browse/MAPREDUCE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071247#comment-14071247
]
Todd Lipcon commented on MAPREDUCE-5974:
----------------------------------------
In the case of the native collector, it's still the same IFile format on disk,
and the same reducer. I'm not sure about whether that's the case with other map
collectors out there (eg from vendors) but I seem to recall some folks working
on a specialized collector for memcmp-able keys. In that case, it might be nice
to have a priority list like
"MemcmpableKeyCollector,NativeCollector,DefaultCollector", and each one would
just throw an exception if it didn't support the types involved.
Implementing this inside the native collector init() method itself might be
messy -- you'd have to essentially write a wrapper collector and have every
method delegate to the real implementation. I would hope that the delegation
would get devirtualized and inlined, but not certain about that. If you're -0
or -1 on the current approach though, I'm willing to give it a go.
> Allow map output collector fallback
> -----------------------------------
>
> Key: MAPREDUCE-5974
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5974
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
> Affects Versions: 2.6.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-5974.txt
>
>
> Currently we only allow specifying a single MapOutputCollector implementation
> class in a job. It would be nice to allow a comma-separated list of classes:
> we should try each collector implementation in the user-specified order until
> we find one that can be successfully instantiated and initted.
> This is useful for cases where a particular optimized collector
> implementation cannot operate on all key/value types, or requires native
> code. The cluster administrator can configure the cluster to try to use the
> optimized collector and fall back to the default collector.
--
This message was sent by Atlassian JIRA
(v6.2#6252)