[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066402#comment-14066402
 ] 

Todd Lipcon commented on MAPREDUCE-5974:
----------------------------------------

Yea, I briefly considered doing that -- introducing a new 
"FallbackMapOutputCollector" and a new configuration like 
"map.output.collector.fallback.list". But it seemed like it introduced extra 
complexity for the user to have to make two config changes instead of just one.

Agreed that the fallback could happen at job-submit time. But, we don't do any 
other "pre-flight config tweaks", so I'm not sure where that would go. Plus, 
like you said, different machines might have different architectures, or one 
machine might be missing native libs due to a config error.

Doing fallback as the records are emitted would be pretty neat, but may also be 
somewhat difficult. Let's say you'd already emitted 100MB of records to one 
collector when you come upon one that can't be handled by the collector you're 
using. How do you take the already-emitted records and "forward them" along to 
the other collector? It seems like it would be pretty difficult and require a 
lot of work.

Compatibility-wise, if someone does go ahead and implement the above (fallback 
at runtime instead of submit time) we could still keep the same configuration 
that I've introduced here, which would be compatible for users, right?

> Allow map output collector fallback
> -----------------------------------
>
>                 Key: MAPREDUCE-5974
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5974
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: task
>    Affects Versions: 2.6.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-5974.txt
>
>
> Currently we only allow specifying a single MapOutputCollector implementation 
> class in a job. It would be nice to allow a comma-separated list of classes: 
> we should try each collector implementation in the user-specified order until 
> we find one that can be successfully instantiated and initted.
> This is useful for cases where a particular optimized collector 
> implementation cannot operate on all key/value types, or requires native 
> code. The cluster administrator can configure the cluster to try to use the 
> optimized collector and fall back to the default collector.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to