[
https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888433#action_12888433
]
Doug Cutting commented on MAPREDUCE-1938:
-----------------------------------------
Two thoughts:
1. In general, we need to better separate the kernel from the library.
CombineFileInputFormat is library code and should be easy to update without
updating the cluster. Long-term, only kernel code should be hardwired on the
classpath of tasks, with library and user code both specified per job. There
should be no default version of library classes for a task: tasks should always
specify their required libraries. Is there a Jira for this? I know Tom's
expressed interest in working on this.
2. We should permit user code to depend on different versions of things than
the kernel does. For example, user code might rely on a different version of
HttpClient or Avro than that used by MapReduce. This should be possible if
instances of classes from these are not a passed between user and kernel code,
e.g., as long as Avro and HttpClient classes are not a part of the MapReduce
API. In this case classloaders (probably via OSGI) could permit this.
> Ability for having user's classes take precedence over the system classes for
> tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify
> for their jobs alternate implementations of classes that are already defined
> in the MapReduce libraries. For example, an alternate implementation for
> CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
