[
https://issues.apache.org/jira/browse/MAPREDUCE-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269586#comment-13269586
]
Jim Donofrio commented on MAPREDUCE-2001:
-----------------------------------------
{noformat}
if (numReduceTasks > 0) {
collector = new MapOutputBuffer(umbilical, job, reporter);
} else {
collector = new DirectMapOutputCollector(umbilical, job, reporter);
}
MapRunnable<INKEY,INVALUE,OUTKEY,OUTVALUE> runner =
ReflectionUtils.newInstance(job.getMapRunnerClass(), job);
{noformat}
Could you give an example of what user code would break if we moved
ReflectionUtils.newInstance(job.getMapRunnerClass(), job); above if
(numReduceTasks > 0) { ? Why would the configure method of the mapper care if
the recordwriter/outputformat had been created yet? I would think we would want
the recordwriter/outputformat to get configured after the configure method to
allow tasks to make task level config changes to a recordwriter/outputformat
> I imagine this working in a much better way. For new API users, they may
> still be able to sneak in changes per map/reduce task, and otherwise (on Old
> API) rely on driver to provide these up.
I am confused by this comment, do you agree with my approach or are you just
disappointed that the behavior will be inconsistent between the old and new api
for map only jobs?
> Enhancement to SequenceFileOutputFormat to allow user to set MetaData
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2001
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2001
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.20.2
> Reporter: David Rosenstrauch
> Priority: Minor
> Attachments: MAPREDUCE-2001.patch
>
>
> The org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat class
> currently does not provide a way for the user to pass in a MetaData object to
> be written to the SequenceFile.
> Currently he only way for a developer to implement this functionality appears
> to be to create a subclass which overrides the SequenceFileOutputFormat's
> getRecordWriter() method, which is a bit of a kludge.
> This seems to be a common enough request to warrant a fix of some sort.
> (It's already been brought up twice in the past year:
> http://www.mail-archive.com/[email protected]/msg02198.html and
> http://www.mail-archive.com/[email protected]/msg00904.html)
> A couple of possible solutions:
> 1) provide a static method SequenceFileOutputFormat.setMetaData(Job, MetaData)
> 2) Provide a (non-static) setMetaData() method on the
> SequenceFileOutputFormat class. The user would create a subclass of
> SequenceFileOutputFormat which, say, implements Configurable. Then in the
> setConf() method, the user could create the MetaData object (using data from
> the Configuration), and then call setMetaData. The SequenceFileOutputFormat
> would then use this MetaData object when creating the SequenceFile. (Note
> that the user would have to create a subclass of SequenceFileOutputFormat to
> make this solution work.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
