[
https://issues.apache.org/jira/browse/MAPREDUCE-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534031#comment-13534031
]
Robert Joseph Evans commented on MAPREDUCE-4868:
------------------------------------------------
Multiple iterations seems like a good idea in principal. The issue is that it
may not be compatible with all input formats. The input format API just
returns a RecordReader.
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RecordReader.html
or
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/RecordReader.html
RecordReader only supports going forward through the data, it does not support
going back through it or resetting it. Adding in support for that is a
non-backwards compatible change. It may be possible to abuse InputFormat to
get a second record reader for the second pass, but that is not exposed through
the current Mapper API/Context. It should work OK for most formats that are
based off of FileInputFormat, but I would be nervous about using it on anything
else without a lot of testing.
Also with HDFS there has been some work recently to support dropping data from
the base OS's file system cache when reading the data sequentially. If this
optimization is enabled, looping through the data twice is likely to only save
on the startup of extra map tasks, but not on the actually data reading time.
Depending on the size of your input block it might just be best for you to
cache the data yourself in memory and send the objects through a second time.
> Allow multiple iteration for map
> --------------------------------
>
> Key: MAPREDUCE-4868
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4868
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 3.0.0, 2.0.3-alpha
> Reporter: Jerry Chen
> Fix For: 3.0.0, 2.0.3-alpha
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, the Mapper class allows advanced users to override "public void
> run(Context context)" method for more control over the execution of the
> mapper, while Context interface limit the operations over the data which is
> the foundation of "more control".
> One of use cases is that when I am considering a hive optimziation problem, I
> want to go two passes over the input data instead of using a another job or
> task ( which may slower the whole process). Each pass do the same thing but
> with a different parameters.
> This is a new paradigm of Map Reduce usage and can be archived easily by
> extend Context interface a little with the more control over the data such as
> reset the input.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
