[
https://issues.apache.org/jira/browse/MAPREDUCE-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777105#action_12777105
]
Neil Conway commented on MAPREDUCE-1211:
----------------------------------------
BTW, there are a few different ways to slice this work:
(1) Basic pipelining support for moving data between tasks
(2) Pipelining data between jobs (do the HDFS write in the background)
(3) Leveraging #1 and #2 to do "online aggregation" (approximate answers as the
query runs)
(4) Leveraging #1 and #2 to do "stream processing" (MR jobs that run
continuously)
These are in increasing order of complexity/invasiveness. If there's any
interest in merging this work, we might begin by just merging #1, which is
relatively low-impact. Intra-job pipelining offers two benefits: (1)
potentially reduced response times (better cluster utilization), by overlapping
map computation with network I/O and reducer-side merging (2) more effective
straggler handling (rather than starting a speculative task from scratch, we
can use pipelining to "checkpoint" the partial work done by the straggler, and
not bother re-sending that portion of the map output over the network again).
> Online aggregation and continuous query support
> -----------------------------------------------
>
> Key: MAPREDUCE-1211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1211
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: task
> Reporter: Tyson Condie
> Priority: Minor
>
> The purpose of this post is to propose a modified MapReduce architecture that
> allows data to be pipelined between operators. This extends the MapReduce
> programming model beyond batch processing, and can reduce completion times
> and improve system utilization for batch jobs as well. We have built a
> modified version of the Hadoop MapReduce framework that supports online
> aggregation, which allows users to see "early returns" from a job as it is
> being computed. Our Hadoop Online Prototype (HOP) also supports continuous
> queries, which enable MapReduce programs to be written for applications such
> as event monitoring and stream processing. HOP retains the fault tolerance
> properties of Hadoop, and can run unmodified user-defined MapReduce programs.
> For more information on the HOP design, please see our technical report.
> http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-136.html
> Further details are discussed in the following blog posts.
> http://databeta.wordpress.com/2009/10/18/mapreduce-online/
> http://radar.oreilly.com/2009/10/pipelining-and-real-time-analytics-with-mapreduce-online.html
> http://dbmsmusings.blogspot.com/2009/10/analysis-of-mapreduce-online-paper.html
> The HOP code has been published at the following location.
> http://code.google.com/p/hop/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
