[
https://issues.apache.org/jira/browse/MAPREDUCE-728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728420#action_12728420
]
Arun C Murthy commented on MAPREDUCE-728:
-----------------------------------------
bq. Clarification question: If the per-job digest from the "source" workload is
generated by a real Hadoop cluster, then that workload would be an artifact of
all the pluggable components used in the cluster used to generate it?
The characteristics of the workload (e.g. for a given job j, runtime for
data-local maps, off-rack maps etc.) are reasonably independent of the
Scheduler in question.
bq. For instance, if I had a cluster running the default scheduler, and I took
those job digests and throw those into Mumak, then the question that Mumak is
supposed to answer is, given a workload of a set of jobs under the Default
Scheduler, how different would the execution times be under some different set
of pluggable components?
Yes. With 'pluggable components' limited to the Scheduler.
Crucially, an equally important (if not more so) role of Mumak is to help us
answer the question: What will be turnaround of the the same workload be if we
added a feature 'X' to the same scheduler 'Y'?
> Mumak: Map-Reduce Simulator
> ---------------------------
>
> Key: MAPREDUCE-728
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-728
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: mumak.png
>
>
> h3. Vision:
> We want to build a Simulator to simulate large-scale Hadoop clusters,
> applications and workloads. This would be invaluable in furthering Hadoop by
> providing a tool for researchers and developers to prototype features (e.g.
> pluggable block-placement for HDFS, Map-Reduce schedulers etc.) and predict
> their behaviour and performance with reasonable amount of confidence,
> there-by aiding rapid innovation.
> ----
> h3. First Cut: Simulator for the Map-Reduce Scheduler
> The Map-Reduce Scheduler is a fertile area of interest with at least four
> schedulers, each with their own set of features, currently in existence:
> Default Scheduler, Capacity Scheduler, Fairshare Scheduler & Priority
> Scheduler.
> Each scheduler's scheduling decisions are driven by many factors, such as
> fairness, capacity guarantee, resource availability, data-locality etc.
> Given that, it is non-trivial to accurately choose a single scheduler or even
> a set of desired features to predict the right scheduler (or features) for a
> given workload. Hence a simulator which can predict how well a particular
> scheduler works for some specific workload by quickly iterating over
> schedulers and/or scheduler features would be quite useful.
> So, the first cut is to implement a simulator for the Map-Reduce scheduler
> which take as input a job trace derived from production workload and a
> cluster definition, and simulates the execution of the jobs in as defined in
> the trace in this virtual cluster. As output, the detailed job execution
> trace (recorded in relation to virtual simulated time) could then be analyzed
> to understand various traits of individual schedulers (individual jobs turn
> around time, throughput, faireness, capacity guarantee, etc). To support
> this, we would need a simulator which could accurately model the conditions
> of the actual system which would affect a schedulers decisions. These include
> very large-scale clusters (thousands of nodes), the detailed characteristics
> of the workload thrown at the clusters, job or task failures, data locality,
> and cluster hardware (cpu, memory, disk i/o, network i/o, network topology)
> etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
