Thanks for the detailed answers. Most of the differences that I mentioned will
be applicable to running MR with “yarn-tez” mode ( except for the split
handling ). I believe the automatic update of number of reducers should also
work if configured to do so.
Let us know if we can help in any othe
Hi Hitesh,
- "When you say run a trace of a map-reduce job against SLS+Tez, could
you clarify what that means?"
Well, I have some "job profiles", where each job profile tells
statistics about a particular map-reduce application, such as: minimum
and maximum Map and Reduce execution time, num
Hi Fabio
The behavior that Tez induces on a cluster for a MapReduce-like job may be
vastly different to what MapReduce does today:
- Tez can do splits calculation on the cluster and makes use of information
such as available cluster resources to decide how many tasks to run
- Tez does con
Thanks for the reply, actually what I was planning to do is to generate
artificial traces of map-reduce jobs and run them against SLS and
SLS+Tez to analyze the differences.
I asked here directly since I am pretty sure that in the Hadoop mailing
list they were going to tell me to ask you about i
Hello Fabio
We do not have a job trace file generated by Tez and therefore no simulator
that can re-run the trace. We do store some historical data for the job but the
level of tooling around it is pretty minimal.
— Hitesh
On Nov 14, 2014, at 3:29 AM, Fabio wrote:
> With SLS (Yarn Schedule
Probably a question for the Hadoop YARN/MR mailing list since the
simulator code lives there.
Bikas
-Original Message-
From: Fabio [mailto:anyte...@gmail.com]
Sent: Friday, November 14, 2014 3:30 AM
To: user@tez.apache.org
Subject: SLS and Tez
With SLS (Yarn Scheduler Load Simulator) I
With SLS (Yarn Scheduler Load Simulator) I can test a MR job trace
against different schedulers, but to do so I see one has to specify
"yarn.sls.am.type.mapreduce" that is "The AMSimulator implementation for
MapReduce-like applications. Users can specify implementations for other
type of applic
