Checked the mesos github(https://github.com/mesos/hadoop). It listed
support for MapReduce V1
How about the MR V2?
Right now we are using cloudera to manage hadoop clusters where uses MRV2.
We are planning to migrate all our services to mesos(still in the initial
investigating stage). Good sugge
Hey Luyi,
That's correct, the Hadoop framework currently only supports Hadoop 2 MRv1.
It also doesn't have great support for the HA jobtracker available in newer
versions of Hadoop, but I've been working on that the past few weeks.
I'm not sure how Hadoop 2 would play with Mesos, but very interes
Hey Tom:
Really nice to get your reply. I am also looking forward that. If your
progress can be shared, that would be great. I will also look into this.
Will report back if I got any progress.
Thanks a lot!
-Luyi.
On Fri, Jul 25, 2014 at 11:54 AM, Tom Arnfeld wrote:
> Hey Luyi,
>
> Th
We run both mesos and yarn in prod and it does not make sense to run yarn
over mesos.
However it would be interesting to find a way to run MRv2 jobs on mesos
with some custom layer to swap yarn with mesos. Not sure how to start
though... MRv2 contains a yarn application master that needs to be
rew
I've not seen any issues pertaining to running many short lived frameworks,
but that's not near the number of frameworks you'd see if each job was a
framework.
We've been pushing all our work on MRv1 High Availability JT upstream on
the github.com/mesos/hadoop repo, though there hasn't been much t
I haven't written yarn app masters myself and browsing the
hadoop-mapreduce-project directories is really not easy. I think it is
feasible to get a prototype to work but it would take time.
>From what I know one difference is that the app master (which is equivalent
to the mesos framework) is run
I am interested in investigating the idea of YARN on top of Mesos. One of
the benefits I can think of is that we can get rid of the static resource
allocation between YARN and Mesos clusters. In that way, Mesos can allocate
those resources that are not used by YARN to other Mesos frameworks like
Au
FWIW I also think the fastest approach here is is porting Yarn onto Mesos.
In a perfect world, writing an implementation layer for the Yarn Interface
on Mesos would certainly be the optimal approach, but looking at the MRv2
code, it is very very coupled to many Yarn modules.
If someone wanted to
When I said that running yarn over mesos did not make sense I meant that
running a resource manager in a resource manager was very sub-optimal. You
will eventually do static allocation of resources for the Yarn framework in
Mesos or have complex logic to determine how much resource should be given
So excuse my naivety in this space, but my ignorance has never really
stopped me from asking questions:
I see YARN (Yet another resource negotiator) as very similar to Mesos. I.e.
something to manage resources on a cluster of machines. So when I hear talk
of running "YARN" on Mesos it's seems very
John, i believe that you are 100% correct. Theoretically we should run MRv2
on Mesos but the current implementation of MRv2 on Yarn seem very complex
and difficult to decouple from the resource manager/negotiator.
It's still something that could be done I guess but maybe as completely
independent
I second john's opinion on the confusing part of different terminology of
hadoop v2. That's the reason I asked the question on if mesos support mr
v2. As maxime's concern, the decoupling part might be difficult. After
reading the mesos mrv1's implementation, I think possibly mrv2 migration
can b
Thanks for the explanation John, that's very useful. I wasn't aware each
"job" in MRv2 was considered it's own entity to the scheduler, that's
interesting... I think Maxime's point about some kind of hadoop compatible
framework would work well, it sounds to me like the
Framework<>Executor<>Task flo
oogle.com/forum/#!topic/docker-dev/6tt1y9FTWKg
Cheers,
Tim
- Original Message -
> From: "Jie Yu"
> To: user@mesos.apache.org
> Cc: "mesos"
> Sent: Saturday, July 26, 2014 5:19:28 PM
> Subject: Re: Does Mesos support Hadoop MR V2
>
> I am interested i
To quote John below,
"So excuse my naivety... but...", I am also confused as to the version/naming
convention going on at the hadoop project.
I would like to run hadoop over mesos as opposed to over yarn. I would also
like to use the "new" mapreduce packages.
https://github.com/mesos/hadoop me
Porting YARN to run atop Mesos is quite reasonable. Some folks at eBay
have started some work on this (https://github.com/mesos/myriad). If
you're interested, you should check it out, and contribute to the project.
On Tue, Oct 28, 2014 at 5:21 AM, Yaneeve Shekel
wrote:
> To quote John below,
16 matches
Mail list logo
