Re: (elastic) Hadoop YARN on mesos?

Andre You can take a look at Apache Ambari. It allows you to spin up clusters very fast, add nodes and remove nodes. If you have an existing cluster and you want to run long running jobs, you can definitely use Apache Slider which is like Marathon in some ways. There is doc https://cwiki.apache

Re: (elastic) Hadoop YARN on mesos?

Hi Andre Myriad allows you to run YARN on Mesos. Since YARN is also a resource manager, imagine this as a kind of a delegation of resources from Mesos to YARN. With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon startup reports back to YARN's Resource Manager and joins the clust

Re: (elastic) Hadoop YARN on mesos?

HI, yes, I looked at it, but from what I understand it is a project to run YARN next to mesos, not on top of it. I am looking for a way to have “ephemeral” hadoop clusters one some hardware, not a static YARN setup sharing resources with mesos. Maybe mesos is not what I need here, I was just wo

Re: (elastic) Hadoop YARN on mesos?

Hey Andre, Have you seen: https://github.com/mesos/myriad JJ. On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe wrote: > Hi, > > we are currently looking for an easy way to spin up Hadoop YARN clusters > on some hardware that we have lying around. I came across mesos and was > wondering if somebody

(elastic) Hadoop YARN on mesos?

Hi, we are currently looking for an easy way to spin up Hadoop YARN clusters on some hardware that we have lying around. I came across mesos and was wondering if somebody here is using it this way. What I am looking for is something where I say: Give me a cluster with n nodes. Run some tests an

Re: hadoop job stuck.

Would need the task logs from the slave which the TaskTracker was launched on, to debug this further. On Wed, Jan 14, 2015 at 1:28 PM, Dan Dong wrote: > Checked /etc/hosts is correct, master and slave can ssh login each other > by hostname without password, and hadoop runs well without mesos, bu

Re: hadoop job stuck.

Checked /etc/hosts is correct, master and slave can ssh login each other by hostname without password, and hadoop runs well without mesos, but it stucks when running on mesos. Cheers, Dan 2015-01-14 15:02 GMT-06:00 Brenden Matthews : > At a first glance, it looks like `/etc/hosts` might be set i

Re: hadoop job stuck.

Hi, All, I'm using hadoop-2.5.0-cdh5.2.0 on mesos 0.21.0, the following are from master log of mesos: I0114 14:47:04.855033 17756 master.cpp:1768] Deactivating framework 20150114-144506-3205108908-5050-17750- (Hadoop: (RPC port: 9001, WebUI port: 50030)) at scheduler-d37097ae-8b55-4f73

Re: hadoop job stuck.

At a first glance, it looks like `/etc/hosts` might be set incorrectly and it cannot resolve the hostname of the worker. See here for more: https://wiki.apache.org/hadoop/UnknownHost On Wed, Jan 14, 2015 at 12:32 PM, Vinod Kone wrote: > What do the master logs say? > > On Wed, Jan 14, 2015 at 1

Re: hadoop job stuck.

What do the master logs say? On Wed, Jan 14, 2015 at 12:21 PM, Dan Dong wrote: > Hi, > When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever: > 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to > process : 8 > 15/01/14 13:59:30 INFO mapred.JobClient: Running jo

Re: hadoop job stuck.

Hi Dan, Can you look at the stdout/stderr logs in the task sandbox for me and share any errors here? Also – What version of Hadoop are you using, and what version of the Hadoop on Mesos framework? Thanks. -- Tom Arnfeld Developer // DueDil On Wednesday, Jan 14, 2015 at 8:2

hadoop job stuck.

Hi, When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever: 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to process : 8 15/01/14 13:59:30 INFO mapred.JobClient: Running job: job_201501141358_0001 15/01/14 13:59:31 INFO mapred.JobClient: map 0% reduce 0% >From j

MesosCon Early-Bird Registration Now Open

A limited number of early-bird tickets are now available for #MesosCon 2015 , an annual conference organized by the Apache Mesos community. MesosCon brings together users and developers to share and learn about the project and its growing ecosystem, and will take place in Seatt