Re: (elastic) Hadoop YARN on mesos?

[Andre Kelpe]

Thanks for the pointers! I will try to get something up and running.

- Andre

On Wed, Jan 14, 2015 at 5:36 PM, mohit soni mohitsoni1...@gmail.com wrote:

 Hi Andre

 Myriad allows you to run YARN on Mesos. Since YARN is also a resource
 manager, imagine this as a kind of a delegation of resources from Mesos to
 YARN.

 With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon
 startup reports back to YARN's Resource Manager and joins the cluster.

 Now, in your scenario you would like to spin up 'ephemeral' hadoop
 clusters. You can certainly do so in Mesos ecosystem, with a little bit of
 work (right now). My suggestion would be to use Marathon (
 https://mesosphere.github.io/marathon/) to launch Myriad/YARN Resource
 Manager, and then trigger Myriad's flexUp API (
 https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup)
 to spin N number of NodeManagers on mesos cluster.

 Regards
 Mohit

 On Wed, Jan 14, 2015 at 4:37 PM, Andre Kelpe ake...@concurrentinc.com
 wrote:

 HI,

 yes, I looked at it, but from what I understand it is a project to run
 YARN next to mesos, not on top of it. I am looking for a way to have
 “ephemeral” hadoop clusters one some hardware, not a static YARN setup
 sharing resources with mesos. Maybe mesos is not what I need here, I was
 just wondering if somebody here does something similar.

 - Andre

  On 14 Jan 2015, at 16:11, Joseph Jacks jacks@gmail.com wrote:
 
  Hey Andre,
 
  Have you seen: https://github.com/mesos/myriad
 
  JJ.
 
  On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe ake...@concurrentinc.com
 wrote:
  Hi,
 
  we are currently looking for an easy way to spin up Hadoop YARN
 clusters on some hardware that we have lying around. I came across mesos
 and was wondering if somebody here is using it this way. What I am looking
 for is something where I say: Give me a cluster with n nodes. Run some
 tests and then destroy the Hadoop cluster. Similar to how EMR would work or
 how you work with vagrant locally. Is somebody here doing this and if so,
 do you have any starter docs for such a setup somewhere?
 
  Note that we def. need YARN and we want to use vanilla Apache Hadoop,
 so no commercial distribution.
 
  Thanks for your help!
 
  - Andre
 
  --
  André Kelpe
  an...@concurrentinc.com
  http://concurrentinc.com
 
 
 
 
 

 --
 André Kelpe
 an...@concurrentinc.com
 http://concurrentinc.com








-- 
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

(elastic) Hadoop YARN on mesos?

[Andre Kelpe]

Hi,

we are currently looking for an easy way to spin up Hadoop YARN clusters on 
some hardware that we have lying around. I came across mesos and was wondering 
if somebody here is using it this way. What I am looking for is something where 
I say: Give me a cluster with n nodes. Run some tests and then destroy the 
Hadoop cluster. Similar to how EMR would work or how you work with vagrant 
locally. Is somebody here doing this and if so, do you have any starter docs 
for such a setup somewhere? 

Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no 
commercial distribution.

Thanks for your help!

- Andre

--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

Re: (elastic) Hadoop YARN on mesos?

[Andre Kelpe]

HI,

yes, I looked at it, but from what I understand it is a project to run YARN 
next to mesos, not on top of it. I am looking for a way to have “ephemeral” 
hadoop clusters one some hardware, not a static YARN setup sharing resources 
with mesos. Maybe mesos is not what I need here, I was just wondering if 
somebody here does something similar.

- Andre

 On 14 Jan 2015, at 16:11, Joseph Jacks jacks@gmail.com wrote:
 
 Hey Andre,
 
 Have you seen: https://github.com/mesos/myriad
 
 JJ.
 
 On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe ake...@concurrentinc.com wrote:
 Hi,
 
 we are currently looking for an easy way to spin up Hadoop YARN clusters on 
 some hardware that we have lying around. I came across mesos and was 
 wondering if somebody here is using it this way. What I am looking for is 
 something where I say: Give me a cluster with n nodes. Run some tests and 
 then destroy the Hadoop cluster. Similar to how EMR would work or how you 
 work with vagrant locally. Is somebody here doing this and if so, do you have 
 any starter docs for such a setup somewhere?
 
 Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no 
 commercial distribution.
 
 Thanks for your help!
 
 - Andre
 
 --
 André Kelpe
 an...@concurrentinc.com
 http://concurrentinc.com
 
 
 
 
 

--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com