Re: What else can be built on top of YARN.

John, On Jun 1, 2013, at 7:02 AM, John Lilley wrote: · Algorithms that are not well-suited to the MR model, such as transitive closure. They are more naturally expressed as MPI-like algorithms. You might be interested in MPICH2 on YARN: https://github.com/clarkyzl/mpich2-yarn

Re: What else can be built on top of YARN.

Thanks a lot for the responses. I now have a better understanding. To answer to Jay's question , I think ZK can be used as for coordination service for a distributed program (you have built it on top of exposed granular api's) and it doesn't have features like resource management (including

RE: What else can be built on top of YARN.

Rahul, This is a very good question, and one we are grappling with currently in our application port. I think there are a lot of legacy data-processing applications like ours which would benefit by a port to Hadoop. However, because we have a great load of C++, it is not necessarily a good

Re: What else can be built on top of YARN.

Hi Rahul, It is at least because of the reasons that Vinod listed that makes my life easy for porting my application on to YARN instead of making it work in the Map Reduce framework. The main purpose of me using YARN is to exploit the resource management capabilities of YARN. Thanks, Kishore

Re: What else can be built on top of YARN.

What is the separation of concerns between YARN and Zookeeper? That is, where does YARN leave off and where does Zookeeper begin? Or is there some overlap On Thu, May 30, 2013 at 2:42 AM, Krishna Kishore Bonagiri write2kish...@gmail.com wrote: Hi Rahul, It is at least because of the

Re: What else can be built on top of YARN.

Hi Rahul, I am porting a distributed application that runs on a fixed set of given resources to YARN, with the aim of being able to run it on a dynamically selected resources whichever are available at the time of running the application. Thanks, Kishore On Wed, May 29, 2013 at 8:04 PM,

Re: What else can be built on top of YARN.

Thanks for the response Krishna. I was wondering if it were possible for using MR to solve you problem instead of building the whole stack on top of yarn. Most likely its not possible , thats why you are building it . I wanted to know why is that ? I am in just trying to find out the need or

Re: What else can be built on top of YARN.

Two scenarios I can think of are re-implementations of Twitter's Storm ( http://storm-project.net/) and DryadLinq ( http://research.microsoft.com/en-us/projects/dryadlinq/). Storm, a distributed realtime computation framework used for analyzing realtime steams of data, doesn't really need to be

Re: What else can be built on top of YARN.

There is a project at Yahoo which makes it possible to run Storm on Yarn. I think the team behind it is going to give a talk at Hadoop Summit and plan to open source it after that. -Viral On Wed, May 29, 2013 at 11:04 AM, John Conwell j...@iamjohn.me wrote: Storm, a distributed realtime

Re: What else can be built on top of YARN.

Historically, many applications/frameworks wanted to take advantage of just the resource management capabilities and failure handling of Hadoop (via JobTracker/TaskTracker), but were forced to used MapReduce even though they didn't have to. Obvious examples are graph processing (Giraph),