Tobias, Let me explain a little more. I want to create a synchronous REST API that will process some data that is passed in as some request. I would imagine that the Spark Streaming Job on YARN is a long running job that waits on requests from something. What that something is is still not clear to me, but I would imagine that it’s some queue. The goal is to be able to push a message onto a queue with some id, and then get the processed results back from Spark Streaming. The goal is for the REST API be able to respond to lots of calls with low latency. Hope that clarifies things...
Thanks, Ron On Sep 8, 2014, at 7:41 PM, Tobias Pfeiffer <t...@preferred.jp> wrote: > Ron, > > On Tue, Sep 9, 2014 at 11:27 AM, Ron's Yahoo! <zlgonza...@yahoo.com.invalid> > wrote: > I’m trying to figure out how I can run Spark Streaming like an API. > The goal is to have a synchronous REST API that runs the spark data flow on > YARN. > > I guess I *may* develop something similar in the future. > > By "a synchronous REST API", do you mean that submitting the job is > synchronous and you would fetch the processing results via a different call? > Or do you want to submit a job and get the processed data back as an HTTP > stream? > > To begin with, is it even possible to have Spark Streaming run as a yarn job? > > I think it is very much possible to run Spark Streaming as a YARN job; at > least it worked well with Mesos. > > Tobias >
