perhaps the author is referring to Spark Streaming applications? they're
examples of long-running applications.
the application/domain-level protocol still needs to be implemented
yourself, as sandy pointed out.
On Wed, Jul 9, 2014 at 11:03 AM, John Omernik wrote:
> So how do I do the "long-l
So how do I do the "long-lived server continually satisfying requests" in
the Cloudera application? I am very confused by that at this point.
On Wed, Jul 9, 2014 at 12:49 PM, Sandy Ryza wrote:
> Spark doesn't currently offer you anything special to do this. I.e. if
> you want to write a Spark
Spark doesn't currently offer you anything special to do this. I.e. if you
want to write a Spark application that fires off jobs on behalf of remote
processes, you would need to implement the communication between those
remote processes and your Spark application code yourself.
On Wed, Jul 9, 20
So basically, I have Spark on Yarn running (spark shell) how do I connect
to it with another tool I am trying to test using the spark://IP:7077 URL
it's expecting? If that won't work with spark shell, or yarn-client mode,
how do I setup Spark on Yarn to be able to handle that?
Thanks!
On Wed,
Thank you for the link. In that link the following is written:
For those familiar with the Spark API, an application corresponds to an
instance of the SparkContext class. An application can be used for a single
batch job, an interactive session with multiple jobs spaced apart, or a
long-lived ser
To add to Ron's answer, this post explains what it means to run Spark
against a YARN cluster, the difference between yarn-client and yarn-cluster
mode, and the reason spark-shell only works in yarn-client mode.
http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-mode
The idea behind YARN is that you can run different application types like
MapReduce, Storm and Spark.
I would recommend that you build your spark jobs in the main method without
specifying how you deploy it. Then you can use spark-submit to tell Spark how
you would want to deploy to it using ya
I am trying to get my head around using Spark on Yarn from a perspective of
a cluster. I can start a Spark Shell no issues in Yarn. Works easily. This
is done in yarn-client mode and it all works well.
In multiple examples, I see instances where people have setup Spark
Clusters in Stand Alone mod
