Thanks Steve, but my goal is to hopefully avoid installing yet another
component into my environment. Yarn is cool, but wouldn't be used for
anything but Spark. We have no hadoop in our ecosystem (or HDFS). Ideally
I'd avoid having to learn about yet another tool.
On Wed, Jul 23, 2014 at 11:12 AM, Steve Nunez snu...@hortonworks.com
wrote:
I’m also in early stages of setting up long running Spark jobs. Easiest
way I’ve found is to set up a cluster and submit the job via YARN. Then I
can come back and check in on progress when I need to. Seems the trick is
tuning the queue priority and YARN preemption to get the job to run in a
reasonable amount of time without disrupting the other jobs.
- SteveN
From: Chris Schneider ch...@christopher-schneider.com
Reply-To: user@spark.apache.org
Date: Wednesday, July 23, 2014 at 7:39
To: user@spark.apache.org
Subject: Cluster submit mode - only supported on Yarn?
We are starting to use Spark, but we don't have any existing
infrastructure related to big-data, so we decided to setup the standalone
cluster, rather than mess around with Yarn or Mesos.
But it appears like the driver program has to stay up on the client for
the full duration of the job (client mode).
What is the simplest way to setup cluster submission mode, to allow our
client boxes to submit jobs and then move on with the other work they need
to do without keeping a potentially long running java process up?
Thanks,
Chris
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
