Yes, I’m pretty sure my YARN and HDFS HA configuration is correct. I can use
the UIs and HDFS command line tools with HA support as expected (failing over
namenodes and resourcemanagers, etc) so I believe this to be a Spark issue.
Like I mentioned earlier, if i manipulate the
On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell matt.narr...@gmail.com wrote:
An “unaccepted” reply to this thread from Dean Chen suggested to build Spark
with a newer version of Hadoop (2.4.1) and this has worked to some extent.
I’m now able to submit jobs (omitting an explicit
Ah, sorry, forgot to talk about the second issue.
On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell matt.narr...@gmail.com wrote:
However, now the Spark jobs running in the ApplicationMaster on a given node
fails to find the active resourcemanager. Below is a log excerpt from one
of the assigned
Marcelo,
Specifying the driver-class-path yields behavior like
https://issues.apache.org/jira/browse/SPARK-2420 and
https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a can
of worms here if I also need to replace the guava dependencies.
Wouldn’t calling
Hi,
On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell matt.narr...@gmail.com wrote:
Specifying the driver-class-path yields behavior like
https://issues.apache.org/jira/browse/SPARK-2420 and
https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a
can of worms here if I also
Ok Marcelo,
Thanks for the quick and thorough replies. I’ll keep an eye on these tickets
and the mailing list to see how things move along.
mn
On Aug 20, 2014, at 1:33 PM, Marcelo Vanzin van...@cloudera.com wrote:
Hi,
On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell matt.narr...@gmail.com
Hi Matt,
I checked in the YARN code and I don't see any references to
yarn.resourcemanager.address. Have you made sure that your YARN client
configuration on the node you're launching from contains the right configs?
-Sandy
On Mon, Aug 18, 2014 at 4:07 PM, Matt Narrell matt.narr...@gmail.com
Hello,
I have an HA enabled YARN cluster with two resource mangers. When submitting
jobs via “spark-submit —master yarn-cluster”. It appears that the driver is
looking explicitly for the yarn.resourcemanager.address” property rather than
round robin-ing through the resource managers via the
