Dan,
Even though you may be adding more nodes to the cluster, the Spark application
has to be requesting additional executors in order to thus use the added
resources. Or the Spark application can be using Dynamic Resource Allocation
Suraj,
Spark History server is running on 18080
(http://spark.apache.org/docs/latest/monitoring.html) which is not going to
give you are real-time update on a running Spark application. Given this is
Spark on YARN, you will need to view the Spark UI from the Application Master
URL which can
We worked it out. There was multiple items (like location of remote metastore
and db user auth) to make HiveContext happy in yarn-cluster mode.
For reference
https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/examples/using-hivecontext-yarn-cluster.md
-Christopher Bozeman
On
You can use Spark 1.4 on EMR AMI 3.8.0 if you install Spark as a 3rd party
application using the bootstrap action directly without the native Spark
inclusion with 1.3.1. See
https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark
Refer to
Sean,
Spark on YARN (https://spark.apache.org/docs/latest/running-on-yarn.html)
follows the logging construct of YARN.
If you are using cluster deployment mode on yarn (master=yarn-cluster) then the
logging performed in the driver (your code) would be picked up by YARN’s logs
in the
Shuai,
What did ps aux | grep spark-submit reveal?
When you compare using _JAVA_OPTIONS and without using it, where do you see the
difference?
Thanks
Christopher
-Original Message-
From: Shuai Zheng [mailto:szheng.c...@gmail.com]
Sent: Wednesday, April 01, 2015 11:12 AM
To: 'Sean
Ankur,
The JavaKinesisWordCountASLYARN is no longer valid and was added just to the
EMR build back in 1.1.0 to demonstrate Spark Streaming with Kinesis in YARN,
just follow the stock example as seen in JavaKinesisWordCountASL as it is
better form anyway given it is best not to hard code the
Thanks to Aniket’s work there is two new options to the EMR install script for
Spark. See
https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/README.md
The “-a” option can be used to bump the spark-assembly to the front of the
classpath.
-Christopher
From: Aniket Bhatnagar
