How Spark HA works

2016-07-05 Thread Akmal Abbasov
Hi,  I'm trying to understand how Spark HA works. I'm using Spark 1.6.1 and Zookeeper 3.4.6. I've add the following line to $SPARK_HOME/conf/spark-env.sh export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=zk1:2181,zk2:2181,zk3:2181

Long(20+ seconds) startup delay for jobs when running Spark on YARN

2016-04-21 Thread Akmal Abbasov
Hi, I'm running Spark(1.6.1) on YARN(2.5.1), cluster mode. It's taking 20+ seconds for application to move from ACCEPTED to RUNNING state, here's logs 16/04/21 09:06:56 INFO impl.YarnClientImpl: Submitted application application_1461229289298_0001 16/04/21 09:06:57 INFO yarn.Client: Application

No event logs in yarn-cluster mode

2015-08-01 Thread Akmal Abbasov
Hi, I am trying to configure a history server for application. When I running locally(./run-example SparkPi), the event logs are being created, and I can start history server. But when I am trying ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster