I am running Spark jobs on YARN cluster. It took ~30 seconds to create a spark context, while it takes only 1-2 seconds running Spark in local mode. The master is set as yarn-client, and both the machine that submits the Spark job and the YARN cluster are in the same domain.
Originally I suspected that the following config might play a role, especially spark.scheduler.maxRegisteredResourcesWaitingTime was set to 30 seconds. However, lowering it to 10 makes no difference. Modifications to other two parameters also has no effect. spark.scheduler.maxRegisteredResourcesWaitingTime=1000 (lowered from 30000 default) spark.yarn.applicationMaster.waitTries=1 (lowered from 10 default) spark.yarn.scheduler.heartbeat.interval-ms = 1000 (lowered from 5000 default) What could be the possible reason? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/It-takes-too-long-30-seconds-to-create-Spark-Context-with-SPARK-YARN-tp22847.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
