Hello, I am working on improving the performance of our Spark on Yarn applications. Scanning through the logs I found the following lines:
[2015-10-07T16:25:17.245-04:00] [DataProcessing] [INFO] [] [org.apache.spark.Logging$class] [tid:main] [userID:yarn] Started progress reporter thread - sleep time : 5000 [2015-10-07T16:25:22.262-04:00] [DataProcessing] [INFO] [] [org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl] [tid:Reporter] [userID:yarn] Received new token for : hostname:8041 As the log says, the main thread sleeps for 5 seconds. Is there a way to configure/eliminate this sleep? thanks, Yael
