Hi, i am running spark jobs with standalone resource manager and i am gathering several performance metrics from my cluster nodes. I am also gathering disk io metrics from my nodes and because many of my jobs are using the same dataset i am trying to prevent the operating system from caching the dataset in memory in every node so as to gather the correct metrics for every job. Therefore before i submit my jobs to spark i clear my caches with the commands: sync ; echo 3 >/proc/sys/vm/drop_caches
The problem is that when i do so i see this error at the beginning of the job: WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory Ultimately the job runs successfully in most cases, but i feel like this error has a significant effect in the overall execution time of the job which i try to avoid. I am also pretty confident that there is nothing wrong in my configurations, because when i run jobs without clearing my nodes' caches the above error doesn't come up. I would really appreciate i anyone could help me with this error. Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Initial-job-has-not-accepted-any-resources-tp22955.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
