Hello All, I need to clean up app folder(include app downloaded jar) in spark under work folder. I have tried to set below configuration in spark env. but it is not working as expected.
SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=10" I cannot clear work folder blindly because of some other job may run on the same time. Is there any way to find out below :- 1) Clear only my app folder under work folder 2) In executor level can we set work directories. So, that i will clean that folder after my job completed. 3) Is there way to find of app id of my application. I tried to get master object in code but it is throwing actor error. 4) Is there any correction required on below configuration SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=10" Thanks in advance Hutashan. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Clean-up-app-folders-in-worker-nodes-tp20889.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
