I recently upgraded to spark 2.3.1 I have had these same settings in my spark submit script, which worked on 2.0.2, and according to the documentation appear to not have changed:
spark.ui.retainedTasks=1 spark.ui.retainedStages=1 spark.ui.retainedJobs=1 However in 2.3.1 the UI doesn't seem to respect this, it still retains a huge number of jobs: [image: Screen Shot 2018-10-16 at 10.31.50 AM.png] Is this a known issue? Any ideas?
