Done:
https://issues.apache.org/jira/browse/SPARK-25837
On Thu, Oct 25, 2018 at 10:21 AM Marcelo Vanzin wrote:
> Ah that makes more sense. Could you file a bug with that information
> so we don't lose track of this?
>
> Thanks
> On Wed, Oct 24, 2018 at 6:13 PM Patrick Brown
> wrote:
> >
> >
Ah that makes more sense. Could you file a bug with that information
so we don't lose track of this?
Thanks
On Wed, Oct 24, 2018 at 6:13 PM Patrick Brown
wrote:
>
> On my production application I am running ~200 jobs at once, but continue to
> submit jobs in this manner for sometimes ~1 hour.
>
When you say many jobs at once, what ballpark are you talking about?
The code in 2.3+ does try to keep data about all running jobs and
stages regardless of the limit. If you're running into issues because
of that we may have to look again at whether that's the right thing to
do.
On Tue, Oct 23,
I believe I may be able to reproduce this now, it seems like it may be
something to do with many jobs at once:
Spark 2.3.1
> spark-shell --conf spark.ui.retainedJobs=1
scala> import scala.concurrent._
scala> import scala.concurrent.ExecutionContext.Implicits.global
scala> for (i <- 0 until
Just tried on 2.3.2 and worked fine for me. UI had a single job and a
single stage (+ the tasks related to that single stage), same thing in
memory (checked with jvisualvm).
On Sat, Oct 20, 2018 at 6:45 PM Marcelo Vanzin wrote:
>
> On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown
> wrote:
> > I
On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown
wrote:
> I recently upgraded to spark 2.3.1 I have had these same settings in my spark
> submit script, which worked on 2.0.2, and according to the documentation
> appear to not have changed:
>
> spark.ui.retainedTasks=1
> spark.ui.retainedStages=1
I have the same problem when I upgrade my application from Spark 2.2.1 to
Spark 2.3.2 and run in Yarn client mode.
Also I noticed that in my Spark driver, org.apache.spark.status.TaskDataWrapper
could take up more than 2G of memory.
Shing
On Tuesday, 16 October 2018, 17:34:02 GMT+1,
I recently upgraded to spark 2.3.1 I have had these same settings in my
spark submit script, which worked on 2.0.2, and according to the
documentation appear to not have changed:
spark.ui.retainedTasks=1
spark.ui.retainedStages=1
spark.ui.retainedJobs=1
However in 2.3.1 the UI doesn't seem to
