Re: LiveListenerBus is occupying most of the Driver Memory and frequent GC is degrading the performance

We did take heap dump from the live job. To our surprise, 85% of the memory is being occupied by `org.apache.spark.scheduler.LiveListenerBus` Here are few pictures for context

Re: LiveListenerBus is occupying most of the Driver Memory and frequent GC is degrading the performance

Sorry for the poor formatting -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail:

Re: [ANNOUNCE] Announcing Apache Spark 3.0.1

Great work, thanks, Ruifeng! On Fri, Sep 11, 2020 at 11:09 PM Gengliang Wang < gengliang.w...@databricks.com> wrote: > Congrats! > Thanks for the work, Ruifeng! > > > On Fri, Sep 11, 2020 at 9:51 PM Takeshi Yamamuro > wrote: > >> Congrats and thanks, Ruifeng! >> >> >> On Fri, Sep 11, 2020 at

Re: [ANNOUNCE] Announcing Apache Spark 3.0.1

Congrats! Thanks for the work, Ruifeng! On Fri, Sep 11, 2020 at 9:51 PM Takeshi Yamamuro wrote: > Congrats and thanks, Ruifeng! > > > On Fri, Sep 11, 2020 at 9:50 PM Dongjoon Hyun > wrote: > >> It's great. Thank you, Ruifeng! >> >> Bests, >> Dongjoon. >> >> On Fri, Sep 11, 2020 at 1:54 AM 郑瑞峰

Re: [ANNOUNCE] Announcing Apache Spark 3.0.1

Congrats and thanks, Ruifeng! On Fri, Sep 11, 2020 at 9:50 PM Dongjoon Hyun wrote: > It's great. Thank you, Ruifeng! > > Bests, > Dongjoon. > > On Fri, Sep 11, 2020 at 1:54 AM 郑瑞峰 wrote: > >> Hi all, >> >> We are happy to announce the availability of Spark 3.0.1! >> Spark 3.0.1 is a

Re: [DISCUSS] Spark cannot identify the problem executor

What do you mean by "read from executor A"? I can think of several paths for an executor to read something from another remote executor: 1. shuffle data If the executor fails to fetch the shuffle data, I think it will result in the FetchFiled for the task. For this case, blacklist can identify

Re: [ANNOUNCE] Announcing Apache Spark 3.0.1

It's great. Thank you, Ruifeng! Bests, Dongjoon. On Fri, Sep 11, 2020 at 1:54 AM 郑瑞峰 wrote: > Hi all, > > We are happy to announce the availability of Spark 3.0.1! > Spark 3.0.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.0 maintenance branch of

Re: [DISCUSS] Spark cannot identify the problem executor

-dev, +user Executors do not communicate directly, so I don't think that's quite what you are seeing. You'd have to clarify. On Fri, Sep 11, 2020 at 12:08 AM 陈晓宇 wrote: > > Hello all, > > We've been using spark 2.3 with blacklist enabled and often meet the problem > that when executor A has

[ANNOUNCE] Announcing Apache Spark 3.0.1

Hi all, We are happy to announce the availability of Spark 3.0.1! Spark 3.0.1 is a maintenance release containing stability fixes. This release is based on the branch-3.0 maintenance branch of Spark. We strongly recommend all 3.0 users to upgrade to this stable release. To download Spark