Unsubscribe

[Ashish Singh]

Unsubscribe

[Ashish]

Sent from my iPhone

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

[Ashish Singh]

Hi Kalyan,

Is this something you are still interested in pursuing? There are some open
discussion threads on the doc you shared.

@Mridul Muralidharan  In what state are your efforts
along this? Is it something that your team is actively pursuing/ building
or are mostly planning right now? Asking so that we can align efforts on
this.

On Sun, Feb 18, 2024 at 10:32 PM xiaoping.huang <1754789...@qq.com> wrote:

> Hi all,
> Any updates on this project? This will be a very useful feature.
>
> xiaoping.huang
> 1754789...@qq.com
>
>  Replied Message 
> From kalyan 
> Date 02/6/2024 10:08
> To Jay Han 
> Cc Ashish Singh ,
>  Mridul Muralidharan ,
>  dev ,
>  
> 
> Subject Re: [Spark-Core] Improving Reliability of spark when Executors
> OOM
> Hey,
> Disk space not enough is also a reliability concern, but might need a diff
> strategy to handle it.
> As suggested by Mridul, I am working on making things more configurable in
> another(new) module… with that, we can plug in new rules for each type of
> error.
>
> Regards
> Kalyan.
>
> On Mon, 5 Feb 2024 at 1:10 PM, Jay Han  wrote:
>
>> Hi,
>> what about supporting for solving the disk space problem of "device space
>> isn't enough"? I think it's same as OOM exception.
>>
>> kalyan  于2024年1月27日周六 13:00写道：
>>
>>> Hi all,
>>>
>>
>>> Sorry for the delay in getting the first draft of (my first) SPIP out.
>>>
>>> https://docs.google.com/document/d/1hxEPUirf3eYwNfMOmUHpuI5dIt_HJErCdo7_yr9htQc/edit?pli=1
>>>
>>> Let me know what you think.
>>>
>>> Regards
>>> kalyan.
>>>
>>> On Sat, Jan 20, 2024 at 8:19 AM Ashish Singh  wrote:
>>>
>>>> Hey all,
>>>>
>>>> Thanks for this discussion, the timing of this couldn't be better!
>>>>
>>>> At Pinterest, we recently started to look into reducing OOM failures
>>>> while also reducing memory consumption of spark applications. We considered
>>>> the following options.
>>>> 1. Changing core count on executor to change memory available per task
>>>> in the executor.
>>>> 2. Changing resource profile based on task failures and gc metrics to
>>>> grow or shrink executor memory size. We do this at application level based
>>>> on the app's past runs today.
>>>> 3. K8s vertical pod autoscaler
>>>> <https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler>
>>>>
>>>> Internally, we are mostly getting aligned on option 2. We would love to
>>>> make this happen and are looking forward to the SPIP.
>>>>
>>>>
>>>> On Wed, Jan 17, 2024 at 9:34 AM Mridul Muralidharan 
>>>> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>   We are internally exploring adding support for dynamically changing
>>>>> the resource profile of a stage based on runtime characteristics.
>>>>> This includes failures due to OOM and the like, slowness due to
>>>>> excessive GC, resource wastage due to excessive overprovisioning, etc.
>>>>> Essentially handles scale up and scale down of resources.
>>>>> Instead of baking these into the scheduler directly (which is already
>>>>> complex), we are modeling it as a plugin - so that the 'business logic' of
>>>>> how to handle task events and mutate state is pluggable.
>>>>>
>>>>> The main limitation I find with mutating only the cores is the limits
>>>>> it places on what kind of problems can be solved with it - and mutating
>>>>> resource profiles is a much more natural way to handle this
>>>>> (spark.task.cpus predates RP).
>>>>>
>>>>> Regards,
>>>>> Mridul
>>>>>
>>>>> On Wed, Jan 17, 2024 at 9:18 AM Tom Graves
>>>>>  wrote:
>>>>>
>>>>>> It is interesting. I think there are definitely some discussion
>>>>>> points around this.  reliability vs performance is always a trade off and
>>>>>> its great it doesn't fail but if it doesn't meet someone's SLA now that
>>>>>> could be as bad if its hard to figure out why.   I think if something 
>>>>>> like
>>>>>> this kicks in, it needs to be very obvious to the user so they can see 
>>>>>> that
>>>>>> it occurred.  Do you have s

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

[Ashish Singh]

Hey all,

Thanks for this discussion, the timing of this couldn't be better!

At Pinterest, we recently started to look into reducing OOM failures while
also reducing memory consumption of spark applications. We considered the
following options.
1. Changing core count on executor to change memory available per task in
the executor.
2. Changing resource profile based on task failures and gc metrics to grow
or shrink executor memory size. We do this at application level based on
the app's past runs today.
3. K8s vertical pod autoscaler


Internally, we are mostly getting aligned on option 2. We would love to
make this happen and are looking forward to the SPIP.


On Wed, Jan 17, 2024 at 9:34 AM Mridul Muralidharan 
wrote:

>
> Hi,
>
>   We are internally exploring adding support for dynamically changing the
> resource profile of a stage based on runtime characteristics.
> This includes failures due to OOM and the like, slowness due to excessive
> GC, resource wastage due to excessive overprovisioning, etc.
> Essentially handles scale up and scale down of resources.
> Instead of baking these into the scheduler directly (which is already
> complex), we are modeling it as a plugin - so that the 'business logic' of
> how to handle task events and mutate state is pluggable.
>
> The main limitation I find with mutating only the cores is the limits it
> places on what kind of problems can be solved with it - and mutating
> resource profiles is a much more natural way to handle this
> (spark.task.cpus predates RP).
>
> Regards,
> Mridul
>
> On Wed, Jan 17, 2024 at 9:18 AM Tom Graves 
> wrote:
>
>> It is interesting. I think there are definitely some discussion points
>> around this.  reliability vs performance is always a trade off and its
>> great it doesn't fail but if it doesn't meet someone's SLA now that could
>> be as bad if its hard to figure out why.   I think if something like this
>> kicks in, it needs to be very obvious to the user so they can see that it
>> occurred.  Do you have something in place on UI or something that indicates
>> this? The nice thing is also you aren't wasting memory by increasing it for
>> all tasks when maybe you only need it for one or two.  The downside is you
>> are only finding out after failure.
>>
>> I do also worry a little bit that in your blog post, the error you
>> pointed out isn't a java OOM but an off heap memory issue (overhead + heap
>> usage).  You don't really address heap memory vs off heap in that article.
>> Only thing I see mentioned is spark.executor.memory which is heap memory.
>> Obviously adjusting to only run one task is going to give that task more
>> overall memory but the reasons its running out in the first place could be
>> different.  If it was on heap memory for instance with more tasks I would
>> expect to see more GC and not executor OOM.  If you are getting executor
>> OOM you are likely using more off heap memory/stack space, etc then you
>> allocated.   Ultimately it would be nice to know why that is happening and
>> see if we can address it to not fail in the first place.  That could be
>> extremely difficult though, especially if using software outside Spark that
>> is using that memory.
>>
>> As Holden said,  we need to make sure this would play nice with the
>> resource profiles, or potentially if we can use the resource profile
>> functionality.  Theoretically you could extend this to try to get new
>> executor if using dynamic allocation for instance.
>>
>> I agree doing a SPIP would be a good place to start to have more
>> discussions.
>>
>> Tom
>>
>> On Wednesday, January 17, 2024 at 12:47:51 AM CST, kalyan <
>> justfors...@gmail.com> wrote:
>>
>>
>> Hello All,
>>
>> At Uber, we had recently, done some work on improving the reliability of
>> spark applications in scenarios of fatter executors going out of memory and
>> leading to application failure. Fatter executors are those that have more
>> than 1 task running on it at a given time concurrently. This has
>> significantly improved the reliability of many spark applications for us at
>> Uber. We made a blog about this recently. Link:
>> https://www.uber.com/en-US/blog/dynamic-executor-core-resizing-in-spark/
>>
>> At a high level, we have done the below changes:
>>
>>1. When a Task fails with the OOM of an executor, we update the core
>>requirements of the task to max executor cores.
>>2. When the task is picked for rescheduling, the new attempt of the
>>task happens to be on an executor where no other task can run 
>> concurrently.
>>All cores get allocated to this task itself.
>>3. This way we ensure that the configured memory is completely at the
>>disposal of a single task. Thus eliminating contention of memory.
>>
>> The best part of this solution is that it's reactive. It kicks in only
>> when the executors fail with the OOM exception.
>>
>> We understand that the problem s

SPARK-9559

[Ashish Soni]

Hi All ,

Just wanted to know if there is any work around or resolution for below
issue in Stand alone mode

https://issues.apache.org/jira/browse/SPARK-9559

Ashish

Re: Mutiple spark contexts

[Ashish Soni]

There is a property you need to set which is
spark.driver.allowMultipleContexts=true

Ashish

On Wed, Jan 27, 2016 at 1:39 PM, Jakob Odersky  wrote:

> A while ago, I remember reading that multiple active Spark contexts
> per JVM was a possible future enhancement.
> I was wondering if this is still on the roadmap, what the major
> obstacles are and if I can be of any help in adding this feature?
>
> regards,
> --Jakob
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: High Availability of Spark Driver

[Ashish Rawat]

Thanks Steve. I had not spent many brain cycles on analysing the Yarn pieces, 
your insights would be extremely useful.

I was also considering Zookeeper and Yarn registry for persisting state and 
sharing information. But for a basic POC, I used the file system and was able to

  1.  Preserve Executors.
  2.  Reconnect Executors back to Driver by storing the Executor endpoints info 
into a local file system. When driver restarts, use this info to send update 
driver message to executor endpoints. Executors can then update all of their 
Akka endpoints and reconnect.
  3.  Reregister Block Manager and report back blocks. This utilises most of 
Spark’s existing code, I only had to update the BlockManagerMaster endpoint in 
executors.

Surprisingly, Spark components took the restart in a much better way than I had 
anticipated and were easy to accept new work :-)

I am still figuring out other complexities around preserving RDD lineage and 
computation. From my initial analysis, preserving the whole computation might 
be complex and may not be required. Perhaps, the lineage of only the cached 
RDDs can be preserved to recover any lost blocks.

I am definitely not underestimating the effort, both within Spark and around 
interfacing with Yarn, but just trying to emphasise that a single node leading 
to full application restart, does not seem right for a long running service. 
Thoughts?

Regards,
Ashish

From: Steve Loughran mailto:ste...@hortonworks.com>>
Date: Thursday, 27 August 2015 4:19 pm
To: Ashish Rawat mailto:ashish.ra...@guavus.com>>
Cc: "dev@spark.apache.org<mailto:dev@spark.apache.org>" 
mailto:dev@spark.apache.org>>
Subject: Re: High Availability of Spark Driver


On 27 Aug 2015, at 08:42, Ashish Rawat 
mailto:ashish.ra...@guavus.com>> wrote:

Hi Patrick,

As discussed in another thread, we are looking for a solution to the problem of 
lost state on Spark Driver failure. Can you please share Spark’s long term 
strategy for resolving this problem.

<-- Original Mail Content Below -->

We have come across the problem of Spark Applications (on Yarn) requiring a 
restart in case of Spark Driver (or application master) going down. This is 
hugely inconvenient for long running applications which are maintaining a big 
state in memory. The repopulation of state in itself may require a downtime of 
many minutes, which is not acceptable for most live systems.

As you would have noticed that Yarn community has acknowledged "long running 
services" as an important class of use cases, and thus identified and removed 
problems in working with long running services in Yarn.
http://hortonworks.com/blog/support-long-running-services-hadoop-yarn-clusters/


Yeah, I spent a lot of time on that, or at least using the features, in other 
work under YARN-896, summarised in 
http://www.slideshare.net/steve_l/yarn-services

It would be great if Spark, which is the most important processing engine on 
Yarn,

I'f you look at the CPU-hours going in to the big hadoop clusters, it's 
actually MR work and things behind Hive. but: these apps don't attempt HA

Why not? It requires whatever maintains the overall app status (spark: the 
driver) to persist that state in a way where it can be rebuilt. A restarted AM 
with the "retain containers" feature turned on gets nothing back from YARN 
except the list of previous allocated containers, and is left to sort itself 
out.

also figures out issues in working with long running Spark applications and 
publishes recommendations or make framework changes for removing those. The 
need to keep the application running in case of Driver and Application Master 
failure, seems to be an important requirement from this perspective. The two 
most compelling use cases being:

  1.  Huge state of historical data in Spark Streaming, required for stream 
processing
  2.  Very large cached tables in Spark SQL (very close to our use case where 
we periodically cache RDDs and query using Spark SQL)


Generally spark streaming is viewed as the big need here, but yes, long-lived 
cached data matters.

Bear in mind that before Spark 1.5, you can't run any spark YARN app for longer 
than the expiry time of your delegation tokens, so in a secure cluster you have 
a limit of a couple of days anyway. Unless your cluster is particularly 
unreliable, AM failures are usually pretty unlikely in such a short timespan. 
Container failure is more likely as 1) you have more of them and 2) if you have 
pre-emption turned on in the scheduler or are pushing the work out to a label 
containing spot VMs, the will fail.

In our analysis, for both of these use cases, a working HA solution can be 
built by

  1.  Preserving the state of executors (not killing them on driver failures)

This is a critical one


  1.  Persisting some meta info required by Spark SQL and Block Manager.

again, needs a failure tolerant storage mechanism. HDFS

Re: High Availability of Spark Driver

[Ashish Rawat]

If anyone else is also facing similar problems and have figured out a good 
workaround within the current design, then please share.

Regards,
Ashish

From: Ashish Rawat mailto:ashish.ra...@guavus.com>>
Date: Thursday, 27 August 2015 1:12 pm
To: "dev@spark.apache.org<mailto:dev@spark.apache.org>" 
mailto:dev@spark.apache.org>>
Subject: FW: High Availability of Spark Driver

Hi Patrick,

As discussed in another thread, we are looking for a solution to the problem of 
lost state on Spark Driver failure. Can you please share Spark's long term 
strategy for resolving this problem.

<-- Original Mail Content Below -->

We have come across the problem of Spark Applications (on Yarn) requiring a 
restart in case of Spark Driver (or application master) going down. This is 
hugely inconvenient for long running applications which are maintaining a big 
state in memory. The repopulation of state in itself may require a downtime of 
many minutes, which is not acceptable for most live systems.

As you would have noticed that Yarn community has acknowledged "long running 
services" as an important class of use cases, and thus identified and removed 
problems in working with long running services in Yarn.
http://hortonworks.com/blog/support-long-running-services-hadoop-yarn-clusters/

It would be great if Spark, which is the most important processing engine on 
Yarn, also figures out issues in working with long running Spark applications 
and publishes recommendations or make framework changes for removing those. The 
need to keep the application running in case of Driver and Application Master 
failure, seems to be an important requirement from this perspective. The two 
most compelling use cases being:

  1.  Huge state of historical data in Spark Streaming, required for stream 
processing
  2.  Very large cached tables in Spark SQL (very close to our use case where 
we periodically cache RDDs and query using Spark SQL)

In our analysis, for both of these use cases, a working HA solution can be 
built by

  1.  Preserving the state of executors (not killing them on driver failures)
  2.  Persisting some meta info required by Spark SQL and Block Manager.
  3.  Restarting and reconnecting the Driver to AM and executors

This would preserve the cached data, enabling the application to come back 
quickly. This can further be extended for preserving and recovering the 
Computation State.

I would request you to share your thoughts on this issue and possible future 
directions.

Regards,
Ashish

FW: High Availability of Spark Driver

[Ashish Rawat]

Hi Patrick,

As discussed in another thread, we are looking for a solution to the problem of 
lost state on Spark Driver failure. Can you please share Spark's long term 
strategy for resolving this problem.

<-- Original Mail Content Below -->

We have come across the problem of Spark Applications (on Yarn) requiring a 
restart in case of Spark Driver (or application master) going down. This is 
hugely inconvenient for long running applications which are maintaining a big 
state in memory. The repopulation of state in itself may require a downtime of 
many minutes, which is not acceptable for most live systems.

As you would have noticed that Yarn community has acknowledged "long running 
services" as an important class of use cases, and thus identified and removed 
problems in working with long running services in Yarn.
http://hortonworks.com/blog/support-long-running-services-hadoop-yarn-clusters/

It would be great if Spark, which is the most important processing engine on 
Yarn, also figures out issues in working with long running Spark applications 
and publishes recommendations or make framework changes for removing those. The 
need to keep the application running in case of Driver and Application Master 
failure, seems to be an important requirement from this perspective. The two 
most compelling use cases being:

  1.  Huge state of historical data in Spark Streaming, required for stream 
processing
  2.  Very large cached tables in Spark SQL (very close to our use case where 
we periodically cache RDDs and query using Spark SQL)

In our analysis, for both of these use cases, a working HA solution can be 
built by

  1.  Preserving the state of executors (not killing them on driver failures)
  2.  Persisting some meta info required by Spark SQL and Block Manager.
  3.  Restarting and reconnecting the Driver to AM and executors

This would preserve the cached data, enabling the application to come back 
quickly. This can further be extended for preserving and recovering the 
Computation State.

I would request you to share your thoughts on this issue and possible future 
directions.

Regards,
Ashish