Re: Spark Issue with Istio in Distributed Mode

[Deepak Sharma]

Was able to resolve the idle connections being terminated issue using
EnvoyFilter

On Sat, 3 Sept 2022 at 18:14, Ilan Filonenko  wrote:

> Must be set in envoy (maybe could passthrough via istio)
>
> https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-httpprotocoloptions-idle-timeout
>
>
> On Sat, Sep 3, 2022 at 4:23 AM Deepak Sharma 
> wrote:
>
>> Thank for the reply IIan .
>> Can we set this in spark conf or does it need to goto istio / envoy conf?
>>
>>
>>
>> On Sat, 3 Sept 2022 at 10:28, Ilan Filonenko  wrote:
>>
>>> This might be a result of the idle_timeout that is configured in envoy.
>>> The default is an hour.
>>>
>>> On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma 
>>> wrote:
>>>
 Hi All,
 In 1 of our cluster , we enabled Istio where spark is running in
 distributed mode.
 Spark works fine when we run it with Istio in standalone mode.
 In spark distributed mode , we are seeing that every 1 hour or so the
 workers are getting disassociated from master and then master is not able
 to spawn any jobs on these workers , until we restart spark rest server.

 Here is the error we see in the worker logs:


 *ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to :
 Driver spark-rest-service:44463 disassociated! Shutting down.*

 For 1 hour or so (until this issue happens) , spark distributed mode
 works just fine.


 Thanks
 Deepak

>>>

Re: Spark Issue with Istio in Distributed Mode

[Deepak Sharma]

Thank for the reply IIan .
Can we set this in spark conf or does it need to goto istio / envoy conf?



On Sat, 3 Sept 2022 at 10:28, Ilan Filonenko  wrote:

> This might be a result of the idle_timeout that is configured in envoy.
> The default is an hour.
>
> On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma 
> wrote:
>
>> Hi All,
>> In 1 of our cluster , we enabled Istio where spark is running in
>> distributed mode.
>> Spark works fine when we run it with Istio in standalone mode.
>> In spark distributed mode , we are seeing that every 1 hour or so the
>> workers are getting disassociated from master and then master is not able
>> to spawn any jobs on these workers , until we restart spark rest server.
>>
>> Here is the error we see in the worker logs:
>>
>>
>> *ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to :
>> Driver spark-rest-service:44463 disassociated! Shutting down.*
>>
>> For 1 hour or so (until this issue happens) , spark distributed mode
>> works just fine.
>>
>>
>> Thanks
>> Deepak
>>
>

Spark Issue with Istio in Distributed Mode

[Deepak Sharma]

Hi All,
In 1 of our cluster , we enabled Istio where spark is running in
distributed mode.
Spark works fine when we run it with Istio in standalone mode.
In spark distributed mode , we are seeing that every 1 hour or so the
workers are getting disassociated from master and then master is not able
to spawn any jobs on these workers , until we restart spark rest server.

Here is the error we see in the worker logs:


*ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Driver
spark-rest-service:44463 disassociated! Shutting down.*

For 1 hour or so (until this issue happens) , spark distributed mode works
just fine.


Thanks
Deepak