Was able to resolve the idle connections being terminated issue using EnvoyFilter
On Sat, 3 Sept 2022 at 18:14, Ilan Filonenko <i...@cornell.edu> wrote: > Must be set in envoy (maybe could passthrough via istio) > > https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-httpprotocoloptions-idle-timeout > > > On Sat, Sep 3, 2022 at 4:23 AM Deepak Sharma <deepakmc...@gmail.com> > wrote: > >> Thank for the reply IIan . >> Can we set this in spark conf or does it need to goto istio / envoy conf? >> >> >> >> On Sat, 3 Sept 2022 at 10:28, Ilan Filonenko <i...@cornell.edu> wrote: >> >>> This might be a result of the idle_timeout that is configured in envoy. >>> The default is an hour. >>> >>> On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma <deepakmc...@gmail.com> >>> wrote: >>> >>>> Hi All, >>>> In 1 of our cluster , we enabled Istio where spark is running in >>>> distributed mode. >>>> Spark works fine when we run it with Istio in standalone mode. >>>> In spark distributed mode , we are seeing that every 1 hour or so the >>>> workers are getting disassociated from master and then master is not able >>>> to spawn any jobs on these workers , until we restart spark rest server. >>>> >>>> Here is the error we see in the worker logs: >>>> >>>> >>>> *ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : >>>> Driver spark-rest-service:44463 disassociated! Shutting down.* >>>> >>>> For 1 hour or so (until this issue happens) , spark distributed mode >>>> works just fine. >>>> >>>> >>>> Thanks >>>> Deepak >>>> >>>
