Must be set in envoy (maybe could passthrough via istio) https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-httpprotocoloptions-idle-timeout
On Sat, Sep 3, 2022 at 4:23 AM Deepak Sharma <deepakmc...@gmail.com> wrote: > Thank for the reply IIan . > Can we set this in spark conf or does it need to goto istio / envoy conf? > > > > On Sat, 3 Sept 2022 at 10:28, Ilan Filonenko <i...@cornell.edu> wrote: > >> This might be a result of the idle_timeout that is configured in envoy. >> The default is an hour. >> >> On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma <deepakmc...@gmail.com> >> wrote: >> >>> Hi All, >>> In 1 of our cluster , we enabled Istio where spark is running in >>> distributed mode. >>> Spark works fine when we run it with Istio in standalone mode. >>> In spark distributed mode , we are seeing that every 1 hour or so the >>> workers are getting disassociated from master and then master is not able >>> to spawn any jobs on these workers , until we restart spark rest server. >>> >>> Here is the error we see in the worker logs: >>> >>> >>> *ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : >>> Driver spark-rest-service:44463 disassociated! Shutting down.* >>> >>> For 1 hour or so (until this issue happens) , spark distributed mode >>> works just fine. >>> >>> >>> Thanks >>> Deepak >>> >>
