Thank for the reply IIan . Can we set this in spark conf or does it need to goto istio / envoy conf?
On Sat, 3 Sept 2022 at 10:28, Ilan Filonenko <i...@cornell.edu> wrote: > This might be a result of the idle_timeout that is configured in envoy. > The default is an hour. > > On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma <deepakmc...@gmail.com> > wrote: > >> Hi All, >> In 1 of our cluster , we enabled Istio where spark is running in >> distributed mode. >> Spark works fine when we run it with Istio in standalone mode. >> In spark distributed mode , we are seeing that every 1 hour or so the >> workers are getting disassociated from master and then master is not able >> to spawn any jobs on these workers , until we restart spark rest server. >> >> Here is the error we see in the worker logs: >> >> >> *ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : >> Driver spark-rest-service:44463 disassociated! Shutting down.* >> >> For 1 hour or so (until this issue happens) , spark distributed mode >> works just fine. >> >> >> Thanks >> Deepak >> >
