This might be a result of the idle_timeout that is configured in envoy. The default is an hour.
On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma <deepakmc...@gmail.com> wrote: > Hi All, > In 1 of our cluster , we enabled Istio where spark is running in > distributed mode. > Spark works fine when we run it with Istio in standalone mode. > In spark distributed mode , we are seeing that every 1 hour or so the > workers are getting disassociated from master and then master is not able > to spawn any jobs on these workers , until we restart spark rest server. > > Here is the error we see in the worker logs: > > > *ERROR CoarseGrainedExecutorBackend: Executor self-exiting due to : Driver > spark-rest-service:44463 disassociated! Shutting down.* > > For 1 hour or so (until this issue happens) , spark distributed mode works > just fine. > > > Thanks > Deepak >
