Does this property spark.kubernetes.executor.deleteontermination checks whether the executor which is deleted have shuffle data or not ?
On Tue, 18 Jan 2022, 11:20 Pralabh Kumar, <pralabhku...@gmail.com> wrote: > Hi spark team > > Have cluster wide property spark.kubernetis.executor.deleteontermination > to true. > During the long running job, some of the executor got deleted which have > shuffle data. Because of this, in the subsequent stage , we get lot of > spark shuffle fetch fail exceptions. > > > Please let me know , is there a way to fix it. Note if setting above > property to false , I face no shuffle fetch exception. > > > Regards > Pralabh >
