Re: What happens when a partition that holds data under a task fails

2022-01-25 Thread Siddhesh Kalgaonkar
g on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Fri, 21 Jan 2022 at 17:54, Siddhesh Kalgaonkar < > kalgaonkarsiddh...@gmail.com> wro

Re: What happens when a partition that holds data under a task fails

2022-01-21 Thread Siddhesh Kalgaonkar
, not > somehow consumed and deleted. > > On Fri, Jan 21, 2022 at 12:07 PM Siddhesh Kalgaonkar < > kalgaonkarsiddh...@gmail.com> wrote: > >> Okay, so suppose I have 10 records distributed across 5 nodes and the >> partition of the first node holding 2 records

Re: What happens when a partition that holds data under a task fails

2022-01-21 Thread Siddhesh Kalgaonkar
sks won't > re-read the whole file; no task does or can do that. Failed partitions are > reprocessed, but as in the first pass, the same partition is processed. > > On Fri, Jan 21, 2022 at 12:00 PM Siddhesh Kalgaonkar < > kalgaonkarsiddh...@gmail.com> wrote: > >> Hello te

What happens when a partition that holds data under a task fails

2022-01-21 Thread Siddhesh Kalgaonkar
Hello team, I am aware that in case of memory issues when a task fails, it will try to restart 4 times since it is a default number and if it still fails then it will cause the entire job to fail. But suppose if I am reading a file that is distributed across nodes in partitions. So, what will