Thank you so much for the detailed explanation. I was able to recollect a
few things about Spark.
Thanks for your time once again :)
On Mon, Jan 24, 2022 at 2:20 PM Mich Talebzadeh
wrote:
> Hm,
>
> I don't see what partition failure means here.
>
> You can have a node or executor failure etc.
Hm,
I don't see what partition failure means here.
You can have a node or executor failure etc. So let us look at a scenario
here irrespective of being a streaming or micro-batch
Spark replicates the partitions among multiple nodes. *If one executor
fails*, it moves the processing over to the
Just couple of points to add:
1. "partition" is more of a logical construct so partitions can not fail. A
task which is reading from persistent storage to RDD can fail, and thus can
be rerun to reprocess the partition. What is Ranadip mentioned above is
true, with a caveat that data will be
Interesting question! I think this goes back to the roots of Spark. You ask
"But suppose if I am reading a file that is distributed across nodes in
partitions. So, what will happen if a partition fails that holds some
data?". Assuming you mean the distributed file system that holds the file
Probably, because Spark prefers locality, but not necessarily.
On Fri, Jan 21, 2022 at 2:10 PM Siddhesh Kalgaonkar <
kalgaonkarsiddh...@gmail.com> wrote:
> Thank you so much for this information, Sean. One more question, that when
> it wants to re-run the failed partition, where does it run? On
Thank you so much for this information, Sean. One more question, that when
it wants to re-run the failed partition, where does it run? On the same
node or some other node?
On Fri, 21 Jan 2022, 23:41 Sean Owen, wrote:
> The Spark program already knows the partitions of the data and where they
>
The Spark program already knows the partitions of the data and where they
exist; that's just defined by the data layout. It doesn't care what data is
inside. It knows partition 1 needs to be processed and if the task
processing it fails, needs to be run again. I'm not sure where you're
seeing data
Okay, so suppose I have 10 records distributed across 5 nodes and the
partition of the first node holding 2 records failed. I understand that it
will re-process this partition but how will it come to know that XYZ
partition was holding XYZ data so that it will pick again only those
records and
In that case, the file exists in parts across machines. No, tasks won't
re-read the whole file; no task does or can do that. Failed partitions are
reprocessed, but as in the first pass, the same partition is processed.
On Fri, Jan 21, 2022 at 12:00 PM Siddhesh Kalgaonkar <
Hello team,
I am aware that in case of memory issues when a task fails, it will try to
restart 4 times since it is a default number and if it still fails then it
will cause the entire job to fail.
But suppose if I am reading a file that is distributed across nodes in
partitions. So, what will
10 matches
Mail list logo
