Just to correct the last sentence, if we end up starting a new instance of Spark, I don't think it will be able to read the shuffle data from storage from another instance, I stand corrected.
Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 22 May 2023 at 15:27, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Maksym. > > Let us understand the basics here first > My thoughtsSpark replicates the partitions among multiple nodes. If one > executor fails, it moves the processing over to the other executor. > However, if the data is lost, it re-executes the processing that generated > the data, > and might have to go back to the source. In case of failure, there will > be delay in getting the results. The amount of delay depends on how much > reprocessing Spark needs to do. > Spark, by itself, doesn't add executors when executors fail. It just moves > the tasks to other executors. If you are installing plain vanilla Spark > on your own cluster, you need to figure out how to bring back executors. > Most of the popular platforms built on top of Spark (Glue, EMR, GKS) will > replace failed nodes. However, I don't think that applies to your case. > > With regard to below point you raised > > "" One of the offerings from the service we use is EBS migration which > basically means if a host is about to get evicted, a new host is created > and the EBS volume is attached to it. When Spark assigns a new executor > to the newly created instance, it basically can recover all the shuffle > files that are already persisted in the migrated EBS volume Is this how > it works? Do executors recover / re-register the shuffle files that they > found?""" > > My understanding is that RDD lineage keeps track of records of what needs > to be re-executed. It uses RDD lineage to figure out what needs to be > re-executed in the same Spark instance. For example, if you have done a > groupBy Key , you will have 2 stages. After the first stage, the data will > be shuffled by hashing the groupBy key , so that data for the same value of > key lands in same partition. Now, if one of those partitions is lost > during execution of second stage, I am guessing Spark will have to go back > and re-execute all the tasks in the first stage. > > > So it your case stating a new instance will not have the same issue, > re-executes the job. > > > HTH > > > Mich Talebzadeh, > Lead Solutions Architect/Engineering Lead > Palantir Technologies Limited > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 22 May 2023 at 13:19, Maksym M <maksy...@nubank.com.br.invalid> > wrote: > >> Hey vaquar, >> >> The link does't explain the crucial detail we're interested in - does >> executor >> re-use the data that exists on a node from previous executor and if not, >> how >> can we configure it to do so? >> >> We are not running on kubernetes, so EKS/Kubernetes-specific advice isn't >> very relevant. >> >> We are running spark standalone mode. >> >> Best regards, >> maksym >> >> On 2023/05/17 12:28:35 vaquar khan wrote: >> > Following link you will get all required details >> > >> > >> https://aws.amazon.com/blogs/containers/best-practices-for-running-spark-on-amazon-eks/ >> > >> > Let me know if you required further informations. >> > >> > >> > Regards, >> > Vaquar khan >> > >> > >> > >> > >> > On Mon, May 15, 2023, 10:14 PM Mich Talebzadeh <mi...@gmail.com> >> > wrote: >> > >> > > Couple of points >> > > >> > > Why use spot or pre-empt intantes when your application as you stated >> > > shuffles heavily. >> > > Have you looked at why you are having these shuffles? What is the >> cause of >> > > these large transformations ending up in shuffle >> > > >> > > Also on your point: >> > > "..then ideally we should expect that when an executor is killed/OOM'd >> > > and a new executor is spawned on the same host, the new executor >> registers >> > > the shuffle files to itself. Is that so?" >> > > >> > > What guarantee is that the new executor with inherited shuffle files >> will >> > > succeed? >> > > >> > > Also OOM is often associated with some form of skewed data >> > > >> > > HTH >> > > . >> > > Mich Talebzadeh, >> > > Lead Solutions Architect/Engineering Lead >> > > Palantir Technologies Limited >> > > London >> > > United Kingdom >> > > >> > > >> > > view my Linkedin profile >> > > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> > > >> > > >> > > https://en.everybodywiki.com/Mich_Talebzadeh >> > > >> > > >> > > >> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any >> > > loss, damage or destruction of data or any other property which may >> arise >> > > from relying on this email's technical content is explicitly >> disclaimed. >> > > The author will in no case be liable for any monetary damages arising >> from >> > > such loss, damage or destruction. >> > > >> > > >> > > >> > > >> > > On Mon, 15 May 2023 at 13:11, Faiz Halde <fa...@nubank.com.br.invalid >> > >> > > wrote: >> > > >> > >> Hello, >> > >> >> > >> We've been in touch with a few spark specialists who suggested us a >> > >> potential solution to improve the reliability of our jobs that are >> shuffle >> > >> heavy >> > >> >> > >> Here is what our setup looks like >> > >> >> > >> - Spark version: 3.3.1 >> > >> - Java version: 1.8 >> > >> - We do not use external shuffle service >> > >> - We use spot instances >> > >> >> > >> We run spark jobs on clusters that use Amazon EBS volumes. The >> > >> spark.local.dir is mounted on this EBS volume. One of the offerings >> from >> > >> the service we use is EBS migration which basically means if a host >> is >> > >> about to get evicted, a new host is created and the EBS volume is >> attached >> > >> to it >> > >> >> > >> When Spark assigns a new executor to the newly created instance, it >> > >> basically can recover all the shuffle files that are already >> persisted in >> > >> the migrated EBS volume >> > >> >> > >> Is this how it works? Do executors recover / re-register the shuffle >> > >> files that they found? >> > >> >> > >> So far I have not come across any recovery mechanism. I can only see >> > >> >> > >> KubernetesLocalDiskShuffleDataIO >> > >> >> > >> that has a pre-init step where it tries to register the available >> > >> shuffle files to itself >> > >> >> > >> A natural follow-up on this, >> > >> >> > >> If what they claim is true, then ideally we should expect that when >> an >> > >> executor is killed/OOM'd and a new executor is spawned on the same >> host, >> > >> the new executor registers the shuffle files to itself. Is that so? >> > >> >> > >> Thanks >> > >> >> > >> ------------------------------ >> > >> Confidentiality note: This e-mail may contain confidential >> information >> > >> from Nu Holdings Ltd and/or its affiliates. If you have received it >> by >> > >> mistake, please let us know by e-mail reply and delete it from your >> system; >> > >> you may not copy this message or disclose its contents to anyone; for >> > >> details about what personal information we collect and why, please >> refer to >> > >> our privacy policy >> > >> < >> https://api.mziq.com/mzfilemanager/v2/d/59a081d2-0d63-4bb5-b786-4c07ae26bc74/6f4939b9-5f74-a528-1835-596b481dca54 >> > >> > >> . >> > >> >> > > >> > >> -- >> >> Confidentiality note: This e-mail may contain confidential information >> from Nu Holdings Ltd and/or its affiliates. If you have received it by >> mistake, please let us know by e-mail reply and delete it from your >> system; >> you may not copy this message or disclose its contents to anyone; for >> details about what personal information we collect and why, please refer >> to >> our privacy policy >> < >> https://api.mziq.com/mzfilemanager/v2/d/59a081d2-0d63-4bb5-b786-4c07ae26bc74/6f4939b9-5f74-a528-1835-596b481dca54 >> >. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
