(Sorry Mehmet, I'm seeing just now your first reply with the link to SO; it had first gone to my spam folder :-/ )
On 2017-09-14 10:02 CEST, GRANDE Johan Ext DTSI/DSI wrote: Well if the order cannot be guaranteed in case of a failure (or at all since failure can happen transparently), what does it mean to sort an RDD (method sortBy)? On 2017-09-14 03:36 CEST mehmet.su...@gmail.com wrote: I think it is one of the conceptual difference in Spark compare to other languages, there is no indexing in plain RDDs, This was the thread with Ankit: Yes. So order preservation can not be guaranteed in the case of failure. Also not sure if partitions are ordered. Can you get the same sequence of partitions in mapPartition? On 13 Sep 2017 19:54, "Ankit Maloo" <ankitmaloo1...@gmail.com> wrote: > > Rdd are fault tolerant as it can be recomputed using DAG without storing the > intermediate RDDs. > > On 13-Sep-2017 11:16 PM, "Suzen, Mehmet" <su...@acm.org> wrote: >> >> But what happens if one of the partitions fail, how fault tolerance recover >> elements in other partitions. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
