Thanks Vipul,
I was looking specifically for documents spark committer use for reference.
Currently I’ve put custom logs in spark-core sources then building and running
jobs on it.
Form printed logs I try to understand execution flows.
From: Vipul Rajan
Sent: Thursday, September 19, 2019 12:23 PM
To: Kamal7 Kumar
Cc: spark-user
Subject: [External]Re: spark 2.x design docs
The e-mail below is from an external source. Please do not open attachments or
click links from an unknown or suspicious origin.
https://github.com/JerryLead/SparkInternals/blob/master/EnglishVersion/2-JobLogicalPlan.md
This is pretty old. but it might help a little bit. I myself am going through
the source code and trying to reverse engineer stuff. Let me know if you'd like
to pool resources sometime.
Regards
On Thu, Sep 19, 2019 at 11:35 AM
mailto:kamal7.ku...@ril.com>> wrote:
Hi ,
Can someone provide documents/links (apart from official documentation) for
understanding internal workings of spark-core,
Document containing components pseudo codes, class diagrams, execution flows ,
etc.
Thanks, Kamal
"Confidentiality Warning: This message and any attachments are intended only
for the use of the intended recipient(s), are confidential and may be
privileged. If you are not the intended recipient, you are hereby notified that
any review, re-transmission, conversion to hard copy, copying, circulation or
other use of this message and any attachments is strictly prohibited. If you
are not the intended recipient, please notify the sender immediately by return
email and delete this message and any attachments from your system.
Virus Warning: Although the company has taken reasonable precautions to ensure
no viruses are present in this email. The company cannot accept responsibility
for any loss or damage arising from the use of this email or attachment."
"Confidentiality Warning: This message and any attachments are intended only
for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient.
you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other
use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the
sender immediately by return email.
and delete this message and any attachments from your system.
Virus Warning: Although the company has taken reasonable precautions to ensure
no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from
the use of this email or attachment."
