Re: is the Lineage of RDD stored as a byte code in memory or a file?

You are saying the RDD lineage must be serialized, otherwise we could not recreate it after a node failure. This is false. The RDD lineage is not serialized. It is only relevant to the driver application and as such it is just kept in memory in the driver application. If the driver application

Re: is the Lineage of RDD stored as a byte code in memory or a file?

can you please elaborate a bit more? On Wed, Aug 24, 2016 12:41 AM, Sean Owen so...@cloudera.com wrote: Byte code, no. It's sufficient to store the information that the RDD represents, which can include serialized function closures, but that's not quite storing byte code. On Wed, Aug 24,

Re: is the Lineage of RDD stored as a byte code in memory or a file?

Byte code, no. It's sufficient to store the information that the RDD represents, which can include serialized function closures, but that's not quite storing byte code. On Wed, Aug 24, 2016 at 2:00 AM, kant kodali wrote: > Hi Guys, > > I have this question for a very long

is the Lineage of RDD stored as a byte code in memory or a file?

Hi Guys, I have this question for a very long time and after diving into the source code(specifically from the links below) I have a feeling that the lineage of an RDD (the transformations) are converted into byte code and stored in memory or disk. or if I were to ask another question on a