Hey Mich,
the question was not about one particular job but rather about general way
how spark functions.
If I do call persist on rdd then the executor which computed the partition
of the rdd would try to save the partition on the memory that executor has
reserved for caching. So my question is
Hi,
Are you caching RDD into storage memory here?
Example
s.persist(org.apache.spark.storage.StorageLevel.MEMORY_ONLY)
Do you have a snapshot of your storage tab?
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
Hey all,
I have a conceptual question which I have hard time finding answer for.
Is the jvm where spark driver is running also used to run computations over
rdd partitions and persist them? The answer is obvious for local mode
(yes). But when it runs on yarn/mesos/standalone with many executors
