unsubscribe

unsubscribe

How executor Understand which RDDs needed to be persist from the submitted Task

Hello all, I want to find when a Task that is sended by Driver to executor contains a call to function persist(). I am trying to read the submitted function that driver send to executor but I could not find any call to persist() method. Do you know how executor understand which RDDs needed to

unsubscribe

unsubscribe

Re: How executor Understand which RDDs needed to be persist from the submitted Task

RDD has a flag `storageLevel` which will be set by calling persist(). RDD will be serialized and sent to executors for running tasks. So executors just look at RDD.storageLevel and store output in its block manager when needed. On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis wrote: > Hello all, >

Re: How executor Understand which RDDs needed to be persist from the submitted Task

You can take a look at ShuffleMapTask.runTask. It's not just a function. On Thu, Jan 9, 2020 at 11:25 PM Jack Kolokasis wrote: > Thanks for the help. I read that driver only send a function (task) to > executors and the executors apply this function to their local RDD > partitions. > > Iacovos

Re: How executor Understand which RDDs needed to be persist from the submitted Task

Thanks for the help. I read that driver only send a function (task) to executors and the executors apply this function to their local RDD partitions. Iacovos On 1/9/20 5:03 PM, Wenchen Fan wrote: RDD has a flag `storageLevel` which will be set by calling persist(). RDD will be serialized and 

jenkins down next tuesday (jan 14th) morning

our colo is performing some electrical work next week, and jenkins will be down between 630-930am PST. sorry for the interruption in service... i'll be following up on this thread w/updates as they come in. -- Shane Knapp Computer Guy / Voice of Reason UC Berkeley EECS Research / RISELab Staff

Question about Datasource V2

Hi devs, I did some investigation on data source v2 feature and have some questions: 1. View is not supported for now, eg, CREATE VIEW, this feature is not finished or datasource v2 won’t support view? 2. Column resolution are failed when use datasource v2, eg, select tbl.col from tbl

Re: How executor Understand which RDDs needed to be persist from the submitted Task

Thanks for your help! Iacovos On 1/9/20 5:49 PM, Wenchen Fan wrote: You can take a look at ShuffleMapTask.runTask. It's not just a function. On Thu, Jan 9, 2020 at 11:25 PM Jack Kolokasis > wrote: Thanks for the help. I read that driver only send a

unsubscribe

Requesting for a support

Hi team, I am very new to the community I have experience in using spark framework. I would like to help the community any way I can. Regards Timmanna