Re: SPIP: Executor Plugin (SPARK-24918)
Both ahead of time, or just in time. Just like a normal Spark closure. On Fri, Aug 31, 2018 at 10:18 AM Nihar Sheth wrote: > Hi @rxin, > > Just to make sure I understand your desired use case, are you suggesting a > way (for the driver) to call, at any given time, a general method that can > be defined ahead of time on the executors? > > On Thu, Aug 30, 2018 at 11:11 PM, Reynold Xin wrote: > >> I actually had a similar use case a while ago, but not entirely the same. >> In my use case, Spark is already up, but I want to make sure all existing >> (and new) executors run some specific code. Can we update the API to >> support that? I think that's doable if we split the design into two: one is >> the ability to do what I just mentioned, and second is the ability to >> register via config class when Spark starts to run the code. >> >> >> On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung >> wrote: >> >>> +1 >>> -- >>> *From:* Mridul Muralidharan >>> *Sent:* Wednesday, August 29, 2018 1:27:27 PM >>> *To:* dev@spark.apache.org >>> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918) >>> >>> +1 >>> I left a couple of comments in NiharS's PR, but this is very useful to >>> have in spark ! >>> >>> Regards, >>> Mridul >>> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid >>> wrote: >>> > >>> > I'd like to propose adding a plugin api for Executors, primarily for >>> instrumentation and debugging ( >>> https://issues.apache.org/jira/browse/SPARK-24918). The changes are >>> small, but as its adding a new api, it might be spip-worthy. I mentioned >>> it as well in a recent email I sent about memory monitoring >>> > >>> > The spip proposal is here (and attached to the jira as well): >>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing >>> > >>> > There are already some comments on the jira and pr, and I hope to get >>> more thoughts and opinions on it. >>> > >>> > thanks, >>> > Imran >>> >>> - >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> >
Re: SPIP: Executor Plugin (SPARK-24918)
Hi @rxin, Just to make sure I understand your desired use case, are you suggesting a way (for the driver) to call, at any given time, a general method that can be defined ahead of time on the executors? On Thu, Aug 30, 2018 at 11:11 PM, Reynold Xin wrote: > I actually had a similar use case a while ago, but not entirely the same. > In my use case, Spark is already up, but I want to make sure all existing > (and new) executors run some specific code. Can we update the API to > support that? I think that's doable if we split the design into two: one is > the ability to do what I just mentioned, and second is the ability to > register via config class when Spark starts to run the code. > > > On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung > wrote: > >> +1 >> -- >> *From:* Mridul Muralidharan >> *Sent:* Wednesday, August 29, 2018 1:27:27 PM >> *To:* dev@spark.apache.org >> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918) >> >> +1 >> I left a couple of comments in NiharS's PR, but this is very useful to >> have in spark ! >> >> Regards, >> Mridul >> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid >> wrote: >> > >> > I'd like to propose adding a plugin api for Executors, primarily for >> instrumentation and debugging (https://issues.apache.org/ >> jira/browse/SPARK-24918). The changes are small, but as its adding a >> new api, it might be spip-worthy. I mentioned it as well in a recent email >> I sent about memory monitoring >> > >> > The spip proposal is here (and attached to the jira as well): >> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5c >> bHBQtyqIA2hgtc/edit?usp=sharing >> > >> > There are already some comments on the jira and pr, and I hope to get >> more thoughts and opinions on it. >> > >> > thanks, >> > Imran >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: SPIP: Executor Plugin (SPARK-24918)
+1 Original message From: Reynold Xin Date: 8/30/18 11:11 PM (GMT-08:00) To: Felix Cheung Cc: dev Subject: Re: SPIP: Executor Plugin (SPARK-24918) I actually had a similar use case a while ago, but not entirely the same. In my use case, Spark is already up, but I want to make sure all existing (and new) executors run some specific code. Can we update the API to support that? I think that's doable if we split the design into two: one is the ability to do what I just mentioned, and second is the ability to register via config class when Spark starts to run the code. On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung wrote: +1 From: Mridul Muralidharan Sent: Wednesday, August 29, 2018 1:27:27 PM To: dev@spark.apache.org Subject: Re: SPIP: Executor Plugin (SPARK-24918) +1 I left a couple of comments in NiharS's PR, but this is very useful to have in spark ! Regards, Mridul On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid wrote: > > I'd like to propose adding a plugin api for Executors, primarily for > instrumentation and debugging > (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small, > but as its adding a new api, it might be spip-worthy. I mentioned it as well in a recent email I sent about memory monitoring > > The spip proposal is here (and attached to the jira as well): https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing > > There are already some comments on the jira and pr, and I hope to get more > thoughts and opinions on it. > > thanks, > Imran - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: SPIP: Executor Plugin (SPARK-24918)
+1 On Fri, Aug 31, 2018 at 8:11 AM, Reynold Xin wrote: > I actually had a similar use case a while ago, but not entirely the same. > In my use case, Spark is already up, but I want to make sure all existing > (and new) executors run some specific code. Can we update the API to > support that? I think that's doable if we split the design into two: one is > the ability to do what I just mentioned, and second is the ability to > register via config class when Spark starts to run the code. > > > On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung > wrote: > >> +1 >> -- >> *From:* Mridul Muralidharan >> *Sent:* Wednesday, August 29, 2018 1:27:27 PM >> *To:* dev@spark.apache.org >> *Subject:* Re: SPIP: Executor Plugin (SPARK-24918) >> >> +1 >> I left a couple of comments in NiharS's PR, but this is very useful to >> have in spark ! >> >> Regards, >> Mridul >> On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid >> wrote: >> > >> > I'd like to propose adding a plugin api for Executors, primarily for >> instrumentation and debugging (https://issues.apache.org/ >> jira/browse/SPARK-24918). The changes are small, but as its adding a >> new api, it might be spip-worthy. I mentioned it as well in a recent email >> I sent about memory monitoring >> > >> > The spip proposal is here (and attached to the jira as well): >> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5c >> bHBQtyqIA2hgtc/edit?usp=sharing >> > >> > There are already some comments on the jira and pr, and I hope to get >> more thoughts and opinions on it. >> > >> > thanks, >> > Imran >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: SPIP: Executor Plugin (SPARK-24918)
I actually had a similar use case a while ago, but not entirely the same. In my use case, Spark is already up, but I want to make sure all existing (and new) executors run some specific code. Can we update the API to support that? I think that's doable if we split the design into two: one is the ability to do what I just mentioned, and second is the ability to register via config class when Spark starts to run the code. On Thu, Aug 30, 2018 at 11:01 PM Felix Cheung wrote: > +1 > -- > *From:* Mridul Muralidharan > *Sent:* Wednesday, August 29, 2018 1:27:27 PM > *To:* dev@spark.apache.org > *Subject:* Re: SPIP: Executor Plugin (SPARK-24918) > > +1 > I left a couple of comments in NiharS's PR, but this is very useful to > have in spark ! > > Regards, > Mridul > On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid > wrote: > > > > I'd like to propose adding a plugin api for Executors, primarily for > instrumentation and debugging ( > https://issues.apache.org/jira/browse/SPARK-24918). The changes are > small, but as its adding a new api, it might be spip-worthy. I mentioned > it as well in a recent email I sent about memory monitoring > > > > The spip proposal is here (and attached to the jira as well): > https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing > > > > There are already some comments on the jira and pr, and I hope to get > more thoughts and opinions on it. > > > > thanks, > > Imran > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: SPIP: Executor Plugin (SPARK-24918)
+1 From: Mridul Muralidharan Sent: Wednesday, August 29, 2018 1:27:27 PM To: dev@spark.apache.org Subject: Re: SPIP: Executor Plugin (SPARK-24918) +1 I left a couple of comments in NiharS's PR, but this is very useful to have in spark ! Regards, Mridul On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid wrote: > > I'd like to propose adding a plugin api for Executors, primarily for > instrumentation and debugging > (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small, > but as its adding a new api, it might be spip-worthy. I mentioned it as well > in a recent email I sent about memory monitoring > > The spip proposal is here (and attached to the jira as well): > https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing > > There are already some comments on the jira and pr, and I hope to get more > thoughts and opinions on it. > > thanks, > Imran - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: SPIP: Executor Plugin (SPARK-24918)
+1 I left a couple of comments in NiharS's PR, but this is very useful to have in spark ! Regards, Mridul On Fri, Aug 3, 2018 at 10:00 AM Imran Rashid wrote: > > I'd like to propose adding a plugin api for Executors, primarily for > instrumentation and debugging > (https://issues.apache.org/jira/browse/SPARK-24918). The changes are small, > but as its adding a new api, it might be spip-worthy. I mentioned it as well > in a recent email I sent about memory monitoring > > The spip proposal is here (and attached to the jira as well): > https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing > > There are already some comments on the jira and pr, and I hope to get more > thoughts and opinions on it. > > thanks, > Imran - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
