The issue is: where do you name the class you want to initialize? In your job?
What if the plugin has nothing to do with your job, e.g. it's some monitoring tool? Are you going to modify all your jobs so you include initialization of that particular plugin? Forcing everybody to change their code, builds, re-deploy stuff, etc? What if a task in an early stage of your job, the one that initializes the plugin, runs in executor A, and a task in a further stage runs on executor B? It's just way more complicated, if possible at all, to write this feature by asking everybody to change their application code. On Tue, Aug 28, 2018 at 9:39 AM, Sean Owen <sro...@apache.org> wrote: > I should be able to force a class to init by just naming it. The issue is > that this means doing that in any job that needs init? not that it doesn't > work, right? I concede that could be annoying if just about all your code > needs this init; I had understood the use cases to be more like "establish > some local config and init for this particular thing I'm doing for this > legacy system". > > On Tue, Aug 28, 2018 at 11:35 AM Marcelo Vanzin <van...@cloudera.com> wrote: >> >> +1 >> >> Class init is not enough because there is nowhere for you to force a >> random class to be initialized. This is basically adding that >> mechanism, instead of forcing people to add hacks using e.g. >> mapPartitions which don't even cover all scenarios. >> >> On Tue, Aug 28, 2018 at 7:09 AM, Sean Owen <sro...@apache.org> wrote: >> > Still +0 on the idea, as I am still not sure it does much over simple >> > JVM >> > mechanisms like a class init. More comments on the JIRA. I can't say >> > it's a >> > bad idea though, so would not object to it. >> > >> > On Tue, Aug 28, 2018 at 8:50 AM Imran Rashid >> > <iras...@cloudera.com.invalid> >> > wrote: >> >> >> >> There has been discussion on jira & the PR, all generally positive, so >> >> I'd >> >> like to call a vote for this spip. >> >> >> >> I'll start with own +1. >> >> >> >> On Fri, Aug 3, 2018 at 11:59 AM Imran Rashid <iras...@cloudera.com> >> >> wrote: >> >>> >> >>> I'd like to propose adding a plugin api for Executors, primarily for >> >>> instrumentation and debugging >> >>> (https://issues.apache.org/jira/browse/SPARK-24918). The changes are >> >>> small, >> >>> but as its adding a new api, it might be spip-worthy. I mentioned it >> >>> as >> >>> well in a recent email I sent about memory monitoring >> >>> >> >>> The spip proposal is here (and attached to the jira as well): >> >>> >> >>> https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit?usp=sharing >> >>> >> >>> There are already some comments on the jira and pr, and I hope to get >> >>> more thoughts and opinions on it. >> >>> >> >>> thanks, >> >>> Imran >> >> >> >> -- >> Marcelo -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
