Hey Pasha, Is your suggestion towards the spark team? I can make use of the plugin system on the driver side of spark but considering spark is distributed, the executor side of spark needs to adapt to the pf4j framework I believe too
Thanks Faiz On Tue, Nov 28, 2023, 16:57 Pasha Finkelshtein <pavel.finkelsht...@gmail.com> wrote: > To me it seems like it's the best possible use case for PF4J. > > > [image: facebook] <https://fb.com/asm0dey> > [image: twitter] <https://twitter.com/asm0di0> > [image: linkedin] <https://linkedin.com/in/asm0dey> > [image: instagram] <https://instagram.com/asm0dey> > > Pasha Finkelshteyn > > Developer Advocate for Data Engineering > > JetBrains > > > > asm0...@jetbrains.com > https://linktr.ee/asm0dey > > Find out more <https://jetbrains.com> > > > > On Tue, 28 Nov 2023 at 12:47, Holden Karau <holden.ka...@gmail.com> wrote: > >> So I don’t think we make any particular guarantees around class path >> isolation there, so even if it does work it’s something you’d need to pay >> attention to on upgrades. Class path isolation is tricky to get right. >> >> On Mon, Nov 27, 2023 at 2:58 PM Faiz Halde <haldef...@gmail.com> wrote: >> >>> Hello, >>> >>> We are using spark 3.5.0 and were wondering if the following is >>> achievable using spark-core >>> >>> Our use case involves spinning up a spark cluster where the driver >>> application loads user jars containing spark transformations at runtime. A >>> single spark application can load multiple user jars ( same cluster ) that >>> can have class path conflicts if care is not taken >>> >>> AFAIK, to get this right requires the Executor to be designed in a way >>> that allows for class path isolation ( UDF, lambda expressions ). Ideally >>> per Spark Session is what we want >>> >>> I know Spark connect has been designed this way but Spark connect is not >>> an option for us at the moment. I had some luck using a private method >>> inside spark called JobArtifactSet.withActiveJobArtifactState >>> >>> Is it sufficient for me to run the user code enclosed >>> within JobArtifactSet.withActiveJobArtifactState to achieve my requirement? >>> >>> Thank you >>> >>> >>> Faiz >>> >>
