Re: Classpath isolation per SparkSession without Spark Connect

the plugin > system on the driver side of spark but considering spark is distributed, > the executor side of spark needs to adapt to the pf4j framework I believe > too > > Thanks > Faiz > > On Tue, Nov 28, 2023, 16:57 Pasha Finkelshtein < > pavel.finkelsht...@gmail

Re: Classpath isolation per SparkSession without Spark Connect

To me it seems like it's the best possible use case for PF4J. [image: facebook] [image: twitter] [image: linkedin] [image: instagram] Pasha Finkelshteyn Developer Advocate

Re: Data ingestion

But not in streaming, right? It will be a usual batch approach, but initial question was about streaming. [image: facebook] [image: twitter] [image: linkedin] [image: instagram]

Re: Data ingestion

Hello Spark does not have any built-in solution for this problem. Most probably you will want to use Debezium+Kafka and read with Spark from Kafka [image: facebook] [image: twitter] [image: linkedin]

Re: What does Apache Spark do?

Hi Mr. Turritopsis Dohrnii Teo En Ming, Spark can perform variety of different tasks, do the most important thing you should know about it is that it's a distributed computation framework. Usually it's used for ETL (extract-transform-load) Pipelines, but also there is a plethora of extensions,