Re: Release Apache Spark 2.4.4 before 3.0.0

Hi Dongjoon, Should we also consider fixing https://issues.apache.org/jira/browse/SPARK-27812 before the cut? Best, Stavros On Mon, Jul 15, 2019 at 7:04 PM Dongjoon Hyun wrote: > Hi, Apache Spark PMC members. > > Can we cut Apache Spark 2.4.4 next Monday (22nd July)? > > Bests, > Dongjoon. >

Re: Spark on K8S - --packages not working for cluster mode?

Hi, This has been fixed here: https://github.com/apache/spark/pull/23546. Will be available with Spark 3.0.0 Best, Stavros On Wed, Jun 5, 2019 at 11:18 PM pacuna wrote: > I'm trying to run a sample code that reads a file from s3 so I need the aws > sdk and aws hadoop dependencies. > If I

Re: K8s-Spark client mode : Executor image not able to download application jar from driver

Yes here is why the initial effort didnt work, explained a bit better. As I mentioned earlier SparkContext will add your jars/files (declared with the related conf properties) to the FileServer. If it is a local to the container's fs jar (has schema local:) it will just be resolved to: file +

Re: K8s-Spark client mode : Executor image not able to download application jar from driver

Hi Nikhil, Application jar by default is added to spark.jars so it is fetched by executors when tasks are launched (behind the scenes SparkContext

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

Awesome! On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: > Indeed! > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun > wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > > On

custom sink & model transformation

Hi, Is it unsfate to do model prediction within a custom sink eg. model.transform(df)? I see that the only transformation done is adding a prediction column AFAIK. Stavros