Unsubscribe

Re: Spark Pair RDD write to Hive

2nd try From: Anil Dasari Date: Sunday, September 5, 2021 at 10:42 AM To: "user@spark.apache.org" Subject: Spark Pair RDD write to Hive Hello, I have a use case where users of group id are persisted to hive table. // pseudo code looks like below usersRDD = sc.parallelize(..) usersPairRDD =

Re: Spark Stream on Kubernetes Cannot Set up JavaSparkContext

Stelios Philippou 16:20 (3 minutes ago) to Mich My local Spark submit : ~/development/SimpleKafkaStream spark-submit --version Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.1.2 /_/ Using Scala

Re: Spark Stream on Kubernetes Cannot Set up JavaSparkContext

1. which version of Spark the docker is built for 2. Which version of spark-submit you are using to submit this job view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss,

Re: Spark Stream on Kubernetes Cannot Set up JavaSparkContext

Yes on Local mode both from intelli and using spark-submit on my machine and on a windows machine work. I have noticed the following error when adding this in the above spark-submit for k8 --packages org.apache.spark:spark-streaming-kafka-0-10_2.12:3.1.2 \ :: resolving dependencies ::

Re: Spark Stream on Kubernetes Cannot Set up JavaSparkContext

Hi, Have you tried this on local mode as opposed to Kubernetes to see if it works? HTH view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or

Re: Get application metric from Spark job

Hi Akshay, Thank you for your reply. Sounds like a good idea, but I unfortunately have a 2.6 cluster. Do you know if there would be another solution that would run on 2.6 or if I have no other choice than migrating to 3? Regards, Aurélien Le jeu. 2 sept. 2021 à 20:12, Haryani, Akshay a écrit

Re: Spark Stream on Kubernetes Cannot Set up JavaSparkContext

Hello Jacek, Yes this is a spark-streaming. I have removed all code and created a new project with just the base code that is enough to open a stream and loop over it to see what i am doing wrong. Not adding the packages would result me in the following error 21/09/06 08:10:41 WARN

Re: Appending a static dataframe to a stream create Parquet file fails

I'd recommend getting in touch with Delta Lake community (Google Groups) https://groups.google.com/forum/#!forum/delta-users to get more feedback from experts about Delta Lake specific issues. On Mon, Sep 6, 2021 at 1:56 AM wrote: > Hi Jungtaek, > thanks for your reply. I was afraid that the