Unsubscribe

-- With Best Regards, Dipayan Dev Author of *Deep Learning with Hadoop * M.Tech (AI), IISc, Bangalore

Re: error trying to save to database (Phoenix)

Sorry for being so Dense and thank you for your help. I was using this version phoenix-spark-5.0.0-HBase-2.0.jar Because it was the latest in this repo https://mvnrepository.com/artifact/org.apache.phoenix/phoenix-spark On Mon, Aug 21, 2023 at 5:07 PM Sean Owen wrote: > It is. But you have a

Re: error trying to save to database (Phoenix)

It is. But you have a third party library in here which seems to require a different version. On Mon, Aug 21, 2023, 7:04 PM Kal Stevens wrote: > OK, it was my impression that scala was packaged with Spark to avoid a > mismatch > https://spark.apache.org/downloads.html > > It looks like spark

Re: error trying to save to database (Phoenix)

OK, it was my impression that scala was packaged with Spark to avoid a mismatch https://spark.apache.org/downloads.html It looks like spark 3.4.1 (my version) uses scala Scala 2.12 How do I specify the scala version? On Mon, Aug 21, 2023 at 4:47 PM Sean Owen wrote: > That's a mismatch in the

Re: error trying to save to database (Phoenix)

That's a mismatch in the version of scala that your library uses vs spark uses. On Mon, Aug 21, 2023, 6:46 PM Kal Stevens wrote: > I am having a hard time figuring out what I am doing wrong here. > I am not sure if I have an incompatible version of something installed or > something else. > I

error trying to save to database (Phoenix)

I am having a hard time figuring out what I am doing wrong here. I am not sure if I have an incompatible version of something installed or something else. I can not find anything relevant in google to figure out what I am doing wrong I am using *spark 3.4.1*, and *python3.10* This is my code to

DataFrame cache keeps growing

Hi Team, While trying to understand/looking out for a problem of "where size of dataframe keeps growing" , I realized that a similar question was asked a couple of years ago. Need your help in resolving this.

Re: Problem with spark 3.4.1 not finding spark java classes

In yours file /home/spark/real-estate/pullhttp/pull_apartments.py replace import org.apache.spark.SparkContext with from pyspark import SparkContext man. 21. aug. 2023 kl. 15:13 skrev Kal Stevens : > I am getting a class not found error > import org.apache.spark.SparkContext > > It sounds

Problem with spark 3.4.1 not finding spark java classes

I am getting a class not found error import org.apache.spark.SparkContext It sounds like this is because pyspark is not installed, but as far as I can tell it is. Pyspark is installed in the correct python verison root@namenode:/home/spark/# pip3.10 install pyspark Requirement already

Spark doesn’t create SUCCESS file when external path is passed

Hi Team, I need some help and if someone can replicate the issue at their end, or let me know if I am doing anything wrong. https://issues.apache.org/jira/browse/SPARK-44884 We have recently upgraded to Spark 3.3.0 in our Production Dataproc. We have a lot of downstream application that relies

Unsubscribe

Re: k8s+ YARN Spark

Interesting. Spark supports the following cluster managers - Standalone: A cluster-manager, limited in features, shipped with Spark. - Apache Hadoop YARN is the most widely used resource manager not just for Spark but for other artefacts as well. On-premise YARN is used extensively.

Re: Problem with spark 3.4.1 not finding spark java classes

This should work check your path. It should pyspark from which pyspark /opt/spark/bin/pyspark And your installation should contain cd $SPARK_HOME /opt/spark> ls LICENSE NOTICE R README.md RELEASE bin conf data examples jars kubernetes licenses logs python sbin yarn You should

Re: Problem with spark 3.4.1 not finding spark java classes

Nevermind I was doing something dumb On Sun, Aug 20, 2023 at 9:53 PM Kal Stevens wrote: > Are there installation instructions for Spark 3.4.1? > > I defined SPARK_HOME as it describes here > > https://spark.apache.org/docs/latest/api/python/getting_started/install.html > > ls

k8s+ YARN Spark

Good afternoon. Perhaps you will be discouraged by what I will write below, but nevertheless, I ask for help in solving my problem. Perhaps the architecture of our solution will not seem correct to you. There are backend services that communicate with a service that implements spark-driver.