conflicting dependencies.
El jue, 2 nov 2023 a las 5:39, Harry Jamison
() escribió:
> I am getting the error below when I try to run a spark job connecting to
> phoneix. It seems like I have the incorrect scala version that some part of
> the code is expecting.
>
> I am us
I am getting the error below when I try to run a spark job connecting to
phoneix. It seems like I have the incorrect scala version that some part of
the code is expecting.
I am using spark 3.5.0, and I have copied these phoenix jars into the spark lib
phoenix-server-hbase-2.5-5.1.3.jar
rc.zip:/usr/local/lib/python3.7/dist-packages/pyspark/python/::/yarn/nm/usercache/spark/appcache/application_1693107150164_0198/container_e03_1693107150164_0198_01_03/__pyfiles__:/yarn/nm/usercache/spark/appcache/application_1693107150164_0198/container_e03_1693107150164_0198_01_03/pyspark.zip:/yarn/nm/usercache/spark/appcac
I am using python3.7 and spark 2.4.7
I am trying to figure out why my job is using the wrong python version
This is how it is starting up the logs confirm that I am using python 3.7But I
later see the error message showing it is trying to us 3.8, and I am not sure
where it is picking that up.
roduce issues
where many partitions are trying to invoke the api.
I found this medium-article which discusses the issue you are facing, but does
not discuss a solution for the same. Do check the comments also
Regards,Varun
On Sat, Aug 26, 2023 at 10:32 AM Harry Jamison
wrote:
I am using
I am using python 3.7 and Spark 2.4.7
I am not sure what the best way to do this is.
I have a dataframe with a url in one of the columns, and I want to download the
contents of that url and put it in a new column.
Can someone point me in the right direction on how to do this?I looked at the
UDFs
