The only change I am making is spark directory name.
It keeps failing in this same cell. df.toPandas()
findspark.init('/home/spark-2.4.6-bin-hadoop2.7') FAIL
findspark.init('/home/spark-3.0.0-bin-hadoop2.7'). PASS
On Sun, 21 Jun 2020, 19:51 randy clinton, wrote:
> You can see from the
That part isn't related to Spark. It means you have some code compiled for
Java 11, but are running Java 8.
On Sun, Jun 21, 2020 at 1:51 PM randy clinton
wrote:
> You can see from the GitHub history for "toPandas()" that the function has
> been in the code for 5 years.
>
>
You can see from the GitHub history for "toPandas()" that the function has
been in the code for 5 years.
https://github.com/apache/spark/blame/a075cd5b700f88ef447b559c6411518136558d78/python/pyspark/sql/dataframe.py#L923
When I google IllegalArgumentException: 'Unsupported class file major
Two versions of Spark running against same code
https://towardsdatascience.com/your-first-apache-spark-ml-model-d2bb82b599dd
version spark-2.4.6-bin-hadoop2.7 is producing error for toPandas(). See
error stack below
Jupyter Notebook
import findspark
I got an illegal argument error with 2.4.6.
I then pointed my Jupiter notebook to 3.0 version and it worked as
expected.
Using same .ipnyb file.
I was following this machine learning example.
“Your First Apache Spark ML Model” by Favio Vázquez
afaik It has been there since Spark 2.0 in 2015. Not certain about Spark
1.5/1.6
On Thu, 18 Jun 2020 at 23:56, Anwar AliKhan
wrote:
> I first ran the command
> df.show()
>
> For sanity check of my dataFrame.
>
> I wasn't impressed with the display.
>
> I then ran
> df.toPandas() in Jupiter
I first ran the command
df.show()
For sanity check of my dataFrame.
I wasn't impressed with the display.
I then ran
df.toPandas() in Jupiter Notebook.
Now the display is really good looking .
Is toPandas() a new function which became available in Spark 3.0 ?
