I got an illegal argument error with 2.4.6. I then pointed my Jupiter notebook to 3.0 version and it worked as expected. Using same .ipnyb file.
I was following this machine learning example. “Your First Apache Spark ML Model” by Favio Vázquez https://towardsdatascience.com/your-first-apache-spark-ml-model-d2bb82b599dd In the example he is using version 3.0 so I assumed I got the error because I am using different version (2.4.6). On Fri, 19 Jun 2020, 08:06 Stephen Boesch, <java...@gmail.com> wrote: > afaik It has been there since Spark 2.0 in 2015. Not certain about > Spark 1.5/1.6 > > On Thu, 18 Jun 2020 at 23:56, Anwar AliKhan <anwaralikhan...@gmail.com> > wrote: > >> I first ran the command >> df.show() >> >> For sanity check of my dataFrame. >> >> I wasn't impressed with the display. >> >> I then ran >> df.toPandas() in Jupiter Notebook. >> >> Now the display is really good looking . >> >> Is toPandas() a new function which became available in Spark 3.0 ? >> >> >> >> >> >>
