Hey Chris and Sean, thanks for taking the time to answer. Perhaps my installation of pyspark is off, although I did use version 2.4.4 When developing in scala and pyspark how do you setup your environment?
I used sbt for scala spark libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.4.4", "org.apache.spark" %% "spark-sql" % "2.4.4", "org.scalactic" %% "scalactic" % "3.1.2", "org.scalatest" %% "scalatest" % "3.1.2" % "test", "org.apache.spark" %% "spark-mllib" % "2.4.4", "org.plotly-scala" %% "plotly-render" % "0.7.2", "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly() ) and pip for pyspark (python 3.6.5) pip3 install pyspark==2.4.4
