Thanks Wenchen. SOLVED! KINDA!
I removed all dependencies from the pom.xml in my IDE so I wouldn't be
picking up any libraries from maven repository.
I *instead* included the libraries (jar) from the *spark download* of
*spark-3.0.0-preview2-bin-hadoop2.7*
This way I am using the *same librarie
Running Spark application with an IDE is not officially supported. It may
work under some cases but there is no guarantee at all. The official way is
to run interactive queries with spark-shell or package your application to
a jar and use spark-submit.
On Thu, Mar 26, 2020 at 4:12 PM Zahid Rahman
Hi,
When I run the code for a user defined data type dataset using case class
in scala and run the code in the interactive spark-shell against parquet
file. The results are as expected.
However I then the same code programmatically in IntelliJ IDE then spark is
give a file corruption error.
Step
