Subject: Re: py4j.protocol.Py4JJavaError: An error occurred while calling
o794.parquet
Caused by: org.apache.spark.SparkException: Task not serializable
That's the answer :)
What are you trying to save? Is it empty or None / null?
On Wed, Jan 10, 2018 at 4:58 PM, Liana Napalkova
<liana.napalk
py4j.protocol.Py4JJavaError: An error occurred while calling o794.parquet.
: org.apache.spark.SparkException: Job aborted.
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:215)
at
org.apache.spark.sql.execution.datasourc
following problem in PySpark? (Python 2.7.12):
>
> df.show() # works fine and shows the first 5 rows of DataFrame
>
> df.write.parquet(outputPath + '/data.parquet', mode="overwrite") #
> throws the error
>
> The last line throws the following error:
>
> py4j.
wing error:
py4j.protocol.Py4JJavaError: An error occurred while calling o794.parquet.
: org.apache.spark.SparkException: Job aborted.
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter
