I'm running a Jupyter-Spark setup and I want to benchmark my cluster with
different input parameters. To make sure the enivorment stays the same I'm
trying to reset(restart) the SparkContext, here is some code:
* temp_result_parquet = os.path.normpath('/home/spark_tmp_parquet') i
= 0 while i < max_i: i += 1 if
os.path.exists(temp_result_parquet):
shutil.rmtree(temp_result_parquet) # I know I could simply overwrite the
parquet My_DF = do_something(i)
My_DF.write.parquet(temp_result_parquet) sc.stop()
time.sleep(10) sc = pyspark.SparkContext(master='spark://ip:here',
appName='PySparkShell')*
when I do this in the first iteration it runs fine but in the second I get
the following error:
* Py4JJavaError: An error occurred while calling o1876.parquet. :
org.apache.spark.SparkException: Job aborted. [...] Caused by:
java.lang.IllegalStateException: SparkContext has been shutdown at
org.apache.spark.SparkContext.runJob(SparkContext.scala:2014) at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:188)*
I tried running the code without the SparkContext restart but this results
in memory issues. So to wipe the slate clean before every iteration I'm
trying this. With the weird result that parquet "thinks" SparkContext is
down.
