Gautam Solanki created SPARK-17493:
--------------------------------------
Summary: Spark Job hangs while DataFrame writing to HDFS path with
parquet mode
Key: SPARK-17493
URL: https://issues.apache.org/jira/browse/SPARK-17493
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 2.0.0
Environment: AWS Cluster
Reporter: Gautam Solanki
While saving a RDD to HDFS path in parquet format with the following
rddout.write.partitionBy("event_date").mode(org.apache.spark.sql.SaveMode.Append).parquet("hdfs:////tmp//rddout_parquet_full_hdfs1//")
, the spark job was hanging as the two write tasks with Shuffle Read of size 0
could not complete. But, the executors notified the driver about the completion
of these two tasks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
