[ https://issues.apache.org/jira/browse/SPARK-24194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469825#comment-16469825 ]
Apache Spark commented on SPARK-24194: -------------------------------------- User 'zheh12' has created a pull request for this issue: https://github.com/apache/spark/pull/21286 > HadoopFsRelation cannot overwrite a path that is also being read from > --------------------------------------------------------------------- > > Key: SPARK-24194 > URL: https://issues.apache.org/jira/browse/SPARK-24194 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.4.0 > Environment: spark master > Reporter: yangz > Priority: Major > Labels: pull-request-available > Fix For: 2.4.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > When > {code:java} > INSERT OVERWRITE TABLE territory_count_compare select * from > territory_count_compare where shop_count!=real_shop_count > {code} > And territory_count_compare is a table with parquet, there will be a error > Cannot overwrite a path that is also being read from > > And in file MetastoreDataSourceSuite.scala, there have a test case > > > {code:java} > table(tableName).write.mode(SaveMode.Overwrite).insertInto(tableName) > {code} > > But when the table territory_count_compare is a common hive table, there is > no error. > So I think the reason is when insert overwrite into hadoopfs relation with > static partition, it first delete the partition in the output. But it should > be the time when the job commited. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org