[ https://issues.apache.org/jira/browse/SPARK-38976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
wesharn updated SPARK-38976: ---------------------------- Description: It occured duplicate records when spark-sql overwrite hive table . when spark job has failure stages,but dateframe has no duplicate records? when I run the job again, the reasult is correct.It confused me.why? eg: dataFrame.write().mode(SaveMode.Overwrite).insertInto("outputTable"); no duplicate records in dataFrame, but duplicate records existed in hive outputTable was:It occured duplicate records when spark-sql overwrite hive table . when spark job has failure stages,but dateframe has any duplicate id? when I run the job again, the reasult is correct.It confused me.why? > spark-sql. overwrite. hive table-duplicate records > -------------------------------------------------- > > Key: SPARK-38976 > URL: https://issues.apache.org/jira/browse/SPARK-38976 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.2.1 > Reporter: wesharn > Priority: Major > > It occured duplicate records when spark-sql overwrite hive table . when spark > job has failure stages,but dateframe has no duplicate records? when I run the > job again, the reasult is correct.It confused me.why? > eg: dataFrame.write().mode(SaveMode.Overwrite).insertInto("outputTable"); > no duplicate records in dataFrame, but duplicate records existed in hive > outputTable -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org