Hi All,
Spark 1.5.2 does not seem to be backward compatible with functionality that
was available in earlier versions, at least in 1.3.1 and 1.4.1. It is not
possible to insert overwrite into an existing table that was read as a
DataFrame initially.
Our existing code base has few internal Hive tables being overwritten after
some join operations.
For e.g.
val PRODUCT_TABLE = "product_dim"
val productDimDF = hiveContext.table(PRODUCT_TABLE)
// Joins, filters ...
productDimDF.write.mode(SaveMode.Overwrite).insertInto(PRODUCT_TABLE)
This results in the exception -
org.apache.spark.sql.AnalysisException: Cannot overwrite table
`product_dim` that is also being read from.;
at
org.apache.spark.sql.execution.datasources.PreWriteCheck.failAnalysis(rules.scala:82)
at
org.apache.spark.sql.execution.datasources.PreWriteCheck$$anonfun$apply$2.apply(rules.scala:155)
at
org.apache.spark.sql.execution.datasources.PreWriteCheck$$anonfun$apply$2.apply(rules.scala:85)
Is there any configuration to disable this particular rule? Any pointers to
solve this would be very helpful.
Thanks,
Ram
