nareshbab commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3026603432
Update: Figured out the behaviour happening here. Spark is committing the
changes after the end of a batch. So during the execution of a specific batch,
though data is written in temp table its reflecting only after batch end. This
is leading to scenario where merge is reading empty stale or empty data from
temp table and hence the data is not reflecting to the snapshot table.
Workaround: Explicitly commit the write for temp table post writing.
Modified the below method to make it work
```
def writeAuditTable(batchDF: Dataset[Row], batchID: Long)(implicit spark:
SparkSession): Unit = {
logInfo(s"Writing audit table for batch $batchID")
ensureTableExists()
try {
// Write the batch to the temp audit table
batchDF
.writeTo(s"$catalogName.$dbName.$tempTableName")
.replace()
// Explicitly refresh the temp table to commit the data
spark.catalog.refreshTable(s"$catalogName.$dbName.$tempTableName")
logInfo(s"Successfully wrote audit table for batch $batchID")
} catch {
case e: Exception =>
logError(s"Failed to write audit table for batch $batchID", e)
throw e
}
```
Thank you for the support here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
