nareshbab opened a new issue, #13431: URL: https://github.com/apache/iceberg/issues/13431
### Query engine Apache Spark ### Question When running Spark Structured Streaming with Apache Iceberg on local, data written in one micro-batch is not immediately visible for subsequent queries until the next micro-batch is processed. This is only applicable in the cases where MERGE is done in the pipeline. For append, replace/overwrite scenarios data is visible as soon as a microbatch is processed. ``` **Pipeline flow:** Kafka -> Microbatch (30s) -> foreachbatch -> write to snapshot table -> merge to main table using snapshot table -> process new batch ``` ``` Environment Java: 17 Scala: 2.13.16 Spark Version: 3.5.1 Iceberg Version: 1.9.1 Catalog Type: Hadoop Catalog Storage: Local filesystem Write Mode: Structured Streaming with foreachBatch ``` **Expected Behavior** After a merge/write operation completes in a micro-batch, the data should be immediately visible to subsequent queries without waiting for the next micro-batch to trigger. Actual Behavior Data written in micro-batch N is only visible after micro-batch N+1 starts processing, causing visibility delays ranging from seconds to minutes depending on trigger interval. **Questions for Iceberg Team** - Is this expected behavior for Iceberg with Spark Structured Streaming? - What's the recommended pattern for achieving immediate visibility in streaming scenarios? - Should we always use REFRESH TABLE after write operations in streaming contexts? NOTE: REFRESH Table is also not working for this scenario - Are there Iceberg-specific configurations that can improve metadata refresh frequency for streaming workloads? - Is there a way to flush/commit transactions explicitly within a micro-batch to ensure immediate visibility? Additional Context This issue is consistently reproducible across different environments (local, Biglake) and appears to be a fundamental interaction between Spark's streaming execution model and Iceberg's transaction/metadata management. Any guidance on best practices or potential improvements would be greatly appreciated! PS: Same code base works with Bigquery MERGE scenarios and data is available immediately post MERGE statement execution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
