MrDerecho opened a new issue, #586: URL: https://github.com/apache/incubator-xtable/issues/586
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 Description I am encountering an issue while using xtable to perform updates from Iceberg to Delta tables. Here is the observed behavior: Snapshot 0: The metadata between the Iceberg and Delta tables reconcile as expected. Snapshot 1: Erroneous metadata is generated that includes "add" and "remove" actions that did not actually occur. This results in a lowered row count in the Delta metadata compared to the source Iceberg table. Snapshot 2: The metadata appears to reconcile again and reflects the updated values accurately. Snapshot 3: The issue is recreated with similar discrepancies in the metadata. Additional Context: This behavior has been observed consistently in a sample of 30 tables. The issue is in the largest of these tables with around 7 million files and 7.3 trillion records. This table object is "append-only", the disappeared or removed files on snapshot 1 are re-added in snapshot 2. The issue seems cyclical, occurring every alternate snapshot. The only error/info found in the logs is: "incremental sync is not safe from instant falling back to snapshot sync" and "truncated the string representation of a plan since it was too large" Steps to Reproduce: Use xtable to perform updates from Iceberg to Delta tables. Observe metadata reconciliation across snapshots. Expected Behavior: The metadata between the Iceberg and Delta tables should reconcile accurately across all snapshots, without erroneous "add" or "remove" actions. Actual Behavior: Alternate snapshots (e.g., snapshots 1 and 3) generate erroneous metadata with inaccurate "add" and "remove" actions, leading to a mismatch in row counts. Environment Tool: xtable Source: Apache Iceberg Destination: Delta Lake Additional Notes The issue might be related to how snapshots are processed or metadata is generated. ### Are you willing to submit PR? - [ ] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
