RussellSpitzer commented on PR #17388: URL: https://github.com/apache/iceberg/pull/17388#issuecomment-5112087941
So the underlying issue here is that the CachingCatalog isn't actually racing, we are intentionally serving bounded-stale results in order to avoid a serializability problem. The basic problem is this: - I have a query that refers to table "X" several times. - Spark can load "X" separately for different references to it (self-joins, CTEs reused across executions, a lazily-resolved plan re-touched at execution time). - If "X" is at a different state when a subsequent load comes in, you essentially get two different "X" tables in the same query plan. - This leads to a broken plan that produces very wrong results. So the CachingCatalog exists (partly) to avoid this scenario. By guaranteeing that for a certain amount of time (the cache TTL) we ignore changes made by other threads, we avoid ending up with the same table read at multiple points in time within a single query. The guarantee here is bounded staleness, not freshness, and that's a deliberate trade, not a bug. This is a structural problem in Spark that folks are still trying to fix even now. Because of this, I don't think we can "fix" this the way the PR proposes. Evicting or reconciling the entry on commit reintroduces exactly the two-snapshots-in-one-plan failure the cache exists to suppress; it isn't a narrower or safer variant of invalidate-on-commit. (Retaining the committing instance doesn't help here either, since callers re-read currentSnapshot() on every load.) For users who understand the risk and want always-fresh reads, they can disable the cache (cache-enabled=false) and have the catalog reload on every access. Notably, the Iceberg source itself and the incremental-query tests already set cache-enabled=false for this reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
