davkim opened a new issue, #17338: URL: https://github.com/apache/iceberg/issues/17338
### Apache Iceberg version 1.10.0 ### Query engine Spark ### Please describe the bug 🐞 Reopening this closed issue https://github.com/apache/iceberg/issues/10493 CachingCatalog is susceptible to a race condition because it uses Cache.put. Cache.put inherently races with Cache.invalidate / Cache.invalidateAll thread A: check cache - empty thread A: compute new value for the cache (not yet stale) thread B: change external state (value computed by A becomes stale at this point thread B: invalidate() or invalidateAll() thread A: put into the cache (the put value is stale) stale value remains visible in the cache until cache expiration This realizes when enabling OpenLineage which calls `loadTable()` in a background thread and when that happens while write is in progress, the writer thread reading back the same table it wrote to ends up reading stale cache reference from the loadTable() call -- this has caused a major production incident on our end. I believe the solution to this is to expire cache on commit - this should be the correct behavior, the table state has changed with the commit so any cache references to this table should be expired. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
