16pierre commented on issue #18242: URL: https://github.com/apache/datafusion/issues/18242#issuecomment-3915285585
Even when matching error pattern [object_store::Error::NotFound](https://docs.rs/object_store/latest/object_store/enum.Error.html#variant.NotFound%5D), silent data regressions are still possible - imagine an object-store implementation with relaxed data consistency guarantees, that can return transient `NotFound` errors for files recently written. From a Datafusion client perspective, it feels reasonable to expect a failfast by default rather than a data-regression in such cases - that was our understanding of the `listing_table` implementation before https://github.com/apache/datafusion/pull/17364. I wonder if there are ways to tweak the implementation of the quality-of-life PR https://github.com/apache/datafusion/pull/17364 so that it doesn't risk major silent data regressions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
