NoahKusaba commented on code in PR #2784:
URL:
https://github.com/apache/datafusion-ballista/pull/2784#discussion_r4648079387
##########
integrations/iceberg/src/catalog.rs:
##########
@@ -109,18 +125,20 @@ impl SchemaProvider for PinningSchema {
let Some(table) = self.0.table(name).await? else {
return Ok(None);
};
- let Some(provider) =
table.downcast_ref::<IcebergCatalogTableProvider>() else {
- return Ok(Some(table));
+ if let Some(provider) =
table.downcast_ref::<IcebergCatalogTableProvider>() {
+ let table = load(provider).await;
Review Comment:
Agreed, that's the common case. #2885 wouldn't reach it yet, though.
ballista-iceberg pins ballista-core at a DataFusion 55 commit until
datafusion-iceberg supports 56, and the composed codec would still flatten the
cause.
So register_iceberg now adds an analyzer rule, RequireLoadedTables. It fails
a query that reads a table its catalog couldn't load, with the load error,
while the query is planned. Direct reads, view reads and subqueries, with or
without a filter, now all fail before anything is encoded:
require_loaded_iceberg_tables
caused by
IcebergError(TableNotFound)
caused by
Execution error: TableNotFound => No such table: …
DataFusion doesn't analyze DDL or DESCRIBE, so dropping, replacing and
describing still work while a table is gone, and #2882 stays fixed.
The rule is a stopgap with a TODO to remove it once #2881 lands, since a
table that can't be loaded will then fail the query where it's loaded.
should_fail_to_read_a_table_that_is_gone now reads t directly, through a view
and in a subquery, each with and without a filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]