pepijnve commented on PR #27210: URL: https://github.com/apache/datafusion/pull/27210#issuecomment-6923562626
> Basically I am trying to figure out what the core rationale is for including time travel queries in the core. I started from what Paimon's DataFusion integration does which can be seen at https://github.com/apache/paimon-rust/blob/main/crates/integrations/datafusion/src/relation_planner.rs#L121. What's done there is that the table name is first looked up using the bare table reference in the prefetched table map. Then a downcast is done, and a new TableProvider is derived that takes the options from the TableFactor into account. This is returned as a logical scan of the bare table reference. I think this is more or less the alternative solution you're talking about. I see a couple of possible issues with that approach. - `RelationPlanner` is not async. As a consequence all metadata you might possible need must have been fetched already during prefetch and there's no clear path to optimising this. This means you pay the cost of fetching all historic metadata regardless of whether you'll actually need it or not. - The returned scan logical plan is in no way different from a scan of the same table without any or with different options. In other words, at the logical plan level `from t version as of 1` and `from t version as of 2` are identical. I'm not sure if this is currently the case, but something like `select * from t version as of 1 union select * from t version as of 2` could end up being optimised incorrectly because of this. - Very practically, time travel queries are quite common across formats (not just Iceberg). It seems practical for the query planner to know how to handle these so that each extension doesn't have to redo this. The alternative is that every extension implements their own 'versioned scan' logical plan. What I'm experimenting with here is if there's some way to add an extension point that lets us improve upon the above without restricting ourselves to just time travel. What came out of a bunch of experimenting is two needs: - During the async prefetch phase, you want some way to extract information out of the `TableFactor` which can then be passed to the `SchemaProvider` so that it knows which metadata is actually needed. It only needs to return a `TableProvider` that can read exactly what was asked for. - If I perform two different lookups like this I need some way to tell the two `TableProvider` instances apart. We could key this on the `TableReference` along with the request value (i.e. the info extracted from the `TableFactor`), but then it's no longer possible for multiple requests that resolve to the same thing to actually be identical. When using `timestamp as of` for instance it's quite possible that two distinct timestamps resolve to the same snapshot. If we want to be able to detect this situation, then we need something that identifies a particular version (or more generally aspect) of a table that may be different from the request parameters. In @alexanderbianchi's suggestion the request and the identity values use two distinct types. Currently in this PR I'm using the same type as representation for both, but the request and identity values are allowed to be different. For lack of a better name this is currently called a 'table read selector'. The 'table read selector planner' is the bit that converts `TableFactor` to something independent of the SQL AST. `SchemaProvider::table_with_read_selector` performs the lookup during prefetch. `TableProvider::read_selector` is the identity part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
