OussamaSaoudi commented on issue #25775:
URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5940260679
Hey folks, chiming in here from a delta-kernel-rs viewpoint. We've also
found that this operator (that we call DynamicScan) is extremely useful in any
table format style workload.
The idea, as Ion suggests is to have a node that computes the list of files
to read using a subquery. This is in contrast to a normal scan that has a fixed
set of files at planning time. IE the files to read are computed at runtime
instead of planning time. This has the advantage that the optimizer/runtime is
aware of the full nature of a table format query, and can apply optimizations
and scheduling decisions that span both metadata and data. Things like
partitioning, pruning, and so on.
Here's a handy example from our [doc
comments](https://github.com/delta-io/delta-kernel-rs/blob/main/kernel/src/plans/ir/nodes.rs#L415)
that showcases what the definition could look like:
```
/// ```text
/// upstream (metadata)
/// path | size | filemod | version | dv
/// -----------------+------+---------+---------+------
/// part-0.parquet | 1024 | 100000 | 7 | NULL
/// part-1.parquet | 2048 | 200000 | 8 | NULL
/// ```
/// ```text
/// DynamicScan {
/// schema: { id: int, name: string, version: long },
/// file_type: Parquet,
/// base_url: "s3://table/",
/// file_constant_columns: ["version"],
/// path_column: "path",
/// file_size_column: "size",
/// last_modified_column: "filemod",
/// dv_column: "dv",
/// }
/// ```
/// The engine opens `s3://table/part-0.parquet` and
`s3://table/part-1.parquet`, reads
/// `{id, name}` from each, sees a NULL DV for each file so all rows
survive, and
/// broadcasts the row's `version` onto every emitted file row. One possible
output
/// (row order is not guaranteed):
/// ```text
/// | id | name | version
/// +----+------+--------
/// | 3 | c | 8
/// | 2 | b | 7
/// | 4 | d | 8
/// | 1 | a | 7
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
