jayshrivastava commented on issue #23817:
URL: https://github.com/apache/datafusion/issues/23817#issuecomment-5073598375
> But the largest problem I see is that the partitioning information will be
duplicated in the ExecutionPlans and in the filters
I'm not 100% sure if that's a problem since dynamic filters already store
partitioning, baked into the `CASE` statement. Storing it separately should
behave similarly. At execution time, I would expect the `DataSource` to take an
optional optimization path where, if the partitioning is equivalent, it can
optionally call a `current_partitioned` API instead of `current` like it does
today.
```rust
struct DynamicFilterPhysicalExpr {
partitioning: Partitioning,
}
impl DynamicFilterPhysicalExpr {
// Same as today. Returns a CASE expression always and can still be
used.
pub fn current(&self) -> Result<Arc<dyn PhysicalExpr>>
// Contract: `DataSource` passes its own partitioning as an argument.
Returns `Some` if the partitioning is equivalent and per-
// partition filters can be generated.
pub fn current_partitioned(&self, partitioning: Partitioning) ->
Option<Vec<Arc<dyn PhysicalExpr>>> {
if self.partitioning != partitioning {
return None;
}
...
}
}
```
The partitioning of the filter is with respect to the producer (ex. hash
join) and doesn't change, so it shouldn't be affected by any repartitions in
between the producer and consumer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
