LiaCastaneda opened a new issue, #17523:
URL: https://github.com/apache/datafusion/issues/17523
### Is your feature request related to a problem or challenge?
Currently, DataFusion only supports bounds dynamic filters, which work well
when the data sources have metadata or are sorted (e.g., Parquet). However, for
unsorted data sources, we would benefit from supporting list-based dynamic
filters.
The idea is to take the build side values, collect them into a List, and
then convert that list into a predicate (similar to an IN clause).
### Describe the solution you'd like
For example if we have the following query:
```
SELECT *
FROM fact_table f
JOIN dim_table d
ON f.key = d.key
WHERE d.region = 'US';
```
With list dynamic filter, we could collect the f.key values into a List and
push down a predicate like:
`d.key IN (1, 42, 77, 103, ...)`
into the probe side.
I understand we should put a limit on how big this List can get, maybe make
this configurable by the users though an option and use a safe default limit.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
