alamb commented on code in PR #18820:
URL: https://github.com/apache/datafusion/pull/18820#discussion_r2549988898
##########
datafusion/common/src/config.rs:
##########
@@ -699,6 +699,12 @@ config_namespace! {
/// the filters are applied in the same order as written in the query
pub reorder_filters: bool, default = false
+ /// (reading) Force the use of RowSelections for filter results, when
Review Comment:
This is an escape valve if we find some issue when using the new adaptive
filter from @hhhizzz
- https://github.com/apache/arrow-rs/pull/8733
##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3106,7 +3106,7 @@ mod tests {
assert_contains!(
&e,
- r#"Error during planning: Can not find compatible types to compare
Boolean with [Struct("foo": Boolean), Utf8]"#
+ r#"Error during planning: Can not find compatible types to compare
Boolean with [Struct("foo": non-null Boolean), Utf8]"#
Review Comment:
these are due the changes from
https://github.com/apache/arrow-rs/issues/8648 to clean up datatype display. It
is a nice improvement in my mind
##########
datafusion/core/tests/parquet/filter_pushdown.rs:
##########
@@ -636,6 +636,27 @@ async fn predicate_cache_pushdown_default() ->
datafusion_common::Result<()> {
config.options_mut().execution.parquet.pushdown_filters = true;
let ctx = SessionContext::new_with_config(config);
// The cache is on by default, and used when filter pushdown is enabled
+ PredicateCacheTest {
+ expected_inner_records: 8,
+ expected_records: 7, // reads more than necessary from the cache as
then another bitmap is applied
Review Comment:
this behavior changed due to adaptive filtering. I added a new test that
turns off adaptive filtering to show doing so restores the old behavior
##########
datafusion-cli/src/main.rs:
##########
@@ -592,9 +592,9 @@ mod tests {
+-----------------------------------+-----------------+---------------------+------+------------------+
| filename | file_size_bytes |
metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
- | alltypes_plain.parquet | 1851 | 6957
| 2 | page_index=false |
- | alltypes_tiny_pages.parquet | 454233 | 267014
| 2 | page_index=true |
- | lz4_raw_compressed_larger.parquet | 380836 | 996
| 2 | page_index=false |
+ | alltypes_plain.parquet | 1851 | 8882
| 2 | page_index=false |
Review Comment:
Update: the size is correct. As @etseidl says "the truth hurts"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
