haohuaijin opened a new issue, #25226:
URL: https://github.com/apache/datafusion/issues/25226
### Describe the bug
When a nonempty row group lacks min/max statistics, DataFusion can still
report another group's bounds as exact for the entire file. MIN/MAX queries can
then return incorrect results.
### To Reproduce
Generate the file with Rust (`parquet` 59.3, re-exported by `datafusion`,
and `tempfile`). Enable statistics for the first row group and disable them for
the second using the public writer APIs. Both groups contain non-null strings.
This explicitly creates missing statistics; long strings alone do not cause
parquet-rs to omit bounds.
```rust
use std::{fs::File, sync::Arc};
fn main() {
use datafusion::parquet::column::writer::ColumnWriterImpl;
use datafusion::parquet::data_type::{ByteArray, ByteArrayType};
use datafusion::parquet::file::properties::{EnabledStatistics,
WriterProperties};
use datafusion::parquet::file::writer::{
SerializedFileWriter, SerializedPageWriter, TrackedWrite,
};
use datafusion::parquet::schema::parser::parse_message_type;
let dir = tempfile::tempdir().unwrap();
let path = "/tmp/parquet_missing_bounds.parquet";
let column_path = dir.path().join("column.pages");
let schema = Arc::new(
parse_message_type("message schema { REQUIRED BINARY a (UTF8);
}").unwrap(),
);
let mut writer = SerializedFileWriter::new(
File::create(path).unwrap(),
schema,
Arc::new(WriterProperties::default()),
)
.unwrap();
let long_value = "z".repeat(8192);
for (values, statistics) in [
(["a", "b"], EnabledStatistics::Chunk),
(
[long_value.as_str(), long_value.as_str()],
EnabledStatistics::None,
),
] {
let properties = Arc::new(
WriterProperties::builder()
.set_statistics_enabled(statistics)
.build(),
);
let mut buffer =
TrackedWrite::new(File::create(&column_path).unwrap());
let mut column = ColumnWriterImpl::<ByteArrayType>::new(
writer.schema_descr().column(0),
properties,
Box::new(SerializedPageWriter::new(&mut buffer)),
);
let values = values.map(ByteArray::from);
column.write_batch(&values, None, None).unwrap();
let result = column.close().unwrap();
assert_eq!(
result.metadata.statistics().is_some(),
statistics == EnabledStatistics::Chunk
);
drop(buffer);
let mut group = writer.next_row_group().unwrap();
group
.append_column(&File::open(&column_path).unwrap(), result)
.unwrap();
group.close().unwrap();
}
writer.close().unwrap();
}
```
Query the generated file in DataFusion:
```sql
CREATE EXTERNAL TABLE t STORED AS PARQUET
LOCATION '/tmp/parquet_missing_bounds.parquet';
SELECT LENGTH(MIN(a)), LENGTH(MAX(a)) FROM t;
```
Before the fix: `1, 1`. Expected: `1, 8192`. The maximum is in the second
row group, whose missing bound must not be ignored.
### Expected behavior
A missing bound in a potentially nonempty, non-null row group must prevent
reporting that file bound as exact.
### Additional context
find this when work on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
