etseidl commented on code in PR #7574:
URL: https://github.com/apache/arrow-rs/pull/7574#discussion_r2122362828
##########
parquet/tests/arrow_reader/mod.rs:
##########
@@ -987,6 +991,33 @@ fn create_data_batch(scenario: Scenario) ->
Vec<RecordBatch> {
make_utf8_batch(vec![Some("e"), Some("f"), Some("g"),
Some("h"), Some("i")]),
]
}
+ Scenario::TruncatedUTF8 => {
+ // Make utf8 batch with strings longer than 64 bytes
+ // to check truncation of row group statistics
+ vec![
+ make_utf8_batch(vec![
+ Some(&("a".repeat(64) + "1")),
+ Some(&("b".repeat(64) + "2")),
+ Some(&("c".repeat(64) + "3")),
Review Comment:
```suggestion
None,
```
Try throwing a null in the first batch, that seems to clear up the issue
with empty string showing up in the stats. It seems like the schema is inferred
from the first batch, so all values are not nullable. Putting a null in the
first batch fixes that.
##########
parquet/tests/arrow_reader/statistics.rs:
##########
@@ -354,7 +376,45 @@ impl Test<'_> {
//
// Remaining cases
// f64::NAN
-// - Using truncated statistics ("exact min value" and "exact max value"
https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_max_and_min_value_truncated() {
+ let reader = TestReader {
+ scenario: Scenario::TruncatedUTF8,
+ row_per_group: 5,
+ }
+ .build()
+ .await;
+
+ Test {
+ reader: &reader,
+ // min is truncated to
+ // 1. `"a".repeate(64)`, original value is `"a".repeat(64) + "1"`
+ // 2. "", since there's a null in the second row group
+ // 3. "j"
+ expected_min: Arc::new(StringArray::from(vec![&("a".repeat(64)), "",
"j"])),
Review Comment:
with the change I proposed, now the second min should be `"f".repeat(64)`.
The expected null counts and `min_value_exact` arrays below will need
modification as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
