Daniel Vanko has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/25008 )
Change subject: IMPALA-15413: Truncate Parquet page index min/max on UTF-8 boundaries ...................................................................... IMPALA-15413: Truncate Parquet page index min/max on UTF-8 boundaries TruncateDown()/TruncateUp() cut the per-page min/max string statistics at an arbitrary byte offset, splitting multi-byte UTF-8 sequences. Since IMPALA-12675 annotates STRING columns as UTF-8 by default, that writes a value that is not valid UTF-8 into a column the file declares as UTF8, which the Parquet spec does not allow. Both functions now leave the value on a character boundary when the truncated prefix is valid UTF-8, and keep the previous byte-wise behaviour otherwise, like parquet-java's BinaryTruncator. The boundary comes from simdutf's trim_partial_utf8(), wrapped as TrimPartialUtf8(). TruncateUp() raises a single byte in place instead of re-encoding an incremented code point, so the result never outgrows the length limit, and revalidating the whole string after each attempt rules out surrogates, overlongs and code points above U+10FFFF. This also fixes TruncateDown() returning the whole string for inputs longer than INT32_MAX, where the old cast of the length to int32_t went negative. Testing: - extended TruncateDownTest and TruncateUpTest - added test_utf8_string_values, which fails without the fix - test_parquet_page_index.py and string-util-test pass Change-Id: Ib50b12a62cfad863825b1994b95a339ef6e5b788 Assisted-by: Claude Opus 5 (Claude Code) --- M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h M be/src/util/utf8-util.cc M be/src/util/utf8-util.h M tests/query_test/test_parquet_page_index.py 6 files changed, 139 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/25008/2 -- To view, visit http://gerrit.cloudera.org:8080/25008 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib50b12a62cfad863825b1994b95a339ef6e5b788 Gerrit-Change-Number: 25008 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Vanko <[email protected]>
