Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24950 )
Change subject: IMPALA-15411: Add is_valid_utf8() builtin ...................................................................... IMPALA-15411: Add is_valid_utf8() builtin IMPALA-12675 turned on UTF-8 validation on the Parquet write path, so an INSERT or CTAS writing non-UTF-8 bytes into an annotated STRING column fails with PARQUET_INVALID_UTF8_STRING. The error points at one byte, but there was no way to find which rows are affected. is_valid_utf8(STRING) -> BOOLEAN closes that gap: SELECT * FROM t WHERE NOT is_valid_utf8(s); It calls impala::IsValidUtf8(), the same simdutf-based primitive the Parquet writer uses, so it accepts exactly what the write path accepts. It validates bytes, not characters, so the result does not depend on UTF8_MODE. NULL returns NULL, the empty string is valid. The error message and the incompatible-changes note now name it. Testing: - extended Utf8Test - added is_valid_utf8() call and error message check to e2e test Change-Id: Ia1c8b537bdfc7c35839d53eb43fbe841c71c4a51 Assisted-by: Claude Opus 5 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24950 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/string-functions.h M common/function-registry/impala_functions.py M common/thrift/generate_error_codes.py M docs/topics/impala_incompatible_changes.xml M docs/topics/impala_string_functions.xml M tests/query_test/test_scanners.py 8 files changed, 72 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia1c8b537bdfc7c35839d53eb43fbe841c71c4a51 Gerrit-Change-Number: 24950 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
