hcrosse opened a new pull request, #1468: URL: https://github.com/apache/arrow-go/pull/1468
### Rationale for this change `starts_with` returns `arrow.ErrNotImplemented` when `MatchSubstringOptions.IgnoreCase` is set. Substrait plans with `CASE_INSENSITIVE` fail for the same reason. I added support for both, following Arrow C++. ### What changes are included in this PR? `string` and `large_string` use Unicode simple folding, without full folding or Turkish special casing. Input is decoded character by character, so folds with different UTF-8 lengths match, such as the Kelvin sign and `k`. An invalid input byte never matches a pattern character. With `IgnoreCase` on string input, an invalid UTF-8 pattern returns `arrow.ErrInvalid`. `binary` and `large_binary` fold ASCII letters only. The issue proposed Latin-1 folds for binary input, but C++ folds only ASCII letters there. In pyarrow `26.0.0`, `b"\xc0"` and `b"\xe0"` don't match each other under `starts_with`, `match_substring` or a `(?i)` regex. This PR follows C++. Substrait `CASE_INSENSITIVE` maps to `IgnoreCase`. `CASE_INSENSITIVE_ASCII` still returns `arrow.ErrNotImplemented`. Case-sensitive matching is unchanged. Folding follows the Go toolchain's `unicode` tables. With Go 1.26, the `go.mod` minimum, those tables are Unicode 15.0 and string fold results matched pyarrow on every oracle case. Go 1.27 uses Unicode 17.0 and differs for 116 characters whose simple folds were added or changed. ### Are these changes tested? I extended two tests: - `TestStartsWith` covers folds with different UTF-8 lengths, no full folding, no Turkish special casing, invalid input bytes versus U+FFFD, ASCII-only binary folding, and empty patterns. Each row runs as arrays and scalars for its input types. The options test checks the invalid string pattern error. - `TestStartsWithExpr` executes `CASE_INSENSITIVE` and checks that `CASE_INSENSITIVE_ASCII` is rejected. I also ran an uncommitted differential comparison against pyarrow `26.0.0`. With Go 1.26, its 224,230 cases had only two differences, both surrogate-encoded string patterns that Go rejects and C++ accepts. On macOS arm64 with Go 1.26.10, `go test ./arrow/compute/...` passed with and without `-tags assert,test`. Tests with `-race` passed on `compute`, `kernels` and `exprs`, along with `gofmt`, `go vet`, `golangci-lint` and the Release Audit Tool. ### Are there any user-facing changes? Yes. `IgnoreCase` now works instead of returning `arrow.ErrNotImplemented`, and Substrait `CASE_INSENSITIVE` plans execute. Invalid UTF-8 string patterns with `IgnoreCase` return `arrow.ErrInvalid`. There is no API change. Closes https://github.com/apache/arrow-go/issues/1460. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
