iabhi4 commented on code in PR #46590:
URL: https://github.com/apache/arrow/pull/46590#discussion_r2110798927
##########
cpp/src/arrow/compute/kernels/scalar_string_test.cc:
##########
@@ -1384,10 +1384,12 @@ TYPED_TEST(TestStringKernels, IsDecimalUnicode) {
}
TYPED_TEST(TestStringKernels, IsDigitUnicode) {
- // These are digits according to Python, but we don't have the information in
- // utf8proc for this
- // this->CheckUnary("utf8_is_digit", "[\"²\", \"①\"]", boolean(), "[true,
- // true]");
+ // Tests for digits across various Unicode scripts.
+ // ٤: Arabic 4, ³: Superscript 3, ५: Devanagari 5, Ⅷ: Roman 8 (not digit),
１２３: Fullwidth 123
+ this->CheckUnary("utf8_is_digit",
+ R"(["٤", "³", "५", "Ⅷ", "٣٣", "१२३", "abc", "１２３"])",
+ boolean(),
+ "[true, true, true, false, true, true, false, true]");
}
TYPED_TEST(TestStringKernels, IsNumericUnicode) {
Review Comment:
Hi @pitrou,
Thanks for the suggestion to expand the test — it helped uncover a subtle
issue. While testing characters in `UTF8PROC_CATEGORY_NO`, I noticed that
values like `'¾'` (vulgar fraction) return true, even though Python's
`str.isdigit()` returns false.
Most edge cases behave as expected, but this seems to be an inconsistency
due to `utf8proc` treating all No characters as numeric. Would it make sense to
document this in a comment as a known limitation of `utf8proc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
