uros-db commented on code in PR #45216:
URL: https://github.com/apache/spark/pull/45216#discussion_r1505471123
##########
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:
##########
@@ -341,6 +342,21 @@ public boolean contains(final UTF8String substring) {
return false;
}
+ public boolean contains(final UTF8String substring, int collationId) throws
SparkException {
+ if (CollationFactory.fetchCollation(collationId).isBinaryCollation) {
+ return this.containsBinary(substring);
+ }
+ if (collationId == CollationFactory.LOWERCASE_COLLATION_ID) {
+ return this.toLowerCase().containsBinary(substring.toLowerCase());
+ }
+ // TODO: enable ICU collation support for "contains"
+ Map<String, String> params = new HashMap<>();
+ params.put("functionName",
Thread.currentThread().getStackTrace()[1].getMethodName());
+ params.put("collationName",
CollationFactory.fetchCollation(collationId).collationName);
+ throw new SparkException("COLLATION_NOT_SUPPORTED_FOR_FUNCTION",
Review Comment:
I think the logic here was that there are gonna be some functions that are
not (yet) supported for some particular collation types (some might get added
at one point in the near future, while others may not), so we will eventually
need a way to let users know. that's why we added the
`COLLATION_NOT_SUPPORTED_FOR_FUNCTION` error class - while it will likely get
removed from here, there are other places where it will appear instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
