[ https://issues.apache.org/jira/browse/SPARK-46841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nikola Mandic updated SPARK-46841: ---------------------------------- Description: Languages and localization for collations are supported by ICU library. Collation naming format is as follows: {code:java} <2-letter language code>_<optional 4-letter script>_<optional 3-letter country code>[_specifier_specifier...]{code} Locale specifier consists of the first part of collation name (language + script + country). Locale specifiers need to be stable across ICU versions; to keep existing ids and names invariant we introduce golden file will locale table which should case CI failure on any silent changes. Currently supported optional specifiers: * CS/CI - case sensitivity, default is case-sensitive; supported by configuring ICU collation levels * AS/AI - accent sensitivity; default is accent-sensitive; supported by configuring ICU collation levels * <unspecified>/LCASE/UCASE - case conversion performed prior to comparisons; supported by internal implementation relying on ICU locale-aware conversions User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names defined in CollationFactory. was: Languages and localization for collations are supported by ICU library. Collation naming format is as follows: {code:java} <2-letter language code>_<optional 4-letter script>_<3-letter country code>[_specifier_specifier...]{code} Locale specifier consists of the first part of collation name (language + script + country). Locale specifiers need to be stable across ICU versions; to keep existing ids and names invariant we introduce golden file will locale table which should case CI failure on any silent changes. Currently supported optional specifiers: * CS/CI - case sensitivity, default is case-sensitive; supported by configuring ICU collation levels * AS/AI - accent sensitivity; default is accent-sensitive; supported by configuring ICU collation levels * <unspecified>/LCASE/UCASE - case conversion performed prior to comparisons; supported by internal implementation relying on ICU locale-aware conversions User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names defined in CollationFactory. > Language support for collations > ------------------------------- > > Key: SPARK-46841 > URL: https://issues.apache.org/jira/browse/SPARK-46841 > Project: Spark > Issue Type: New Feature > Components: Spark Core, SQL > Affects Versions: 4.0.0 > Reporter: Aleksandar Tomic > Priority: Major > Labels: pull-request-available > > Languages and localization for collations are supported by ICU library. > Collation naming format is as follows: > {code:java} > <2-letter language code>_<optional 4-letter script>_<optional 3-letter > country code>[_specifier_specifier...]{code} > Locale specifier consists of the first part of collation name (language + > script + country). Locale specifiers need to be stable across ICU versions; > to keep existing ids and names invariant we introduce golden file will locale > table which should case CI failure on any silent changes. > Currently supported optional specifiers: > * CS/CI - case sensitivity, default is case-sensitive; supported by > configuring ICU collation levels > * AS/AI - accent sensitivity; default is accent-sensitive; supported by > configuring ICU collation levels > * <unspecified>/LCASE/UCASE - case conversion performed prior to > comparisons; supported by internal implementation relying on ICU locale-aware > conversions > User can use collation specifiers in any order except of locale which is > mandatory and must go first. There is a one-to-one mapping between collation > ids and collation names defined in CollationFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org