[
https://issues.apache.org/jira/browse/SPARK-46841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikola Mandic updated SPARK-46841:
----------------------------------
Description:
Languages and localization for collations are supported by ICU library.
Collation naming format is as follows:
{code:java}
<2-letter language code>[_<4-letter script>][_<3-letter country
code>][_specifier_specifier...]{code}
Locale specifier consists of the first part of collation name (language +
script + country). Locale specifiers need to be stable across ICU versions; to
keep existing ids and names invariant we introduce golden file will locale
table which should case CI failure on any silent changes.
Currently supported optional specifiers:
* CS/CI - case sensitivity, default is case-sensitive; supported by
configuring ICU collation levels
* AS/AI - accent sensitivity; default is accent-sensitive; supported by
configuring ICU collation levels
* <unspecified>/LCASE/UCASE - case conversion performed prior to comparisons;
supported by internal implementation relying on ICU locale-aware conversions
User can use collation specifiers in any order except of locale which is
mandatory and must go first. There is a one-to-one mapping between collation
ids and collation names defined in CollationFactory.
was:
Languages and localization for collations are supported by ICU library.
Collation naming format is as follows:
{code:java}
<2-letter language code>_<optional 4-letter script>_<optional 3-letter country
code>[_specifier_specifier...]{code}
Locale specifier consists of the first part of collation name (language +
script + country). Locale specifiers need to be stable across ICU versions; to
keep existing ids and names invariant we introduce golden file will locale
table which should case CI failure on any silent changes.
Currently supported optional specifiers:
* CS/CI - case sensitivity, default is case-sensitive; supported by
configuring ICU collation levels
* AS/AI - accent sensitivity; default is accent-sensitive; supported by
configuring ICU collation levels
* <unspecified>/LCASE/UCASE - case conversion performed prior to comparisons;
supported by internal implementation relying on ICU locale-aware conversions
User can use collation specifiers in any order except of locale which is
mandatory and must go first. There is a one-to-one mapping between collation
ids and collation names defined in CollationFactory.
> Language support for collations
> -------------------------------
>
> Key: SPARK-46841
> URL: https://issues.apache.org/jira/browse/SPARK-46841
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core, SQL
> Affects Versions: 4.0.0
> Reporter: Aleksandar Tomic
> Priority: Major
> Labels: pull-request-available
>
> Languages and localization for collations are supported by ICU library.
> Collation naming format is as follows:
> {code:java}
> <2-letter language code>[_<4-letter script>][_<3-letter country
> code>][_specifier_specifier...]{code}
> Locale specifier consists of the first part of collation name (language +
> script + country). Locale specifiers need to be stable across ICU versions;
> to keep existing ids and names invariant we introduce golden file will locale
> table which should case CI failure on any silent changes.
> Currently supported optional specifiers:
> * CS/CI - case sensitivity, default is case-sensitive; supported by
> configuring ICU collation levels
> * AS/AI - accent sensitivity; default is accent-sensitive; supported by
> configuring ICU collation levels
> * <unspecified>/LCASE/UCASE - case conversion performed prior to
> comparisons; supported by internal implementation relying on ICU locale-aware
> conversions
> User can use collation specifiers in any order except of locale which is
> mandatory and must go first. There is a one-to-one mapping between collation
> ids and collation names defined in CollationFactory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
