[
https://issues.apache.org/jira/browse/FLINK-40350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102698#comment-18102698
]
sepuri sai krishna commented on FLINK-40350:
--------------------------------------------
[~raminqaf] [~Sergey Nuyanzin] you have both worked on this file recently
(FLINK-39866 touched SqlFunctionUtils directly), so flagging it to you. This is
the same class of issue as FLINK-36267, which moved SPLIT to code-point
iteration.
I have a fix ready with red/green verified tests — both pad tests fail on the
first supplementary-plane case before it, and the full ScalarFunctionsTest
class passes after (99/99, including all the existing BMP cases):
https://github.com/SEPURI-SAI-KRISHNA/flink/tree/lpad-rpad-smp
Could a committer assign this? Happy to open a PR as soon as it is.
> LPAD/RPAD split supplementary-plane characters into unpaired surrogates
> -----------------------------------------------------------------------
>
> Key: FLINK-40350
> URL: https://issues.apache.org/jira/browse/FLINK-40350
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
> Reporter: sepuri sai krishna
> Priority: Major
>
> h3. Problem
> {{LPAD}} and {{RPAD}} measure length in UTF-16 code units instead of
> characters. When the requested length falls in the middle of a
> supplementary-plane character, the function splits the surrogate pair and
> returns a string containing an unpaired surrogate, which is not valid Unicode.
> In the examples below, {{E}} stands for the single-character string U+1F600
> GRINNING FACE, which UTF-16 encodes as the surrogate pair U+D83D U+DE00. It
> is written as {{E}} rather than inline so that this description stays within
> the Basic Multilingual Plane.
> || Expression || Returned, as UTF-16 code units || Note ||
> | {{LPAD(E, 1, 'x')}} | {{U+D83D}} | unpaired high surrogate: the first half
> of {{E}} |
> | {{RPAD(E, 1, 'x')}} | {{U+D83D}} | unpaired high surrogate: the first half
> of {{E}} |
> | {{RPAD('a', 4, E)}} | {{U+0061 U+D83D U+DE00 U+D83D}} | {{'a'}}, then an
> intact {{E}}, then a trailing unpaired high surrogate |
> The third case is the least obvious: the padding is copied one code unit at a
> time, so the final repetition of the pad string can stop between the two
> halves of its character.
> The documentation states that the length is measured in characters:
> {quote}Returns a new string from string1 left-padded with string2 to a length
> of integer *characters*.{quote}
> {{E}} is one character, so {{LPAD(E, 1, 'x')}} should return {{E}} unchanged
> rather than half of it.
> h3. Impact
> Silent corruption of string data. The result is invalid UTF-16 and does not
> round-trip: encoding it yields a replacement character or malformed bytes,
> and comparisons and downstream string functions then operate on a value the
> query never produced. No exception is thrown and nothing is logged.
> Affects any query where {{LPAD}} or {{RPAD}} truncates, or where the pad
> string contains supplementary-plane characters - emoji, historic scripts, and
> less common CJK ideographs.
> h3. Affects
> Present in every branch checked from release-1.19 through release-2.3, and on
> master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
