0z5a opened a new pull request, #402:
URL: https://github.com/apache/dubbo-go-hessian2/pull/402
<!-- Thanks for sending a pull request!
-->
**What this PR does**:
When `decode2utf8` sees a high surrogate (U+D800 to U+DBFF), it always reads
3 more bytes to look for a low surrogate. It does this without checking how
many chars are left in the current string chunk. #393 added a check on the
value of those 3 bytes, but not on whether they belong to the string at all.
A Java `String` can end with a lone high surrogate, for example when
`substring` cuts an emoji in half while truncating a name or a title. As the
comment on the existing `TestDecodeStringLoneHighSurrogateRealWorld` test
notes, Java writes a lone surrogate as 3-byte CESU-8 (U+D83D becomes `0xED 0xA0
0xBD`). When such a char is the last char of the chunk, or is followed by only
one or two ASCII chars, the decoder reads past the end of the string.
There are two symptoms on master:
1. The lone high surrogate is the last char: the read goes past the
`charTotal*3` buffer and the decoder panics.
```
[]byte{0x02, 'a', 0xED, 0xA0, 0xBD} // "a\ud83d"
panics with: runtime error: slice bounds out of range [:7] with capacity 6
```
2. The lone high surrogate is followed by one or two ASCII chars and then
another value: the string takes the first byte(s) of the next value, and
decoding the rest of the stream goes wrong.
```
[]byte{0x7a, 0x03, 0xED, 0xA0, 0xBD, 'a', 'b', 0x02, 'c', 'd'} //
["\ud83dab", "cd"]
error: illegal class index @idx 3
[]byte{0x7a, 0x02, 0xED, 0xA0, 0xBD, 'a', 0x91} // ["\ud83da", 1]
error: unexpected EOF
```
The fix passes the number of chars left in the chunk to `decode2utf8`. A low
surrogate is only looked for when at least one more char is left. The decoder
then reads one byte first, and reads the other two only if that byte starts a
3-byte sequence. Otherwise the high surrogate is kept as a lone 3-byte char,
the same as #393 already does when the next char is not a low surrogate. Valid
surrogate pairs are decoded as before.
The new `TestDecodeStringLoneHighSurrogateAtChunkEnd` test covers both
symptoms, plus a valid surrogate pair as a control. On master, 4 of its 5 cases
fail: two panic, and two return `illegal class index @idx 3` and `unexpected
EOF`. The valid pair case passes. With the fix, all 5 cases pass.
I also ran every Go test in the module one by one. With the fix, 88 pass and
none fail. Another 81 tests need the `test_hessian` jar, which I did not build
locally. That set is the same on master and with the fix, and only the new test
differs between the two runs. I also round-tripped random CJK and emoji strings
through the encoder and decoder, including lengths at the 4096-char chunk
boundary, with `-race`.
**Which issue(s) this PR fixes**:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related
issues/tests in a comment and do not use `Fixes`_*
-->
None filed.
**Special notes for your reviewer**:
The change is limited to `decode2utf8` and its only caller,
`readStringChunkData`. The surrogate-pair branch keeps its existing code; it
only gets the two new guards (a char left in the chunk, and a 3-byte lead byte).
**Does this PR introduce a user-facing change?**:
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires
additional action from users switching to the new release, include the string
"action required".
-->
```release-note
Fix a panic, or wrong decoding of the following values, when a string ends
with a lone high surrogate.
```
This change was drafted with Claude Code (claude-opus-5-5); the test results
above are from the runs described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]