allthingssecurity opened a new pull request, #27107: URL: https://github.com/apache/camel/pull/27107
# Description [CAMEL-25152](https://issues.apache.org/jira/browse/CAMEL-25152) Since CAMEL-14521 (3.1.0) a fixed-length record is cut into fields with `UnicodeHelper`: a field of `length = 5` is 5 code points, or 5 graphemes with `@FixedLengthRecord(countGrapheme = true)`. Marshal was not changed: `BindyFixedLengthFactory.generateFixedLengthPositionMap` still pads a field with `length - result.length()` padding characters and clips it with `result.substring(0, length)`, both in UTF-16 chars, and the record length check in `BindyFixedLengthDataFormat.createModel` also uses `String.length()`. For text in the Basic Multilingual Plane with `countGrapheme = false` the counts are the same. A character outside the BMP (an emoji, CJK Extension B) is one code point but two UTF-16 chars, and with `countGrapheme = true` a letter with a combining mark (`e` + U+0301) is one grapheme but two code points. Such a field is written one padding character short for each extra char, and unmarshal of the record takes the missing characters from the next field, so every following field is shifted: ``` @FixedLengthRecord(length = 12), fields of length 5, 4, 3 (align R, trim = true), values "ok😀", "x", "y" marshal today: " ok😀 x y" (4 code points for the first field, 11 in all) unmarshal: "ok😀 ", "x ", "y" marshal fixed: " ok😀 x y" -> "ok😀", "x", "y" the correctly padded record " ok😀 x y" (12 code points) from another system: unmarshal today: IllegalArgumentException: Size of the record: 13 is not equal to the value provided in the model: 12 ``` With `clip = true` the clip can also cut a surrogate pair in half and write an invalid character. This change: - `generateFixedLengthPositionMap` measures and clips the formatted value with `UnicodeHelper` and the same method as unmarshal (a new `BindyFixedLengthFactory.unicodeMethod()`: `CODEPOINTS`, or `GRAPHEME` with `countGrapheme`), so the padding, the clip and the "must not be larger than allowed" check use the unmarshal count. - `createModel` checks the record length, and cuts it for `ignoreTrailingChars`, with the same count. Compatibility: text in the BMP without combining characters (with `countGrapheme = true`: without characters that form one grapheme together) is written and read exactly as before. For other text the output changes, because it is now padded to the field length that unmarshal reads, and a record with such characters that an older version wrote is now too short by that count, so it fails the record `length` check (unless `ignoreMissingChars = true`) instead of being read with shifted fields. There is a note in the 4.23 upgrade guide. Tests: - New `BindyFixedLengthMarshalUnicodeTest`: the marshal output of `ok😀` (padded to 5 code points), the marshal/unmarshal round trip of `ok😀`, two CJK Extension B characters and `café` with `@FixedLengthRecord(length = 12)`, the clip of `abcd😀f` to 5, and the round trip of a decomposed `café` with `countGrapheme = true`. - Without the change, all 4 tests fail (for example `expected: <ok😀> but was: <ok😀 >`). The existing `BindyCountGraphemeTest`, `BindyMultiBytesTest` and `UnicodeHelperTest` pass unchanged. - With the change, the whole camel-bindy module: 232 tests, 0 failures, 0 errors, 3 skipped. Found with a Lean 4 model of the fixed-length padding and slicing (every value with a non-BMP character followed by another field is read back with extra characters today; with the change every record whose values fit their lengths round trips, and for BMP text the output is the same as today), then reproduced with the real classes on main. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested the affected module, including the formatter and import-sort plugins. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
