fxbing opened a new pull request, #4622: URL: https://github.com/apache/flink-cdc/pull/4622
## What is the purpose of this pull request? Fixes [FLINK-41434](https://issues.apache.org/jira/browse/FLINK-41434). After the Fluss source reader observes a newer schema, every record with an older schema rebuilds two Fluss schemas to derive the same column mapping. On wide tables, repeated column-name validation makes this path CPU-intensive. ## Brief change log - Cache the column index mapping by source schema ID for each table's current target schema. Records using the target schema continue to bypass projection. - Discard the table's mappings when its target schema advances during reading or split restoration, and release them when the table is removed. The cache is transient and does not change the checkpoint format. ## Verifying this change - Added a regression test that checks mapping reuse and verifies that an older record is projected to the new target after either schema evolution or split restoration. - All 174 targeted tests passed on both Flink 1.20.3 and Flink 2.2.0. - Separately verified two consecutive column additions with 453-, 454-, and 455-column records. After both live schema evolution and restoration, interleaved records retained their values and emitted null for columns absent from their source schema. The local benchmark deserialized records after the reader had observed the 454-column schema. Each value is the median across three warmed-up JVM runs. | Input records | Thread CPU before (µs/record) | Thread CPU after (µs/record) | Allocation before (B/record) | Allocation after (B/record) | | --- | ---: | ---: | ---: | ---: | | Latest schema only (454 columns) | 3.26 | 3.29 | 13,240 | 13,240 | | Alternating 453 and 454 columns (50/50) | 551.27 | 3.54 | 168,376 | 13,180 | | Older schema only (453 columns) | 1,233.34 | 3.49 | 323,512 | 13,120 | ## Documentation - Does this pull request introduce a new feature? No. ##### Was generative AI tooling used to co-author this PR? - [X] Yes, OpenAI Codex. Generated-by: OpenAI Codex 0.167.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
