khaledh opened a new pull request, #40491: URL: https://github.com/apache/beam/pull/40491
Use backing maps for ordinary unknown-column collector keys to avoid repeated reflective lookup and string interning during Storage Write API conversion. ## Why Unknown-column capture, nested collector attachment, and empty-STRUCT pruning currently use `TableRow` map operations. Those operations consult `GenericData` model metadata and intern field names, even for ordinary keys. With automatic schema updates, collector bookkeeping also runs for known nested fields, so deeply repeated records pay this cost even without unknown columns. This patch uses the existing backing map for ordinary keys while retaining `TableRow` collectors and their allocation behavior. Annotated model fields still use the original setter and map operations. The emptiness shortcut treats every non-null `f` value as an entry, including an empty cell list, and falls back to the original behavior if future model fields appear. ## Context Related to #40203 (descriptor lookup) and #40204 (required-field validation), but independent of both. This patch changes collector access only; it does not change descriptor lookup, message validation, timestamp conversion, or public converter signatures. ## What changed - Use guarded backing-map access for unknown-column capture, nested attachment, lookup, and removal. - Keep model-field behavior, repeated positions, and empty-container pruning unchanged. - Add regression tests for protobuf bytes, encoded unknown JSON, preexisting collectors, model-field errors, and cell-format conversion. ## Validation - Spotless and Checkstyle pass. - All 42 converter tests pass before and after the patch, including 17 new collector tests, in an isolated Java 11-target build against published Beam 2.77.0 dependencies. - The normal Beam Gradle test task remains blocked locally: the configured `protoc-gen-grpc-java` executable is x86-only on macOS ARM. The isolated build does not replace a full master build or upstream CI. - A downstream Beam 2.68.0 backport showed 27.1% more committed offsets/s and 20.3% lower estimated CPU per committed offset in one matched staging pair. This is supporting evidence for the collector change, not a benchmark of this master port or a production guarantee. Full pipeline payload parity and long-run validation remain open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
