xiangfu0 commented on PR #19568:
URL: https://github.com/apache/pinot/pull/19568#issuecomment-5804053077
@yashmayya thanks for the review. Both notes are addressed in `ba32c703af`:
1. **Stale `worker.proto` comment**: it now points to
`pinot.broker.mse.enable.proto.segment.list` instead of the removed
`protoSegmentList` option.
2. **Serial legacy JSON parse**: good catch, and I made it a fix rather than
leaving it as a known cost. Because the proto encoding ships disabled, this was
the default path, and on master each worker parsed its own list lazily on its
own thread. `WorkerMetadata` now keeps a legacy-encoded segment map as the raw
JSON and parses it on first `getTableSegmentsMap()` /
`getLogicalTableSegmentsMap()`, memoized through a `volatile` field. So the
parse happens when each worker compiles its leaf stage, in parallel again,
rather than serially in `deserializePlan` before any worker of the stage
starts. `isLeafStageWorker()` counts an unparsed map, and malformed JSON fails
at first access again, as on master.
The proto path stays eager on purpose: protobuf has already materialized
the strings while parsing the request, so converting them is just a list copy.
Two new tests pin this down: `testLegacyJsonIsParsedOnFirstAccessOnly`
uses malformed JSON, so decoding must succeed and only the first access may
fail, and `testLegacyJsonIsParsedOnce` checks the memoization. The decode
benchmark now includes the first access, so the legacy numbers still measure
the parse; allocation per worker is unchanged (10.88 MB legacy, 7.46 MB proto
at 60k segments).
This push also re-runs CI, which replaces the manual re-run I mentioned for
the RAT `ConcurrentModificationException` above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
