gortiz commented on issue #18375: URL: https://github.com/apache/pinot/issues/18375#issuecomment-4421346630
Implementation is up in PR #18458. A few notes on how the implementation maps to the proposal: **Implemented as proposed:** - `SubmitWithStream` bidi RPC with `BrokerToServer` (submit + cancel) / `ServerToBroker` (submit_ack + OpChainComplete) messages - `MultiStageStatsTree` / `StageStatsNode` structured tree-shaped payload with operator type, plan-node ids, stat bytes, and children (fixes N-ary set-op reconstruction) - Op→PlanNode mapping captured via the existing `BiConsumer` tracker in `PlanNodeToOpChain` - Mailbox stats suppressed when stream mode is active - Broker per-stage accumulator with `StatMap.merge`, `mergeFailed` on shape mismatch - Wait window after data-mailbox EOS - Fan-out cancel on first peer error - Per-stage `streamStatsCoverage[]` in broker response - Cluster-level + per-query config, default off; `UNIMPLEMENTED` fallback to legacy **Notable divergences / additions:** - `ServerDone` is implicit (server half-closes the outbound stream) rather than a distinct message — equivalent, one fewer round-trip - **Cancel via stream (Phase B) is included.** `BrokerToServer.cancel` + stream-close-as-cancel are both handled; the separate unary `Cancel` RPC is simplified and kept for one release of overlap - **O(1) cancel in `OpChainSchedulerService` for all modes** — replaces the previous O(n) `_opChainCache` scan with a `_executionContextByRequest` map; write lock acquired before `compute()` eviction to close a race window - `tryRecoverWithStream` waits for remaining `OpChainComplete` stats up to the query deadline on error, so error-path coverage is as complete as possible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
