gortiz commented on code in PR #18519:
URL: https://github.com/apache/pinot/pull/18519#discussion_r3272023196
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/MailboxService.java:
##########
@@ -160,13 +210,18 @@ public InstanceType getInstanceType() {
* not open the underlying channel or acquire any additional resources.
Instead, it will initialize lazily when the
* data is sent for the first time.
*/
+ // TODO: Consider adding an application-level global byte budget for sender
outbound. The
+ // transport-layer WriteBufferWaterMark already caps sender direct memory
per (host, port)
+ // channel, so the OOM the original PR fixed is bounded by watermark.high ×
#peers. A global
+ // byte budget would tighten the bound to a single configurable cap across
all peers, but is
+ // unnecessary unless fan-outs hit hundreds of peers per query.
Review Comment:
I originally included this on the PR, but it just made it more complex. It
is also difficult to trace total in-flight memory, as the natural way to track
this is to increase a counter when a block is sent and reduce it when the
receiver sends a status message, but that means we measure at _block_ level,
not _chunk/message_ level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
