zhuqi-lucas commented on PR #16322:
URL: https://github.com/apache/datafusion/pull/16322#issuecomment-2961694324
> I just checked, and SortExec is also setting up RecordBatchReceiverStream.
The worst case scenario in terms of elapsed time in the poll_next call is that
all 10k streams are ready in one cycle. This will trigger 10k cursor
initialization which does some non-trivial work converting the record batch.
But the current code is doing exactly the same thing today already so it's no
worse than the status quo as far as I can tell.
@pepijnve This is a good explain, so even 10k streams are behind
RecordBatchReceiverStream, it's linux epoll based, it's very efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
