Dandandan commented on code in PR #16322:
URL: https://github.com/apache/datafusion/pull/16322#discussion_r2137449500
##########
datafusion/physical-plan/src/sorts/merge.rs:
##########
@@ -216,36 +212,50 @@ impl<C: CursorValues> SortPreservingMergeStream<C> {
// Once all partitions have set their corresponding cursors for the
loser tree,
// we skip the following block. Until then, this function may be
called multiple
// times and can return Poll::Pending if any partition returns
Poll::Pending.
+
if self.loser_tree.is_empty() {
- while let Some(&partition_idx) =
self.uninitiated_partitions.front() {
+ // Manual indexing since we're iterating over the vector and
shrinking it in the loop
+ let mut idx = 0;
+ while idx < self.uninitiated_partitions.len() {
Review Comment:
> I'm new to this, so maybe my mental model is completely wrong, but I don't
think that's correct. This is not an async function. It will iterate over all
the partitions once per call to poll_next_inner and return at line 257 unless
uninitiated_partitions is empty.
AFAIK, `uninitiated_partitions` is only constructed once for each input
partition.
So, if I understand correctly, during polling SPM, all input partitions are
polled once (after this change) when they are needed first and removed from the
list.
So, I think we could slightly simplify it by just tracking a partition index
rather than a `Vec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
