Github user pwendell commented on a diff in the pull request:
https://github.com/apache/incubator-spark/pull/624#discussion_r9959040
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
---
@@ -302,12 +304,14 @@ private[spark] class ExternalAppendOnlyMap[K, V, C](
mergedBuffers += newBuffer
}
- // Repopulate each visited stream buffer and add it back to the
merge heap
+ // Repopulate each visited stream buffer and add it back to the
queue if it is non-empty
mergedBuffers.foreach { buffer =>
- if (buffer.pairs.length == 0) {
+ if (buffer.isEmpty) {
buffer.pairs ++= getMorePairs(buffer.iterator)
}
- mergeHeap.enqueue(buffer)
+ if (!buffer.isEmpty) {
--- End diff --
shouldn't this just be an `else`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
[email protected] or file a JIRA ticket with INFRA.
---
