Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9383#discussion_r43828243
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
---
@@ -183,31 +180,30 @@ class UnsafeFixedWidthAggregationMapSuite
false // disable perf metrics
)
+ var map = createMap()
val keys = randomStrings(1024).take(512)
keys.foreach { keyString =>
val buf =
map.getAggregationBuffer(InternalRow(UTF8String.fromString(keyString)))
buf.setInt(0, keyString.length)
assert(buf != null)
}
-
- // Convert the map into a sorter
val sorter = map.destructAndCreateExternalSorter()
// Add more keys to the sorter and make sure the results come out
sorted.
val additionalKeys = randomStrings(1024)
- val keyConverter = UnsafeProjection.create(groupKeySchema)
- val valueConverter = UnsafeProjection.create(aggBufferSchema)
-
+ map = createMap()
additionalKeys.zipWithIndex.foreach { case (str, i) =>
- val k = InternalRow(UTF8String.fromString(str))
- val v = InternalRow(str.length)
- sorter.insertKV(keyConverter.apply(k), valueConverter.apply(v))
+ val buf =
map.getAggregationBuffer(InternalRow(UTF8String.fromString(str)))
+ buf.setInt(0, str.length)
if ((i % 100) == 0) {
- memoryManager.markExecutionAsOutOfMemoryOnce()
- sorter.closeCurrentPage()
+ val sorter2 = map.destructAndCreateExternalSorter()
+ sorter.merge(sorter2)
+ map = createMap()
}
}
+ val sorter2 = map.destructAndCreateExternalSorter()
+ sorter.merge(sorter2)
--- End diff --
Looks like this updated test is not doing the exactly same thing with the
previous version? In the old version, after we get the sorter we add new
records to the sorter directly. In this updated version, we will always use
merge to add spilled files to the sorter. Should we just create new tests for
this new behavior (using `merge`)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
