markj-db commented on code in PR #59190:
URL: https://github.com/apache/spark/pull/59190#discussion_r4168132323
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala:
##########
@@ -489,7 +489,10 @@ final class SlidingWindowFunctionFrame(
// Only recalculate and update when the buffer changes.
if (processor != null && bufferUpdated) {
- processor.initialize(input.length)
+ // The window driver rejects partitions larger than `Int.MaxValue` rows
for this frame
+ // (see `supportsLargePartition`); `toIntExact` is a defensive backstop
that fails loudly
Review Comment:
Fixed in 4897472. Repointed all four `toIntExact` backstop comments and the
`WindowSegmentTree.build` check at
`WindowEvaluatorFactoryBase.checkPartitionSizeLimit` and dropped the "for this
frame" wording, since `supportsLargePartition` no longer exists.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArraySuite.scala:
##########
@@ -116,6 +116,20 @@ class ExternalAppendOnlyUnsafeRowArraySuite extends
SparkFunSuite with LocalSpar
assert(getNumBytesSpilled > 0)
}
+ test("length is tracked as a Long") {
Review Comment:
Fixed in 4897472. Replaced the value check with a direct return-type
assertion:
`classOf[ExternalAppendOnlyUnsafeRowArray].getMethod("length").getReturnType ==
java.lang.Long.TYPE`, so it now fails if `length` reverts to `Int`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
