This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new cf43623 [SPARK-36900][SPARK-36464][CORE][TEST] Refactor `: size
returns correct positive number even with over 2GB data` to pass with Java 8,
11 and 17
cf43623 is described below
commit cf436233072b75e083a4455dc53b22edba0b3957
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Sat Oct 16 09:10:06 2021 -0500
[SPARK-36900][SPARK-36464][CORE][TEST] Refactor `: size returns correct
positive number even with over 2GB data` to pass with Java 8, 11 and 17
### What changes were proposed in this pull request?
Refactor `SPARK-36464: size returns correct positive number even with over
2GB data` in `ChunkedByteBufferOutputStreamSuite` to reduce the total use of
memory for this test case, then this case can pass with Java 8, Java 11 and
Java 17 use `-Xmx4g`.
### Why are the changes needed?
`SPARK-36464: size returns correct positive number even with over 2GB data`
pass with Java 8 but OOM with Java 11 and Java 17.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test
```
mvn clean install -pl core -am -Dtest=none
-DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite
```
with Java 8, Java 11 and Java 17, all tests passed.
Closes #34284 from LuciferYang/SPARK-36900.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Sean Owen <sro...@gmail.com>
---
.../spark/util/io/ChunkedByteBufferOutputStreamSuite.scala | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
b/core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
index 29443e2..0a61488 100644
---
a/core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
+++
b/core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
@@ -121,12 +121,13 @@ class ChunkedByteBufferOutputStreamSuite extends
SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB
data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
+ val data4M = 1024 * 1024 * 4
+ val writeTimes = 513
+ val ref = new Array[Byte](data4M)
+ val o = new ChunkedByteBufferOutputStream(data4M, ByteBuffer.allocate)
+ (0 until writeTimes).foreach(_ => o.write(ref))
o.close()
assert(o.size > 0L) // make sure it is not overflowing
- assert(o.size == ref.length.toLong * 2)
+ assert(o.size == ref.length.toLong * writeTimes)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
