[
https://issues.apache.org/jira/browse/SPARK-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-16193.
-------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1
1.6.3
Issue resolved by pull request 13896
[https://github.com/apache/spark/pull/13896]
> Address flaky ExternalAppendOnlyMapSuite spilling tests
> -------------------------------------------------------
>
> Key: SPARK-16193
> URL: https://issues.apache.org/jira/browse/SPARK-16193
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Tests
> Affects Versions: 2.0.0
> Reporter: Sean Owen
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 1.6.3, 2.0.1
>
>
> We've seen tests fail, for different codecs and operations, like so, most
> recently from 2.0.0 RC1:
> {code}
> - spilling with compression *** FAILED ***
> java.lang.Exception: Test failed with compression using codec
> org.apache.spark.io.LZ4CompressionCodec:
> assertion failed: expected groupByKey to spill, but did not
> at scala.Predef$.assert(Predef.scala:170)
> at org.apache.spark.TestUtils$.assertSpilled(TestUtils.scala:170)
> at
> org.apache.spark.util.collection.ExternalAppendOnlyMapSuite.org$apache$spark$util$collection$ExternalAppendOnlyMapSuite$$testSimpleSpilling(ExternalAppendOnlyMapSuite.scala:253)
> at
> org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:218)
> at
> org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:216)
> at scala.collection.immutable.Stream.foreach(Stream.scala:594)
> ...
> {code}
> My theory is that the listener doesn't receive notification of the spilled
> stages early enough to show up in the test. It should wait until the job is
> done before letting the test proceed to query the number of spilled stages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
