[ https://issues.apache.org/jira/browse/SPARK-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312415#comment-14312415 ]
Ilya Ganelin commented on SPARK-5079: ------------------------------------- I can work on this - can you please assign it to me? Thank you. > Detect failed jobs / batches in Spark Streaming unit tests > ---------------------------------------------------------- > > Key: SPARK-5079 > URL: https://issues.apache.org/jira/browse/SPARK-5079 > Project: Spark > Issue Type: Bug > Components: Streaming > Reporter: Josh Rosen > > Currently, it is possible to write Spark Streaming unit tests where Spark > jobs fail but the streaming tests succeed because we rely on wall-clock time > plus output comparision in order to check whether a test has passed, and > hence may miss cases where errors occurred if they didn't affect these > results. We should strengthen the tests to check that no job failures > occurred while processing batches. > See https://github.com/apache/spark/pull/3832#issuecomment-68580794 for > additional context. > The StreamingTestWaiter in https://github.com/apache/spark/pull/3801 might > also fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org