[ https://issues.apache.org/jira/browse/SPARK-14833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu resolved SPARK-14833. ---------------------------------- Resolution: Fixed Fix Version/s: 2.0.0 > Refactor StreamTests to test for source fault-tolerance correctly. > ------------------------------------------------------------------ > > Key: SPARK-14833 > URL: https://issues.apache.org/jira/browse/SPARK-14833 > Project: Spark > Issue Type: Sub-task > Components: SQL, Streaming > Reporter: Tathagata Das > Assignee: Tathagata Das > Fix For: 2.0.0 > > > Current StreamTest allows testing of a streaming Dataset generated explicitly > wraps a source. This is different from the actual production code path where > the source object is dynamically created through a DataSource object every > time a query is started. So all the fault-tolerance testing in > FileSourceSuite and FileSourceStressSuite is not really testing the actual > code path as they are just reusing the FileStreamSource object. > Instead of maintaining a mapping of source --> expected offset in StreamTest > (which requires reuse of source object), it should maintain a mapping of > source index --> offset, so that it is independent of the source object. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org