[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470552#comment-16470552 ]
Jose Torres commented on SPARK-24036: ------------------------------------- My concern isn't that we'll have to write more code, but that changing scheduler internals expands the surface area of interactions that need to be considered. For example, can we confidently enumerate all the ways in which the scheduler assumes a Dependency defines a stage boundary? If so, can we change all of them in a way that doesn't impact non-continuous-processing code at all? We'd have to consider a lot of questions like that, and I don't see any large benefit we'd get from doing so. Glad to take a look at your preview PR. > Stateful operators in continuous processing > ------------------------------------------- > > Key: SPARK-24036 > URL: https://issues.apache.org/jira/browse/SPARK-24036 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.4.0 > Reporter: Jose Torres > Priority: Major > > The first iteration of continuous processing in Spark 2.3 does not work with > stateful operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org