[
https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470552#comment-16470552
]
Jose Torres commented on SPARK-24036:
-------------------------------------
My concern isn't that we'll have to write more code, but that changing
scheduler internals expands the surface area of interactions that need to be
considered. For example, can we confidently enumerate all the ways in which the
scheduler assumes a Dependency defines a stage boundary? If so, can we change
all of them in a way that doesn't impact non-continuous-processing code at all?
We'd have to consider a lot of questions like that, and I don't see any large
benefit we'd get from doing so.
Glad to take a look at your preview PR.
> Stateful operators in continuous processing
> -------------------------------------------
>
> Key: SPARK-24036
> URL: https://issues.apache.org/jira/browse/SPARK-24036
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Jose Torres
> Priority: Major
>
> The first iteration of continuous processing in Spark 2.3 does not work with
> stateful operators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
