[
https://issues.apache.org/jira/browse/SPARK-30576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-30576:
---------------------------------
Component/s: (was: Structured Streaming)
> Whether to block streaming batch commit, merge all blocking batchs as one
> batch commit
> --------------------------------------------------------------------------------------
>
> Key: SPARK-30576
> URL: https://issues.apache.org/jira/browse/SPARK-30576
> Project: Spark
> Issue Type: New Feature
> Components: DStreams
> Affects Versions: 2.4.4
> Reporter: liushuo
> Priority: Major
> Attachments: pr-30576.patch
>
>
> When the current job is not completed block streaming batch commit, until
> completed。The next job will merge all batch which during the blocking.
> for example:
> the input seq [1, 2, 3, 4, 5, 6]。
> batch duration: 1s。
> The 3th batch will take a long time。Normally the other batches will be
> completed quickly.
> We expect:
> 1. the 4th batch will not be commited during the 3th batch computing,
> and 4th batch will be merge in the next batch. So that
> the size of jobSets is always less than 1。
> 2. the num completedBatches less than the size of seq。
> 3. the data is not lost
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
