[
https://issues.apache.org/jira/browse/KAFKA-7566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692501#comment-16692501
]
Boyang Chen commented on KAFKA-7566:
------------------------------------
[~guozhang] Yep, for example our KStream application publishes data snapshots
every 10 minute to S3. Periodically we want to gc the outdated ones which could
be done on a separate thread. The caveat is that the gc job will be scheduled
on all instances and sometimes cause race conditions. We could use zk to
mitigate the issue, but just want to reach out to see if there would be some
similar request on stream side, which is a leader-only, non-blocking main
processing feature.
> Add sidecar job to leader (or a random single follower) only
> ------------------------------------------------------------
>
> Key: KAFKA-7566
> URL: https://issues.apache.org/jira/browse/KAFKA-7566
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Boyang Chen
> Priority: Minor
>
> Hey there,
> recently we need to add an archive job to a streaming application. The caveat
> is that we need to make sure only one instance is doing this task to avoid
> potential race condition, and we also don't want to schedule it as a regular
> stream task so that we will be blocking normal streaming operation.
> Although we could do so by doing a zk lease, I'm raising the case here since
> this could be some potential use case for streaming job also. For example,
> there are some `leader specific` operation we could schedule in DSL instead
> of adhoc manner.
> Let me know if you think this makes sense to you, thank you!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
