[
https://issues.apache.org/jira/browse/SAMZA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Kumarasubramanian updated SAMZA-2393:
---------------------------------------------
Parent: SAMZA-2394
Issue Type: Sub-task (was: Improvement)
> Side input streams should respect offset reset and broadcast properties
> -----------------------------------------------------------------------
>
> Key: SAMZA-2393
> URL: https://issues.apache.org/jira/browse/SAMZA-2393
> Project: Samza
> Issue Type: Sub-task
> Reporter: Bharath Kumarasubramanian
> Priority: Major
>
> Side inputs are treated like every other stream (mostly) in Samza today.
> However, some of the properties of the stream like offset.reset policies,
> broadcast etc don't apply to side inputs. As part of this ticket, we want to
> explore the option of supporting these for side input streams too.
> The current limitations are we don't have checkpointing for side inputs and
> use the local checkpoint on the disk to register the starting offsets. We
> need to tie down the offset manager into side inputs consumers which is
> capable of handling offset reset policies.
> Also, side input consumer updates the chooser every time when it chooses the
> message before waiting for the messages to be processed by the side input
> processor. This limits us to use priorities in side inputs and also implement
> broadcast correctly. We would need to do something similar to the RunLoop to
> support the above.
> As part of this ticket, we need to explore if we can use a new run loop
> instance for side inputs or extract some of the runloop logic to make it
> applicable for side inputs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
