[
https://issues.apache.org/jira/browse/SAMZA-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15551235#comment-15551235
]
Nicolas Colomer commented on SAMZA-974:
---------------------------------------
Should have take a look at the user group before commenting, sorry for this
comment.
> Build an end-of-stream concept into Samza
> -----------------------------------------
>
> Key: SAMZA-974
> URL: https://issues.apache.org/jira/browse/SAMZA-974
> Project: Samza
> Issue Type: Bug
> Reporter: Jagadish
> Assignee: Jagadish
> Fix For: 0.12.0
>
> Attachments: ProposalforEndofStreaminSamza (2).pdf,
> ProposalforEndofStreaminSamza.pdf
>
>
> Samza currently works with unbounded data sources. However, for bounded data
> sources like HDFS files, snapshot files which are not infinite, we need a
> notion of 'end-of-stream'.
> The following are the logical tasks:
> 1.SystemConsumer will indicate to Samza that the end of stream has been
> reached for an SSP.
> 2. Samza will shut down the task if all SSPs in the task are at end of stream.
> 3. Samza will provide a callback to the task so that it can perform cleanups/
> commits once tasks are at end of stream.
> 4. Samza will shut down the container if all tasks in the container have been
> shut down.
> 5. Samza will ultimately shut down the job if all containers in the job have
> been shut down.
> This is a step towards realizing a 'finite' Samza job that terminates (as
> opposed to an infinite stream job that keeps running) once data processing is
> complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
