[
https://issues.apache.org/jira/browse/SAMZA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989703#comment-13989703
]
Chris Riccomini commented on SAMZA-252:
---------------------------------------
Updated comments on RB. Responding to JIRA here.
bq. So far it doesn't talk about the details of implementing a two-job system
(where one job processes real-time updates and the other reprocesses history).
We could explain the "A/B streams" approach that was discussed offline, and how
a downstream job can use timestamps of jobs and/or version numbers of jobs to
ensure that old data doesn't overwrite new data.
Yea, I agree. I think [~jkreps] had some good comments on the RB about this.
bq. Anyone have thoughts on where we should strike the balance between
comprehensiveness and conciseness in this doc?
I think we should be as thorough as possible with these docs. For shorter/more
concise docs, we should do tutorials that illustrate a specific method of
reprocessing (using the hello-samza project), and refer back to this document.
Also, some drawings/diagrams might help to explain the different strategies.
> Document stream reprocessing
> ----------------------------
>
> Key: SAMZA-252
> URL: https://issues.apache.org/jira/browse/SAMZA-252
> Project: Samza
> Issue Type: Improvement
> Components: docs
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Martin Kleppmann
> Fix For: 0.7.0, 0.8.0
>
> Attachments: SAMZA-252.1.patch
>
>
> A need with stream processing is to want to re-process prior messages at some
> later date. An example of this is having a stream processing job that is
> classifying messages in some way using a machine learning algorithm. At some
> point, the algorithm will be updated with a more accurate vector of weights.
> When this happens, usually you wish to re-process past messages to get more
> accurate results. Usually this is solved by running a parallel pipeline from
> Hadoop.
> We have thought extensively about this use case, and should document how to
> use Samza in a re-processing use case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
