[
https://issues.apache.org/jira/browse/SAMZA-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jake Maes reassigned SAMZA-1393:
--------------------------------
Assignee: Daniel Nishimura
> Asynchronous SystemProducer API
> -------------------------------
>
> Key: SAMZA-1393
> URL: https://issues.apache.org/jira/browse/SAMZA-1393
> Project: Samza
> Issue Type: Bug
> Reporter: Jake Maes
> Assignee: Daniel Nishimura
> Priority: Major
>
> We've encountered a number of issues with the current SystemProducer
> interface with async processors. For example, see SAMZA-1392.
> Most of the issues arise from the fact that there's no way for us to notify
> the user when there is an async error in from the producer. Consider the
> KafkaSystemProducer. If the KafkaProducer callback returns an exception, the
> KafkaSystemProducer has to track it and find an appropriate time to handle
> the exception. Depending on where the exception is handled and whether it is
> swallowed the OffsetManager will need to be overly conservative to ensure we
> do not checkpoint offsets until the corresponding send() was successful.
> Sometimes whole batches of messages will fail, but only a single exception is
> thrown, and there's no context from which send() the exception originated.
> This will always be unintuitive to the user unless they are synchronously
> notified of the exception.
> There hasn't been much planning on this yet, but the idea is to have a send()
> method that takes a callback which is called directly from the producer
> callback. So the callback would flow something like this:
> producer.callback -> user.callback -> asynctask.callback -> (throw error |
> offsetmanager.update)
> Some care needs to be taken to ensure that there are no gaps in the offsets
> recorded by the OffsetManager. Also, it's not yet clear how to enable users
> to retry a failed send.
> NOTE: depending on the implementation, the changes in SAMZA-1384 may need to
> be reverted after this feature is implemented to ensure that the latest
> offsets are checkpointed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
