[
https://issues.apache.org/jira/browse/APEXMALHAR-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandesh updated APEXMALHAR-2086:
--------------------------------
Description:
Goal : 2 Operartors for Kafka Output
1. Simple Kafka Output Operator
- Supports Atleast Once
- Expose most used producer properties as class properties
2. Exactly Once Kafka Output ( Not possible in all the cases, will be
documented later )
Design for Exactly Once
Window Data Manager - Stores the Kafka partitions offsets.
Kafka Key - Used by the operator = AppID#OperatorId
During recovery. Partially written window is re-created using the following
approach:
Tuples between the largest recovery offsets and the current offset are checked.
Based on the key, tuples written by the other entities are discarded.
Only tuples which are not in the recovered set are emitted.
was:
Goal : 2 Operartors for Kafka Output
1. Simple Kafka Output Operator
- Supports Atleast Once
- Expose most used producer properties as class properties
2. Exactly Once Kafka Output ( Not possible in all the cases, will be
documented later )
> Kafka Output Operator with Kafka 0.9 API
> ----------------------------------------
>
> Key: APEXMALHAR-2086
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2086
> Project: Apache Apex Malhar
> Issue Type: New Feature
> Reporter: Sandesh
> Assignee: Sandesh
>
> Goal : 2 Operartors for Kafka Output
> 1. Simple Kafka Output Operator
> - Supports Atleast Once
> - Expose most used producer properties as class properties
> 2. Exactly Once Kafka Output ( Not possible in all the cases, will be
> documented later )
>
> Design for Exactly Once
> Window Data Manager - Stores the Kafka partitions offsets.
> Kafka Key - Used by the operator = AppID#OperatorId
> During recovery. Partially written window is re-created using the following
> approach:
> Tuples between the largest recovery offsets and the current offset are
> checked. Based on the key, tuples written by the other entities are
> discarded.
> Only tuples which are not in the recovered set are emitted.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
