[
https://issues.apache.org/jira/browse/IGNITE-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601823#comment-14601823
]
Andrey Gura edited comment on IGNITE-428 at 6/25/15 9:44 PM:
-------------------------------------------------------------
[~Vishal.garg.ignite], thanks for contributing!
I found one more issue that important from my point of view. When
{{KafkaStreamer}} instance starts it submits {{Runnable}} instances to
executor. Each instance iterates through messages and adds them to
{{IgniteDataSreamer}}. But what will happen if at some moment
{{ConsumerIterator}} throw {{ConsumerTimeoutException}}? Eventually all
consuming threads will terminated and streamer will not consume messages from
Kafka topic when it will available again. I think we should provide some retry
logic with backoff.
We can discuss this problem on dev-list or here.
BTW, your changes committed to ignite-428 branch.
was (Author: agura):
[~Vishal.garg.ignite], thanks for contributing1
I found one more issue that important from my point of view. When
{{KafkaStreamer}} instance starts it submits {{Runnable}} instances to
executor. Each instance iterates through messages and adds them to
{{IgniteDataSreamer}}. But what will happen if at some moment
{{ConsumerIterator}} throw {{ConsumerTimeoutException}}? Eventually all
consuming threads will terminated and streamer will not consume messages from
Kafka topic when it will available again. I think we should provide some retry
logic with backoff.
We can discuss this problem on dev-list or here.
BTW, your changes committed to ignite-428 branch.
> Implement IgniteKafkaStreamer to stream data from Apache Kafka
> --------------------------------------------------------------
>
> Key: IGNITE-428
> URL: https://issues.apache.org/jira/browse/IGNITE-428
> Project: Ignite
> Issue Type: Sub-task
> Components: streaming
> Affects Versions: sprint-5
> Reporter: Dmitriy Setrakyan
> Assignee: Andrey Gura
> Fix For: sprint-7
>
> Attachments: ignite-sprint-5_ignite-428.patch,
> ignite-sprint-5_ignite-428.patch, ignite-sprint-5_ignite-428.patch
>
>
> We have {{IgniteDataStreamer}} which is used to load data into Ignite under
> high load. It was previously named {{IgniteDataLoader}}, see ticket
> IGNITE-394.
> See [Apache Kafka|http://kafka.apache.org/] for more information.
> We should create {{IgniteKafkaStreamer}} which will consume messages from
> Kafka and stream them into Ignite caches.
> More details to follow, but to the least we should be able to:
> - Convert Kafka messages to Ignite data using an optional pluggable converter.
> - Specify the cache name for the Ignite cache to load data into.
> - Specify other flags available on {{IgniteDataStreamer}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
