[ https://issues.apache.org/jira/browse/IGNITE-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215398#comment-15215398 ]
Denis Magda commented on IGNITE-2730: ------------------------------------- Roman, _Does it mean duplicate events will be ignored when we have a primary node check? Or does it mean that with a primary node check events can be lost? If we receive events both from primary and backup nodes, are there chances that events can be lost?_ By default a *cache* event is sent to both primary and all backup nodes. This will lead to the situation that the same event is sent to Kafka. To avoid this the following code snippet can be added to the remote filter {noformat} Affinity affinity = ignite.affinity("eventsCache"); ClusterNode localNode = ignite.cluster().localNode(); if (affinity.isPrimary(localNode, event.key())) { // Process this event. Will be ignored on backups. } {noformat} When I say that an event may be missed is if to apply the code snippet above and then a primary node can fail somewhere in the middle - before doing the check or somewhere in the "if(affinity.)" block and this will lead to the situation that the event won't be delivered to Kafka. _batching is needed to efficiently send data to Kafka. If we batch it in memory locally, the accumulated data can be lost if something goes wrong with the IgniteSourceTask. Therefore I chose to use IgniteQueue to make sure the data is safe._ >From what I see if something goes wrong with the IgniteSourceTask then data >batched in IgniteQueue would be useless because no other task will poll it and >send to Kafka. It means that you can use the local listener that is triggered if the remote filter returns "true" and from the local listener body you can add an event to a local queue to support batching. _2. Connector requires a de/serializer (Please see makeWorkerProps()). I can implement it but JdkMarshaller has already the functionality._ I see. Can't you use Ignite default marshaller for this task - ignite.configuration().getMarshaller()? > Ignite Events Source Streaming to Kafka > --------------------------------------- > > Key: IGNITE-2730 > URL: https://issues.apache.org/jira/browse/IGNITE-2730 > Project: Ignite > Issue Type: New Feature > Components: streaming > Reporter: Roman Shtykh > Assignee: Roman Shtykh > Labels: community > > Streaming specified Ignite events > (https://apacheignite.readme.io/docs/events) to Kafka via Kafka Connect. > It has to be added to org.apache.ignite.stream.kafka.connect package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)