[
https://issues.apache.org/jira/browse/IGNITE-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215398#comment-15215398
]
Denis Magda commented on IGNITE-2730:
-------------------------------------
Roman,
_Does it mean duplicate events will be ignored when we have a primary node
check? Or does it mean that with a primary node check events can be lost? If we
receive events both from primary and backup nodes, are there chances that
events can be lost?_
By default a *cache* event is sent to both primary and all backup nodes. This
will lead to the situation that the same event is sent to Kafka.
To avoid this the following code snippet can be added to the remote filter
{noformat}
Affinity affinity = ignite.affinity("eventsCache");
ClusterNode localNode = ignite.cluster().localNode();
if (affinity.isPrimary(localNode, event.key())) {
// Process this event. Will be ignored on backups.
}
{noformat}
When I say that an event may be missed is if to apply the code snippet above
and then a primary node can fail somewhere in the middle - before doing the
check or somewhere in the "if(affinity.)" block and this will lead to the
situation that the event won't be delivered to Kafka.
_batching is needed to efficiently send data to Kafka. If we batch it in memory
locally, the accumulated data can be lost if something goes wrong with the
IgniteSourceTask. Therefore I chose to use IgniteQueue to make sure the data is
safe._
>From what I see if something goes wrong with the IgniteSourceTask then data
>batched in IgniteQueue would be useless because no other task will poll it and
>send to Kafka.
It means that you can use the local listener that is triggered if the remote
filter returns "true" and from the local listener body you can add an event to
a local queue to support batching.
_2. Connector requires a de/serializer (Please see makeWorkerProps()). I can
implement it but JdkMarshaller has already the functionality._
I see. Can't you use Ignite default marshaller for this task -
ignite.configuration().getMarshaller()?
> Ignite Events Source Streaming to Kafka
> ---------------------------------------
>
> Key: IGNITE-2730
> URL: https://issues.apache.org/jira/browse/IGNITE-2730
> Project: Ignite
> Issue Type: New Feature
> Components: streaming
> Reporter: Roman Shtykh
> Assignee: Roman Shtykh
> Labels: community
>
> Streaming specified Ignite events
> (https://apacheignite.readme.io/docs/events) to Kafka via Kafka Connect.
> It has to be added to org.apache.ignite.stream.kafka.connect package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
