[
https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498693#comment-16498693
]
ASF GitHub Bot commented on METRON-1594:
----------------------------------------
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1045
> This might actually explain why some of the topologies just won't die
sometimes.
The more I think about it, this is very likely what is happening in the
current code base when topologies just won't die. The queue builds a giant
backlog, then it will block forever trying to clear the queue.
But with these code changes, we should not carry any messages in the
internal Kafka producer queue between calls to `KafkaWriter.write`. So this
*should* no longer be a problem; me thinks.
> KafkaWriter is asynchronous and may lose data on node failure
> -------------------------------------------------------------
>
> Key: METRON-1594
> URL: https://issues.apache.org/jira/browse/METRON-1594
> Project: Metron
> Issue Type: Bug
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
> Priority: Major
>
> Currently, we do not block for data to be sent from kafka writer and we do
> not batch. Unfortunately, the send call is asynchronous, so if the node dies
> before the data is actually sent from kafka then it will drop data. It is
> highly unlikely that we will be able to make kafkawriter synchronous in a
> performant way, so we will likely need to batch in the parser and enrichment
> topologies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
