[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498611#comment-16498611 ]
ASF GitHub Bot commented on METRON-1594: ---------------------------------------- Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1045 I noticed that we should probably use a timeout when we call `KafkaProducer.close`. If we get a huge backlog of messages, it will just block until the backlog is cleared. We can get in situations where we build-up such a large backlog that there just is no hope of clearing it. This might actually explain why some of the topologies just won't die sometimes. > KafkaWriter is asynchronous and may lose data on node failure > ------------------------------------------------------------- > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug > Reporter: Michael Miklavcic > Assignee: Michael Miklavcic > Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)