[
https://issues.apache.org/jira/browse/NIFI-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565958#comment-16565958
]
ASF GitHub Bot commented on NIFI-3531:
--------------------------------------
Github user mosermw commented on the issue:
https://github.com/apache/nifi/pull/2931
The JIRA ticket mentioned a dev list conversation where Oleg said it was to
limit the possibility of message loss in the event of a processor and/or NiFi
crash. However, it has proven to cause problems. I have tried to kill -9 NiFi
several times while ConsumeJMS is under heavy load and haven't lost a message,
though perhaps I've just been lucky. The javadoc of the session.recover method
doesn't seem to indicate that it should be called regularly.
The decision regarding what to do with this PR might become a risk
management one, where the benefits of session.recover as it is now are unknown
but the problems associated with it are known and significant.
> modify session.recover behaviour in nifi-jms-processors to cope with
> high-traffic JMS
> -------------------------------------------------------------------------------------
>
> Key: NIFI-3531
> URL: https://issues.apache.org/jira/browse/NIFI-3531
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Dominik Benz
> Assignee: Michael Moser
> Priority: Major
>
> As described in this mailing list post
> http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-td14940.html
> the current implementation of nifi-jms-processor, especially of JMSConsumer
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSConsumer.java
> causes frequent re-delivery of JMS messages due to the following behaviour:
> 1) several threads perform the JMS session callback in parallel
> 2) each callback performs a session.recover
> 3) during high traffic, the situation arises that the ACKs from another
> thread may not (yet) have arrived at the JMS server
> 4) this implies that the pointer of session.recover will reconsume the
> not-yet-acked message from another thread
> I understood Nifi prefers message duplication over message loss - however, in
> our case the number of re-delivered messages is "piling up" over time,
> leading to a growing number of un-acked messages in JMS and finally to
> storage problems on the JMS server side.
> It would be great to modify the nifi-jms-processor package to reliably cope
> with higher-traffic JMS sources. Ideas from my side would be / include:
> 1) make synchronous / asynchronous delivery configurable
> 2) add configuration option for custom ACKing modes (i.e. non-standard JMS
> modes like individual ACKing or NO_ACK)
> 3) add configuration option for JMS message selectors
> From other projects, I found the JMS communication options in this
> spark-jms-receiver package
> https://github.com/tbfenet/spark-jms-receiver
> very helpful (in which situations are synchronous / asynchronous approaches
> reliable, how to buffer messages before acking them, how/when to ack, ...).
> Here's also a java port of a subset:
> https://github.com/bernhardschaefer/spark-jms-receiver-java
> For any mentioned option I'm also happy to help!
> Best & thanks for a great piece of software,
> Dominik
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
