[ https://issues.apache.org/jira/browse/NIFI-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15031845#comment-15031845 ]
Mark Payne commented on NIFI-1221: ---------------------------------- [~bbende] - you're right - we should add some logic to handle the case of an Exception being thrown when calling session.write / session.append. I think I lost that during the refactoring because i didn't understand what the intent was there, since it was catching ProcessException (which isn't thrown in the case of a failure to session.write/append). We should add that back in and catch the general Exception when calling write/append, and then handle accordingly. Good catch! > ListenSyslog should support batching > ------------------------------------ > > Key: NIFI-1221 > URL: https://issues.apache.org/jira/browse/NIFI-1221 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions > Reporter: Mark Payne > Assignee: Bryan Bende > Fix For: 0.4.0 > > Attachments: > 0001-NIFI-1221-Support-batching-of-Syslog-messages.patch, > 0001-NIFI-1221-Updating-unit-tests.patch, > 0001-NIFI-1221-fixing-test-failures-contrib-failed-line-m.patch > > > Currently, performance of ListenSyslog is pretty reasonable. If I configure > logging to WARN level, and I use a Yield Duration of 0 ms, 3 threads, 25 ms > run duration, then I can push about 23,000 messages per second to a single > NiFi node without any loss, with message parsing enabled. > However, I think we can do a lot better than that. Since these Syslog events > are just log messages, they lend themselves very well to concatenation. We > should have a Max Batch Size property as well as a Message Delimiter property. > If using batch, though, it's important that we do not allow the Parse > Messages property to be true, since it doesn't really make sense to add > attributes if we have multiple messages. > However, since we cannot parse the messages if bundled together, we should > have a separate ParseSyslog processor that does parse them. This way, we can > route specific events to a ParseSyslog processor. For instance, using > RouteText to pull out events of interest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)