[ https://issues.apache.org/jira/browse/NIFI-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027799#comment-15027799 ]
Joseph Witt commented on NIFI-1221: ----------------------------------- will try to review this tonight. The performance difference will be quite important for some use cases. Going from 23K to 100K + persistent messages per second is helpful. > ListenSyslog should support batching > ------------------------------------ > > Key: NIFI-1221 > URL: https://issues.apache.org/jira/browse/NIFI-1221 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions > Reporter: Mark Payne > Assignee: Mark Payne > Fix For: 0.4.0 > > Attachments: 0001-NIFI-1221-Support-batching-of-Syslog-messages.patch > > > Currently, performance of ListenSyslog is pretty reasonable. If I configure > logging to WARN level, and I use a Yield Duration of 0 ms, 3 threads, 25 ms > run duration, then I can push about 23,000 messages per second to a single > NiFi node without any loss, with message parsing enabled. > However, I think we can do a lot better than that. Since these Syslog events > are just log messages, they lend themselves very well to concatenation. We > should have a Max Batch Size property as well as a Message Delimiter property. > If using batch, though, it's important that we do not allow the Parse > Messages property to be true, since it doesn't really make sense to add > attributes if we have multiple messages. > However, since we cannot parse the messages if bundled together, we should > have a separate ParseSyslog processor that does parse them. This way, we can > route specific events to a ParseSyslog processor. For instance, using > RouteText to pull out events of interest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)