Mark Payne created NIFI-2851: -------------------------------- Summary: Improve performance of SplitText Key: NIFI-2851 URL: https://issues.apache.org/jira/browse/NIFI-2851 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.1.0
SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 1.4 million line text file into 5k line chunks and then splits those 5k line chunks into 1 line chunks is only capable of pushing through about 10k lines per second. This equates to about 10 MB/sec. JVisualVM shows that the majority of the time is spent in the locateSplitPoint() method. Isolating this code and inspecting how it works, and using some micro-benchmarking, it appears that if we refactor the calls to InputStream.read() to instead read into a byte array, we can improve performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)