Re: Processing multiple lines per flowfile with ExtractGrok

2017-09-27 Thread Adam Lamar
Thanks Joe and Bryan. The setup is little more involved but I was able to get ConvertRecord running with a grok reader and a json writer. And I can confirm that setup splits records as expected by newline. Nice touch to have multiple records contained in the same flow file! Thanks for the tip and e

Re: Processing multiple lines per flowfile with ExtractGrok

2017-09-26 Thread Bryan Bende
Adam, I'm only a little bit familiar with Grok, but the ExtactGrok processor reads the entire content of the flow file into memory and then performs the match with the grok expression against the entire content, so it seems like this processor wasn't intended to perform the match line-by-line. Th

Re: Processing multiple lines per flowfile with ExtractGrok

2017-09-25 Thread Joe Witt
Adam, I'm not very familiar with that specific processor but I think you'll find your case is probably far better handled using the Record reader/writer processors anyway. There is a GrokReader which you can use to read each line of a given input as grok expressions to parse out key fields agains

Processing multiple lines per flowfile with ExtractGrok

2017-09-25 Thread Adam Lamar
Hi there, I've been playing with the ExtractGrok processor and noticed I was missing some data that I expected to be extracted. After some investigation, it seems that ExtractGrok extracts only the first line of the flowfile content, and ignores the rest. Is this expected behavior? I should be ab