[ANNOUNCE] Apache NiFi 1.21.0 release.

2023-04-07 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.21.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation,

Re: Handling CSVs dynamically with NiFi

2023-04-07 Thread James McMahon
Hello Bryan. Thank you for your question. A downstream consumer requires the complete set in json. So that's part of why I convert. Other downstream tools require json input, but not the entire set. The data needs to be routed based on certain features. Geographic location. Sales data by zip

Re: Handling CSVs dynamically with NiFi

2023-04-07 Thread Bryan Bende
James, I'm not sure what the end goal is, but why do you need to use EvaluateJsonPath and SplitJson? Generally you don't want to split a flow file of multiple records into 1 record per flow file, this is an anti-pattern that leads to poor performance in the flow. Thanks, Bryan On Fri, Apr 7,

Re: Handling CSVs dynamically with NiFi

2023-04-07 Thread James McMahon
Very interesting, very helpful insights. Thank you again, Mike. Late last night I decided to punt on a pure NiFi solution. I knew I could do this easily with Groovy scripting, and I knew that was well-within my wheelhouse. So that's what I did: Groovy from an ExecuteScript processor. I'm 90% of

RE: Handling CSVs dynamically with NiFi

2023-04-07 Thread Mike Sofen
This is where I felt Nifi wasn’t the right tool for the job and Postgres was. After I imported the CSV directly into a staging table in the database (using Nifi), I converted the payload part of the columns into jsonb and stored that into the final table in a column with additional columns as