Iteratively concatenate record fields?

2020-07-22 Thread Mike Thomsen
Here's our use case: Schema is - addresses:list - street:string - city:string ... - clean_addresses:list Something like this using pseudo recordpath: foreach(/addresses[*], concat(./street, ' ', ./city, ...)) => /clean_addresses Can't seem to get that iterative element in there. Is

Re: Monitoring FlowFiles - Lineage Duration

2020-07-22 Thread Pierre Villard
Hi, Both [1] and [2] could give you some ideas based on your needs and what you're using as external systems for monitoring. Overall we're working on making NiFi better on everything around monitoring, but most of the time, for specific requirements I advocate for Monitoring Driven Development

Monitoring FlowFiles - Lineage Duration

2020-07-22 Thread Firlefanz Development
Hello fellow NiFi users, we have some fairly large NiFi flows and would like to monitor whether there are FlowFiles stucked, e.g. because of failing processors or being stuck in a loop. Is there a way to find any FlowFiles being "stuck" in the Flow for longer than a time period of X, e.g. an

IBM MQ message selector

2020-07-22 Thread Marc Recht
Hi, I try to use a message selector with NiFi and IBM MQ and the ConsumeJMS processor. Unfortunately, I am not sure how... Is there an example available somewhere how to set this up? Any help appreciated, Marc

Re: NiFi Registry Bundles - Integration into NiFi

2020-07-22 Thread Firlefanz Development
Thank you Bryan and Juan Pablo for the quick and very helpful answers! I'm looking forward to the future changes regarding the seamless integration of NAR-files in the NiFi registry with NiFi. Until then I'll stick to my current artifact repository and deployment pipeline. Thanks once more!

Re: MiNiFi version

2020-07-22 Thread Pierre Villard
I don't recall the specific details but when I used MQTT processors [1] with latest version of MiNiFi Java, I *think* I could not use the latest version of NiFi NARs, I had to use code from a previous NiFi version and cherry-pick some code (open pull request) I needed for my use case [2]. Having

Re: Urgent: HDFS processors throwing OOM - Compressed class space exception

2020-07-22 Thread Joe Witt
The files arent read into memory anyway. The class space filling up is what to focus on. On Wed, Jul 22, 2020 at 2:34 AM Mohit Jain wrote: > Those are the small files not more than 100 MB each. > Nifi is configured to 16g. > > Thanks > > Get Outlook for iOS >

Re: Urgent: HDFS processors throwing OOM - Compressed class space exception

2020-07-22 Thread Mohit Jain
Those are the small files not more than 100 MB each. Nifi is configured to 16g. Thanks Get Outlook for iOS From: Jorge Machado Sent: Wednesday, July 22, 2020 2:55:26 PM To: users@nifi.apache.org Subject: Re: Urgent: HDFS processors

Re: Urgent: HDFS processors throwing OOM - Compressed class space exception

2020-07-22 Thread Jorge Machado
How big are the files that you are trying to store? How much memory did you configure nifi ? > On 22. Jul 2020, at 06:13, Mohit Jain wrote: > > Hi team, > > I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS > throws the error - > Failed to write to HDFS due to