Nifi throws an error when reading a large csv file

2021-04-13 Thread Vibhath Ileperuma
Hi All, I'm using a SplitRecord processor with a CSV Reader and a CSV RecordSetWriter to split a large csv file (5.5GB-6GB) into multiple small csv files. When I start the processor, the below exception is thrown. "failed to process session due to Requested array size exceeds VM limit; Processor

Release from holding queue after timeout

2021-04-13 Thread Jeremy Pemberton-Pigott
Hi, I have a parallel update process in 2 different flows running in a 3 node cluster running 1.6.0, if the insert side has not completed yet the update side moves the flow file to a waiting queue for retry. I want to retry every 20s all the flow files in the queue that have been waiting that lon

Re: NiFi 1.11.4 Custom Processor Development

2021-04-13 Thread Russell Bateman
I'm sorry. I mistyped (numbers soup). I meant to say NiFi 1.2.0, a version several years ago already. On 4/13/21 9:04 AM, Russell Bateman wrote: There shouldn't be any problem. I have many custom processors in a NAR I haven't rebuilt since 1.12 and we use them successfully. They're just missin

Re: NiFi 1.11.4 Custom Processor Development

2021-04-13 Thread Joe Witt
Nathan Nothing has occurred that should result in issues with using custom processors. The challenge comes in for those trying to build on older lines as bintray ages out and as tools like Maven (rightfully) push to https only based repositories. Thanks On Tue, Apr 13, 2021 at 8:04 AM Russell B

Re: NiFi 1.11.4 Custom Processor Development

2021-04-13 Thread Russell Bateman
There shouldn't be any problem. I have many custom processors in a NAR I haven't rebuilt since 1.12 and we use them successfully. They're just missing versions because we had not yet upgraded the Maven we used to build the NAR (not exactly relevant, I know, but I thought I'd I would point this

Re: NiFi 1.11.4 Custom Processor Development

2021-04-13 Thread nathan.english
Hi Joe, Thanks for that information. Will there be any issue in running a Custom NAR file built against 1.9.2 on a different versioned cluster. We haven’t experienced anything yet, but would hate to spend ages in debugging an issue that is the result of this! Kind Regards, Nathan

Re: NiFi 1.11.4 Custom Processor Development

2021-04-13 Thread Joe Witt
Hello In moving to support maven 3.8 we found only https based repos are allowed by default. In addressing this we updated the build to move away from bintray and another location as well. This is now the case when you build master. You could try to cherry pick and edit the key commits or hang

NiFi 1.11.4 Custom Processor Development

2021-04-13 Thread nathan.english
Hi There, I am in the process of upgrading the version of the nifi-nar-bundle from 1.9.2 to 1.12.1 however I’ve hit a unexpected issue. We develop on an offline platform, with Maven Central Mirrored using a Nexus Repository Manager, with all dependencies delivered through this mechanism. Today