I could be totally barking up the wrong tree, but I think this is our clue: Requested array size exceeds VM limit
That means that something is causing the reader to try to allocate an array with a number of entries greater than the VM allows. Without seeing the schema, a sample of the CSV and a stacktrace it's pretty hard to guess what's going on. For what it's worth, I've split 55GB JSON sets using a custom streaming JSON reader without a hiccup on a NiFi instance with only 4-8GB of RAM allocated, so I'm fairly confident we've got some quirky edge case here. If you want to sanitize some inputs and share along with a schema that might help. On Wed, Apr 14, 2021 at 1:07 PM Vibhath Ileperuma <vibhatharunapr...@gmail.com> wrote: > > Hi Chris, > > As you have mentioned, I am trying to split the large csv file in multiple > stages. But this error is thrown at the first stage even without creating a > single flow file. > It seems like the issue is not with the processor, but with the CSV record > reader. This error is thrown while reading the csv file. I tried to write the > data in the large csv file into a kudu table using a putKudu processor with > the same CSV reader. Then also I got the same error message. > > Hi Otto, > > Only following information is available in log file related to the exception > > 2021-04-14 17:48:28,628 ERROR [Timer-Driven Process Thread-1] > o.a.nifi.processors.standard.SplitRecord > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] failed to process > session due to java.lang.OutOfMemoryError: Requested array size exceeds VM > limit; Processor Administratively Yielded for 1 sec: > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > 2021-04-14 17:48:28,628 WARN [Timer-Driven Process Thread-1] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] due to uncaught > Exception: java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > Thanks & Regards > > Vibhath Ileperuma > > > > > On Wed, Apr 14, 2021 at 7:47 PM Otto Fowler <ottobackwa...@gmail.com> wrote: >> >> What is the complete stack trace of that exception? >> >> On Apr 14, 2021, at 02:36, Vibhath Ileperuma <vibhatharunapr...@gmail.com> >> wrote: >> >> Requested array size exceeds VM limit >> >>
