How large is each line expected to be? You could have a massive line or much larger than thought of. Or you could be creating far more flowfiles than intended. If you cut the file in size does it work better? Will need more data to help narrow in but obviously we're all very interested to know what is happening. These processors and the readers/writers are meant to be quite bullet proof and handle very very large data easily in most cases.
On Wed, Apr 14, 2021 at 10:07 AM Vibhath Ileperuma <vibhatharunapr...@gmail.com> wrote: > > Hi Chris, > > As you have mentioned, I am trying to split the large csv file in multiple > stages. But this error is thrown at the first stage even without creating a > single flow file. > It seems like the issue is not with the processor, but with the CSV record > reader. This error is thrown while reading the csv file. I tried to write the > data in the large csv file into a kudu table using a putKudu processor with > the same CSV reader. Then also I got the same error message. > > Hi Otto, > > Only following information is available in log file related to the exception > > 2021-04-14 17:48:28,628 ERROR [Timer-Driven Process Thread-1] > o.a.nifi.processors.standard.SplitRecord > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] failed to process > session due to java.lang.OutOfMemoryError: Requested array size exceeds VM > limit; Processor Administratively Yielded for 1 sec: > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > 2021-04-14 17:48:28,628 WARN [Timer-Driven Process Thread-1] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] due to uncaught > Exception: java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > Thanks & Regards > > Vibhath Ileperuma > > > > > On Wed, Apr 14, 2021 at 7:47 PM Otto Fowler <ottobackwa...@gmail.com> wrote: >> >> What is the complete stack trace of that exception? >> >> On Apr 14, 2021, at 02:36, Vibhath Ileperuma <vibhatharunapr...@gmail.com> >> wrote: >> >> Requested array size exceeds VM limit >> >>
