It would be good to get the stack trace, or rather a better one to see where the array is being created.
How many columns does the file have? How are you doing the schema? Which csv parser have you configured? Can you reproduce with a smaller file size ( see split command on linux )? > On Apr 14, 2021, at 13:07, Vibhath Ileperuma <[email protected]> > wrote: > > Hi Chris, > > As you have mentioned, I am trying to split the large csv file in multiple > stages. But this error is thrown at the first stage even without creating a > single flow file. > It seems like the issue is not with the processor, but with the CSV record > reader. This error is thrown while reading the csv file. I tried to write the > data in the large csv file into a kudu table using a putKudu processor with > the same CSV reader. Then also I got the same error message. > > Hi Otto, > > Only following information is available in log file related to the exception > > 2021-04-14 17:48:28,628 ERROR [Timer-Driven Process Thread-1] > o.a.nifi.processors.standard.SplitRecord > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] failed to process > session due to java.lang.OutOfMemoryError: Requested array size exceeds VM > limit; Processor Administratively Yielded for 1 sec: > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > 2021-04-14 17:48:28,628 WARN [Timer-Driven Process Thread-1] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] due to uncaught > Exception: java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > Thanks & Regards > Vibhath Ileperuma > > > > On Wed, Apr 14, 2021 at 7:47 PM Otto Fowler <[email protected] > <mailto:[email protected]>> wrote: > What is the complete stack trace of that exception? > >> On Apr 14, 2021, at 02:36, Vibhath Ileperuma <[email protected] >> <mailto:[email protected]>> wrote: >> >> Requested array size exceeds VM limit >
