For splitting large files, it's often recommended to use a multi-stage approach.
For example, if your file contains 1_000_000 records and you want to split it into 1 record per FlowFile, it would be better to split into batches of, say 1_000 records using SplitRecord and then split each of those FlowFiles again into files of 1 record each. But bear in mind that this is going to result in 1_000_000 FlowFiles in the Flow, which is unlikely to be very performant. While you may not be trying to split into individual records, the error suggests you're trying to create too many FlowFiles from an incoming file in a single operation. All FlowFiles created by a processor in a single session (i.e. run of the processor) are held in memory until the session is committed - each FlowFile uses an operating file descriptor, so it's common to see OS/VM level errors like these in such scenarios. The general recommendation is to try and use Record-based processors throughout your Flow in order to avoid the need to Split/Merge file content (but this isn't always possible, depending upon your use case and the processors available in your version). --- *Chris Sampson* IT Consultant [email protected] <https://www.naimuri.com/> On Wed, 14 Apr 2021 at 07:36, Vibhath Ileperuma <[email protected]> wrote: > Hi All, > > I'm using a SplitRecord processor with a CSV Reader and a CSV > RecordSetWriter to split a large csv file (5.5GB-6GB) into multiple small > csv files. When I start the processor, the below exception is thrown. > > "failed to process session due to Requested array size exceeds VM limit; > Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: > Requested array size exceeds VM limit" > > > I would be grateful if someone can suggest a way to overcome this error. > > Thanks & Regards > > *Vibhath Ileperuma* >
