I'm creating a processor that will read a customer csv and will create a new flowfile for each line in the form of XML. The CSV file will be quite large (100s of thousands of lines). I would like to commit a reasonable amount from time to time so that they can flow down to other processors. But looking at similar processors SplitText and SplitXml they save up all the created flowfiles and release them all at the end. In some trials, I'm running out of memory doing that. But I can't commit the session early because I'm still reading the original CSV file. Is there a workflow where I can read the incoming CSV flowfile but still release created flowfiles? I'm thinking of not using AbstractProcessor and instead use AbstractSessionFactoryProcessor and create two different sessions but is that advisable or possible?
Devin
