Monica, Are you asking what does NiFi do when it picks up a large file from the filesystem using a processor like GetFile?
If so, it will stream the content of that file into NiFi's content repository, and create a FlowFile pointing to that content. As far as NiFi is concerned the content is just bytes at this point and has not been changed in anyway from the original file. The content is not held in memory, and the FlowFile can move through many processors without ever accessing the content, unless the processor needs to, and then when accessing the content it is typically done in a streaming fashion (when possible) to avoid loading the large content into memory. There are processors that can then split up the content based on specific data formats, for example SplitText, SplitJSON, SplitAvro, etc.. but it is up to the designer of the flow to do that. -Bryan On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini < [email protected]> wrote: > Hi, > I'm figuring out how does Nifi ingest large files: does it split them into > chunks or is it a massive load?Can you please, explain the behavior? > Kind regards, > Monica > -- > > *Monica Franceschini* > Solution Architecture Manager > > *Big Data Competence Center Engineering Group* > Corso Stati Uniti 23/C, 35127 Padova, Italia > Tel: +39 049.8283547 > Fax: +39 049.8692566 > Twitter: @twittmonique > www.spagobi.org - www.eng.it <http://www.eng.it/web/eng_en/home> *proud > SpagoBI supporter and contributor* > [image: SpagoBI] > Respect the environment. Please don't print this e-mail unless you > really need to. > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. >
