I should add that if you're comfortable with that sort of volatile behavior a better path to consider is to setup a RAM-Disk and just run a persistent content repository on that. It will survive process restarts, give better memory/heap behavior (by a lot), but you'll lose data on system restarts.
Thanks Joe On Thu, Oct 27, 2016 at 11:58 AM, Joe Witt <joe.w...@gmail.com> wrote: > That is correct. > > Thanks > Joe > > On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota <jfarb...@payoff.com> > wrote: > >> Bryan, >> >> If I have the content repo implementation set to >> org.apache.nifi.controller.repository.VolatileContentRepository, it will >> stream the content in memory, correct? >> >> On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende <bbe...@gmail.com> wrote: >> >>> Monica, >>> >>> Are you asking what does NiFi do when it picks up a large file from the >>> filesystem using a processor like GetFile? >>> >>> If so, it will stream the content of that file into NiFi's content >>> repository, and create a FlowFile pointing to that content. As far as NiFi >>> is concerned the content is just bytes at this point and has not been >>> changed in anyway from the original file. >>> >>> The content is not held in memory, and the FlowFile can move through >>> many processors without ever accessing the content, unless the processor >>> needs to, and then when accessing the content it is typically done in a >>> streaming fashion (when possible) to avoid loading the large content into >>> memory. >>> >>> There are processors that can then split up the content based on >>> specific data formats, for example SplitText, SplitJSON, SplitAvro, etc.. >>> but it is up to the designer of the flow to do that. >>> >>> -Bryan >>> >>> >>> On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini < >>> monica.francesch...@eng.it> wrote: >>> >>>> Hi, >>>> I'm figuring out how does Nifi ingest large files: does it split them >>>> into chunks or is it a massive load?Can you please, explain the behavior? >>>> Kind regards, >>>> Monica >>>> -- >>>> >>>> *Monica Franceschini* >>>> Solution Architecture Manager >>>> >>>> *Big Data Competence Center Engineering Group* >>>> Corso Stati Uniti 23/C, 35127 Padova, Italia >>>> Tel: +39 049.8283547 >>>> Fax: +39 049.8692566 >>>> Twitter: @twittmonique >>>> www.spagobi.org - www.eng.it <http://www.eng.it/web/eng_en/home> *proud >>>> SpagoBI supporter and contributor* >>>> [image: SpagoBI] >>>> Respect the environment. Please don't print this e-mail unless you >>>> really need to. >>>> >>>> The information transmitted is intended only for the person or entity >>>> to which it is addressed and may contain confidential and/or privileged >>>> material. Any review, retransmission, dissemination or other use of, or >>>> taking of any action in reliance upon, this information by persons or >>>> entities other than the intended recipient is prohibited. If you received >>>> this in error, please contact the sender and delete the material from any >>>> computer. >>>> >>> >>> >> >> >> -- >> >> [image: Payoff, Inc.] <http://www.payoff.com/> >> >> Jeremy Farbota >> Software Engineer, Data >> jfarb...@payoff.com <em...@payoff.com> • (217) 898-8110 <(949)+430-0630> >> >> I'm a Storyteller. Discover your Financial Personality! >> <https://www.payoff.com/quiz> >> >> [image: Facebook] <https://www.facebook.com/payoff> [image: Twitter] >> <https://www.twitter.com/payoff> [image: Linkedin] >> <https://www.linkedin.com/company/payoff-com> >> > >
