All, This is something I think we shouldn't dismiss so easily. While the FlowFile repo is lighter than the content repo, allowing it to grow too large can cause major problems.
Specifically, an "overgrown" FlowFile repo may prevent a NiFi instance from coming back up after a restart due to the way in which records are held in memory. If there is more memory available to give to the JVM, this can sometimes be worked around... but if there isn't you may just be out of luck. For that matter, allowing the FlowFile repo to grow so large that it consumes all the heap isn't going to be good for system health in general (OOM is probably never where you want to be...). To Pierre's point "you don't want to limit that repository in size since it would prevent the workflows to create new flow files"... that's exactly why I would want to limit the size of the repo. You do then get into questions of how exactly to do this. For example, you may not want to simply block all transactions that create a FlowFile, because it may remove even more (e.g. MergeContent). Additionally, you have to be concerned about deadlocks (e.g. a "Wait" that hangs forever because its "Notify" is being starved). Or, perhaps that's all you can do... freeze everything at some threshold prior to actual damage being done, and alert operators that manual intervention is necessary (e.g. bring up the graph with autoResume=false, and bleed off data in a controlled fashion). In summary, I believe this is a problem. Even if it doesn't come up often, when it does it is significant. While the solution likely isn't simple, it's worth putting some thought towards. Brandon On Wed, Apr 25, 2018 at 9:43 AM Sivaprasanna <sivaprasanna...@gmail.com> wrote: > No, he actually had mentioned “like content repository”. The answer is, > there aren’t any properties that support this, AFAIK. Pierre’s response > pretty much sums up why there aren’t any properties. > > Thanks, > Sivaprasanna > > On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen <mikerthom...@gmail.com> > wrote: > > > I have a feeling that what Ben meant was how to limit the content > > repository size. > > > > On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard < > > pierre.villard...@gmail.com> > > wrote: > > > > > Hi Ben, > > > > > > Since the flow file repository contains the information of the flow > files > > > currently being processed by NiFi, you don't want to limit that > > repository > > > in size since it would prevent the workflows to create new flow files. > > > > > > Besides this repository is very lightweight, I'm not sure it'd need to > be > > > limited in size. > > > Do you have a specific use case in mind? > > > > > > Pierre > > > > > > > > > 2018-04-25 9:15 GMT+02:00 尹文才 <batman...@gmail.com>: > > > > > > > Hi guys, I checked NIFI's system administrator guide trying to find a > > > > configuration item so that the size of the flowfile repository could > be > > > > limited similar to the other repositories(e.g. content repository), > > but I > > > > didn't find such configuration items, is there currently any > > > configuration > > > > to limit the flowfile repository's size? thanks. > > > > > > > > Regards, > > > > Ben > > > > > > > > > >
