Ryan, Elli, Do you by chance have any error messages in your logs from the FileSystemRepository?
I.e., if you perform: grep FileSystemRepository logs/* Do you get anything interesting in there? Thanks -Mark > On Oct 23, 2015, at 2:38 PM, Elli Schwarz <eliezer_schw...@yahoo.com.INVALID> > wrote: > > I've been working with Ryan. There appear to be a few issues here: > > - We upgraded from 0.2.0 to 0.3.0 and it appears that content_repository > archive is now true by default. In 0.2.0 it was false, and the documentation > still states it is false by default. > - When we ran out of disk space overnight, the problem was solved by me > simply restarting nifi, and that cleared out the archive by itself. > > - In order to clear up the archive, I had to set archive to true, and set > max usage to 1%, and restart nifi. That cleared it up, and then I set archive > to false and restarted again so we don't run out of space. > - Based on the above, it appears that something happened yesterday that > prevented Nifi from clearing out the archive even though disk usage reached > 100%. However, restarting nifi apparently enabled it to perform the clearing > of the archive. So apparently the max usage setting doesn't work under some > conditions, but we don't know what conditions occurred overnight to cause > this problem. > > Thanks!-Elli > > > > On Friday, October 23, 2015 2:29 PM, Ryan H <rhendrickson.w...@gmail.com> > wrote: > > > > Agree, they concern the archive... although it sounds like there are 2 > archives? > > Within the content_repository folder, there are subfolders with the name > 'archive' and files inside them. > > Example: > ./nfii/content_repository/837/archive/1445611320767-837 > > Settings: > nifi.content.repository.archive.max.retention.period=12 hours > nifi.content.repository.archive.max.usage.percentage=50% > nifi.content.repository.archive.enabled=true > > Last night, our server ran out of disk space because the content_repository > grew too large. Nifi didn't crash, but the log file contained errors > saying the disk was full. > > We're not sure how, but the content_repository did not respect the above > settings. > > We restarted Nifi, and it only then started to remove files, such as: > ./nfii/content_repository/837/archive/1445611320767-837 > > We've turned off archiving for now. > > Ryan > > > > > On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinp...@gmail.com> wrote: > >> Ryan, >> >> Those items only concern the archive. Did you have data enqueued in >> connections in your flow? If so, these items are not eligible and could >> explain why your disk was filled. Otherwise, can you please provide some >> additional information so we can dig into why this may have arisen. >> >> Thanks! >> >> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rhendrickson.w...@gmail.com> >> wrote: >> >>> I've got the following set: >>> >>> nifi.content.repository.archive.max.retention.period=12 hours >>> nifi.content.repository.archive.max.usage.percentage=50% >>> nifi.content.repository.archive.enabled=true >>> >>> Yet, the content repo filled my disk last night... >>> >>> >>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinp...@gmail.com> >> wrote: >>> >>>> Ryan, >>>> >>>> Those archive folders map to the >> nifi.content.repository.archive.enabled >>>> property. >>>> >>>> What this property provides is a retention of files no longer in the >>> system >>>> for historical context of your flow's processing and the ability for >>>> viewing this in conjunction with provenance events as well as allowing >>>> replay. The amount of the archive when enabled is bounded by the >>>> properties nifi.content.repository.archive.max.retention.period and >>>> nifi.content.repository.archive.max.usage.percentage. >>>> >>>> Additional detail is available in the system properties of our >>>> Administration Guide [1] >>>> >>>> Let us know if you have additional questions. >>>> >>>> --aldrin >>>> >>>> [1] >>>> >>>> >>> >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties >>>> >>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.w...@gmail.com> >>>> wrote: >>>> >>>>> Interesting.. So what would >>>>> >>>>> ./nfii/content_repository/837/archive/1445611320767-837 >>>>> >>>>> typically be? >>>>> >>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande < >>> agra...@hortonworks.com >>>>> >>>>> wrote: >>>>> >>>>>> Attachments don't go through, view at imagebin: >>>>>> http://ibin.co/2K3SwR0z8yWX >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <agra...@hortonworks.com> >>>> wrote: >>>>>> >>>>>>> Ryan, >>>>>>> >>>>>>> ./conf/archive is to create a snapshot of your entire flow, not >> the >>>>>> content repository data. See the attached screenshot (Settings menu >>> on >>>>> the >>>>>> right). >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrick...@gmail.com on >> behalf >>>> of >>>>>> Ryan H" <ryan.andrew.hendrick...@gmail.com on behalf of >>>>>> rhendrickson.w...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> I'm noticing my Content Repo growing large. There's a number >> of >>>>>> files... >>>>>>>> >>>>>>>> content_repo/837/archive/144...-837 >>>>>>>> >>>>>>>> Is this new in 3.0? My conf file says any archiving should be >>>> going >>>>>>>> into ./conf/archive, but i don't see anything in there. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ryan >>>>>> >>>>> >>>> >>> >> > > >
