I've been working with Ryan. There appear to be a few issues here: - We upgraded from 0.2.0 to 0.3.0 and it appears that content_repository archive is now true by default. In 0.2.0 it was false, and the documentation still states it is false by default. - When we ran out of disk space overnight, the problem was solved by me simply restarting nifi, and that cleared out the archive by itself.
- In order to clear up the archive, I had to set archive to true, and set max usage to 1%, and restart nifi. That cleared it up, and then I set archive to false and restarted again so we don't run out of space. - Based on the above, it appears that something happened yesterday that prevented Nifi from clearing out the archive even though disk usage reached 100%. However, restarting nifi apparently enabled it to perform the clearing of the archive. So apparently the max usage setting doesn't work under some conditions, but we don't know what conditions occurred overnight to cause this problem. Thanks!-Elli On Friday, October 23, 2015 2:29 PM, Ryan H <rhendrickson.w...@gmail.com> wrote: Agree, they concern the archive... although it sounds like there are 2 archives? Within the content_repository folder, there are subfolders with the name 'archive' and files inside them. Example: ./nfii/content_repository/837/archive/1445611320767-837 Settings: nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true Last night, our server ran out of disk space because the content_repository grew too large. Nifi didn't crash, but the log file contained errors saying the disk was full. We're not sure how, but the content_repository did not respect the above settings. We restarted Nifi, and it only then started to remove files, such as: ./nfii/content_repository/837/archive/1445611320767-837 We've turned off archiving for now. Ryan On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinp...@gmail.com> wrote: > Ryan, > > Those items only concern the archive. Did you have data enqueued in > connections in your flow? If so, these items are not eligible and could > explain why your disk was filled. Otherwise, can you please provide some > additional information so we can dig into why this may have arisen. > > Thanks! > > On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rhendrickson.w...@gmail.com> > wrote: > > > I've got the following set: > > > > nifi.content.repository.archive.max.retention.period=12 hours > > nifi.content.repository.archive.max.usage.percentage=50% > > nifi.content.repository.archive.enabled=true > > > > Yet, the content repo filled my disk last night... > > > > > > On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinp...@gmail.com> > wrote: > > > > > Ryan, > > > > > > Those archive folders map to the > nifi.content.repository.archive.enabled > > > property. > > > > > > What this property provides is a retention of files no longer in the > > system > > > for historical context of your flow's processing and the ability for > > > viewing this in conjunction with provenance events as well as allowing > > > replay. The amount of the archive when enabled is bounded by the > > > properties nifi.content.repository.archive.max.retention.period and > > > nifi.content.repository.archive.max.usage.percentage. > > > > > > Additional detail is available in the system properties of our > > > Administration Guide [1] > > > > > > Let us know if you have additional questions. > > > > > > --aldrin > > > > > > [1] > > > > > > > > > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties > > > > > > On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.w...@gmail.com> > > > wrote: > > > > > > > Interesting.. So what would > > > > > > > > ./nfii/content_repository/837/archive/1445611320767-837 > > > > > > > > typically be? > > > > > > > > On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande < > > agra...@hortonworks.com > > > > > > > > wrote: > > > > > > > > > Attachments don't go through, view at imagebin: > > > > > http://ibin.co/2K3SwR0z8yWX > > > > > > > > > > > > > > > > > > > > > > > > > On 10/23/15, 12:52 PM, "Andrew Grande" <agra...@hortonworks.com> > > > wrote: > > > > > > > > > > >Ryan, > > > > > > > > > > > >./conf/archive is to create a snapshot of your entire flow, not > the > > > > > content repository data. See the attached screenshot (Settings menu > > on > > > > the > > > > > right). > > > > > > > > > > > >Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On 10/23/15, 12:47 PM, "ryan.andrew.hendrick...@gmail.com on > behalf > > > of > > > > > Ryan H" <ryan.andrew.hendrick...@gmail.com on behalf of > > > > > rhendrickson.w...@gmail.com> wrote: > > > > > > > > > > > >>Hi, > > > > > >> I'm noticing my Content Repo growing large. There's a number > of > > > > > files... > > > > > >> > > > > > >>content_repo/837/archive/144...-837 > > > > > >> > > > > > >> Is this new in 3.0? My conf file says any archiving should be > > > going > > > > > >>into ./conf/archive, but i don't see anything in there. > > > > > >> > > > > > >>Thanks, > > > > > >>Ryan > > > > > > > > > > > > > > >