Ryan What version are you using? I do think we had an issue that kept items around longer than intended that has been addressed.
Thanks On Thu, Sep 17, 2020 at 7:58 AM Ryan Hendrickson < ryan.andrew.hendrick...@gmail.com> wrote: > Hello, > I've got ~15 million FlowFiles, each roughly 4KB, totally in about 55GB of > data on my canvas. > > However, the content repository (on it's own partition) is completely full > with 350GB of data. I'm pretty certain the way Content Claims store the > data is responsible for this. In previous experience, we've had files that > are larger, and haven't seen this as much. > > My guess is that as data was streaming through and being added to a claim, > it isn't always released as the small files leaves the canvas. > > We've run into this issue enough times that I figure there's probably a > "best practice for small files" for the content claims settings. > > These are our current settings: > > nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository > nifi.content.claim.max.appendable.size=1 MB > nifi.content.claim.max.flow.files=100 > nifi.content.repository.directory.default=/var/nifi/repositories/content > nifi.content.repository.archive.max.retention.period=12 hours > nifi.content.repository.archive.max.usage.percentage=50% > nifi.content.repository.archive.enabled=true > nifi.content.repository.always.sync=false > > > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#content-repository > > > There's 1024 folders on the disk (0-1023) for the Content Claims. > Each file inside the folders are roughly 2MB to 8 MB (Which is odd > because I thought the max appendable size would make this no larger than > 1MB.) > > Is there a way to expand the number of folders and/or reduce the amount of > individual FlowFiles that are stored in the claims? > > I'm hoping there might be a best practice out there though. > > Thanks, > Ryan > >
