This is indicating that it's unable to write to the FileSystemRepository - presumably because it is out of disk space.
Any other error messages that you find? I'm wondering specifically if perhaps you received an OutOfMemoryError or anything of that nature? If it's not cleaning up after itself, that would tend to indicate that the cleanup thread is no longer running. So looking for anything that could explain that. Also of note, you're seeing that the Content Repo is out of space, and you also mentioned an issue writing to the Provenance Repository. Was your FlowFile Repository also out of disk space? Looking through the code now I'm seeing a case where if the FlowFile Repository is also out of disk space you may start running into issues with the content repo filling up as well. Thanks -Mark > On Oct 23, 2015, at 3:00 PM, Ryan H <rhendrickson.w...@gmail.com> wrote: > > Random mentions of FileSystemRepo > > nifi-app_2015-10-23_10.0.log: > org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import > data from java.io.ByteArrayInputStream@639fe470 for > StandardFlowFileRecord[uuid=881506e8-c62f-442f-a564-88675e4f0372,claim=,offset=0,name=914750081936080,size=0] > due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable > to create ContentClaim due to java.io.IOException: Failed to write to > FileSystemRepository Stream [StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1445594395635-74075, > container=default, section=347], offset=0, length=-1]] > nifi-app_2015-10-23_10.0.log:Caused by: > org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create > ContentClaim due to java.io.IOException: Failed to write to > FileSystemRepository Stream [StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1445594395635-74075, > container=default, section=347], offset=0, length=-1]] > nifi-app_2015-10-23_10.0.log:Caused by: java.io.IOException: Failed to write > to FileSystemRepository Stream [StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1445594395635-74075, > container=default, section=347], offset=0, length=-1]] > nifi-app_2015-10-23_10.0.log: at > org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:913) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > nifi-app_2015-10-23_10.0.log: at > org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:689) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > nifi-app_2015-10-23_10.0.log: at > org.apache.nifi.controller.repository.FileSystemRepository$2.write(FileSystemRepository.java:910) > ~[nifi-framework-core-0.3.0.jar:0.3.0] > nifi-app_2015-10-23_10.0.log:org.apache.nifi.processor.exception.FlowFileAccessException: > Failed to import data from /opt/nifi/logs/nifi-app_2015-10-23_05.0.log for > StandardFlowFileRecord[uuid=1b365988-1287-4d12-a31c-75c99dfc12ad,claim=,offset=0,name=914754786738219,size=0] > due to java.io.IOException: Failed to write to FileSystemRepository Stream > [StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1445594400001-74079, > container=default, section=351], offset=0, length=-1]] > > > > On Fri, Oct 23, 2015 at 2:59 PM, Ryan H <rhendrickson.w...@gmail.com > <mailto:rhendrickson.w...@gmail.com>> wrote: > Here's our provenance settings: > > # Persistent Provenance Repository Properties > nifi.provenance.repository.directory.default=./provenance_repository > nifi.provenance.repository.max.storage.time=24 hours > nifi.provenance.repository.max.storage.size=1 GB > nifi.provenance.repository.rollover.time=30 secs > nifi.provenance.repository.rollover.size=100 MB > nifi.provenance.repository.query.threads=2 > nifi.provenance.repository.index.threads=1 > nifi.provenance.repository.compress.on.rollover=true > nifi.provenance.repository.always.sync=false > nifi.provenance.repository.journal.count=16 > # Comma-separated list of fields. Fields that are not indexed will not be > searchable. Valid fields are: > # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, > AlternateIdentifierURI, ContentType, Relationship, Details > nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, > ProcessorID, Relationship > # FlowFile Attributes that should be indexed and made searchable > nifi.provenance.repository.indexed.attributes= > # Large values for the shard size will result in more Java heap usage when > searching the Provenance Repository > # but should provide better performance > nifi.provenance.repository.index.shard.size=500 MB > # Indicates the maximum length that a FlowFile attribute can be when > retrieving a Provenance Event from > # the repository. If the length of any attribute exceeds this value, it will > be truncated when the event is retrieved. > nifi.provenance.repository.max.attribute.length=65536 > > # Volatile Provenance Respository Properties > nifi.provenance.repository.buffer.size=100000 > > On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz > <eliezer_schw...@yahoo.com.invalid > <mailto:eliezer_schw...@yahoo.com.invalid>> wrote: > We had max storage size of 1GB, but that's for provenance repo and our > problem was with content_repo. Our disk was 60GB, all on one partition, and > 55GB were taken up by content_repo. Now, it only contains 233MB. > > > > On Friday, October 23, 2015 2:50 PM, Mark Payne <marka...@hotmail.com > <mailto:marka...@hotmail.com>> wrote: > > > > OK, so this is interesting. Do you have your content repository and > provenance repository > both pointing to the same partition? What do you have the > "nifi.provenance.repository.max.storage.size" > property set to? How large is the actual disk? > > Thanks > -Mark > > > > On Oct 23, 2015, at 2:45 PM, Ryan H <rhendrickson.w...@gmail.com > > <mailto:rhendrickson.w...@gmail.com>> wrote: > > > > I've got this one... let me look for that > > > > 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1] > > o.a.n.p.PersistentProvenanceRepository > > java.io.IOException: No space left on device > > at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51] > > at java.io.FileOutputStream.write(FileOutputStream.java:326) > > ~[na:1.8.0_51] > > at > > org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) > > ~[na:1.8.0_51] > > at > > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) > > ~[na:1.8.0_51] > > at > > org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66) > > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0] > > at > > org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906) > > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0] > > at > > org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260) > > [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0] > > > > On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <marka...@hotmail.com > > <mailto:marka...@hotmail.com>> wrote: > > > >> Ryan, Elli, > >> > >> Do you by chance have any error messages in your logs from the > >> FileSystemRepository? > >> > >> I.e., if you perform: > >> > >> grep FileSystemRepository logs/* > >> > >> Do you get anything interesting in there? > >> > >> Thanks > >> -Mark > >> > >> > >>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz > >> <eliezer_schw...@yahoo.com.INVALID> wrote: > >>> > >>> I've been working with Ryan. There appear to be a few issues here: > >>> > >>> - We upgraded from 0.2.0 to 0.3.0 and it appears that > >> content_repository archive is now true by default. In 0.2.0 it was false, > >> and the documentation still states it is false by default. > >>> - When we ran out of disk space overnight, the problem was solved by > >> me simply restarting nifi, and that cleared out the archive by itself. > >>> > >>> - In order to clear up the archive, I had to set archive to true, and > >> set max usage to 1%, and restart nifi. That cleared it up, and then I set > >> archive to false and restarted again so we don't run out of space. > >>> - Based on the above, it appears that something happened yesterday > >> that prevented Nifi from clearing out the archive even though disk usage > >> reached 100%. However, restarting nifi apparently enabled it to perform the > >> clearing of the archive. So apparently the max usage setting doesn't work > >> under some conditions, but we don't know what conditions occurred overnight > >> to cause this problem. > >>> > >>> Thanks!-Elli > >>> > >>> > >>> > >>> On Friday, October 23, 2015 2:29 PM, Ryan H < > >> rhendrickson.w...@gmail.com <mailto:rhendrickson.w...@gmail.com>> wrote: > >>> > >>> > >>> > >>> Agree, they concern the archive... although it sounds like there are 2 > >>> archives? > >>> > >>> Within the content_repository folder, there are subfolders with the name > >>> 'archive' and files inside them. > >>> > >>> Example: > >>> ./nfii/content_repository/837/archive/1445611320767-837 > >>> > >>> Settings: > >>> nifi.content.repository.archive.max.retention.period=12 hours > >>> nifi.content.repository.archive.max.usage.percentage=50% > >>> nifi.content.repository.archive.enabled=true > >>> > >>> Last night, our server ran out of disk space because the > >> content_repository > >>> grew too large. Nifi didn't crash, but the log file contained errors > >>> saying the disk was full. > >>> > >>> We're not sure how, but the content_repository did not respect the above > >>> settings. > >>> > >>> We restarted Nifi, and it only then started to remove files, such as: > >>> ./nfii/content_repository/837/archive/1445611320767-837 > >>> > >>> We've turned off archiving for now. > >>> > >>> Ryan > >>> > >>> > >>> > >>> > >>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <aldrinp...@gmail.com > >>> <mailto:aldrinp...@gmail.com>> > >> wrote: > >>> > >>>> Ryan, > >>>> > >>>> Those items only concern the archive. Did you have data enqueued in > >>>> connections in your flow? If so, these items are not eligible and could > >>>> explain why your disk was filled. Otherwise, can you please provide > >> some > >>>> additional information so we can dig into why this may have arisen. > >>>> > >>>> Thanks! > >>>> > >>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <rhendrickson.w...@gmail.com > >>>> <mailto:rhendrickson.w...@gmail.com>> > >>>> wrote: > >>>> > >>>>> I've got the following set: > >>>>> > >>>>> nifi.content.repository.archive.max.retention.period=12 hours > >>>>> nifi.content.repository.archive.max.usage.percentage=50% > >>>>> nifi.content.repository.archive.enabled=true > >>>>> > >>>>> Yet, the content repo filled my disk last night... > >>>>> > >>>>> > >>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <aldrinp...@gmail.com > >>>>> <mailto:aldrinp...@gmail.com>> > >>>> wrote: > >>>>> > >>>>>> Ryan, > >>>>>> > >>>>>> Those archive folders map to the > >>>> nifi.content.repository.archive.enabled > >>>>>> property. > >>>>>> > >>>>>> What this property provides is a retention of files no longer in the > >>>>> system > >>>>>> for historical context of your flow's processing and the ability for > >>>>>> viewing this in conjunction with provenance events as well as allowing > >>>>>> replay. The amount of the archive when enabled is bounded by the > >>>>>> properties nifi.content.repository.archive.max.retention.period and > >>>>>> nifi.content.repository.archive.max.usage.percentage. > >>>>>> > >>>>>> Additional detail is available in the system properties of our > >>>>>> Administration Guide [1] > >>>>>> > >>>>>> Let us know if you have additional questions. > >>>>>> > >>>>>> --aldrin > >>>>>> > >>>>>> [1] > >>>>>> > >>>>>> > >>>>> > >>>> > >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties > >> > >> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties> > >>>>>> > >>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <rhendrickson.w...@gmail.com > >>>>>> <mailto:rhendrickson.w...@gmail.com> > >>> > >>>>>> wrote: > >>>>>> > >>>>>>> Interesting.. So what would > >>>>>>> > >>>>>>> ./nfii/content_repository/837/archive/1445611320767-837 > >>>>>>> > >>>>>>> typically be? > >>>>>>> > >>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande < > >>>>> agra...@hortonworks.com <mailto:agra...@hortonworks.com> > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Attachments don't go through, view at imagebin: > >>>>>>>> http://ibin.co/2K3SwR0z8yWX <http://ibin.co/2K3SwR0z8yWX> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <agra...@hortonworks.com > >>>>>>>> <mailto:agra...@hortonworks.com>> > >>>>>> wrote: > >>>>>>>> > >>>>>>>>> Ryan, > >>>>>>>>> > >>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not > >>>> the > >>>>>>>> content repository data. See the attached screenshot (Settings menu > >>>>> on > >>>>>>> the > >>>>>>>> right). > >>>>>>>>> > >>>>>>>>> Andrew > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 10/23/15, 12:47 PM, "ryan.andrew.hendrick...@gmail.com > >>>>>>>>> <mailto:ryan.andrew.hendrick...@gmail.com> on > >>>> behalf > >>>>>> of > >>>>>>>> Ryan H" <ryan.andrew.hendrick...@gmail.com > >>>>>>>> <mailto:ryan.andrew.hendrick...@gmail.com> on behalf of > >>>>>>>> rhendrickson.w...@gmail.com <mailto:rhendrickson.w...@gmail.com>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi, > >>>>>>>>>> I'm noticing my Content Repo growing large. There's a number > >>>> of > >>>>>>>> files... > >>>>>>>>>> > >>>>>>>>>> content_repo/837/archive/144...-837 > >>>>>>>>>> > >>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be > >>>>>> going > >>>>>>>>>> into ./conf/archive, but i don't see anything in there. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Ryan > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >>> > >> > >> > > > > > >
