Thank you Joe. That does help. And I know that I can get to 0.7.1. I'll start working on that first thing tomorrow.
Jim On Mon, Mar 13, 2017 at 6:23 PM, Joe Witt <joe.w...@gmail.com> wrote: > Actually in following the referenced PR to the JIRA it references it > appears it went out in Apache NiFi 0.7.1 as > https://issues.apache.org/jira/browse/NIFI-2429. So if you can get > onto that version it should address that condition. > > Thanks > Joe > > On Mon, Mar 13, 2017 at 3:20 PM, Joe Witt <joe.w...@gmail.com> wrote: > > James > > > > Yes the problem is likely to reappear most likely. I suspect you're > > being impacted by https://issues.apache.org/jira/browse/NIFI-2395. > > > > It shows as though it was not addressed in any 0.x release, however, > > in looking at the JIRA notes it does appear that relevant aspects of > > this was merged into 0.x. Joe Skora/Mike Moser: can you please review > > this JIRA and get it tagged to the proper fix version since it is now > > on master in 0.x? > > > > Thanks > > Joe > > > > On Mon, Mar 13, 2017 at 2:39 PM, James McMahon <jsmcmah...@gmail.com> > wrote: > >> Sent you that info Joe. > >> > >> Just to be certain I understood clearly: I should go ahead and rm -rf > >> ./provenance_repository? Won't the problem just reappear again as soon > as it > >> recreates provenance_repository at start up? > >> > >> On Mon, Mar 13, 2017 at 5:04 PM, Joe Witt <joe.w...@gmail.com> wrote: > >>> > >>> Jim > >>> > >>> Blow away the prov repo and restart to get back going. But to > >>> troubleshoot why it isn't cleaning up it seems as though there is some > >>> reason nifi is failing to rollover. > >>> > >>> What version precisely are you running? > >>> > >>> Thanks > >>> Joe > >>> > >>> On Mar 13, 2017 2:01 PM, "James McMahon" <jsmcmah...@gmail.com> wrote: > >>>> > >>>> Here is all the info Joe. I had to manually recreate this here. Tried > to > >>>> avoid typos. Anything that looks bizarre or oddly misspelled is on me. > >>>> > >>>> Certainly Joe. I need to do whatever is humanly possible to get this > up > >>>> and > >>>> rolling again. Here is what I can provide: > >>>> > >>>> (Background) > >>>> /mnt/provenance_repo 50G size, 47G used, 17M available, 100% capacity > >>>> under this is > >>>> database_repository > >>>> provenance_repository > >>>> du -h output: > >>>> 8.5M ./database_repository > >>>> 16K ./lost+found > >>>> 4.0K ./provenance_repository/index-1489...124 > >>>> 92K ./provenance_repository/toc > >>>> 47G ./provenance_repository/journals > >>>> 47G ./provenance_repository > >>>> 47G . > >>>> > >>>> In my nifi.properties: > >>>> #H2 Settings > >>>> nifi.database.directory=/mnt/provenance_repo/database_repository > >>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0; > AUTO_SERVER=FALSE > >>>> > >>>> nifi.provenance.repository.directory.default=/mnt/ > provenance_repo/provenance_repos > >>>> itory > >>>> nifi.provenance.repository.max.storage.time=24 hours > >>>> nifi.provenance.repository.max.storage.size=1 GB > >>>> nifi.provenance.repository.rollover.time=30 secs > >>>> nifi.provenance.repository.rollover.size=100 MB > >>>> nifi.provenance.repository.query.threads=2 > >>>> nifi.provenance.repository.index.threads=1 > >>>> nifi.provenance.repository.compress.on.rollover=true > >>>> nifi.provenance.repository.always.sync=false > >>>> nifi.provenance.repository.journal.count=16 > >>>> > >>>> nifi.provenance.repository.indexed.fields=EventType, > FlofFileUUID,Filename,ProcessorID,Relationship > >>>> nifi.provenance.repository.indexed.attributes= > >>>> nifi.provenance.repository.index.shard.size=500 MB > >>>> nifi.provenance.repository.max.attributes.length=65536 > >>>> nifi.provenance.repository.buffer.size=100000 > >>>> (Errors) > >>>> in nifi-bootstrap.log, java.io.IOException No space left on device > >>>> /mnt/provenance_repo/database_repository/nifi-users.h2.db > >>>> similar error reported in nifi-app.log, fails when trying to create > >>>> org.springframework.beans.factory.BeanCreationException > >>>> > >>>> I think both are just symptoms of the true problem, which is to say > too > >>>> much stuff in > >>>> journals under provenance_repository > >>>> Jim > >>>> > >>>> On Mon, Mar 13, 2017 at 4:10 PM, Joe Witt <joe.w...@gmail.com> wrote: > >>>>> > >>>>> Jim, > >>>>> > >>>>> Processors create provenance events which get written to the > >>>>> provenenance repo but that is the end of their relationship. The > >>>>> provenance repository is then self managing. Can you share your > >>>>> nifi.properties provenance configuration? > >>>>> > >>>>> Thanks > >>>>> Joe > >>>>> > >>>>> On Mon, Mar 13, 2017 at 1:05 PM, James McMahon <jsmcmah...@gmail.com > > > >>>>> wrote: > >>>>> > Good evening. I have a number of process groups in my NiFi instance > >>>>> > that run > >>>>> > concurrently. Evidently I have filled up my provenance repo. How > do I > >>>>> > recover from this? NiFi will not start back up after shutdown. > >>>>> > > >>>>> > I am in a dev environment right now so I can afford to lose what > had > >>>>> > been in > >>>>> > my flows. Does that give me other options to recover? > >>>>> > > >>>>> > However I must determine what process group and processor are not > >>>>> > freeing > >>>>> > resources from the provenance repo. I can't allow this workflow to > >>>>> > promote > >>>>> > to production until I determine how to prevent this from > reoccurring. > >>>>> > How > >>>>> > can I do that? > >>>>> > > >>>>> > My provenance, content, and flow repos are all on separate disk > >>>>> > devices, > >>>>> > each of which is 50GB in capacity. > >>>>> > > >>>>> > Thank you for your help. -Jim > >>>> > >>>> > >> >
