[
https://issues.apache.org/jira/browse/NIFI-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740963#comment-14740963
]
Bryan Bende commented on NIFI-936:
----------------------------------
Mark, after applying this patch, what this expected behavior?
Should you be able to fill up the provenance repo, stop and start nifi with the
repo still full, and expect nifi to start successfully (successful meaning you
can at least get to the UI)?
I've tried this test three times and actually got three different results...
- First time when I restarted nifi, it encountered some kind of EOF exception
trying to read a provenance file which unfortunately I did not capture, and it
ended up in a weird state where I could not access the UI but the processes
were there, and "/bin/nifi.sh stop" timed out and had to kill the processes. I
will try to reproduce this and get the stacktrace.
- Second time nifi fully started and was basically back in the same state as
when it was stopped with a full prov repo, I assume this was the behavior we
want
- Third time nifi failed to start and got this...
{code}
Caused by: java.lang.RuntimeException: Unable to create Provenance Repository
at
org.apache.nifi.controller.FlowController.<init>(FlowController.java:410)
~[na:na]
at
org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
~[na:na]
at
org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
~[na:na]
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[na:na]
... 130 common frames omitted
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.7.0_65]
at java.io.FileOutputStream.write(FileOutputStream.java:345) ~[na:1.7.0_65]
at
org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:49)
~[na:na]
at
org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
~[na:na]
at
org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
~[na:na]
at
org.apache.nifi.stream.io.ByteCountingOutputStream.flush(ByteCountingOutputStream.java:59)
~[na:na]
at
org.apache.nifi.stream.io.DataOutputStream.flush(DataOutputStream.java:104)
~[na:na]
at
org.apache.nifi.provenance.StandardRecordWriter.writeHeader(StandardRecordWriter.java:87)
~[na:na]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.createWriters(PersistentProvenanceRepository.java:363)
~[na:na]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:222)
~[na:na]
at
org.apache.nifi.controller.FlowController.<init>(FlowController.java:406)
~[na:na]
... 133 common frames omitted
{code}
> Provenance Repository can become corrupt if it runs out of disk space,
> causing NiFi not to restart
> --------------------------------------------------------------------------------------------------
>
> Key: NIFI-936
> URL: https://issues.apache.org/jira/browse/NIFI-936
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-936-Ensure-that-if-we-fail-to-write-a-record-to.patch
>
>
> I caused the Provenance Repository to run out of disk space. When I attempted
> to restart, I got a huge stack trace that ended with the following:
> {noformat}
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowController': FactoryBean threw exception on
> object creation; nested exception is java.lang.OutOfMemoryError: Java heap
> space
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ~[na:na]
> ... 123 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.nifi.provenance.StandardRecordReader.readLongString(StandardRecordReader.java:427)
> ~[na:na]
> at
> org.apache.nifi.provenance.StandardRecordReader.readAttributes(StandardRecordReader.java:372)
> ~[na:na]
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:314)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1646)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1357)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:577)
> ~[na:na]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:219)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.<init>(FlowController.java:406)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
> ~[na:na]
> at
> org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[na:na]
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
> ~[na:na]
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ~[na:na]
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
> ~[na:na]
> 2015-09-08 07:48:28,165 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server
> shutdown completed (nicely or otherwise).
> {noformat}
> Looking it over, it appears to be the same issue that caused problems with
> the FlowFile repository in NIFI-902.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
