[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737509#comment-17737509 ] Giovanni commented on NIFI-11530: - I got it. Thanks for your help [~joewitt] > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737230#comment-17737230 ] Joe Witt commented on NIFI-11530: - Right you will need some buffer for the rollover/shard development. I dont think you'll want to be so tight such that if you have 10GB you allow a max of 9GB. You will need to experiment a bit but at this point I do not believe you're dealing with a bug but rather finding an optimal configuration. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[
https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737221#comment-17737221
]
Giovanni commented on NIFI-11530:
-
After I configured the volumes I also set the provenance max size to 9GB:
{code:java}
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/var/nifi/provenance_repo/data
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=9 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false {code}
So 8.98GB could be a normal size according to the setting or should I expect
the repo size to be lower?
> Disk full even with nifi.content.repository.archive.max.usage.percentage set
> to 50%
> ---
>
> Key: NIFI-11530
> URL: https://issues.apache.org/jira/browse/NIFI-11530
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
> Environment: Ubuntu 20.04.5 LTS
>Reporter: Giovanni
>Priority: Major
> Attachments: 20230605_disk_usage.jpg, content_archive.jpg,
> flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png,
> image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png,
> image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png,
> jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg,
> provenance_archive.jpg
>
>
> Nifi primary node reports disk full causing all nodes to stop working.
> Restarting nifi service does not resolve.
> Restarting the VM does not resolve.
> The only way to fix is to clean te content_repository dir:
> rm -rf ./nifi/content_repository/*
>
> Unfortunately I have no logs of the issue ongoing.
>
> UPDATE:
> I'm having the problem again.
> Every archive size is more than 50% on each node, with 70%+ peak on
> coordinator node (see attachments).
> I'm also attaching nifi-app.log this time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737209#comment-17737209 ] Joe Witt commented on NIFI-11530: - The provenance settings from nifi.properties you shared a few weeks ago and the disk size for provenance shown would not work out. It says you want to keep 10GB but you have only 9.75GB of disk. Try cutting provenance storage in half. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737060#comment-17737060 ] Giovanni commented on NIFI-11530: - Update: I reconfigured the repository to have a logical volume for each: !image-2023-06-26-11-13-19-289.png! The overall performance are improved. However the provenance repository is still problematic: !image-2023-06-26-11-15-00-379.png! The other repos are ok though: !image-2023-06-26-11-16-21-096.png! !image-2023-06-26-11-16-35-791.png! !image-2023-06-26-11-16-48-186.png! > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, image-2023-06-26-11-13-19-289.png, > image-2023-06-26-11-15-00-379.png, image-2023-06-26-11-16-21-096.png, > image-2023-06-26-11-16-35-791.png, image-2023-06-26-11-16-48-186.png, > jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, nifi_bug.jpg, > provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731412#comment-17731412 ] Joe Witt commented on NIFI-11530: - I suspect what you're hitting is based on 'nifi.content.repository.archive.backpressure.percentage' which in your case is defaulting to 52% which you're pretty much always already at since all the repos are shared and always already at 50%. Read more about this in the admin guide here https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html You could disable archiving which will let it keep going but you'll run into other space issues at it approaches 100%. The better path is to ensure each key repository has its own physical/logical volume to ensure its space is manageable and not shared (on each node). The best place to start with these docs is here https://nifi.apache.org/docs.html > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730426#comment-17730426 ] Giovanni commented on NIFI-11530: - Hi Joe, so what is the recommended configuration? Consider my nifi setup is a 3 nodes cluster. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[
https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729762#comment-17729762
]
Giovanni commented on NIFI-11530:
-
Hi Joe,
thanks for your reply.
Here are my settings:
{code:java}
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=./flowfile_repository
nifi.flowfile.repository.checkpoint.interval=20 secs
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.retain.orphaned.flowfiles=true
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=2{code}
{code:java}
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.directory.default=./content_repository
nifi.content.repository.archive.max.retention.period=7 days
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/{code}
{code:java}
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some
examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving
a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be
truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=10{code}
In response to your point, I confirm each node has only a single disk.
> Disk full even with nifi.content.repository.archive.max.usage.percentage set
> to 50%
> ---
>
> Key: NIFI-11530
> URL: https://issues.apache.org/jira/browse/NIFI-11530
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
> Environment: Ubuntu 20.04.5 LTS
>Reporter: Giovanni
>Priority: Major
> Attachments: 20230605_disk_usage.jpg, content_archive.jpg,
> flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log,
> nifi_bug.jpg, provenance_archive.jpg
>
>
> Nifi primary node reports disk full causing all nodes to stop working.
> Restarting nifi service does not resolve.
> Restarting the VM does not resolve.
> The only way to fix is to clean te content_repository dir:
> rm -rf ./nifi/content_repository/*
>
> Unfortunately I have no logs of the issue ongoing.
>
> UPDATE:
> I'm having the problem again.
> Every archive size is more than 50% on each node, with 70%+ peak on
> coordinator node (see attachments).
> I'm also attaching nifi-app.log this time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11530) Disk full even with nifi.content.repository.archive.max.usage.percentage set to 50%
[ https://issues.apache.org/jira/browse/NIFI-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729366#comment-17729366 ] Joe Witt commented on NIFI-11530: - Hello All of the details that matter for this will come from how it is configured. It appears you are using a single shared disk on each node whereby that disk has content, flowfile, and provenance repositories. While this is possible it is not recommended. It being possible means you will need to have specific configurations that work for it as well. What are your settings for content storage? Provenance storage? And flowfile storage? Those are all in nifi.properties. So far looking at this I see no sign of a bug. Just a sign of a configuration that won't likely work well or as you likely intend. > Disk full even with nifi.content.repository.archive.max.usage.percentage set > to 50% > --- > > Key: NIFI-11530 > URL: https://issues.apache.org/jira/browse/NIFI-11530 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 > Environment: Ubuntu 20.04.5 LTS >Reporter: Giovanni >Priority: Major > Attachments: 20230605_disk_usage.jpg, content_archive.jpg, > flowfile_archive.jpg, jvm.jpg, nifi1-app.log, nifi2-app.log, nifi3-app.log, > nifi_bug.jpg, provenance_archive.jpg > > > Nifi primary node reports disk full causing all nodes to stop working. > Restarting nifi service does not resolve. > Restarting the VM does not resolve. > The only way to fix is to clean te content_repository dir: > rm -rf ./nifi/content_repository/* > > Unfortunately I have no logs of the issue ongoing. > > UPDATE: > I'm having the problem again. > Every archive size is more than 50% on each node, with 70%+ peak on > coordinator node (see attachments). > I'm also attaching nifi-app.log this time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
