Thank you for your reply Mark, Somehow it went to junk…. And it’s definitely not. 😊 I better understand how NiFi looks at Content Storage.
Here are the 3 options I see which I asked about yesterday as well: - change nifi.content.repository.archive.max.usage.percentage=50% but I’m hesitant to do that due to the nature of the services using this storage, it does fluctuate quite a bit. - I could look into adding another content repository on all 3 nodes but that will get cumbersome to some extent. - I’m also looking into a File System Content Repository Property to fix the content repo directory to a pre-established size, say 1TB for instance. I can’t find it in the guide, but is there one? I think this would be the best option for our use-case. Thanks, Roland Rosso AdventHealth Big Data Administrator | Corporate Analytics O: 407-805-8532 From: Mark Payne <marka...@hotmail.com> Sent: Wednesday, January 20, 2021 9:25 AM To: users@nifi.apache.org Subject: Re: [EXTERNAL] NiFi Cluster 1.9.2 Content Repository Roland, So based on the configuration below, that makes sense. The content repo will keep archived data as long as the disk usage is below 50%, and for up to 12 hours. So if you have 3 TB free, that’s 67% used, approximately, so data will be aged off of the archive very quickly. So note that the configuration is not saying “Use up to 50% of the drive for archive.” It’s saying “Keep archiving until 50% of the drive is used.” Thanks -Mark On Jan 19, 2021, at 11:23 PM, Rosso, Roland <roland.ro...@adventhealth.com<mailto:roland.ro...@adventhealth.com>> wrote: Joe, The disks are 9TB drives, with over 3Tb available. This is true for each node Ideally, we would be able to replay up to 12 hours for any of the flows in order to troubleshoot/investigate data issues. Thanks, Roland Rosso AdventHealth Big Data Administrator | Corporate Analytics O: 407-805-8532 From: Joe Witt <joe.w...@gmail.com<mailto:joe.w...@gmail.com>> Sent: Tuesday, January 19, 2021 10:26 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: [EXTERNAL] Re: NiFi Cluster 1.9.2 Content Repository Hello The key value to watch is the 50% value. That means we will work to remove content no longer reachable in the flow until we are at 50% of the available disk space for that volume. So how big is the disk there for each node? Can you share more about what you are hoping to see happen against what is happening? Thanks On Tue, Jan 19, 2021 at 5:18 PM Rosso, Roland <roland.ro...@adventhealth.com<mailto:roland.ro...@adventhealth.com>> wrote: Hello all, We have a 3 node NiFi 1.9.2 cluster for which the content repository config is below. I don’t have the entire history of this install, however we are only able to retrieve the content from the flows that ran within the past couple of minutes. All others when trying to view NiFi Data Provenance -> Content will show Replay Content is no longer available in Content Repository Checking all 3 nodes at intervals, the content repository size on disk is: Node 1: 1.1G, goes up to 2.1G, back down to 1.1G. This is currently the coordinator Node 2: 5.9G, static Node 3: 201M, static Is there a default size for the content repository set around 4GB? Looking at the documentation, I can’t seem to find the answer to that question. # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=1 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.directory.default=/u12/nifi/data/content_repository nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=../nifi-content-viewer/ Thank you for your help, Roland This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
