Re: How to backup NiFi flow files when running on Kubernetes?
> Yes, that's the problem; recommend configuring persistent storage for the >Kubernetes containers running NiFi. I have already configured Persistent Volume Claims (PVC's) for NiFi on Kubernetes. They work fine for persisting data between pod restarts. The problem the limitation of PVCs on Kubernetes is that they are not replicated between availability zones. So if there is a permanent loss of the availability zone hosting the PVC then it results in permanent data loss of the flow files. My questions was about what options are available for backing up NiFi flow files in Kubernetes? I don't seem to be able to find a solution where NiFi flow files can be either stored in PostgreSQL (which we already backup) or to S3-compatible object storage. I'm hoping someone with experience of running NiFi on Kubernetes may have the answer. Thanks. On Monday, 23 October 2023 at 14:35:25 BST, Patrick Timmins wrote: "The problem is that our PVCs aren't backed up, nor are they resilient to an availability zone failure. So I'm trying to find a method for backing up the flow files, but I've not found anything for Kubernetes." Yes, that's the problem; recommend configuring persistent storage for the Kubernetes containers running NiFi. On 10/23/2023 7:41 AM, David Dean via users wrote: Hi everyone. I'm running a single instance of NiFi on a Kubernetes cluster using the the Helm chart from Cetic (I can't share the link here, it's on GitHub). This Helm chart uses PVCs for flow file storage, which works fine for persisting flows between restarts of NiFi. The problem is that our PVCs aren't backed up, nor are they resilient to an availability zone failure. So I'm trying to find a method for backing up the flow files, but I've not found anything for Kubernetes. Ideally I would like to write the flow files to a PostgreSQL database, which is already present in the Kubernetes cluster and is backed up. But from what I can read (and I couldn't easily be wrong!) I don't think NiFi can persist flow files in a PostgreSQL database? The other option would be to write the flow files to object storage, like an S3 or Azure blob storage bucket, but I don't think NiFi can do that either? My question for anyone else running NiFi on Kubernetes - how do you back up the flow files so they can be resolved in the event that the PVCs are permanently lost? Thanks.
Re: How to backup NiFi flow files when running on Kubernetes?
"The problem is that our PVCs aren't backed up, nor are they resilient to an availability zone failure. So I'm trying to find a method for backing up the flow files, but I've not found anything for Kubernetes." Yes, that's the problem; recommend configuring persistent storage for the Kubernetes containers running NiFi. On 10/23/2023 7:41 AM, David Dean via users wrote: Hi everyone. I'm running a single instance of NiFi on a Kubernetes cluster using the the Helm chart from Cetic (I can't share the link here, it's on GitHub). This Helm chart uses PVCs for flow file storage, which works fine for persisting flows between restarts of NiFi. The problem is that our PVCs aren't backed up, nor are they resilient to an availability zone failure. So I'm trying to find a method for backing up the flow files, but I've not found anything for Kubernetes. Ideally I would like to write the flow files to a PostgreSQL database, which is already present in the Kubernetes cluster and is backed up. But from what I can read (and I couldn't easily be wrong!) I don't think NiFi can persist flow files in a PostgreSQL database? The other option would be to write the flow files to object storage, like an S3 or Azure blob storage bucket, but I don't think NiFi can do that either? My question for anyone else running NiFi on Kubernetes - how do you back up the flow files so they can be resolved in the event that the PVCs are permanently lost? Thanks.
How to backup NiFi flow files when running on Kubernetes?
Hi everyone. I'm running a single instance of NiFi on a Kubernetes cluster using the the Helm chart from Cetic (I can't share the link here, it's on GitHub). This Helm chart uses PVCs for flow file storage, which works fine for persisting flows between restarts of NiFi. The problem is that our PVCs aren't backed up, nor are they resilient to an availability zone failure. So I'm trying to find a method for backing up the flow files, but I've not found anything for Kubernetes. Ideally I would like to write the flow files to a PostgreSQL database, which is already present in the Kubernetes cluster and is backed up. But from what I can read (and I couldn't easily be wrong!) I don't think NiFi can persist flow files in a PostgreSQL database? The other option would be to write the flow files to object storage, like an S3 or Azure blob storage bucket, but I don't think NiFi can do that either? My question for anyone else running NiFi on Kubernetes - how do you back up the flow files so they can be resolved in the event that the PVCs are permanently lost? Thanks.
