Hi, At Stackable we have Kubernetes operators and images for Apache NiFi and Apache ZooKeeper, so if you want avoid the effort of building all of this yourself it’s worth taking a look. There’s an experimental NiFi 2.0 Docker image in our repository if you’re keen to try that out, but that won’t be officially supported until we see a final NiFi 2.0 release.
Kind regards, Jim On 9 Jul 2024, at 15:01, Joe Obernberger <[email protected]> wrote: > > Hi James - I too would like more information on this. I've been using NiFi > in a kubernetes cluster for over a year now using cetic > (https://github.com/cetic/helm-nifi) and have it working with NiFi 1.25.0. > This works well with a 3 node cluster, and can successfully scale up. > Scaling down can work, but (if I understand correctly), you'll need to make > sure the queues on the node you are removing are empty, otherwise you'll > lose that data. Nifi 2.0.0 looks like it has much better integration with > kubernetes with the KubernetesLeaderElectionManager > (https://issues.apache.org/jira/browse/NIFI-10757), which appears to take the > place of zookeeper, but I've not seen any helm charts or nice ways to deploy > a NiFi 2.0.0 cluster yet. > Anyone? > > -Joe > > On 7/6/2024 2:09 PM, James McMahon wrote: >> I would like to build a containerized, scalable, highly available NiFi >> architecture - likely docker, likely on EC2s. I intend to have dev, int, and >> prod containerized groups, version controlled through NiFi Registry. I am >> trying to understand how NiFi clusters, Registry, and containerization get >> used in such a complex architecture. >> >> Does one first containerize each and every NiFi node in the cluster >> independently, each and every zookeeper node independently, and then group >> them together in NiFi clusters as described in the Apache NiFi Admin Guide, >> here: >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering? >> There doesn’t seem to be much in there about clustering and containerizing. >> >> Am I thinking about this wrong? Does one abandon traditional nifi clustering >> and instead achieve scalability somehow through dockerized containers >> running single NiFi node instances, employing a load balancer in front of >> the containers to distribute load - so no traditional NiFi cluster, no >> zookeeper cluster, no coordination between NiFi nodes? >> >> How well does NiFi Registry function when the underlying NiFi clusters are >> containerized? >> >> Has anyone accomplished anything similar? Could you help me understand how >> to build out such a complex architecture: NiFi nodes in containers, >> zookeeper nodes in containers, all clustered, NiFi Registry lording over all >> for version control and process group promotion? >> >> This may be helpful to get started: >> >> https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04 >> >> But it doesn't speak to clustering or to NiFi Registry in a containerized, >> clustered architecture. >> >> Thanks in advance for any thoughts. > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free.www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > <x-msg://1/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
