+1 to this. Would like to know if we can scale out and scale in based on the
load.
On 9 Jul 2024, 7:32 PM +0530, Joe Obernberger <[email protected]>,
wrote:
> Hi James - I too would like more information on this. I've been using NiFi in
> a kubernetes cluster for over a year now using cetic
> (https://github.com/cetic/helm-nifi) and have it working with NiFi 1.25.0.
> This works well with a 3 node cluster, and can successfully scale up. Scaling
> down can work, but (if I understand correctly), you'll need to make sure the
> queues on the node you are removing are empty, otherwise you'll lose that
> data. Nifi 2.0.0 looks like it has much better integration with kubernetes
> with the KubernetesLeaderElectionManager
> (https://issues.apache.org/jira/browse/NIFI-10757), which appears to take the
> place of zookeeper, but I've not seen any helm charts or nice ways to deploy
> a NiFi 2.0.0 cluster yet.
> Anyone?
> -Joe
> On 7/6/2024 2:09 PM, James McMahon wrote:
> > I would like to build a containerized, scalable, highly available NiFi
> > architecture - likely docker, likely on EC2s. I intend to have dev, int,
> > and prod containerized groups, version controlled through NiFi Registry. I
> > am trying to understand how NiFi clusters, Registry, and containerization
> > get used in such a complex architecture.
> >
> > Does one first containerize each and every NiFi node in the cluster
> > independently, each and every zookeeper node independently, and then group
> > them together in NiFi clusters as described in the Apache NiFi Admin Guide,
> > here:
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering?
> > There doesn’t seem to be much in there about clustering and containerizing.
> >
> > Am I thinking about this wrong? Does one abandon traditional nifi
> > clustering and instead achieve scalability somehow through dockerized
> > containers running single NiFi node instances, employing a load balancer in
> > front of the containers to distribute load - so no traditional NiFi
> > cluster, no zookeeper cluster, no coordination between NiFi nodes?
> >
> > How well does NiFi Registry function when the underlying NiFi clusters are
> > containerized?
> >
> > Has anyone accomplished anything similar? Could you help me understand how
> > to build out such a complex architecture: NiFi nodes in containers,
> > zookeeper nodes in containers, all clustered, NiFi Registry lording over
> > all for version control and process group promotion?
> >
> > This may be helpful to get started:
> >
> > https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
> >
> > But it doesn't speak to clustering or to NiFi Registry in a containerized,
> > clustered architecture.
> >
> > Thanks in advance for any thoughts.
>
> Virus-free.www.avg.com
