Hi James - I too would like more information on this. I've been using
NiFi in a kubernetes cluster for over a year now using cetic
(https://github.com/cetic/helm-nifi) and have it working with NiFi
1.25.0. This works well with a 3 node cluster, and can successfully
scale up. Scaling down can work, but (if I understand correctly),
you'll need to make sure the queues on the node you are removing are
empty, otherwise you'll lose that data. Nifi 2.0.0 looks like it has
much better integration with kubernetes with the
KubernetesLeaderElectionManager
(https://issues.apache.org/jira/browse/NIFI-10757), which appears to
take the place of zookeeper, but I've not seen any helm charts or nice
ways to deploy a NiFi 2.0.0 cluster yet.
Anyone?
-Joe
On 7/6/2024 2:09 PM, James McMahon wrote:
I would like to build a containerized, scalable, highly available NiFi
architecture - likely docker, likely on EC2s. I intend to have dev,
int, and prod containerized groups, version controlled through NiFi
Registry. I am trying to understand how NiFi clusters, Registry, and
containerization get used in such a complex architecture.
Does one first containerize each and every NiFi node in the cluster
independently, each and every zookeeper node independently, and then
group them together in NiFi clusters as described in the Apache NiFi
Admin Guide, here:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering?
There doesn’t seem to be much in there about clustering and
containerizing.
Am I thinking about this wrong? Does one abandon traditional nifi
clustering and instead achieve scalability somehow through
dockerized containers running single NiFi node instances, employing a
load balancer in front of the containers to distribute load - so no
traditional NiFi cluster, no zookeeper cluster, no coordination
between NiFi nodes?
How well does NiFi Registry function when the underlying NiFi clusters
are containerized?
Has anyone accomplished anything similar? Could you help me understand
how to build out such a complex architecture: NiFi nodes in
containers, zookeeper nodes in containers, all clustered, NiFi
Registry lording over all for version control and process group promotion?
This may be helpful to get started:
https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04
But it doesn't speak to clustering or to NiFi Registry in a
containerized, clustered architecture.
Thanks in advance for any thoughts.
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
