I would like to build a containerized, scalable, highly available NiFi architecture - likely docker, likely on EC2s. I intend to have dev, int, and prod containerized groups, version controlled through NiFi Registry. I am trying to understand how NiFi clusters, Registry, and containerization get used in such a complex architecture.
Does one first containerize each and every NiFi node in the cluster independently, each and every zookeeper node independently, and then group them together in NiFi clusters as described in the Apache NiFi Admin Guide, here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering? There doesn’t seem to be much in there about clustering and containerizing. Am I thinking about this wrong? Does one abandon traditional nifi clustering and instead achieve scalability somehow through dockerized containers running single NiFi node instances, employing a load balancer in front of the containers to distribute load - so no traditional NiFi cluster, no zookeeper cluster, no coordination between NiFi nodes? How well does NiFi Registry function when the underlying NiFi clusters are containerized? Has anyone accomplished anything similar? Could you help me understand how to build out such a complex architecture: NiFi nodes in containers, zookeeper nodes in containers, all clustered, NiFi Registry lording over all for version control and process group promotion? This may be helpful to get started: https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04 But it doesn't speak to clustering or to NiFi Registry in a containerized, clustered architecture. Thanks in advance for any thoughts.
