James, I run my own project at home that does this. Individual containers for each NiFi node, one for Registry, a zookeeper cluster of 3 containers and one container for nginx to proxy front end access. This to me is the Docker way of doing things. This then does look like a traditional NiFi cluster, but the emphasis is on using well designed and implemented images to create the containers, then use a management tool such as Compose, Swarm or Kubernetes to compose those containers into a single cluster.
For my own needs I have added in Kafka and Redis nodes and connected the cluster to a separate Elasticsearch cluster running on a different Docker network. I only use it for developing flow ideas, so it lacks production features such as volume persistence and secure connections. I have seen in the past a few people on this forum mention they have containerised solutions in production. They may wish to share their ideas too. For what it is worth, I have it all on my personal GitHub account as a public repo. GitHub - hindmasj/my-nifi-cluster: Quick project to create a NiFi cluster in Docker<https://github.com/hindmasj/my-nifi-cluster> Regards Steve Hindmarch From: James McMahon <[email protected]> Sent: Saturday, July 6, 2024 7:10 PM To: users <[email protected]> Subject: Containerizing, clustering, Registry - how to begin? I would like to build a containerized, scalable, highly available NiFi architecture - likely docker, likely on EC2s. I intend to have dev, int, and prod containerized groups, version controlled through NiFi Registry. I am trying to understand how NiFi clusters, Registry, and containerization get used in such a complex architecture. Does one first containerize each and every NiFi node in the cluster independently, each and every zookeeper node independently, and then group them together in NiFi clusters as described in the Apache NiFi Admin Guide, here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering? There doesn't seem to be much in there about clustering and containerizing. Am I thinking about this wrong? Does one abandon traditional nifi clustering and instead achieve scalability somehow through dockerized containers running single NiFi node instances, employing a load balancer in front of the containers to distribute load - so no traditional NiFi cluster, no zookeeper cluster, no coordination between NiFi nodes? How well does NiFi Registry function when the underlying NiFi clusters are containerized? Has anyone accomplished anything similar? Could you help me understand how to build out such a complex architecture: NiFi nodes in containers, zookeeper nodes in containers, all clustered, NiFi Registry lording over all for version control and process group promotion? This may be helpful to get started: https://sandundayananda.medium.com/deploy-apache-nifi-on-docker-with-aws-ec2-instance-and-connect-to-web-interface-3e516e06fe04 But it doesn't speak to clustering or to NiFi Registry in a containerized, clustered architecture. Thanks in advance for any thoughts.
