Hi Dnyaneshwar, as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then: - You should use a StatefulSet to manage NiFi pods - Probably Helm charts are the most efficient way to get started - I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml. - Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3] - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies. - You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use)
[1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh [2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23 [3] https://issues.apache.org/jira/browse/NIFI-5542 [4] https://github.com/helm/charts/tree/master/incubator/zookeeper On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <varun.to...@zaplabs.com> wrote: > Hi Dnyaneshwar, > > > > We have nifi running on k8s for around 8-10 months. We create nifi cluster > as part of CICD and then there is a stage which does the template > deployment. Haven’t faced any major issues. Just sometime if a node reboots > the old cluster member in nifi does not gets cleaned up. > > > > Regards, > > Varun > > > > *From: *Andy LoPresto <alopre...@apache.org> > *Reply-To: *<users@nifi.apache.org> > *Date: *Thursday, August 30, 2018 at 10:23 AM > *To: *<users@nifi.apache.org> > *Subject: *Re: Secure NiFi cluster on kubernetes. > > > > Hi Dnyaneshwar, > > > > I know other users are working on the same thing, so yes, NiFi + > Kubernetes will allow you to stand up secure clusters. There is ongoing > work targeted for upcoming releases to make this easier and more performant > (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter > Wilcsinszky has done a lot of great work here, and he may be able to share > some resources he used/created. > > > > [1] > https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC > <https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20=%20%22Apache%20NiFi%22%20and%20resolution%20%20=%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20=%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC> > > > > Andy LoPresto > > alopre...@apache.org > > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar < > dnyaneshwar_pa...@persistent.com> wrote: > > > > Hi, > > > > We have requirement of deploying NiFi on cloud platforms and we are > considering kubernetes as orchestrator. I have knowledge on configuring > nifi cluster, however, I am not sure on how things would go on kubernetes. > Further, we are using Apache DS as LDAP server for authentication and > planning to use embedded zookeeper instance to make the zookeeper cluster. > > Any help or pointer to documentation would be appreciated. > > > > Thank You. > > > > Regards, > > Dnyaneshwar Pawar > > > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is > the property of Persistent Systems Ltd. It is intended only for the use of > the individual or entity to which it is addressed. If you are not the > intended recipient, you are not authorized to read, retain, copy, print, > distribute or use this message. If you have received this communication in > error, please notify the sender and delete all copies of this message. > Persistent Systems Ltd. does not accept any liability for virus infected > mails. > > > > ------------------------------ > This email may be confidential. If you are not the intended recipient, > please notify us immediately and delete this copy from your system. > >
