This is my service.yaml (2 services one headless and one regular) and depoloyment.yaml, let me know if this helps.:
kind: Service apiVersion: v1 metadata: name: nifi-sync namespace: nifi labels: app: nifi spec: ports: - name: prometheus-jmx port: 8079 - name: web port: 8080 - name: cluster port: 8082 - name: misc port: 9001 type: NodePort selector: app: nifi --- kind: Service apiVersion: v1 metadata: name: nifi spec: clusterIP: None selector: app: nifi ports: - protocol: TCP port: 8081 targetPort: 8081 ################ apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: nifi spec: serviceName: nifi replicas: 3 podManagementPolicy: Parallel updateStrategy: type: RollingUpdate template: metadata: labels: app: nifi spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - nifi topologyKey: "kubernetes.io/hostname" imagePullSecrets: - name: us-west-2-ecr-registry containers: - name: nifi image: XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful command: - "/bin/sh" - "-c" - > wget http://xxxxxxxx/build/deploy/configure_statefulset.sh -O /tmp/configure_statefulset.sh; chmod +x /tmp/configure_statefulset.sh; /tmp/configure_statefulset.sh; apt-get -y install dnsutils; sed -i -e "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties; sed -i -e "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties; sed -i -e "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties; sed -i -e "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1 min|" $path/conf/nifi.properties; mkdir -p $path/state/zookeeper; touch $path/state/zookeeper/myid; echo `hostname |rev | cut -d'-' -f 1 | rev` > $path/state/zookeeper/myid; sed -i -e "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|" /opt/nifi/nifi-1.7.0/conf/nifi.properties; sed -i -e "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" /opt/nifi/nifi-1.7.0/conf/nifi.properties; sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf; sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" /opt/nifi/nifi-1.7.0/conf/bootstrap.conf; /opt/nifi/nifi-1.7.0/bin/nifi.sh run securityContext: privileged: true imagePullPolicy: Always ports: - containerPort: 8080 - containerPort: 8081 From: Peter Wilcsinszky <peterwilcsins...@gmail.com> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> Date: Sunday, September 23, 2018 at 6:33 AM To: "users@nifi.apache.org" <users@nifi.apache.org> Subject: Re: Secure NiFi cluster on kubernetes. Hi Varun, hard to tell without seeing your statefulset config. How do you add the new nodes? If you add them through the statefulset (kubectl scale statefulset <your statefulset's name> --replicas <desired replica count>) the nodes should have the names nifi-3, nifi-4 instead of the ones on your screenshot. But again, this is going to be hard to debug without seeing your config. Peter On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote: Hi Peter, I tried your suggestion of using statefulset in k8s. The problem is still there. The new nodes join the cluster but the old nodes still remains am I missing something. I am guessing each nodes get an Id which is the deciding factor in cluster config and not the node address, that’s the reason I am seeing 3/5. "address": "nifi-1.nifi.nifi.svc.cluster.local", "status": "CONNECTED", "message": "Connection requested from existing node. Setting status to connecting." "address": "nifi-2.nifi.nifi.svc.cluster.local", "status": "CONNECTED", "message": "Connection requested from existing node. Setting status to connecting." "address": "nifi-0.nifi.nifi.svc.cluster.local", "status": "CONNECTED", "address": "nifi-2.nifi.nifi.svc.cluster.local", "status": "DISCONNECTED", "message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds" "address": "nifi-1.nifi.nifi.svc.cluster.local", "status": "DISCONNECTED", "message": "Node disconnected from cluster due to Have not received a heartbeat from node in 44 seconds" [cid:image001.png@01D452E1.9D0F93B0] From: Peter Wilcsinszky <peterwilcsins...@gmail.com<mailto:peterwilcsins...@gmail.com>> Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>> Date: Friday, August 31, 2018 at 10:01 AM To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>> Subject: Re: Secure NiFi cluster on kubernetes. On Fri, 31 Aug 2018, 16:51 Varun Tomar, <varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote: Hi Peter, We started using nifi as statefulset last year you but moved to deployment. -CICD tool Spinnaker does not support statefulsets. - We have also customized logback.xml as it was log within log issue which was not getting parsed properly in ELK - For ports and cluster IP I pass them as argument so even if the pod reboot we don't have any issues. Why do you need to pass an IP? - we also use external zookeeper. I dint find any benefit of running statefulset . The only issue as I said is if we restart any undeying node we extra node and old nodes does not get deleted. With a statefulset you wouldnt have issues with that and you would have stable persistent volumes as well. Regards, Varun ________________________________ From: Peter Wilcsinszky <peterwilcsins...@gmail.com<mailto:peterwilcsins...@gmail.com>> Sent: Friday, August 31, 2018 2:50 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: Secure NiFi cluster on kubernetes. Hi Dnyaneshwar, as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure when it will be available publicly. Some pointers that can help by then: - You should use a StatefulSet to manage NiFi pods - Probably Helm charts are the most efficient way to get started - I recommend using the official NiFi image and wrapping the original nifi.sh script from the Kubernetes pod spec similarly how we do it in the Docker image [1]. Caveats: setting dynamic properties like nifi.web.http.host from the wrapper script is a good idea, but for more static properties like nifi.web.http.port you may want to use the config files directly as configmaps and do templating using Helm. This is especially true for more complex configurations like the authorizers.xml or the login-identity-providers.xml. - Authorizations in NiFi can be configured for the initial cluster setup, but needs to be done manually when you add a new Node to the cluster above the initial cluster size. Also these extra nodes should have a vanilla authorizations.xml to avoid conflicts when joining to the existing ones. You can use the wrapper script to decide which configmap to use when starting the container. Once the pod has started you still have to add the node and authorize it manually using the UI. There is ongoing work to make this more dynamic: [3] - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server mode. The NiFi pods have an init container that uses tls-toolkit in client mode to request and receive certificates from the CA server. The communication is protected using a shared secret that is generated inside the cluster on the fly, also you can further protect access to the CA using NetworkPolicies. - You should avoid using the embedded Zookeeper, but you can use an already existing helm chart as a dependency to install it [4] (caveat: the image used by that chart is not recommended for production use) [1] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=> [2] https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=> [3] https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote: Hi Dnyaneshwar, We have nifi running on k8s for around 8-10 months. We create nifi cluster as part of CICD and then there is a stage which does the template deployment. Haven’t faced any major issues. Just sometime if a node reboots the old cluster member in nifi does not gets cleaned up. Regards, Varun From: Andy LoPresto <alopre...@apache.org<mailto:alopre...@apache.org>> Reply-To: <users@nifi.apache.org<mailto:users@nifi.apache.org>> Date: Thursday, August 30, 2018 at 10:23 AM To: <users@nifi.apache.org<mailto:users@nifi.apache.org>> Subject: Re: Secure NiFi cluster on kubernetes. Hi Dnyaneshwar, I know other users are working on the same thing, so yes, NiFi + Kubernetes will allow you to stand up secure clusters. There is ongoing work targeted for upcoming releases to make this easier and more performant (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a lot of great work here, and he may be able to share some resources he used/created. [1] https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=> Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <dnyaneshwar_pa...@persistent.com<mailto:dnyaneshwar_pa...@persistent.com>> wrote: Hi, We have requirement of deploying NiFi on cloud platforms and we are considering kubernetes as orchestrator. I have knowledge on configuring nifi cluster, however, I am not sure on how things would go on kubernetes. Further, we are using Apache DS as LDAP server for authentication and planning to use embedded zookeeper instance to make the zookeeper cluster. Any help or pointer to documentation would be appreciated. Thank You. Regards, Dnyaneshwar Pawar DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. ________________________________ This email may be confidential. If you are not the intended recipient, please notify us immediately and delete this copy from your system. *Wire Fraud is Real*. Before wiring any money, call the intended recipient at a number you know is valid to confirm the instructions. Additionally, please note that the sender does not have authority to bind a party to a real estate contract via written or verbal communication.