I beleive the problem is that you don't use PersistentVolumeClaims with
your statefulset. Do you have a specific reason not using persistent
volumes for your data and/or mutable config (authorizations.xml, users.xml,
flow.xml.gz)?
On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky <
peterwilcsins...@gmail.com> wrote:
> Sorry I did not understand the problem first, but now I do and will check
> this.
>
> (Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
> $2}'|head -1` you could simply use `hostname -f`, but that is not the
> problem. )
>
>
>
> On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <varun.to...@zaplabs.com>
> wrote:
>
>> This is my service.yaml (2 services one headless and one regular) and
>> depoloyment.yaml, let me know if this helps.:
>>
>>
>>
>> *kind: *Service
>> *apiVersion: *v1
>>
>> *metadata: name: *nifi-sync
>> *namespace: *nifi
>>
>> *labels: app: *nifi
>>
>>
>> *spec: ports: *- *name: *prometheus-jmx
>> *port: *8079
>> - *name: *web
>> *port: *8080
>> - *name: *cluster
>> *port: *8082
>> - *name: *misc
>> *port: *9001
>> *type: *NodePort
>>
>> *selector: app: *nifi
>>
>>
>> ---
>>
>> *kind: *Service
>> *apiVersion: *v1
>>
>> *metadata: name: *nifi
>>
>> *spec: clusterIP: *None
>>
>> *selector: app: *nifi
>>
>> *ports: *- *protocol: *TCP
>> *port: *8081
>> *targetPort: *8081
>>
>>
>>
>>
>>
>> ################
>>
>>
>>
>> *apiVersion: *apps/v1beta1
>> *kind: *StatefulSet
>>
>> *metadata: name: *nifi
>>
>> *spec: serviceName: *nifi
>> *replicas: *3
>> *podManagementPolicy: *Parallel
>>
>> *updateStrategy: type: *RollingUpdate
>>
>>
>>
>> *template: metadata: labels: app: *nifi
>>
>>
>>
>>
>> *spec: affinity: podAntiAffinity:
>> requiredDuringSchedulingIgnoredDuringExecution: *-
>>
>> *labelSelector: matchExpressions: *- *key: *
>> *"app" **operator: *In
>>
>> *values: *- nifi
>> *topologyKey: *
>> *"kubernetes.io/hostname <http://kubernetes.io/hostname>" *
>> *imagePullSecrets: *- *name: *us-west-2-ecr-registry
>>
>> *containers: *- *name: *nifi
>> *image: *XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful
>>
>> *command: *-
>> *"/bin/sh" *-
>> *"-c" *- >
>> wget http://xxxxxxxx/build/deploy/configure_statefulset.sh -O
>> /tmp/configure_statefulset.sh;
>> chmod +x /tmp/configure_statefulset.sh;
>> /tmp/configure_statefulset.sh;
>> apt-get -y install dnsutils;
>> sed -i -e
>> "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep
>> -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e
>> "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup
>> nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e
>> "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup
>> nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e
>> "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1
>> min|" $path/conf/nifi.properties;
>> mkdir -p $path/state/zookeeper;
>> touch $path/state/zookeeper/myid;
>> echo `hostname |rev | cut -d'-' -f 1 | rev` >
>> $path/state/zookeeper/myid;
>> sed -i -e
>> "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
>> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>> sed -i -e
>> "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|"
>> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|"
>> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|"
>> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>>
>> *securityContext: privileged: *true
>> *imagePullPolicy: *Always
>>
>> *ports: *- *containerPort: *8080
>> - *containerPort: *8081
>>
>>
>>
>>
>>
>> *From: *Peter Wilcsinszky <peterwilcsins...@gmail.com>
>> *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org>
>> *Date: *Sunday, September 23, 2018 at 6:33 AM
>> *To: *"users@nifi.apache.org" <users@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Varun,
>>
>>
>>
>> hard to tell without seeing your statefulset config. How do you add the
>> new nodes? If you add them through the statefulset (kubectl scale
>> statefulset <your statefulset's name> --replicas <desired replica count>)
>>
>> the nodes should have the names nifi-3, nifi-4 instead of the ones on
>> your screenshot. But again, this is going to be hard to debug without
>> seeing your config.
>>
>>
>>
>> Peter
>>
>>
>>
>> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <varun.to...@zaplabs.com>
>> wrote:
>>
>> Hi Peter,
>>
>>
>>
>> I tried your suggestion of using statefulset in k8s. The problem is still
>> there. The new nodes join the cluster but the old nodes still remains am I
>> missing something. I am guessing each nodes get an Id which is the deciding
>> factor in cluster config and not the node address, that’s the reason I am
>> seeing 3/5.
>>
>>
>>
>>
>>
>> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>>
>> "status": "CONNECTED",
>>
>> "message": "Connection requested from existing node. Setting
>> status to connecting."
>>
>> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>>
>> "status": "CONNECTED",
>>
>> "message": "Connection requested from existing node. Setting
>> status to connecting."
>>
>> "address": "nifi-0.nifi.nifi.svc.cluster.local",
>>
>> "status": "CONNECTED",
>>
>> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>>
>> "status": "DISCONNECTED",
>>
>> "message": "Node disconnected from cluster due to Have not
>> received a heartbeat from node in 44 seconds"
>>
>> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>>
>> "status": "DISCONNECTED",
>>
>> "message": "Node disconnected from cluster due to Have not
>> received a heartbeat from node in 44 seconds"
>>
>>
>>
>> [image: cid:image001.png@01D452E1.9D0F93B0]
>>
>> *From: *Peter Wilcsinszky <peterwilcsins...@gmail.com>
>> *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org>
>> *Date: *Friday, August 31, 2018 at 10:01 AM
>> *To: *"users@nifi.apache.org" <users@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>>
>>
>> On Fri, 31 Aug 2018, 16:51 Varun Tomar, <varun.to...@zaplabs.com> wrote:
>>
>> Hi Peter,
>>
>>
>>
>> We started using nifi as statefulset last year you but moved to
>> deployment.
>>
>>
>>
>> -CICD tool Spinnaker does not support statefulsets.
>>
>> - We have also customized logback.xml as it was log within log issue
>> which was not getting parsed properly in ELK
>>
>> - For ports and cluster IP I pass them as argument so even if the pod
>> reboot we don't have any issues.
>>
>> Why do you need to pass an IP?
>>
>>
>>
>> - we also use external zookeeper.
>>
>>
>>
>> I dint find any benefit of running statefulset .
>>
>>
>>
>> The only issue as I said is if we restart any undeying node we extra node
>> and old nodes does not get deleted.
>>
>> With a statefulset you wouldnt have issues with that and you would have
>> stable persistent volumes as well.
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Varun
>>
>>
>> ------------------------------
>>
>> *From:* Peter Wilcsinszky <peterwilcsins...@gmail.com>
>> *Sent:* Friday, August 31, 2018 2:50 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> as Andy mentioned we are working on running NiFi in Kubernetes but I'm
>> not sure when it will be available publicly. Some pointers that can help by
>> then:
>>
>> - You should use a StatefulSet to manage NiFi pods
>>
>> - Probably Helm charts are the most efficient way to get started
>>
>> - I recommend using the official NiFi image and wrapping the original
>> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
>> Docker image [1]. Caveats: setting dynamic properties like
>> nifi.web.http.host from the wrapper script is a good idea, but for more
>> static properties like nifi.web.http.port you may want to use the config
>> files directly as configmaps and do templating using Helm. This is
>> especially true for more complex configurations like the authorizers.xml or
>> the login-identity-providers.xml.
>>
>> - Authorizations in NiFi can be configured for the initial cluster
>> setup, but needs to be done manually when you add a new Node to the cluster
>> above the initial cluster size. Also these extra nodes should have a
>> vanilla authorizations.xml to avoid conflicts when joining to the existing
>> ones. You can use the wrapper script to decide which configmap to use when
>> starting the container. Once the pod has started you still have to add the
>> node and authorize it manually using the UI. There is ongoing work to make
>> this more dynamic: [3]
>>
>> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
>> server mode. The NiFi pods have an init container that uses tls-toolkit in
>> client mode to request and receive certificates from the CA server. The
>> communication is protected using a shared secret that is generated inside
>> the cluster on the fly, also you can further protect access to the CA using
>> NetworkPolicies.
>>
>> - You should avoid using the embedded Zookeeper, but you can use an
>> already existing helm chart as a dependency to install it [4] (caveat: the
>> image used by that chart is not recommended for production use)
>>
>>
>>
>> [1]
>> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
>>
>> [2]
>> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
>>
>> [3] https://issues.apache.org/jira/browse/NIFI-5542
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
>>
>> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>>
>>
>>
>> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <varun.to...@zaplabs.com>
>> wrote:
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> We have nifi running on k8s for around 8-10 months. We create nifi
>> cluster as part of CICD and then there is a stage which does the template
>> deployment. Haven’t faced any major issues. Just sometime if a node reboots
>> the old cluster member in nifi does not gets cleaned up.
>>
>>
>>
>> Regards,
>>
>> Varun
>>
>>
>>
>> *From: *Andy LoPresto <alopre...@apache.org>
>> *Reply-To: *<users@nifi.apache.org>
>> *Date: *Thursday, August 30, 2018 at 10:23 AM
>> *To: *<users@nifi.apache.org>
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Dnyaneshwar,
>>
>>
>>
>> I know other users are working on the same thing, so yes, NiFi +
>> Kubernetes will allow you to stand up secure clusters. There is ongoing
>> work targeted for upcoming releases to make this easier and more performant
>> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
>> Wilcsinszky has done a lot of great work here, and he may be able to share
>> some resources he used/created.
>>
>>
>>
>> [1]
>> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>>
>>
>>
>> Andy LoPresto
>>
>> alopre...@apache.org
>>
>> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>
>>
>> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
>> dnyaneshwar_pa...@persistent.com> wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> We have requirement of deploying NiFi on cloud platforms and we are
>> considering kubernetes as orchestrator. I have knowledge on configuring
>> nifi cluster, however, I am not sure on how things would go on kubernetes.
>> Further, we are using Apache DS as LDAP server for authentication and
>> planning to use embedded zookeeper instance to make the zookeeper cluster.
>>
>> Any help or pointer to documentation would be appreciated.
>>
>>
>>
>> Thank You.
>>
>>
>>
>> Regards,
>>
>> Dnyaneshwar Pawar
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>>
>> ------------------------------
>>
>> This email may be confidential. If you are not the intended recipient,
>> please notify us immediately and delete this copy from your system.
>>
>> *Wire Fraud is Real*. Before wiring any money, call the intended
>> recipient at a number you know is valid to confirm the instructions.
>> Additionally,
>> please note that the sender does not have authority to bind a party to a
>> real estate contract via written or verbal communication.
>>
>>
