Sorry I did not understand the problem first, but now I do and will check
this.
(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
$2}'|head -1` you could simply use `hostname -f`, but that is not the
problem. )
On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar <varun.to...@zaplabs.com> wrote:
> This is my service.yaml (2 services one headless and one regular) and
> depoloyment.yaml, let me know if this helps.:
>
>
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata: name: *nifi-sync
> *namespace: *nifi
>
> *labels: app: *nifi
>
>
> *spec: ports: *- *name: *prometheus-jmx
> *port: *8079
> - *name: *web
> *port: *8080
> - *name: *cluster
> *port: *8082
> - *name: *misc
> *port: *9001
> *type: *NodePort
>
> *selector: app: *nifi
>
>
> ---
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata: name: *nifi
>
> *spec: clusterIP: *None
>
> *selector: app: *nifi
>
> *ports: *- *protocol: *TCP
> *port: *8081
> *targetPort: *8081
>
>
>
>
>
> ################
>
>
>
> *apiVersion: *apps/v1beta1
> *kind: *StatefulSet
>
> *metadata: name: *nifi
>
> *spec: serviceName: *nifi
> *replicas: *3
> *podManagementPolicy: *Parallel
>
> *updateStrategy: type: *RollingUpdate
>
>
>
> *template: metadata: labels: app: *nifi
>
>
>
>
> *spec: affinity: podAntiAffinity:
> requiredDuringSchedulingIgnoredDuringExecution: *-
>
> *labelSelector: matchExpressions: *- *key: *
> *"app" **operator: *In
>
> *values: *- nifi
> *topologyKey: *
> *"kubernetes.io/hostname <http://kubernetes.io/hostname>" *
> *imagePullSecrets: *- *name: *us-west-2-ecr-registry
>
> *containers: *- *name: *nifi
> *image: *XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful
>
> *command: *-
> *"/bin/sh" *-
> *"-c" *- >
> wget http://xxxxxxxx/build/deploy/configure_statefulset.sh -O
> /tmp/configure_statefulset.sh;
> chmod +x /tmp/configure_statefulset.sh;
> /tmp/configure_statefulset.sh;
> apt-get -y install dnsutils;
> sed -i -e
> "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep
> -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e
> "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup
> nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e
> "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup
> nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e
> "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1
> min|" $path/conf/nifi.properties;
> mkdir -p $path/state/zookeeper;
> touch $path/state/zookeeper/myid;
> echo `hostname |rev | cut -d'-' -f 1 | rev` >
> $path/state/zookeeper/myid;
> sed -i -e
> "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e
> "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|"
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|"
> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|"
> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>
> *securityContext: privileged: *true
> *imagePullPolicy: *Always
>
> *ports: *- *containerPort: *8080
> - *containerPort: *8081
>
>
>
>
>
> *From: *Peter Wilcsinszky <peterwilcsins...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org>
> *Date: *Sunday, September 23, 2018 at 6:33 AM
> *To: *"users@nifi.apache.org" <users@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Varun,
>
>
>
> hard to tell without seeing your statefulset config. How do you add the
> new nodes? If you add them through the statefulset (kubectl scale
> statefulset <your statefulset's name> --replicas <desired replica count>)
>
> the nodes should have the names nifi-3, nifi-4 instead of the ones on your
> screenshot. But again, this is going to be hard to debug without seeing
> your config.
>
>
>
> Peter
>
>
>
> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar <varun.to...@zaplabs.com>
> wrote:
>
> Hi Peter,
>
>
>
> I tried your suggestion of using statefulset in k8s. The problem is still
> there. The new nodes join the cluster but the old nodes still remains am I
> missing something. I am guessing each nodes get an Id which is the deciding
> factor in cluster config and not the node address, that’s the reason I am
> seeing 3/5.
>
>
>
>
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
> status to connecting."
>
> "address": "nifi-0.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "address": "nifi-2.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "DISCONNECTED",
>
> "message": "Node disconnected from cluster due to Have not
> received a heartbeat from node in 44 seconds"
>
>
>
> [image: cid:image001.png@01D452E1.9D0F93B0]
>
> *From: *Peter Wilcsinszky <peterwilcsins...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org>
> *Date: *Friday, August 31, 2018 at 10:01 AM
> *To: *"users@nifi.apache.org" <users@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
>
>
> On Fri, 31 Aug 2018, 16:51 Varun Tomar, <varun.to...@zaplabs.com> wrote:
>
> Hi Peter,
>
>
>
> We started using nifi as statefulset last year you but moved to deployment.
>
>
>
> -CICD tool Spinnaker does not support statefulsets.
>
> - We have also customized logback.xml as it was log within log issue which
> was not getting parsed properly in ELK
>
> - For ports and cluster IP I pass them as argument so even if the pod
> reboot we don't have any issues.
>
> Why do you need to pass an IP?
>
>
>
> - we also use external zookeeper.
>
>
>
> I dint find any benefit of running statefulset .
>
>
>
> The only issue as I said is if we restart any undeying node we extra node
> and old nodes does not get deleted.
>
> With a statefulset you wouldnt have issues with that and you would have
> stable persistent volumes as well.
>
>
>
>
>
>
>
> Regards,
>
> Varun
>
>
> ------------------------------
>
> *From:* Peter Wilcsinszky <peterwilcsins...@gmail.com>
> *Sent:* Friday, August 31, 2018 2:50 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> as Andy mentioned we are working on running NiFi in Kubernetes but I'm not
> sure when it will be available publicly. Some pointers that can help by
> then:
>
> - You should use a StatefulSet to manage NiFi pods
>
> - Probably Helm charts are the most efficient way to get started
>
> - I recommend using the official NiFi image and wrapping the original
> nifi.sh script from the Kubernetes pod spec similarly how we do it in the
> Docker image [1]. Caveats: setting dynamic properties like
> nifi.web.http.host from the wrapper script is a good idea, but for more
> static properties like nifi.web.http.port you may want to use the config
> files directly as configmaps and do templating using Helm. This is
> especially true for more complex configurations like the authorizers.xml or
> the login-identity-providers.xml.
>
> - Authorizations in NiFi can be configured for the initial cluster setup,
> but needs to be done manually when you add a new Node to the cluster above
> the initial cluster size. Also these extra nodes should have a vanilla
> authorizations.xml to avoid conflicts when joining to the existing ones.
> You can use the wrapper script to decide which configmap to use when
> starting the container. Once the pod has started you still have to add the
> node and authorize it manually using the UI. There is ongoing work to make
> this more dynamic: [3]
>
> - We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in
> server mode. The NiFi pods have an init container that uses tls-toolkit in
> client mode to request and receive certificates from the CA server. The
> communication is protected using a shared secret that is generated inside
> the cluster on the fly, also you can further protect access to the CA using
> NetworkPolicies.
>
> - You should avoid using the embedded Zookeeper, but you can use an
> already existing helm chart as a dependency to install it [4] (caveat: the
> image used by that chart is not recommended for production use)
>
>
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
>
> [2]
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
>
> [3] https://issues.apache.org/jira/browse/NIFI-5542
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
>
> [4] https://github.com/helm/charts/tree/master/incubator/zookeeper
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
>
>
>
> On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar <varun.to...@zaplabs.com>
> wrote:
>
> Hi Dnyaneshwar,
>
>
>
> We have nifi running on k8s for around 8-10 months. We create nifi cluster
> as part of CICD and then there is a stage which does the template
> deployment. Haven’t faced any major issues. Just sometime if a node reboots
> the old cluster member in nifi does not gets cleaned up.
>
>
>
> Regards,
>
> Varun
>
>
>
> *From: *Andy LoPresto <alopre...@apache.org>
> *Reply-To: *<users@nifi.apache.org>
> *Date: *Thursday, August 30, 2018 at 10:23 AM
> *To: *<users@nifi.apache.org>
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Dnyaneshwar,
>
>
>
> I know other users are working on the same thing, so yes, NiFi +
> Kubernetes will allow you to stand up secure clusters. There is ongoing
> work targeted for upcoming releases to make this easier and more performant
> (dynamic scaling, certificate interaction & provisioning, etc.) [1]. Peter
> Wilcsinszky has done a lot of great work here, and he may be able to share
> some resources he used/created.
>
>
>
> [1]
> https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
>
>
>
> Andy LoPresto
>
> alopre...@apache.org
>
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar <
> dnyaneshwar_pa...@persistent.com> wrote:
>
>
>
> Hi,
>
>
>
> We have requirement of deploying NiFi on cloud platforms and we are
> considering kubernetes as orchestrator. I have knowledge on configuring
> nifi cluster, however, I am not sure on how things would go on kubernetes.
> Further, we are using Apache DS as LDAP server for authentication and
> planning to use embedded zookeeper instance to make the zookeeper cluster.
>
> Any help or pointer to documentation would be appreciated.
>
>
>
> Thank You.
>
>
>
> Regards,
>
> Dnyaneshwar Pawar
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
>
> ------------------------------
>
> This email may be confidential. If you are not the intended recipient,
> please notify us immediately and delete this copy from your system.
>
> *Wire Fraud is Real*. Before wiring any money, call the intended
> recipient at a number you know is valid to confirm the instructions.
> Additionally,
> please note that the sender does not have authority to bind a party to a
> real estate contract via written or verbal communication.
>
>
