I figured it out finally, it was the state management file.
From: Varun Tomar <varun.to...@zaplabs.com>
Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
Date: Monday, September 24, 2018 at 4:07 PM
To: "users@nifi.apache.org" <users@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Same issue even with persistent disks I don’t have users.xml any other
suggestion.
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image: xxxxxxxx.amazonaws.com/devops/nifi-1.7.0:v12-stateful
command:
- "/bin/sh"
- "-c"
- >
set -x;
sed -i -e
"s|^nifi.authorizer.configuration.file=.*$|nifi.authorizer.configuration.file=/mnt/conf/authorizers.xml|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e
"s|^nifi.flow.configuration.file=.*$|nifi.flow.configuration.file=/mnt/conf/flow.xml.gz|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e
"s|^nifi.state.management.configuration.file=.*$|nifi.state.management.configuration.file=/mnt/conf/state-management.xml|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties
sed -i -e
"s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e
"s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test002|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|"
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|"
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
ports:
- name: prometheus-jmx
containerPort: 8079
- name: web
containerPort: 8080
- name: cluster
containerPort: 8082
volumeMounts:
- name: repositories
mountPath: /mnt
volumeClaimTemplates:
- metadata:
name: repositories
annotations:
volume.beta.kubernetes.io/storage-class: thick-disk
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 25Gi
From: Peter Wilcsinszky <peterwilcsins...@gmail.com>
Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
Date: Sunday, September 23, 2018 at 11:21 AM
To: "users@nifi.apache.org" <users@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
I beleive the problem is that you don't use PersistentVolumeClaims with your
statefulset. Do you have a specific reason not using persistent volumes for
your data and/or mutable config (authorizations.xml, users.xml, flow.xml.gz)?
On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky
<peterwilcsins...@gmail.com<mailto:peterwilcsins...@gmail.com>> wrote:
Sorry I did not understand the problem first, but now I do and will check this.
(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head
-1` you could simply use `hostname -f`, but that is not the problem. )
On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar
<varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote:
This is my service.yaml (2 services one headless and one regular) and
depoloyment.yaml, let me know if this helps.:
kind: Service
apiVersion: v1
metadata:
name: nifi-sync
namespace: nifi
labels:
app: nifi
spec:
ports:
- name: prometheus-jmx
port: 8079
- name: web
port: 8080
- name: cluster
port: 8082
- name: misc
port: 9001
type: NodePort
selector:
app: nifi
---
kind: Service
apiVersion: v1
metadata:
name: nifi
spec:
clusterIP: None
selector:
app: nifi
ports:
- protocol: TCP
port: 8081
targetPort: 8081
################
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nifi
spec:
serviceName: nifi
replicas: 3
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nifi
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- nifi
topologyKey:
"kubernetes.io/hostname<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_hostname&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=jlf9tOXtU4c4NhVwcht24WDM_u7SLaQ6DXhr6OlLlaY&e=>"
imagePullSecrets:
- name: us-west-2-ecr-registry
containers:
- name: nifi
image:
XXXXXXXXX.amazonaws.com/devops/nifi-1.7.0:v11-stateful<https://urldefense.proofpoint.com/v2/url?u=http-3A__XXXXXXXXX.amazonaws.com_devops_nifi-2D1.7.0-3Av11-2Dstateful&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=eB6ndvXtCzcQEPUu0OnZUPDk9IcZPTdpqO79YF5K8gc&e=>
command:
- "/bin/sh"
- "-c"
- >
wget
http://xxxxxxxx/build/deploy/configure_statefulset.sh<https://urldefense.proofpoint.com/v2/url?u=http-3A__xxxxxxxx_build_deploy_configure-5Fstatefulset.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=5bGpIYBB3jSE7DDMuldXhx4VkaclPu5a5JZZaH2_ZMY&s=kgjjTsPFUolM5uXzxaKCFVHGtiqZA_L9wQycuBKSfq0&e=>
-O /tmp/configure_statefulset.sh;
chmod +x /tmp/configure_statefulset.sh;
/tmp/configure_statefulset.sh;
apt-get -y install dnsutils;
sed -i -e
"s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i
name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e
"s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup
nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e
"s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup
nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e
"s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1
min|" $path/conf/nifi.properties;
mkdir -p $path/state/zookeeper;
touch $path/state/zookeeper/myid;
echo `hostname |rev | cut -d'-' -f 1 | rev` >
$path/state/zookeeper/myid;
sed -i -e
"s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e
"s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|"
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|"
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|"
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
privileged: true
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 8081
From: Peter Wilcsinszky
<peterwilcsins...@gmail.com<mailto:peterwilcsins...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Sunday, September 23, 2018 at 6:33 AM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Varun,
hard to tell without seeing your statefulset config. How do you add the new
nodes? If you add them through the statefulset (kubectl scale statefulset <your
statefulset's name> --replicas <desired replica count>)
the nodes should have the names nifi-3, nifi-4 instead of the ones on your
screenshot. But again, this is going to be hard to debug without seeing your
config.
Peter
On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar
<varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote:
Hi Peter,
I tried your suggestion of using statefulset in k8s. The problem is still
there. The new nodes join the cluster but the old nodes still remains am I
missing something. I am guessing each nodes get an Id which is the deciding
factor in cluster config and not the node address, that’s the reason I am
seeing 3/5.
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status
to connecting."
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status
to connecting."
"address": "nifi-0.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received
a heartbeat from node in 44 seconds"
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received
a heartbeat from node in 44 seconds"
Error! Filename not specified.
From: Peter Wilcsinszky
<peterwilcsins...@gmail.com<mailto:peterwilcsins...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Friday, August 31, 2018 at 10:01 AM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
<users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
On Fri, 31 Aug 2018, 16:51 Varun Tomar,
<varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote:
Hi Peter,
We started using nifi as statefulset last year you but moved to deployment.
-CICD tool Spinnaker does not support statefulsets.
- We have also customized logback.xml as it was log within log issue which was
not getting parsed properly in ELK
- For ports and cluster IP I pass them as argument so even if the pod reboot we
don't have any issues.
Why do you need to pass an IP?
- we also use external zookeeper.
I dint find any benefit of running statefulset .
The only issue as I said is if we restart any undeying node we extra node and
old nodes does not get deleted.
With a statefulset you wouldnt have issues with that and you would have stable
persistent volumes as well.
Regards,
Varun
________________________________
From: Peter Wilcsinszky
<peterwilcsins...@gmail.com<mailto:peterwilcsins...@gmail.com>>
Sent: Friday, August 31, 2018 2:50 AM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
as Andy mentioned we are working on running NiFi in Kubernetes but I'm not sure
when it will be available publicly. Some pointers that can help by then:
- You should use a StatefulSet to manage NiFi pods
- Probably Helm charts are the most efficient way to get started
- I recommend using the official NiFi image and wrapping the original nifi.sh
script from the Kubernetes pod spec similarly how we do it in the Docker image
[1]. Caveats: setting dynamic properties like nifi.web.http.host from the
wrapper script is a good idea, but for more static properties like
nifi.web.http.port you may want to use the config files directly as configmaps
and do templating using Helm. This is especially true for more complex
configurations like the authorizers.xml or the login-identity-providers.xml.
- Authorizations in NiFi can be configured for the initial cluster setup, but
needs to be done manually when you add a new Node to the cluster above the
initial cluster size. Also these extra nodes should have a vanilla
authorizations.xml to avoid conflicts when joining to the existing ones. You
can use the wrapper script to decide which configmap to use when starting the
container. Once the pod has started you still have to add the node and
authorize it manually using the UI. There is ongoing work to make this more
dynamic: [3]
- We use a Kubernetes deployment to run NiFi Toolkit's tls-toolkit in server
mode. The NiFi pods have an init container that uses tls-toolkit in client mode
to request and receive certificates from the CA server. The communication is
protected using a shared secret that is generated inside the cluster on the
fly, also you can further protect access to the CA using NetworkPolicies.
- You should avoid using the embedded Zookeeper, but you can use an already
existing helm chart as a dependency to install it [4] (caveat: the image used
by that chart is not recommended for production use)
[1]
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=0Es97SkM4avrgOlxitQWc7Qk5qNWuxFU34qcOs9kSiE&e=>
[2]
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L23<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Ddocker_dockerhub_sh_start.sh-23L23&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=PZZ1OI3R84HcnuobOOz3iVAa7HZf9xAX134C_KZ6miU&e=>
[3]
https://issues.apache.org/jira/browse/NIFI-5542<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_NIFI-2D5542&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=B1o5V-PrLz1BMR8cKCMwweSsqtLiO6Bl0FhbWkDXQsY&e=>
[4]
https://github.com/helm/charts/tree/master/incubator/zookeeper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_helm_charts_tree_master_incubator_zookeeper&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=Hei21t3rRzIAJ8_6bsCjUnMsv3l1-zBcEMa2rS-VLLw&e=>
On Thu, Aug 30, 2018 at 10:42 PM Varun Tomar
<varun.to...@zaplabs.com<mailto:varun.to...@zaplabs.com>> wrote:
Hi Dnyaneshwar,
We have nifi running on k8s for around 8-10 months. We create nifi cluster as
part of CICD and then there is a stage which does the template deployment.
Haven’t faced any major issues. Just sometime if a node reboots the old cluster
member in nifi does not gets cleaned up.
Regards,
Varun
From: Andy LoPresto <alopre...@apache.org<mailto:alopre...@apache.org>>
Reply-To: <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Thursday, August 30, 2018 at 10:23 AM
To: <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: Secure NiFi cluster on kubernetes.
Hi Dnyaneshwar,
I know other users are working on the same thing, so yes, NiFi + Kubernetes
will allow you to stand up secure clusters. There is ongoing work targeted for
upcoming releases to make this easier and more performant (dynamic scaling,
certificate interaction & provisioning, etc.) [1]. Peter Wilcsinszky has done a
lot of great work here, and he may be able to share some resources he
used/created.
[1]
https://issues.apache.org/jira/issues/?filter=12338912&jql=project%20%20%3D%20%22Apache%20NiFi%22%20and%20resolution%20%20%3D%20Unresolved%20AND%20(text%20~%20kubernetes%20OR%20description%20~%20kubernetes%20OR%20labels%20%3D%20kubernetes)%20ORDER%20BY%20updatedDate%20DESC<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Ffilter-3D12338912-26jql-3Dproject-2520-2520-3D-2520-2522Apache-2520NiFi-2522-2520and-2520resolution-2520-2520-3D-2520Unresolved-2520AND-2520-28text-2520-7E-2520kubernetes-2520OR-2520description-2520-7E-2520kubernetes-2520OR-2520labels-2520-3D-2520kubernetes-29-2520ORDER-2520BY-2520updatedDate-2520DESC&d=DwMFaQ&c=fie8CffxQEyLNW7eyn-hJg&r=fFC22egstNBV-rEaKPyjN2mHRNLPz6LGSHZuTWaa1_s&m=6V2tvsLte3-eBxUi3ip9KQPonsgu0qdzCZHRkEwTrLg&s=MUrvXLXdWsFFeEJLeKxgOy0RERC-mhlUyWbemkDNxLI&e=>
Andy LoPresto
alopre...@apache.org<mailto:alopre...@apache.org>
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 30, 2018, at 2:53 AM, Dnyaneshwar Pawar
<dnyaneshwar_pa...@persistent.com<mailto:dnyaneshwar_pa...@persistent.com>>
wrote:
Hi,
We have requirement of deploying NiFi on cloud platforms and we are
considering kubernetes as orchestrator. I have knowledge on configuring nifi
cluster, however, I am not sure on how things would go on kubernetes. Further,
we are using Apache DS as LDAP server for authentication and planning to use
embedded zookeeper instance to make the zookeeper cluster.
Any help or pointer to documentation would be appreciated.
Thank You.
Regards,
Dnyaneshwar Pawar
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Persistent Systems Ltd. It is intended only for the use of the
individual or entity to which it is addressed. If you are not the intended
recipient, you are not authorized to read, retain, copy, print, distribute or
use this message. If you have received this communication in error, please
notify the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
________________________________
This email may be confidential. If you are not the intended recipient, please
notify us immediately and delete this copy from your system.
*Wire Fraud is Real*. Before wiring any money, call the intended recipient at
a number you know is valid to confirm the instructions. Additionally, please
note that the sender does not have authority to bind a party to a real estate
contract via written or verbal communication.
