date:20180923

Nifi with docker and LDAP

2018-09-23 Thread Juan Pablo Gardella

Hi all,

I'm using Nifi with docker and it's secure.

I'm facing an issue when I bounce my LAPTOP (I'm running it locally). After
bouncing my lap, I cannot access to it (the container is running). The only
workaround it's restart the service. I suppose it's something related to
the host name. Any thoughts?

Configuration:
   nifi:
build:
  context: .
  dockerfile: Dockerfile-nifi
image: myimageid
container_name: nifi-d
restart: always
ports:
  - 8443:8443
depends_on:
  - ldap
environment:
  AUTH: ldap
(other variables)

I think maybe it's related to the hostname. It's changed after bounce
maybe.

Juan

Re: Secure NiFi cluster on kubernetes.

2018-09-23 Thread Peter Wilcsinszky

I beleive the problem is that you don't use PersistentVolumeClaims with
your statefulset. Do you have a specific reason not using persistent
volumes for your data and/or mutable config (authorizations.xml, users.xml,
flow.xml.gz)?

On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky <
peterwilcsins...@gmail.com> wrote:

> Sorry I did not understand the problem first, but now I do and will check
> this.
>
> (Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
> $2}'|head -1` you could simply use `hostname -f`, but that is not the
> problem. )
>
>
>
> On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar 
> wrote:
>
>> This is my service.yaml (2 services one headless and one regular) and
>> depoloyment.yaml, let me know if this helps.:
>>
>>
>>
>> *kind: *Service
>> *apiVersion: *v1
>>
>> *metadata:   name: *nifi-sync
>>   *namespace: *nifi
>>
>> *labels: app: *nifi
>>
>>
>> *spec:   ports: *- *name: *prometheus-jmx
>>   *port: *8079
>> - *name: *web
>>   *port: *8080
>> - *name: *cluster
>>   *port: *8082
>> - *name: *misc
>>   *port: *9001
>>   *type: *NodePort
>>
>> *selector: app: *nifi
>>
>>
>> ---
>>
>> *kind: *Service
>> *apiVersion: *v1
>>
>> *metadata:   name: *nifi
>>
>> *spec:   clusterIP: *None
>>
>> *selector: app: *nifi
>>
>> *ports: *- *protocol: *TCP
>>   *port: *8081
>>   *targetPort: *8081
>>
>>
>>
>>
>>
>> 
>>
>>
>>
>> *apiVersion: *apps/v1beta1
>> *kind: *StatefulSet
>>
>> *metadata:  name: *nifi
>>
>> *spec:  serviceName: *nifi
>>   *replicas: *3
>>   *podManagementPolicy: *Parallel
>>
>> *updateStrategy:type: *RollingUpdate
>>
>>
>>
>> *template:metadata:  labels:app: *nifi
>>
>>
>>
>>
>> *spec:  affinity:podAntiAffinity:  
>> requiredDuringSchedulingIgnoredDuringExecution:*-
>>
>> *labelSelector:matchExpressions:  *- *key: *
>> *"app"**operator: *In
>>
>> *values:*- nifi
>>   *topologyKey: *
>> *"kubernetes.io/hostname "  *
>> *imagePullSecrets:  *- *name: *us-west-2-ecr-registry
>>
>> *containers:  *- *name: *nifi
>> *image: *X.amazonaws.com/devops/nifi-1.7.0:v11-stateful
>>
>> *command:  *-
>> *"/bin/sh"  *-
>> *"-c"  *- >
>> wget http:///build/deploy/configure_statefulset.sh -O 
>> /tmp/configure_statefulset.sh;
>> chmod +x /tmp/configure_statefulset.sh;
>> /tmp/configure_statefulset.sh;
>> apt-get -y install dnsutils;
>> sed -i -e 
>> "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep 
>> -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e 
>> "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup 
>> nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e 
>> "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup
>>  nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
>> sed -i -e 
>> "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1
>>  min|" $path/conf/nifi.properties;
>> mkdir -p $path/state/zookeeper;
>> touch $path/state/zookeeper/myid;
>> echo `hostname |rev | cut -d'-' -f 1 | rev` > 
>> $path/state/zookeeper/myid;
>> sed -i -e 
>> "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
>>  /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>> sed -i -e 
>> "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" 
>> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
>> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" 
>> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" 
>> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
>> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>>
>> *securityContext:  privileged: *true
>>   *imagePullPolicy: *Always
>>
>> *ports:*- *containerPort: *8080
>> - *containerPort: *8081
>>
>>
>>
>>
>>
>> *From: *Peter Wilcsinszky 
>> *Reply-To: *"users@nifi.apache.org" 
>> *Date: *Sunday, September 23, 2018 at 6:33 AM
>> *To: *"users@nifi.apache.org" 
>> *Subject: *Re: Secure NiFi cluster on kubernetes.
>>
>>
>>
>> Hi Varun,
>>
>>
>>
>> hard to tell without seeing your statefulset config. How do you add the
>> new nodes? If you add them through the statefulset (kubectl scale
>> statefulset  --replicas )
>>
>> the nodes should have the names nifi-3, nifi-4 instead of the ones on
>> your screenshot. But again, this is going to be hard to debug without
>> seeing your config.
>>
>>
>>
>> Peter
>>
>>
>>
>> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar 
>> wrote:
>>
>> Hi Peter,
>>
>>
>>
>>

Re: Secure NiFi cluster on kubernetes.

2018-09-23 Thread Peter Wilcsinszky

Sorry I did not understand the problem first, but now I do and will check
this.

(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print
$2}'|head -1` you could simply use `hostname -f`, but that is not the
problem. )



On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar  wrote:

> This is my service.yaml (2 services one headless and one regular) and
> depoloyment.yaml, let me know if this helps.:
>
>
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata:   name: *nifi-sync
>   *namespace: *nifi
>
> *labels: app: *nifi
>
>
> *spec:   ports: *- *name: *prometheus-jmx
>   *port: *8079
> - *name: *web
>   *port: *8080
> - *name: *cluster
>   *port: *8082
> - *name: *misc
>   *port: *9001
>   *type: *NodePort
>
> *selector: app: *nifi
>
>
> ---
>
> *kind: *Service
> *apiVersion: *v1
>
> *metadata:   name: *nifi
>
> *spec:   clusterIP: *None
>
> *selector: app: *nifi
>
> *ports: *- *protocol: *TCP
>   *port: *8081
>   *targetPort: *8081
>
>
>
>
>
> 
>
>
>
> *apiVersion: *apps/v1beta1
> *kind: *StatefulSet
>
> *metadata:  name: *nifi
>
> *spec:  serviceName: *nifi
>   *replicas: *3
>   *podManagementPolicy: *Parallel
>
> *updateStrategy:type: *RollingUpdate
>
>
>
> *template:metadata:  labels:app: *nifi
>
>
>
>
> *spec:  affinity:podAntiAffinity:  
> requiredDuringSchedulingIgnoredDuringExecution:*-
>
> *labelSelector:matchExpressions:  *- *key: *
> *"app"**operator: *In
>
> *values:*- nifi
>   *topologyKey: *
> *"kubernetes.io/hostname "  *
> *imagePullSecrets:  *- *name: *us-west-2-ecr-registry
>
> *containers:  *- *name: *nifi
> *image: *X.amazonaws.com/devops/nifi-1.7.0:v11-stateful
>
> *command:  *-
> *"/bin/sh"  *-
> *"-c"  *- >
> wget http:///build/deploy/configure_statefulset.sh -O 
> /tmp/configure_statefulset.sh;
> chmod +x /tmp/configure_statefulset.sh;
> /tmp/configure_statefulset.sh;
> apt-get -y install dnsutils;
> sed -i -e 
> "s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep 
> -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e 
> "s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup 
> nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e 
> "s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup
>  nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
> sed -i -e 
> "s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1
>  min|" $path/conf/nifi.properties;
> mkdir -p $path/state/zookeeper;
> touch $path/state/zookeeper/myid;
> echo `hostname |rev | cut -d'-' -f 1 | rev` > 
> $path/state/zookeeper/myid;
> sed -i -e 
> "s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
>  /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e 
> "s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" 
> /opt/nifi/nifi-1.7.0/conf/nifi.properties;
> sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" 
> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" 
> /opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
> /opt/nifi/nifi-1.7.0/bin/nifi.sh run
>
> *securityContext:  privileged: *true
>   *imagePullPolicy: *Always
>
> *ports:*- *containerPort: *8080
> - *containerPort: *8081
>
>
>
>
>
> *From: *Peter Wilcsinszky 
> *Reply-To: *"users@nifi.apache.org" 
> *Date: *Sunday, September 23, 2018 at 6:33 AM
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: Secure NiFi cluster on kubernetes.
>
>
>
> Hi Varun,
>
>
>
> hard to tell without seeing your statefulset config. How do you add the
> new nodes? If you add them through the statefulset (kubectl scale
> statefulset  --replicas )
>
> the nodes should have the names nifi-3, nifi-4 instead of the ones on your
> screenshot. But again, this is going to be hard to debug without seeing
> your config.
>
>
>
> Peter
>
>
>
> On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar 
> wrote:
>
> Hi Peter,
>
>
>
> I tried your suggestion of using statefulset in k8s. The problem is still
> there. The new nodes join the cluster but the old nodes still remains am I
> missing something. I am guessing each nodes get an Id which is the deciding
> factor in cluster config and not the node address, that’s the reason I am
> seeing 3/5.
>
>
>
>
>
> "address": "nifi-1.nifi.nifi.svc.cluster.local",
>
> "status": "CONNECTED",
>
> "message": "Connection requested from existing node. Setting
>

Re: FetchS3 not fetching all objects?

2018-09-23 Thread James Wing

If you wish to consume the latest CloudTrail events as a stream, have you
considered subscribing to CloudTrail notifications via SNS and SQS?  NiFi
can read the SQS messages with the GetSQS processor, and extract the S3 key
with EvaluateJsonPath.  In contrast, I expect ListS3 would be more useful
for reprocessing historical events.

Configuring CloudTrail to Send Notifications
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-cloudtrail-to-send-notifications.html


On Fri, Sep 21, 2018 at 11:55 PM Vets, Laurens  wrote:

> Is there a way to troubleshoot this further? I did an upgrade to 1.7.1.
> Changed the 'Minimum Object Age' to 1 minute, but I _think_ NiFi is
> somehow still not picking up all objects from the S3 bucket.
>
> On 20-Sep-18 19:55, Mark Rachelski wrote:
> > The S3Fetch processor is a stateful processor using NiFi state storage
> > to track the time of the most recent object name list extracted from
> > the bucket. On subsequent runs, it will only pull objects that have an
> > update time newer than the internally stored time stored in state (the
> > time of the last S3Fetch run).
> >
> > Is this possibly contributing to your missing files?
> >
> > mark.
> >
> > On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens  > > wrote:
> >
> > Hello,
> >
> > I'm using NiFi to read an S3 bucket containing all our AWS CloudTrail
> > logs. While debugging an issue, I noticed that not all objects are
> > fetched or listed. Basically, some events which I can find manually
> by
> > grepping the S3 files, I can't find in our Kibana dashboard. Is it
> > therefor possible that there might be an issue with the S3 processors
> > whereby it doesn't pick up all S3 objects?
> >
> > I'm using NiFi 1.3.0. While reading the release notes for the
> > newer NiFi
> > versions, I found https://issues.apache.org/jira/browse/NIFI-4876
> and
> > wondering whether this might be related?
> >
> > Can anyone shed some light on this?
> >
>

Re: Secure NiFi cluster on kubernetes.

2018-09-23 Thread Varun Tomar

This is my service.yaml (2 services one headless and one regular) and 
depoloyment.yaml, let me know if this helps.:

kind: Service
apiVersion: v1
metadata:
  name: nifi-sync
  namespace: nifi
 labels:
app: nifi
spec:
  ports:
- name: prometheus-jmx
  port: 8079
- name: web
  port: 8080
- name: cluster
  port: 8082
- name: misc
  port: 9001
  type: NodePort
  selector:
app: nifi


---

kind: Service
apiVersion: v1
metadata:
  name: nifi
spec:
  clusterIP: None
  selector:
app: nifi
  ports:
- protocol: TCP
  port: 8081
  targetPort: 8081





apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nifi
spec:
  serviceName: nifi
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
type: RollingUpdate
  template:
metadata:
  labels:
app: nifi
spec:
  affinity:
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
  - key: "app"
operator: In
values:
- nifi
  topologyKey: "kubernetes.io/hostname"
  imagePullSecrets:
  - name: us-west-2-ecr-registry
  containers:
  - name: nifi
image: X.amazonaws.com/devops/nifi-1.7.0:v11-stateful
command:
  - "/bin/sh"
  - "-c"
  - >
wget http:///build/deploy/configure_statefulset.sh -O 
/tmp/configure_statefulset.sh;
chmod +x /tmp/configure_statefulset.sh;
/tmp/configure_statefulset.sh;
apt-get -y install dnsutils;
sed -i -e 
"s|^nifi.web.http.host=.*$|nifi.web.http.host=`hostname`.`nslookup nifi|grep -i 
name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e 
"s|^nifi.remote.input.host=.*$|nifi.remote.input.host=`hostname`.`nslookup 
nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e 
"s|^nifi.cluster.node.address=.*$|nifi.cluster.node.address=`hostname`.`nslookup
 nifi|grep -i name |awk '{print $2}'|head -1`|" $path/conf/nifi.properties;
sed -i -e 
"s|^nifi.cluster.flow.election.max.wait.time=.*$|nifi.cluster.flow.election.max.wait.time=1
 min|" $path/conf/nifi.properties;
mkdir -p $path/state/zookeeper;
touch $path/state/zookeeper/myid;
echo `hostname |rev | cut -d'-' -f 1 | rev` > 
$path/state/zookeeper/myid;
sed -i -e 
"s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test|" 
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" 
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" 
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
  privileged: true
  imagePullPolicy: Always
  ports:
- containerPort: 8080
- containerPort: 8081


From: Peter Wilcsinszky 
Reply-To: "users@nifi.apache.org" 
Date: Sunday, September 23, 2018 at 6:33 AM
To: "users@nifi.apache.org" 
Subject: Re: Secure NiFi cluster on kubernetes.

Hi Varun,

hard to tell without seeing your statefulset config. How do you add the new 
nodes? If you add them through the statefulset (kubectl scale statefulset  --replicas )
the nodes should have the names nifi-3, nifi-4 instead of the ones on your 
screenshot. But again, this is going to be hard to debug without seeing your 
config.

Peter

On Sun, Sep 23, 2018 at 11:03 AM Varun Tomar 
mailto:varun.to...@zaplabs.com>> wrote:
Hi Peter,

I tried your suggestion of using statefulset in k8s. The problem is still 
there. The new nodes join the cluster but the old nodes still remains am I 
missing something. I am guessing each nodes get an Id which is the deciding 
factor in cluster config and not the node address, that’s the reason I am 
seeing 3/5.


"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status 
to connecting."
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"message": "Connection requested from existing node. Setting status 
to connecting."
"address": "nifi-0.nifi.nifi.svc.cluster.local",
"status": "CONNECTED",
"address": "nifi-2.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from cluster due to Have not received 
a heartbeat from node in 44 seconds"
"address": "nifi-1.nifi.nifi.svc.cluster.local",
"status": "DISCONNECTED",
"message": "Node disconnected from

Nifi with docker and LDAP

Re: Secure NiFi cluster on kubernetes.

Re: Secure NiFi cluster on kubernetes.

Re: FetchS3 not fetching all objects?

Re: Secure NiFi cluster on kubernetes.

5 matches

Site Navigation

Mail list logo

Footer information