Re: Listing S3

[Koji Kawamura]

Hi Martijn,

I'm not an expert on Jython, but if you already have a python script
using boto3 working fine, then I'd suggest using ExecuteStreamCommand
instead.
For example:
- you can design the python script to print out JSON formatted string
about listed files
- then connect the outputs to SplitJson
- and use EvaluateJsonPath to extract required values to FlowFile attribute
- finally, use FetchS3Object

Thanks,
Koji

Re: High volume data with ExecuteSQL processor

[Koji Kawamura]

Hello,

Did you try setting 'Max Rows Per Flow File' at ExecuteSQL processor?
If the OOM happened when NiFi writes all results into a single
FlowFile, then the property can help breaking the result set into
several FlowFiles to avoid that.

Thanks,
Koji
On Fri, Sep 21, 2018 at 3:56 PM Dnyaneshwar Pawar
 wrote:
>
> Hi,
>
>
>
> How to execute/process High volume data with ExecuteSQL processor:
>
>
>
> We tried to execute query for db2 database which has around 10 lakh records. 
> While executing this query
>
> we are getting OutOfMemory error and that request(flowfile) is stuck in 
> queue. When we restart nifi, it still stuck in queue and as soon as we start 
> nifi,
>
> we are again getting same error as it is stuck in queue. Is there any way to 
> configure retry for queue(connection to 2 processor).
>
>
>
> We also tried to change property for Flow File repository in nifi.properties 
> (nifi.flowfile.repository.implementation) to 
> 'org.apache.nifi.controller.repository.VolatileFlowFileRepository',
>
> This is removing flowfile in query while restarting nifi. But it has risk of 
> data loss in the event of power/machine failure for other processes.
>
> So please suggest how to execute high volume data query execution or any 
> retry mechanism available for queued flowfile.
>
>
>
>
>
> Regards,
>
> Dnyaneshwar Pawar
>
>

Re: Secure NiFi cluster on kubernetes.

[Varun Tomar]

I figured it out finally, it was the state management file.

From: Varun Tomar 
Reply-To: "users@nifi.apache.org" 
Date: Monday, September 24, 2018 at 4:07 PM
To: "users@nifi.apache.org" 
Subject: Re: Secure NiFi cluster on kubernetes.

Same issue even with persistent disks I don’t have users.xml any other 
suggestion.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nifi
spec:
  serviceName: nifi
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
type: RollingUpdate
  template:
metadata:
  labels:
app: nifi
  imagePullSecrets:
  - name: us-west-2-ecr-registry
  containers:
  - name: nifi
image: .amazonaws.com/devops/nifi-1.7.0:v12-stateful
command:
  - "/bin/sh"
  - "-c"
  - >
set -x;
sed -i -e 
"s|^nifi.authorizer.configuration.file=.*$|nifi.authorizer.configuration.file=/mnt/conf/authorizers.xml|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.flow.configuration.file=.*$|nifi.flow.configuration.file=/mnt/conf/flow.xml.gz|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.state.management.configuration.file=.*$|nifi.state.management.configuration.file=/mnt/conf/state-management.xml|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties
sed -i -e 
"s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test002|" 
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" 
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" 
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
  privileged: true
ports:
 - name: prometheus-jmx
   containerPort: 8079
 - name: web
   containerPort: 8080
 - name: cluster
   containerPort: 8082
volumeMounts:
  - name: repositories
mountPath: /mnt
  volumeClaimTemplates:
  - metadata:
  name: repositories
  annotations:
volume.beta.kubernetes.io/storage-class: thick-disk
spec:
  accessModes: [ "ReadWriteOnce" ]
  resources:
requests:
  storage: 25Gi


From: Peter Wilcsinszky 
Reply-To: "users@nifi.apache.org" 
Date: Sunday, September 23, 2018 at 11:21 AM
To: "users@nifi.apache.org" 
Subject: Re: Secure NiFi cluster on kubernetes.

I beleive the problem is that you don't use PersistentVolumeClaims with your 
statefulset. Do you have a specific reason not using persistent volumes for 
your data and/or mutable config (authorizations.xml, users.xml, flow.xml.gz)?

On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky 
mailto:peterwilcsins...@gmail.com>> wrote:
Sorry I did not understand the problem first, but now I do and will check this.

(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head 
-1` you could simply use `hostname -f`, but that is not the problem. )



On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar 
mailto:varun.to...@zaplabs.com>> wrote:
This is my service.yaml (2 services one headless and one regular) and 
depoloyment.yaml, let me know if this helps.:

kind: Service
apiVersion: v1
metadata:
  name: nifi-sync
  namespace: nifi
 labels:
app: nifi
spec:
  ports:
- name: prometheus-jmx
  port: 8079
- name: web
  port: 8080
- name: cluster
  port: 8082
- name: misc
  port: 9001
  type: NodePort
  selector:
app: nifi


---

kind: Service
apiVersion: v1
metadata:
  name: nifi
spec:
  clusterIP: None
  selector:
app: nifi
  ports:
- protocol: TCP
  port: 8081
  targetPort: 8081





apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nifi
spec:
  serviceName: nifi
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
type: RollingUpdate
  template:
metadata:
  labels:
app: nifi
spec:
  affinity:
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
  - key: "app"
operator: In
values:
- nifi
  topologyKey: 
"kubernetes.io/hostname"
  imagePullSecrets:
  - name: us-west-2-ecr-registry
  containers:
  - name: nifi
image: 

Re: Secure NiFi cluster on kubernetes.

[Varun Tomar]

Same issue even with persistent disks I don’t have users.xml any other 
suggestion.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nifi
spec:
  serviceName: nifi
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
type: RollingUpdate
  template:
metadata:
  labels:
app: nifi
  imagePullSecrets:
  - name: us-west-2-ecr-registry
  containers:
  - name: nifi
image: .amazonaws.com/devops/nifi-1.7.0:v12-stateful
command:
  - "/bin/sh"
  - "-c"
  - >
set -x;
sed -i -e 
"s|^nifi.authorizer.configuration.file=.*$|nifi.authorizer.configuration.file=/mnt/conf/authorizers.xml|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.flow.configuration.file=.*$|nifi.flow.configuration.file=/mnt/conf/flow.xml.gz|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.state.management.configuration.file=.*$|nifi.state.management.configuration.file=/mnt/conf/state-management.xml|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties
sed -i -e 
"s|^nifi.zookeeper.connect.string=.*$|nifi.zookeeper.connect.string=qazknifi.com:2181|"
 /opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e 
"s|^nifi.zookeeper.root.node=.*$|nifi.zookeeper.root.node=/test002|" 
/opt/nifi/nifi-1.7.0/conf/nifi.properties;
sed -i -e "s|^java.arg.2=.*$|java.arg.2=-Xms2g|" 
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
sed -i -e "s|^java.arg.3=.*$|java.arg.3=-Xmx2g|" 
/opt/nifi/nifi-1.7.0/conf/bootstrap.conf;
/opt/nifi/nifi-1.7.0/bin/nifi.sh run
securityContext:
  privileged: true
ports:
 - name: prometheus-jmx
   containerPort: 8079
 - name: web
   containerPort: 8080
 - name: cluster
   containerPort: 8082
volumeMounts:
  - name: repositories
mountPath: /mnt
  volumeClaimTemplates:
  - metadata:
  name: repositories
  annotations:
volume.beta.kubernetes.io/storage-class: thick-disk
spec:
  accessModes: [ "ReadWriteOnce" ]
  resources:
requests:
  storage: 25Gi


From: Peter Wilcsinszky 
Reply-To: "users@nifi.apache.org" 
Date: Sunday, September 23, 2018 at 11:21 AM
To: "users@nifi.apache.org" 
Subject: Re: Secure NiFi cluster on kubernetes.

I beleive the problem is that you don't use PersistentVolumeClaims with your 
statefulset. Do you have a specific reason not using persistent volumes for 
your data and/or mutable config (authorizations.xml, users.xml, flow.xml.gz)?

On Sun, Sep 23, 2018 at 7:25 PM Peter Wilcsinszky 
mailto:peterwilcsins...@gmail.com>> wrote:
Sorry I did not understand the problem first, but now I do and will check this.

(Also, instead of `hostname`.`nslookup nifi|grep -i name |awk '{print $2}'|head 
-1` you could simply use `hostname -f`, but that is not the problem. )



On Sun, Sep 23, 2018 at 4:57 PM Varun Tomar 
mailto:varun.to...@zaplabs.com>> wrote:
This is my service.yaml (2 services one headless and one regular) and 
depoloyment.yaml, let me know if this helps.:

kind: Service
apiVersion: v1
metadata:
  name: nifi-sync
  namespace: nifi
 labels:
app: nifi
spec:
  ports:
- name: prometheus-jmx
  port: 8079
- name: web
  port: 8080
- name: cluster
  port: 8082
- name: misc
  port: 9001
  type: NodePort
  selector:
app: nifi


---

kind: Service
apiVersion: v1
metadata:
  name: nifi
spec:
  clusterIP: None
  selector:
app: nifi
  ports:
- protocol: TCP
  port: 8081
  targetPort: 8081





apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: nifi
spec:
  serviceName: nifi
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
type: RollingUpdate
  template:
metadata:
  labels:
app: nifi
spec:
  affinity:
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
  - key: "app"
operator: In
values:
- nifi
  topologyKey: 
"kubernetes.io/hostname"
  imagePullSecrets:
  - name: us-west-2-ecr-registry
  containers:
  - name: nifi
image: 
X.amazonaws.com/devops/nifi-1.7.0:v11-stateful
command:
  - "/bin/sh"
  - "-c"
 

RE: A sensible approach to scheduling via the API?

[Cardinal, Alexandre]

You could do it with an external scheduler, but my gut would tell me that there 
is probably a way to structure your flow in a way that satisfies your batch 
requirement, without having to manage a scheduler.

-Alexandre

-Original Message-
From: Nathan Gough [mailto:thena...@gmail.com]
Sent: September 24, 2018 12:53 PM
To: users@nifi.apache.org
Subject: Re: A sensible approach to scheduling via the API?

Typically I would not expect to schedule dataflows in NiFi as it's not the 
ideal place for data to stay sitting. For running scheduled batch jobs as you 
describe I would expect the data to be constantly flowing to date/time based 
directories on HDFS. This allows data to be stored in a place meant for storing 
data and allows jobs to run for specified time periods with any data that 
arrived during that period.

In the past I have used a directory structure of year/month/day/hour. Eg. 
2018/09/24/12. Any data arriving during that time will be placed in those 
directories. Depending on your requirements you can bucket files into these 
directories based on collected date/time or arrival time (when it's received by 
NiFi). The scheduled batch jobs can then be configured to use the directory 
structure.

Let us know if this helps at all.
Nathan


On 9/24/18, 6:13 AM, "Vos, Walter"  wrote:

Hi,

I don't know what the etiquette on a mailing list is for this, but I'd like 
to bump my original question.

Perhaps it's good to add that many of our flows are batch loads and 
therefore depend on a schedule to run, once.

Does anyone have experience with remote scheduling in NiFi or do you think 
you have a smart take on this? Please let me know :)

Cheers,

Walter

-Oorspronkelijk bericht-
Van: Vos, Walter
Verzonden: woensdag 5 september 2018 10:02
Aan: users@nifi.apache.org
Onderwerp: A sensible approach to scheduling via the API?

Hi,

In our big data environment one of the architectural principles is to 
schedule jobs with Azure Automation (runbooks). A scheduling database is used 
to decide when to start which jobs. NiFi flows however are currently being 
scheduled in NiFi itself. We're looking for a good approach to move this over 
to runbooks. I see a couple of options:

* Have each flow start with a timer driven processor, where the run 
schedule is an hour or so. This processor will be stopped by default, and can 
be turned on via the API. It is then stopped at some point before the run 
schedule ends, preventing the processor from running twice.
* Use a ListenHTTP processor that we can POST a message to that specifies 
which flow to start. Do something like RouteOnAttribute to choose the right 
flow. I imagine this as being one ListenHTTP processor that is connected to all 
flows.
* Translate the schedule from the scheduling database to a ChronTrigger 
expression. Check if the CRON schedule on the processor is indeed set to that 
schedule. If not, stop the processor, change the schedule and start it again. 
If it is, do nothing and assume it'll run. This one seems convoluted on the one 
hand, but requires the least architecture within NiFi itself I imagine.

What do you think? Has anyone had to deal with something like this? How did 
you solve it? I can't find much information about this on the web, although I 
could be using the wrong terms.

Kind regards,

Walter Vos




Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor 
(gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke 
informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of 
verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden 
is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap



Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor 
(gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke 
informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of 
verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden 
is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap




CONFIDENTIALITÉ : Ce document est destiné uniquement à la personne ou à 
l'entité à qui il est adressé. L'information apparaissant dans ce document est 
de nature légalement privilégiée et confidentielle. Si vous n'êtes pas le 
destinataire visé ou la personne chargée de le 

Re: A sensible approach to scheduling via the API?

[Nathan Gough]

Typically I would not expect to schedule dataflows in NiFi as it's not the 
ideal place for data to stay sitting. For running scheduled batch jobs as you 
describe I would expect the data to be constantly flowing to date/time based 
directories on HDFS. This allows data to be stored in a place meant for storing 
data and allows jobs to run for specified time periods with any data that 
arrived during that period. 

In the past I have used a directory structure of year/month/day/hour. Eg. 
2018/09/24/12. Any data arriving during that time will be placed in those 
directories. Depending on your requirements you can bucket files into these 
directories based on collected date/time or arrival time (when it's received by 
NiFi). The scheduled batch jobs can then be configured to use the directory 
structure.

Let us know if this helps at all.
Nathan


On 9/24/18, 6:13 AM, "Vos, Walter"  wrote:

Hi,

I don't know what the etiquette on a mailing list is for this, but I'd like 
to bump my original question.

Perhaps it's good to add that many of our flows are batch loads and 
therefore depend on a schedule to run, once.

Does anyone have experience with remote scheduling in NiFi or do you think 
you have a smart take on this? Please let me know :)

Cheers,

Walter

-Oorspronkelijk bericht-
Van: Vos, Walter
Verzonden: woensdag 5 september 2018 10:02
Aan: users@nifi.apache.org
Onderwerp: A sensible approach to scheduling via the API?

Hi,

In our big data environment one of the architectural principles is to 
schedule jobs with Azure Automation (runbooks). A scheduling database is used 
to decide when to start which jobs. NiFi flows however are currently being 
scheduled in NiFi itself. We're looking for a good approach to move this over 
to runbooks. I see a couple of options:

* Have each flow start with a timer driven processor, where the run 
schedule is an hour or so. This processor will be stopped by default, and can 
be turned on via the API. It is then stopped at some point before the run 
schedule ends, preventing the processor from running twice.
* Use a ListenHTTP processor that we can POST a message to that specifies 
which flow to start. Do something like RouteOnAttribute to choose the right 
flow. I imagine this as being one ListenHTTP processor that is connected to all 
flows.
* Translate the schedule from the scheduling database to a ChronTrigger 
expression. Check if the CRON schedule on the processor is indeed set to that 
schedule. If not, stop the processor, change the schedule and start it again. 
If it is, do nothing and assume it'll run. This one seems convoluted on the one 
hand, but requires the least architecture within NiFi itself I imagine.

What do you think? Has anyone had to deal with something like this? How did 
you solve it? I can't find much information about this on the web, although I 
could be using the wrong terms.

Kind regards,

Walter Vos




Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor 
(gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke 
informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of 
verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden 
is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap



Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor 
(gebruik door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke 
informatie bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of 
verstrekking van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden 
is uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap

Re: Nifi with docker and LDAP

[Juan Pablo Gardella]

I will check, thanks. But what about if I would like to run as a cluster? I
cannot follow that approach right?

On Mon, 24 Sep 2018 at 12:59 Mike Thomsen  wrote:

> hostname: nifi
>
> Under the nifi declaration
> On Mon, Sep 24, 2018 at 11:07 AM Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
>> How?
>>
>> On Mon, 24 Sep 2018 at 11:31 David Gallagher <
>> dgallag...@cleverdevices.com> wrote:
>>
>>> Hi – not sure if it helps, but you can set a static hostname in your
>>> docker-compose.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Dave
>>>
>>>
>>>
>>> *From:* Juan Pablo Gardella 
>>> *Sent:* Sunday, September 23, 2018 3:43 PM
>>> *To:* users@nifi.apache.org
>>> *Subject:* Nifi with docker and LDAP
>>>
>>>
>>>
>>> Hi all,
>>>
>>>
>>>
>>> I'm using Nifi with docker and it's secure.
>>>
>>>
>>>
>>> I'm facing an issue when I bounce my LAPTOP (I'm running it locally).
>>> After bouncing my lap, I cannot access to it (the container is running).
>>> The only workaround it's restart the service. I suppose it's something
>>> related to the host name. Any thoughts?
>>>
>>>
>>>
>>> Configuration:
>>>
>>>nifi:
>>> build:
>>>   context: .
>>>   dockerfile: Dockerfile-nifi
>>> image: myimageid
>>> container_name: nifi-d
>>> restart: always
>>> ports:
>>>   - 8443:8443
>>> depends_on:
>>>   - ldap
>>> environment:
>>>   AUTH: ldap
>>>
>>> (other variables)
>>>
>>>
>>>
>>> I think maybe it's related to the hostname. It's changed after bounce
>>> maybe.
>>>
>>>
>>>
>>> Juan
>>>
>>

Re: Nifi with docker and LDAP

[Mike Thomsen]

hostname: nifi

Under the nifi declaration
On Mon, Sep 24, 2018 at 11:07 AM Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:

> How?
>
> On Mon, 24 Sep 2018 at 11:31 David Gallagher 
> wrote:
>
>> Hi – not sure if it helps, but you can set a static hostname in your
>> docker-compose.
>>
>>
>>
>> Thanks,
>>
>>
>> Dave
>>
>>
>>
>> *From:* Juan Pablo Gardella 
>> *Sent:* Sunday, September 23, 2018 3:43 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Nifi with docker and LDAP
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I'm using Nifi with docker and it's secure.
>>
>>
>>
>> I'm facing an issue when I bounce my LAPTOP (I'm running it locally).
>> After bouncing my lap, I cannot access to it (the container is running).
>> The only workaround it's restart the service. I suppose it's something
>> related to the host name. Any thoughts?
>>
>>
>>
>> Configuration:
>>
>>nifi:
>> build:
>>   context: .
>>   dockerfile: Dockerfile-nifi
>> image: myimageid
>> container_name: nifi-d
>> restart: always
>> ports:
>>   - 8443:8443
>> depends_on:
>>   - ldap
>> environment:
>>   AUTH: ldap
>>
>> (other variables)
>>
>>
>>
>> I think maybe it's related to the hostname. It's changed after bounce
>> maybe.
>>
>>
>>
>> Juan
>>
>

Re: Nifi with docker and LDAP

[Juan Pablo Gardella]

How?

On Mon, 24 Sep 2018 at 11:31 David Gallagher 
wrote:

> Hi – not sure if it helps, but you can set a static hostname in your
> docker-compose.
>
>
>
> Thanks,
>
>
> Dave
>
>
>
> *From:* Juan Pablo Gardella 
> *Sent:* Sunday, September 23, 2018 3:43 PM
> *To:* users@nifi.apache.org
> *Subject:* Nifi with docker and LDAP
>
>
>
> Hi all,
>
>
>
> I'm using Nifi with docker and it's secure.
>
>
>
> I'm facing an issue when I bounce my LAPTOP (I'm running it locally).
> After bouncing my lap, I cannot access to it (the container is running).
> The only workaround it's restart the service. I suppose it's something
> related to the host name. Any thoughts?
>
>
>
> Configuration:
>
>nifi:
> build:
>   context: .
>   dockerfile: Dockerfile-nifi
> image: myimageid
> container_name: nifi-d
> restart: always
> ports:
>   - 8443:8443
> depends_on:
>   - ldap
> environment:
>   AUTH: ldap
>
> (other variables)
>
>
>
> I think maybe it's related to the hostname. It's changed after bounce
> maybe.
>
>
>
> Juan
>

RE: Nifi with docker and LDAP

[David Gallagher]

Hi – not sure if it helps, but you can set a static hostname in your 
docker-compose.

Thanks,

Dave

From: Juan Pablo Gardella 
Sent: Sunday, September 23, 2018 3:43 PM
To: users@nifi.apache.org
Subject: Nifi with docker and LDAP

Hi all,

I'm using Nifi with docker and it's secure.

I'm facing an issue when I bounce my LAPTOP (I'm running it locally). After 
bouncing my lap, I cannot access to it (the container is running). The only 
workaround it's restart the service. I suppose it's something related to the 
host name. Any thoughts?

Configuration:
   nifi:
build:
  context: .
  dockerfile: Dockerfile-nifi
image: myimageid
container_name: nifi-d
restart: always
ports:
  - 8443:8443
depends_on:
  - ldap
environment:
  AUTH: ldap
(other variables)

I think maybe it's related to the hostname. It's changed after bounce maybe.

Juan

RE: A sensible approach to scheduling via the API?

[Vos, Walter]

Hi,

I don't know what the etiquette on a mailing list is for this, but I'd like to 
bump my original question.

Perhaps it's good to add that many of our flows are batch loads and therefore 
depend on a schedule to run, once.

Does anyone have experience with remote scheduling in NiFi or do you think you 
have a smart take on this? Please let me know :)

Cheers,

Walter

-Oorspronkelijk bericht-
Van: Vos, Walter
Verzonden: woensdag 5 september 2018 10:02
Aan: users@nifi.apache.org
Onderwerp: A sensible approach to scheduling via the API?

Hi,

In our big data environment one of the architectural principles is to schedule 
jobs with Azure Automation (runbooks). A scheduling database is used to decide 
when to start which jobs. NiFi flows however are currently being scheduled in 
NiFi itself. We're looking for a good approach to move this over to runbooks. I 
see a couple of options:

* Have each flow start with a timer driven processor, where the run schedule is 
an hour or so. This processor will be stopped by default, and can be turned on 
via the API. It is then stopped at some point before the run schedule ends, 
preventing the processor from running twice.
* Use a ListenHTTP processor that we can POST a message to that specifies which 
flow to start. Do something like RouteOnAttribute to choose the right flow. I 
imagine this as being one ListenHTTP processor that is connected to all flows.
* Translate the schedule from the scheduling database to a ChronTrigger 
expression. Check if the CRON schedule on the processor is indeed set to that 
schedule. If not, stop the processor, change the schedule and start it again. 
If it is, do nothing and assume it'll run. This one seems convoluted on the one 
hand, but requires the least architecture within NiFi itself I imagine.

What do you think? Has anyone had to deal with something like this? How did you 
solve it? I can't find much information about this on the web, although I could 
be using the wrong terms.

Kind regards,

Walter Vos




Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik 
door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie 
bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking 
van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is 
uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap



Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor (gebruik 
door) de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie 
bevatten. Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking 
van (de inhoud van) deze e-mail (en eventuele bijlagen) aan derden is 
uitdrukkelijk niet toegestaan. Indien u niet de bedoelde geadresseerde bent, 
wordt u vriendelijk verzocht degene die de e-mail verzond hiervan direct op de 
hoogte te brengen en de e-mail (en eventuele bijlagen) te vernietigen.

Informatie vennootschap