Nifi and ldap connector

[LEPRINCE Martine]

Hello

I am a NIFI new User.

I'm looking for a NIFI connector to get LDAP information (for example, the list 
of LDAP groups whose name begins with xxx, the list of members in a particular 
LDAP group, etc).

Can anyone help me ?

Best regards




Martine LEPRINCE


[cid:image001.jpg@01D54C3E.761AD150]

www.thalesaleniaspace.com


[cid:image003.png@01D80867.D02A0CA0]
  [cid:image004.png@01D80867.D02A0CA0]    
[cid:image005.png@01D80867.D02A0CA0] 
   
[cid:image006.png@01D80867.D02A0CA0] 
   
[cid:image007.png@01D80867.D02A0CA0] 

Re: apache/nifi convenience Docker Image - publish JDK 11 version?

[Kevin Doran]

Thanks, Chris! I’ve incorporated these ideas into the Confluence page. I’ll 
look into getting you edit permissions - what is your username there?

> On Dec 18, 2021, at 13:35, Chris Sampson  
> wrote:
> 
> Kevin,
> 
> Good idea to write up a list of ideas - unfortunately I don't appear to
> have permission to edit the page, but a few other ideas are:
> 
>   - Provide an "alpine" based image (could be the
>   "openjdk:-alpine" image used as a base) to reduce image size and
>   O/S footprint of the container
>   - Change the Docker start scripts to allow for all nifi.properties (and
>   other config file properties) to be set by Environment Variables - this
>   could be a case of using a common Env Var prefix to and iterating through
>   them within the script, e.g. anything that is NIFI_* is assumed to be a
>   nifi.properties entry, etc. - this would make the convenience image all the
>   more convenient for people and follow suit with some other applications
>   using Docker wrappers
>   - Docker-focussed default config, for example logback.xml that directs
>   output to STDOUT/STDERR by default, possibly in JSON format, allowing
>   easier out-of-the-box use of the logs in Docker-based environments that
>   already process Docker log output
>   - Allow for the nifi.properties/config file changes during start.sh to
>   be skipped (some people inject the config files into their containers but
>   want them to be read-only, which then means the start.sh script fails
>   because the files can't be edited)
>   - Change the default config file locations within the Docker images,
>   e.g. the flow.xml.gz and other system generated config that should be
>   persisted through container restart would be better in a different folder
>   to things like nifi.properties, then the directory can be externalised as a
>   Volume (without running into read-only/mutability issues like the start.sh
>   point above)
>   - Make it easier for Docker-based instances to join an existing cluster
>   - this might be a case of relocating some of the files to make better use
>   of ephemeral vs. persistent directories/Volumes and/or having a
>   configurable mechanism that allows for the cluster config files to be
>   deleted during startup if the instance expects to join a cluster but cannot
>   because it is unable to overwrite existing files with those being offered
>   by the cluster (not sure how much of an issue this still is after some
>   recent changes)
> 
> Please feel free to add any/all of the above to the Confluence page that
> you think would be worthwhile to add to the pile for consideration.
> 
> 
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
> 
> On Fri, 17 Dec 2021 at 18:01, Kevin Doran  wrote:
> 
>> Hi Chris,
>> 
>> I like this idea and have been thinking of a number of other improvements
>> we could make for container images of Apache NiFi, including (I think
>> discussed on another thread) consolidating Dockerfiles for dockerhub /
>> dockermaven, as the only significant difference is where the nifi assembly
>> archive comes from, arm64 images that run native on ARM platforms,
>> including newer M1 Macs (requires some minor changes to NiFi source itself
>> to compile).
>> 
>> I’ve started a Wiki page [1] to collect ideas, and once we reach consensus
>> on the improvements we’d like to make and possible approaches, we can turn
>> these into Jiras.
>> 
>> [1]
>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
>> 
>> Thanks!
>> Kevin
>> 
>>> On Nov 26, 2021, at 07:46, Chris Sampson 
>> wrote:
>>> 
>>> I'm not sure whether I've seen this discussed previously, but I thought
>> it
>>> worth asking whether we could/should publish a JDK 11 based version of
>> the
>>> apache/nifi Docker Image to Docker Hub as part of the release process in
>>> future?
>>> 
>>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
>> gone
>>> into enabling JDK 11 compilation and deployment, along with the planned
>>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
>> start
>>> making it easier for people to move towards using JDK 11 now that it's
>>> available.
>>> 
>>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi or
>>> Registry) can be given a build ARG to use different base images/versions,
>>> so it should be relatively straightforward to do this and maybe publish
>> the
>>> Image to Docker Hub as *apache/nifi:-jdk11*?
>>> 
>>> Users can, of course, download the NiFi source to build this image
>>> themselves, but to make it easier for the community it would seem
>> sensible
>>> to consider the additional release artifact. This avoids lots of people
>>> having to pull all of the code (it's not a small repo) or build the
>> image,
>>> which can take a long time, etc.
>>> 
>>> 
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.samp...@naimuri.com
>> 
>> 

Re: apache/nifi convenience Docker Image - publish JDK 11 version?

[Chris Sampson]

Thanks Kevin, that's great.

My username is "chriss".


---
*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com


On Thu, 13 Jan 2022 at 16:38, Kevin Doran  wrote:

> Thanks, Chris! I’ve incorporated these ideas into the Confluence page.
> I’ll look into getting you edit permissions - what is your username there?
>
> > On Dec 18, 2021, at 13:35, Chris Sampson 
> wrote:
> >
> > Kevin,
> >
> > Good idea to write up a list of ideas - unfortunately I don't appear to
> > have permission to edit the page, but a few other ideas are:
> >
> >   - Provide an "alpine" based image (could be the
> >   "openjdk:-alpine" image used as a base) to reduce image size
> and
> >   O/S footprint of the container
> >   - Change the Docker start scripts to allow for all nifi.properties (and
> >   other config file properties) to be set by Environment Variables - this
> >   could be a case of using a common Env Var prefix to and iterating
> through
> >   them within the script, e.g. anything that is NIFI_* is assumed to be a
> >   nifi.properties entry, etc. - this would make the convenience image
> all the
> >   more convenient for people and follow suit with some other applications
> >   using Docker wrappers
> >   - Docker-focussed default config, for example logback.xml that directs
> >   output to STDOUT/STDERR by default, possibly in JSON format, allowing
> >   easier out-of-the-box use of the logs in Docker-based environments that
> >   already process Docker log output
> >   - Allow for the nifi.properties/config file changes during start.sh to
> >   be skipped (some people inject the config files into their containers
> but
> >   want them to be read-only, which then means the start.sh script fails
> >   because the files can't be edited)
> >   - Change the default config file locations within the Docker images,
> >   e.g. the flow.xml.gz and other system generated config that should be
> >   persisted through container restart would be better in a different
> folder
> >   to things like nifi.properties, then the directory can be externalised
> as a
> >   Volume (without running into read-only/mutability issues like the
> start.sh
> >   point above)
> >   - Make it easier for Docker-based instances to join an existing cluster
> >   - this might be a case of relocating some of the files to make better
> use
> >   of ephemeral vs. persistent directories/Volumes and/or having a
> >   configurable mechanism that allows for the cluster config files to be
> >   deleted during startup if the instance expects to join a cluster but
> cannot
> >   because it is unable to overwrite existing files with those being
> offered
> >   by the cluster (not sure how much of an issue this still is after some
> >   recent changes)
> >
> > Please feel free to add any/all of the above to the Confluence page that
> > you think would be worthwhile to add to the pile for consideration.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.samp...@naimuri.com
> >
> >
> > On Fri, 17 Dec 2021 at 18:01, Kevin Doran  wrote:
> >
> >> Hi Chris,
> >>
> >> I like this idea and have been thinking of a number of other
> improvements
> >> we could make for container images of Apache NiFi, including (I think
> >> discussed on another thread) consolidating Dockerfiles for dockerhub /
> >> dockermaven, as the only significant difference is where the nifi
> assembly
> >> archive comes from, arm64 images that run native on ARM platforms,
> >> including newer M1 Macs (requires some minor changes to NiFi source
> itself
> >> to compile).
> >>
> >> I’ve started a Wiki page [1] to collect ideas, and once we reach
> consensus
> >> on the improvements we’d like to make and possible approaches, we can
> turn
> >> these into Jiras.
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements
> >>
> >> Thanks!
> >> Kevin
> >>
> >>> On Nov 26, 2021, at 07:46, Chris Sampson  .INVALID>
> >> wrote:
> >>>
> >>> I'm not sure whether I've seen this discussed previously, but I thought
> >> it
> >>> worth asking whether we could/should publish a JDK 11 based version of
> >> the
> >>> apache/nifi Docker Image to Docker Hub as part of the release process
> in
> >>> future?
> >>>
> >>> I realise NiFi 1.x is/will remain based on JDK 8 but with all the work
> >> gone
> >>> into enabling JDK 11 compilation and deployment, along with the planned
> >>> move to JDK 11 in the future for NiFi 2.x, it would seem sensible to
> >> start
> >>> making it easier for people to move towards using JDK 11 now that it's
> >>> available.
> >>>
> >>> The Dockerfiles used to build NiFi (I've not looked at Toolkit, MiNiFi
> or
> >>> Registry) can be given a build ARG to use different base
> images/versions,
> >>> so it should be relatively straightforward to do this and maybe publish
> >> the
> >>> Image to Docker Hub as *apache/nifi:-jdk11*?
> >>>
> >>> Users can, of course, download the NiFi source to build this image
> >>> themselves, but to make i

NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments

[sanjeet rath]

Hi,

I encounter one   "java.net.NoRouteToHostException: No route to host (Host
unreachable)" in *Nifi UI login page  *.
The after debugging i realised when i am setting the proxy & port  in
*System.setProperty("proxy address") & System.setProperty("port address") *in
my custom processor. then this issue is appearing .

The other way i also replicated   when i am setting at the* jvm lable(in
bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
exception is coming in nifi ui login page:, After removal of this argument
it works fine.

Could someone help me to understand what could be the issue.

Nifi version: 1.12.1
cluster : 3 node (amazon EC2 linux cluster)

*Detail exception from nifi-app.log:*

o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/current-user to "exampledumyserveraddress" due to
java.net.NoRouteToHostException: No route to host (Host unreachable)

2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator

java.net.NoRouteToHostException: No route to host (Host unreachable)

at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)

at
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)

at
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)

at
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)

at
java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)

at java.base/java.net.Socket.connect(Socket.java:609)

at
okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)

at
okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)

at
okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)

at
okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)

at
okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)

at
okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)

at
okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)

at
okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)

at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)

at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)

at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)

at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)

at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)

at
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)

at okhttp3.RealCall.execute(RealCall.java:81)

at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132)

at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:126)

at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)

at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)

at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:834)





Thanks & Regards
-- 
Sanjeet Kumar Rath,
mob- +91 8777577470

Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments

[David Handermann]

Hi Sanjeet,

Thank you for providing the stack trace and details of your configuration.
Setting Java System properties in custom code is not a safe or supported
operation in NiFi.  As you have observed, setting system properties alters
the behavior of other components, leading to unexpected results.

If you need to support access through a proxy server, the
ProxyConfigurationService interface and StandardProxyConfigurationService
implementation provide a way to specify proxy server properties:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html

Regards,
David Handermann

On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath 
wrote:

> Hi,
>
> I encounter one   "java.net.NoRouteToHostException: No route to host (Host
> unreachable)" in *Nifi UI login page  *.
> The after debugging i realised when i am setting the proxy & port  in
> *System.setProperty("proxy address") & System.setProperty("port address")
> *in
> my custom processor. then this issue is appearing .
>
> The other way i also replicated   when i am setting at the* jvm lable(in
> bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
> exception is coming in nifi ui login page:, After removal of this argument
> it works fine.
>
> Could someone help me to understand what could be the issue.
>
> Nifi version: 1.12.1
> cluster : 3 node (amazon EC2 linux cluster)
>
> *Detail exception from nifi-app.log:*
>
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> java.net.NoRouteToHostException: No route to host (Host unreachable)
>
> 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>
> java.net.NoRouteToHostException: No route to host (Host unreachable)
>
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
>
> at
> java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
>
> at java.base/java.net.Socket.connect(Socket.java:609)
>
> at
> okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
>
> at
>
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
>
> at
>
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
>
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
>
> at
>
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
>
> at
>
> okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
>
> at
> okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
>
> at
> okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
>
> at
>
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
>
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
>
> at okhttp3.RealCall.execute(RealCall.java:81)
>
> at
>
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132)
>
> at
>
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:126)
>
> at
>
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
>
> at
>
> org.apache.nifi.cluster.coordinati

Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments

[sanjeet rath]

Thanks a lot for the quick response.

So u r suggesting we should not use proxy  in jvm argument(nifi
bootstrap.conf file) lable also as it will impact other component.like in
my case its impacting nifi-api.

Regards,
Sanjeet

On Fri, 14 Jan 2022, 12:19 am David Handermann, 
wrote:

> Hi Sanjeet,
>
> Thank you for providing the stack trace and details of your configuration.
> Setting Java System properties in custom code is not a safe or supported
> operation in NiFi.  As you have observed, setting system properties alters
> the behavior of other components, leading to unexpected results.
>
> If you need to support access through a proxy server, the
> ProxyConfigurationService interface and StandardProxyConfigurationService
> implementation provide a way to specify proxy server properties:
>
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
>
> Regards,
> David Handermann
>
> On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath 
> wrote:
>
> > Hi,
> >
> > I encounter one   "java.net.NoRouteToHostException: No route to host
> (Host
> > unreachable)" in *Nifi UI login page  *.
> > The after debugging i realised when i am setting the proxy & port  in
> > *System.setProperty("proxy address") & System.setProperty("port address")
> > *in
> > my custom processor. then this issue is appearing .
> >
> > The other way i also replicated   when i am setting at the* jvm lable(in
> > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
> > exception is coming in nifi ui login page:, After removal of this
> argument
> > it works fine.
> >
> > Could someone help me to understand what could be the issue.
> >
> > Nifi version: 1.12.1
> > cluster : 3 node (amazon EC2 linux cluster)
> >
> > *Detail exception from nifi-app.log:*
> >
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > java.net.NoRouteToHostException: No route to host (Host unreachable)
> >
> > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> >
> > java.net.NoRouteToHostException: No route to host (Host unreachable)
> >
> > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> Method)
> >
> > at
> > java.base/java.net
> > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> >
> > at
> > java.base/java.net
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> >
> > at
> > java.base/java.net
> > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
> >
> > at
> > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
> >
> > at java.base/java.net.Socket.connect(Socket.java:609)
> >
> > at
> > okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
> >
> > at
> >
> >
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
> >
> > at
> >
> >
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
> >
> > at
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
> >
> > at
> >
> >
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
> >
> > at
> >
> >
> okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
> >
> > at
> > okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
> >
> > at
> > okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
> >
> > at
> >
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
> >
> > at
> >
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
> >
> > at
> >
> >
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> >
> > at
> >
> >
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
> >
> > at
> > okhttp3.RealCall.getResponseW

Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments

[David Handermann]

Hi Sanjeet,

That's correct.  Setting the JVM arguments for proxy server access can
create the problems you observed, and it is unlikely to work as expected
when it comes to proxy access.  Various NiFi components use different
methods for creating network connections.  For this reason, enabling
outgoing proxy server access will be more reliable using the
ProxyConfigurationService.  If there specific components that have issues
with proxy server connectivity, it would be worth creating a Jira issue for
those components.  If proxy server access is limited to your custom
component, then the best approach is to integrate the
ProxyConfigurationService and determine how that should be wired to your
custom component library.

Regards,
David Handermann

On Thu, Jan 13, 2022 at 1:02 PM sanjeet rath  wrote:

> Thanks a lot for the quick response.
>
> So u r suggesting we should not use proxy  in jvm argument(nifi
> bootstrap.conf file) lable also as it will impact other component.like in
> my case its impacting nifi-api.
>
> Regards,
> Sanjeet
>
> On Fri, 14 Jan 2022, 12:19 am David Handermann, <
> exceptionfact...@apache.org>
> wrote:
>
> > Hi Sanjeet,
> >
> > Thank you for providing the stack trace and details of your
> configuration.
> > Setting Java System properties in custom code is not a safe or supported
> > operation in NiFi.  As you have observed, setting system properties
> alters
> > the behavior of other components, leading to unexpected results.
> >
> > If you need to support access through a proxy server, the
> > ProxyConfigurationService interface and StandardProxyConfigurationService
> > implementation provide a way to specify proxy server properties:
> >
> >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath 
> > wrote:
> >
> > > Hi,
> > >
> > > I encounter one   "java.net.NoRouteToHostException: No route to host
> > (Host
> > > unreachable)" in *Nifi UI login page  *.
> > > The after debugging i realised when i am setting the proxy & port  in
> > > *System.setProperty("proxy address") & System.setProperty("port
> address")
> > > *in
> > > my custom processor. then this issue is appearing .
> > >
> > > The other way i also replicated   when i am setting at the* jvm
> lable(in
> > > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application
> this
> > > exception is coming in nifi ui login page:, After removal of this
> > argument
> > > it works fine.
> > >
> > > Could someone help me to understand what could be the issue.
> > >
> > > Nifi version: 1.12.1
> > > cluster : 3 node (amazon EC2 linux cluster)
> > >
> > > *Detail exception from nifi-app.log:*
> > >
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET
> > > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > >
> > > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > >
> > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > >
> > > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> > Method)
> > >
> > > at
> > > java.base/java.net
> > > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> > >
> > > at
> > > java.base/java.net
> > >
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> > >
> > > at
> > > java.base/java.net
> > > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
> > >
> > > at
> > > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
> > >
> > > at java.base/java.net.Socket.connect(Socket.java:609)
> > >
> > > at
> > > okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
> > >
> > > at
> > >
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
> > >
> > > at
> > > okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
> > >
> > > at
> > >
> okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
> > >
> > > at
> > >
> > >
> >
> okhttp3.i

Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments

[sanjeet rath]

Thanks David,

Really appreciate it.
It clarified everything.

On Fri, 14 Jan 2022, 12:40 am David Handermann, 
wrote:

> Hi Sanjeet,
>
> That's correct.  Setting the JVM arguments for proxy server access can
> create the problems you observed, and it is unlikely to work as expected
> when it comes to proxy access.  Various NiFi components use different
> methods for creating network connections.  For this reason, enabling
> outgoing proxy server access will be more reliable using the
> ProxyConfigurationService.  If there specific components that have issues
> with proxy server connectivity, it would be worth creating a Jira issue for
> those components.  If proxy server access is limited to your custom
> component, then the best approach is to integrate the
> ProxyConfigurationService and determine how that should be wired to your
> custom component library.
>
> Regards,
> David Handermann
>
> On Thu, Jan 13, 2022 at 1:02 PM sanjeet rath 
> wrote:
>
> > Thanks a lot for the quick response.
> >
> > So u r suggesting we should not use proxy  in jvm argument(nifi
> > bootstrap.conf file) lable also as it will impact other component.like in
> > my case its impacting nifi-api.
> >
> > Regards,
> > Sanjeet
> >
> > On Fri, 14 Jan 2022, 12:19 am David Handermann, <
> > exceptionfact...@apache.org>
> > wrote:
> >
> > > Hi Sanjeet,
> > >
> > > Thank you for providing the stack trace and details of your
> > configuration.
> > > Setting Java System properties in custom code is not a safe or
> supported
> > > operation in NiFi.  As you have observed, setting system properties
> > alters
> > > the behavior of other components, leading to unexpected results.
> > >
> > > If you need to support access through a proxy server, the
> > > ProxyConfigurationService interface and
> StandardProxyConfigurationService
> > > implementation provide a way to specify proxy server properties:
> > >
> > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I encounter one   "java.net.NoRouteToHostException: No route to host
> > > (Host
> > > > unreachable)" in *Nifi UI login page  *.
> > > > The after debugging i realised when i am setting the proxy & port  in
> > > > *System.setProperty("proxy address") & System.setProperty("port
> > address")
> > > > *in
> > > > my custom processor. then this issue is appearing .
> > > >
> > > > The other way i also replicated   when i am setting at the* jvm
> > lable(in
> > > > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application
> > this
> > > > exception is coming in nifi ui login page:, After removal of this
> > > argument
> > > > it works fine.
> > > >
> > > > Could someone help me to understand what could be the issue.
> > > >
> > > > Nifi version: 1.12.1
> > > > cluster : 3 node (amazon EC2 linux cluster)
> > > >
> > > > *Detail exception from nifi-app.log:*
> > > >
> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> > GET
> > > > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > > >
> > > > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > > >
> > > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > > >
> > > > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> > > Method)
> > > >
> > > > at
> > > > java.base/java.net
> > > > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> > > >
> > > > at
> > > > java.base/java.net
> > > >
> > >
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> > > >
> > > > at
> > > > java.base/java.net
> > > > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
> > > >
> > > > at
> > > > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
> > > >
> > > > at java.base/java.net.Socket.connect(Socket.java:609)
> > > >
> > > > at
> > > > okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
> > > >
> > > > at
> > > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> okhttp3.internal.connection.ExchangeFinder.findH

[VOTE] Release Apache NiFi 1.15.3 (rc1)

[Joe Witt]

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.15.3.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1194

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.3/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.15.3-RC1
The Git commit ID is 753c311382005acadc16c64952d7b1eaaf2550d5
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=753c311382005acadc16c64952d7b1eaaf2550d5

Checksums of nifi-1.15.3-source-release.zip:
SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b
SHA512: 
cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

11 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351203

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3

The vote will be open
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test.
Then please vote:

[ ] +1 Release this package as nifi-1.15.3
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: Backfilling in Nfi

[Bryan Bende]

Hello,

The GetSplunk processor has 3 time range strategies...

- Managed from beginning
- Managed from current
- Provided

The "managed" strategies store the last successful time range in
processor state, so when splunk is down it should keep trying to query
Splunk and failing and never updating the state, so each next
execution would still have the start time as the end of the last
success, and therefore would capture the whole range that Splunk was
down.

It looks like you may be using the "provided" strategy since you are
specifying values for Earliest and Latest. The provided strategy was
meant to just query exactly what is provided, so it does not store
state or create incremental time ranges automatically, which means you
won't get the backfill in this case.

Thanks,

Bryan


On Wed, Jan 12, 2022 at 4:53 PM Chirthani, Deepak Reddy
 wrote:
>
> Hey Mark,
>
> Good to see you. I'm specifically talking about GetSplunk Processor which we 
> use to pull data previous 15 minutes data(earliest time: -15m@m and latest 
> time: now()) from the Splunk Enterprise every 15 mins. The processor 
> absolutely works fine. But it is Splunk with which we have issues of getting 
> down so that we miss data when it is down. So trying to find a way of 
> backfilling the data. Does this makes sense? Let me know if you have further 
> questions.
>
>
>
> On 2022/01/11 22:36:39 "Chirthani, Deepak Reddy" wrote:
>
> > Hello,
>
> >
>
> > Is Nifi good at Backfilling the data? Let's say that I have a dataflow 
> > which is cron scheduled to run every 15 mins. Lets say that the source was 
> > down and the dataflow did not pulled any data for a couple of runs and 
> > after some time the source is up and running. If this happens frequently, 
> > how do you handle backfilling in Nifi? Please let me know if there is any 
> > automated way of doing this?
>
> >
>
> > Thanks
>
> > E-MAIL CONFIDENTIALITY NOTICE:
>
> > The contents of this e-mail message and any attachments are intended solely 
> > for the addressee(s) and may contain confidential and/or legally privileged 
> > information. If you are not the intended recipient of this message or if 
> > this message has been addressed to you in error, please immediately alert 
> > the sender by reply e-mail and then delete this message and any 
> > attachments. If you are not the intended recipient, you are notified that 
> > any use, dissemination, distribution, copying, or storage of this message 
> > or any attachment is strictly prohibited.
>
> >
> E-MAIL CONFIDENTIALITY NOTICE:
> The contents of this e-mail message and any attachments are intended solely 
> for the addressee(s) and may contain confidential and/or legally privileged 
> information. If you are not the intended recipient of this message or if this 
> message has been addressed to you in error, please immediately alert the 
> sender by reply e-mail and then delete this message and any attachments. If 
> you are not the intended recipient, you are notified that any use, 
> dissemination, distribution, copying, or storage of this message or any 
> attachment is strictly prohibited.

Re: [VOTE] Release Apache NiFi 1.15.3 (rc1)

[David Handermann]

+1 (binding)

- Verified hashes and signatures
- Built from source on Ubuntu 20.04 using Azul Zulu 1.8.0.312
- Ran NiFi on Azul Zulu 1.8.0.312 and 11.0.13
- Verified Single User Authentication and credentials configuration

- NIFI-7089 Verified successful build without SFTP test failures
- NIFI-7749 Verified ListSFTP operation using authenticated HTTP proxy
- NIFI-7835 Verified ListSFTP and FetchSFTP operation using authentication
SOCKS proxy
- NIFI-9530 Verified absence of google-pubsublite library and successful
invocation of ConsumeGCPubSub
- NIFI-9534 Verified Log4j 2 API libraries with version 2.17.1 in
nifi-elasticsearch-5-nar
- NIFI-9566 Verified Logback version 1.2.10 included in NiFi library
directory
- Started NiFi Registry on Java 11 and created sample bucket for testing
- Ran NiFi Encrypt Config Toolkit and verified successful NiFi properties
encryption

Thanks Joe!

Regards,
David Handermann

On Thu, Jan 13, 2022 at 2:44 PM Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.15.3.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1194
>
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.3/
>
> A helpful reminder on how the release candidate verification process works:
>
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>
> The Git tag is nifi-1.15.3-RC1
> The Git commit ID is 753c311382005acadc16c64952d7b1eaaf2550d5
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=753c311382005acadc16c64952d7b1eaaf2550d5
>
> Checksums of nifi-1.15.3-source-release.zip:
> SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b
> SHA512:
> cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 11 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351203
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3
>
> The vote will be open
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
>
> [ ] +1 Release this package as nifi-1.15.3
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>