Re: External Access using InvokeHTTP_Test processor and StandardSSLContextService

[Andy LoPresto]

Hi Valentina,

I am not sure why it would have worked on a different NiFi 1.10 instance 
previously unless the remote host introduced TLS/HSTS during that time and 
that’s what you’re encountering. If you are connecting to an HTTPS endpoint, 
you will need to provide an SSLContextService with a truststore containing 
either the exact certificate or one of the signing certificates for the remote 
endpoint. If it’s a commonly-accessible commercially-signed endpoint, use the 
cacerts truststore as referenced in this thread. If it’s an internal endpoint 
or you want to be more restrictive, obtain the public certificate of that 
endpoint and manually create a truststore containing that cert. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 7, 2020, at 1:14 AM, Valentina Ivanova  wrote:
> 
> Hello everyone!
> 
> I am facing the same problem using InvokeHTTP 1.11.0 configured with the 
> default values (except for the Remote URL field): 
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
> 
> The SSL Context Service property in the processor is not configured at all. 
> Previously the same workflow worked on another machine with NiFi 1.10.
> 
> Many thanks in advance for your help!
> 
> Valentina
> 
> 
> 
> From: Andy LoPresto 
> Sent: Thursday, 6 August 2020 18:01
> To: users@nifi.apache.org 
> Subject: Re: External Access using InvokeHTTP_Test processor and 
> StandardSSLContextService
>  
> This is not a JVM issue. Josef is correct that the external site you are 
> trying to communicate with is presenting a certificate which the configured 
> NiFi truststore has no way to verify (it can’t find the “path” between cert X 
> and any of its signing certs to one already known by NiFi). 
> 
> The solution is to acquire the external server public certificate or a 
> signing certificate and import it directly into the truststore you have 
> configured for the InvokeHTTP processor via the SSLStandardContextService, or 
> reference a different truststore which already has the certificate present. 
> 
> If you are pointing at the same truststore you use for NiFi as an 
> application, it’s not suggested to import the external cert directly, as this 
> will have an impact on authentication mechanisms. Rather, use a new 
> truststore explicitly for this use case, or reference the JRE provided 
> “cacerts” truststore [1] directly from the SSLContextService (default 
> password is “changeit” and it comes with many commercial/public CAs imported 
> automatically, just like your browser or OS). 
> 
> [1] https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html 
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Aug 6, 2020, at 6:32 AM, Jorge Machado > > wrote:
>> 
>> Hi Dan, 
>> 
>> Seems like this is a jvm issue. 
>> 
>> Try this: 
>> https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html
>>  
>> 
>> Diagnosis
>> Use SSL Poke to verify connectivity
>> Try the Java class SSLPoke to see if your truststore contains the right 
>> certificates. This will let you connect to a SSL service, send a byte of 
>> input, and watch the output.
>> Download SSLPoke.class 
>> 
>> Execute the class as per the below, changing the URL and port appropriately. 
>> Take care that you are running the same Java as what Confluence is running 
>> with. If you used the installer you will need to use 
>> /jre/java
>> $JAVA_HOME/bin/java SSLPoke jira.example.com  443
>> A mail server may be mail.example.com  465 .
>> 
>> The jira.example.com  is your custom site. Add the 
>> CA Certs 
>> 
>> 
>> 
>>> On 6. Aug 2020, at 14:08, >> > >> > wrote:
>>> 
>>> It tells you most probably that the CA cert from the remote HTTPS server 
>>> hasn’t been found in the truststore you’ve defined to access the site. So 
>>> please check again the CA cert and the truststore…
>>>  
>>> Cheers Josef
>>>  
>>>  
>>> From: "White, Daniel" mailto:daniel.wh...@lgim.com>>
>>> Reply to: "users@nifi.apache.org " 
>>> mailto:users@nifi.apache.org>>
>>> Date: Thursday, 6 August 2020 at 13:07
>>> To: "users@nifi.apache.org " 
>

Re: Get all available variables in the InvokeScriptedProcessor

[Andy LoPresto]

I think we need additional clarification on what you mean by “variables”. If 
you are referring to actual Groovy variables, you can enumerate them using the 
binding available in the context (see below). If you mean the attributes 
available on a flowfile, you can access them similarly. 

Find all variables starting with prefix: 

def varsStartingWithABC = this.binding.variables.findAll { k,v -> 
k.startsWith(“a.b.c”) }

Find all attributes starting with prefix:

def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v -> 
k.startsWith(“a.b.c”) }



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 7, 2020, at 2:11 AM, Saloni Udani  wrote:
> 
> Hi,
> We use NiFi 1.5.0.
> Our use case is to get particular key pattern variables (key and value) in 
> the groovy InvokeScriptedProcessor. E.g I want all variables whose key starts 
> with "a.b.c". By this I can write a generic logic on certain categories of 
> variables for further use.
> 
> Is there a way programmatically to get all variables with a certain key 
> pattern? Or for that matter is there a way programmatically to get all 
> available variables Map ? In NiFi 1.5.0 or further versions.
> 
> 
> Thanks
> Saloni Udani

Error when attempting to load balance

[James McMahon]

We are running 1.8 processors, and in the queue following our ListFile and
GetFile processors are trying to configure "Round robin" load balancing.

We have four nodes in our cluster. At present let's say the primary has a
UI URL of https://8.93.20.100:8443. The three other nodes are similar, with
101, 102, and 103.

I have load balancing in nifi.properties configured to use port 6342, which
is open on my hosts.

Load balancing is failing. All the flowfiles stay on the primary node. The
log says this, once for each partner node we are trying to balance with:
Unable to connect to 8.93.20.101:8443 for load balancing
java.net.NoRouteToHostException: No route to host

In nifi.properties I have tried three ways of setting
nifi.cluster.load.balance.host, none of which resolved the problem:
1. I left it blank, and let it use its default which is
nifi.cluster.node.address
2. I attempted to use the host name, which for each is something like this:
abc-def-nic01i [02i, 03i, and 04i on those hosts] . My host names resolve
to IPs in local node /etc/host files that have 6342 open on them.
3. I tried using the IP value the name alias maps to, putting it directly
in this config parm

What is this error telling me, and how do I fix this so that I can employ
load balancing properly in my cluster configuration?

Thank you in advance for any help.

Re: early eof

[Chris Sampson]

Worth looking at the sender's settings as well maybe - are they closing the
connection prematurely (for example, some kind of timeout)?


*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com



On Fri, 7 Aug 2020 at 10:50, Pierre Villard 
wrote:

> Can you share the full stack trace? Anything specific between your client
> and the ListenHTTP? Load balancer?
>
> I just did some tests and I could upload a 5GB file with no issue. I even
> tried with the throttling parameters so that the upload takes a very long
> time and it was fine as well.
>
> Le jeu. 6 août 2020 à 18:21, Michael Di Domenico 
> a écrit :
>
>> On Thu, Aug 6, 2020 at 12:15 PM Joe Witt  wrote:
>> >
>> > ...I am confident far larger objects are sent via this mechanism
>> reliably.  Can you share what happens on the nifi side specifically?  There
>> should be a stack trace and log entry for this in the logs.
>> > What are the settings you have in NiFi for this processor?  How much
>> storage space does your content repository have access to?
>>
>> I have no doubt.  something is odd i just can't figure it out.  There
>> absolutely is a stack trace, but it's not easy for me to post.
>>
>> on my nifi side, i see the file start to down, up until it hits
>> between 400-500MB, and then it stops and restarts.  i'll retype some
>> of the trace
>>
>> ERROR o.a.nifi.processors.standard.ListenHTTP Unable to receive file
>> from Remote Host
>> org.apache.nifi.process.exception.Process Exception: IOException
>> thrown from ListenHTTP java.nio.channels.ClosedChannelException
>> ...then a bunch of java routine errors...
>> Suppressed: java.lang.Throwable: HttpInput failure
>>
>> there's plenty of storage, i only found this because nifi filled the
>> partition with incomplete files.
>>
>> which settings do you want from listenhttp?
>>
>

Re: early eof

[Pierre Villard]

Can you share the full stack trace? Anything specific between your client
and the ListenHTTP? Load balancer?

I just did some tests and I could upload a 5GB file with no issue. I even
tried with the throttling parameters so that the upload takes a very long
time and it was fine as well.

Le jeu. 6 août 2020 à 18:21, Michael Di Domenico  a
écrit :

> On Thu, Aug 6, 2020 at 12:15 PM Joe Witt  wrote:
> >
> > ...I am confident far larger objects are sent via this mechanism
> reliably.  Can you share what happens on the nifi side specifically?  There
> should be a stack trace and log entry for this in the logs.
> > What are the settings you have in NiFi for this processor?  How much
> storage space does your content repository have access to?
>
> I have no doubt.  something is odd i just can't figure it out.  There
> absolutely is a stack trace, but it's not easy for me to post.
>
> on my nifi side, i see the file start to down, up until it hits
> between 400-500MB, and then it stops and restarts.  i'll retype some
> of the trace
>
> ERROR o.a.nifi.processors.standard.ListenHTTP Unable to receive file
> from Remote Host
> org.apache.nifi.process.exception.Process Exception: IOException
> thrown from ListenHTTP java.nio.channels.ClosedChannelException
> ...then a bunch of java routine errors...
> Suppressed: java.lang.Throwable: HttpInput failure
>
> there's plenty of storage, i only found this because nifi filled the
> partition with incomplete files.
>
> which settings do you want from listenhttp?
>

Get all available variables in the InvokeScriptedProcessor

[Saloni Udani]

Hi,
We use NiFi 1.5.0.
Our use case is to get particular key pattern variables (key and value) in
the groovy InvokeScriptedProcessor. E.g I want all variables whose key
starts with "a.b.c". By this I can write a generic logic on certain
categories of variables for further use.

Is there a way programmatically to get all variables with a certain key
pattern? Or for that matter is there a way programmatically to get all
available variables Map ? In NiFi 1.5.0 or further versions.


Thanks
Saloni Udani

RE: External Access using InvokeHTTP_Test processor and StandardSSLContextService

[White, Daniel]

Hi All,

Got this working fine by using the cacerts truststore.

Thanks for all the input.

Regards
Dan

From: Andy LoPresto 
Sent: 06 August 2020 17:02
To: users@nifi.apache.org
Subject: Re: External Access using InvokeHTTP_Test processor and 
StandardSSLContextService

CAUTION: This email originated from outside of the organisation. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.

This is not a JVM issue. Josef is correct that the external site you are trying 
to communicate with is presenting a certificate which the configured NiFi 
truststore has no way to verify (it can’t find the “path” between cert X and 
any of its signing certs to one already known by NiFi).

The solution is to acquire the external server public certificate or a signing 
certificate and import it directly into the truststore you have configured for 
the InvokeHTTP processor via the SSLStandardContextService, or reference a 
different truststore which already has the certificate present.

If you are pointing at the same truststore you use for NiFi as an application, 
it’s not suggested to import the external cert directly, as this will have an 
impact on authentication mechanisms. Rather, use a new truststore explicitly 
for this use case, or reference the JRE provided “cacerts” truststore [1] 
directly from the SSLContextService (default password is “changeit” and it 
comes with many commercial/public CAs imported automatically, just like your 
browser or OS).

[1] https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 6, 2020, at 6:32 AM, Jorge Machado mailto:jom...@me.com>> 
wrote:

Hi Dan,

Seems like this is a jvm issue.

Try this: 
https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html
Diagnosis
Use SSL Poke to verify connectivity
Try the Java class SSLPoke to see if your truststore contains the right 
certificates. This will let you connect to a SSL service, send a byte of input, 
and watch the output.
1. Download 
SSLPoke.class
2. Execute the class as per the below, changing the URL and port 
appropriately. Take care that you are running the same Java as what Confluence 
is running with. If you used the installer you will need to use 
/jre/java

$JAVA_HOME/bin/java SSLPoke jira.example.com 443
A mail server may be mail.example.com 465 .

The jira.example.com is your custom site. Add the CA 
Certs




On 6. Aug 2020, at 14:08, 
mailto:josef.zahn...@swisscom.com>> 
mailto:josef.zahn...@swisscom.com>> wrote:

It tells you most probably that the CA cert from the remote HTTPS server hasn’t 
been found in the truststore you’ve defined to access the site. So please check 
again the CA cert and the truststore…

Cheers Josef


From: "White, Daniel" mailto:daniel.wh...@lgim.com>>
Reply to: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Thursday, 6 August 2020 at 13:07
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: External Access using InvokeHTTP_Test processor and 
StandardSSLContextService

Confidential

Hi All,

We’ve setup the truststore from the NiFi processor. However we get the 
following error when trying to connect to an external HTTPS location

The error I get is: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

Any ideas? Assume this is a cert issue on the Nifi server.

Thanks

Dan White
Lead Technical Architect
Legal & General Investment Management
One Coleman Street, London, EC2R 5AA
Tel: +44 203 124 4048
Mob: +44 7980 027 656
www.lgim.com

This e-mail (and any attachments) may contain privileged and/or confidential 
information. If you are not the intended recipient please do not disclose, 
copy, distribute, disseminate or take any action in reliance on it. If you have 
received this message in error please reply and tell us and then delete it. 
Should you wish to communicate with us by e-mail we cannot guarantee the 
security of any data outside our own computer systems.

Any information contained in this message may be subject to applicable terms 
and conditions and must not be construed as giving investment advice within or 
outside the United Kingdom or Republic of Ireland.

Telephone Conversations may be recorded for your protection and to ensure 
quality of service

Legal & General Investment Management Limited (no 2091894), LGIM Real Assets 
(Operator) Limited (no 05522016), LGIM (Int

Re: External Access using InvokeHTTP_Test processor and StandardSSLContextService

[Valentina Ivanova]

Hello everyone!

I am facing the same problem using InvokeHTTP 1.11.0 configured with the 
default values (except for the Remote URL field):
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

The SSL Context Service property in the processor is not configured at all. 
Previously the same workflow worked on another machine with NiFi 1.10.

Many thanks in advance for your help!

Valentina




From: Andy LoPresto 
Sent: Thursday, 6 August 2020 18:01
To: users@nifi.apache.org 
Subject: Re: External Access using InvokeHTTP_Test processor and 
StandardSSLContextService

This is not a JVM issue. Josef is correct that the external site you are trying 
to communicate with is presenting a certificate which the configured NiFi 
truststore has no way to verify (it can’t find the “path” between cert X and 
any of its signing certs to one already known by NiFi).

The solution is to acquire the external server public certificate or a signing 
certificate and import it directly into the truststore you have configured for 
the InvokeHTTP processor via the SSLStandardContextService, or reference a 
different truststore which already has the certificate present.

If you are pointing at the same truststore you use for NiFi as an application, 
it’s not suggested to import the external cert directly, as this will have an 
impact on authentication mechanisms. Rather, use a new truststore explicitly 
for this use case, or reference the JRE provided “cacerts” truststore [1] 
directly from the SSLContextService (default password is “changeit” and it 
comes with many commercial/public CAs imported automatically, just like your 
browser or OS).

[1] https://docs.oracle.com/cd/E19830-01/819-4712/ablqw/index.html


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 6, 2020, at 6:32 AM, Jorge Machado mailto:jom...@me.com>> 
wrote:

Hi Dan,

Seems like this is a jvm issue.

Try this: 
https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html
Diagnosis
Use SSL Poke to verify connectivity

Try the Java class SSLPoke to see if your truststore contains the right 
certificates. This will let you connect to a SSL service, send a byte of input, 
and watch the output.

  1.  Download 
SSLPoke.class
  2.
Execute the class as per the below, changing the URL and port appropriately. 
Take care that you are running the same Java as what Confluence is running 
with. If you used the installer you will need to use /jre/java

$JAVA_HOME/bin/java SSLPoke jira.example.com 443

A mail server may be mail.example.com 465 .


The jira.example.com is your custom site. Add the CA 
Certs



On 6. Aug 2020, at 14:08, 
mailto:josef.zahn...@swisscom.com>> 
mailto:josef.zahn...@swisscom.com>> wrote:

It tells you most probably that the CA cert from the remote HTTPS server hasn’t 
been found in the truststore you’ve defined to access the site. So please check 
again the CA cert and the truststore…

Cheers Josef


From: "White, Daniel" mailto:daniel.wh...@lgim.com>>
Reply to: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Thursday, 6 August 2020 at 13:07
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: External Access using InvokeHTTP_Test processor and 
StandardSSLContextService

Confidential

Hi All,

We’ve setup the truststore from the NiFi processor. However we get the 
following error when trying to connect to an external HTTPS location

The error I get is: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

Any ideas? Assume this is a cert issue on the Nifi server.

Thanks

Dan White
Lead Technical Architect
Legal & General Investment Management
One Coleman Street, London, EC2R 5AA
Tel: +44 203 124 4048
Mob: +44 7980 027 656
www.lgim.com

This e-mail (and any attachments) may contain privileged and/or confidential 
information. If you are not the intended recipient please do not disclose, 
copy, distribute, disseminate or take any action in reliance on it. If you have 
received this message in error please reply and tell us and then delete it. 
Should you wish to communicate with us by e-mail we cannot guarantee the 
security of any data outside our own computer systems.

Any information contained in this message may be subject to applicable terms 
and conditions and must not be construed as giving investment advice within or 
o