Re: flowfiles stuck in load balanced queue; nifi 1.8
Dan,
You would want to get diagnostics for the processor that is the
source/destination of the connection - not the FlowFile. But if you connection
is connecting 2 process groups then both its source and destination are Ports,
not Processors. So the easiest thing to do would be to drop a “dummy processor”
into the flow between the 2 groups, drag the Connection to that processor, get
diagnostics for the processor, and then drag it back to where it was. Does that
make sense? Sorry for the hassle.
Thanks
-Mark
Sent from my iPhone
On Dec 24, 2018, at 11:40 AM, dan young
mailto:danoyo...@gmail.com>> wrote:
Hello Bryan,
Thank you, that was the ticket!
Mark, I was able to run the diagnostics for a processor that's downstream from
the connection where the flowfile appears to be "stuck". I'm not sure what
processor is the source of this particular "stuck" flowfile since we have a
number of upstream processor groups (PG) that feed into a funnel. This funnel
is then connected to a downstream PG. It is this connection between the funnel
and a downstream PG where the flowfile is stuck. I might reduce the upstream
"load balanced connections" between the various PGs to just one so I can narrow
where we need to run diagnostics If this isn't the correct processor to be
gathering diagnostics, please LMK where else I should look or other diagnostics
to run...
I've also attached the output (nifi-api/connections/{id}) of the get for that
connection where the flowfile appears to be "stuck"
On Sun, Dec 23, 2018 at 8:36 PM Bryan Bende
mailto:bbe...@gmail.com>> wrote:
You’ll need to get the token that was obtained when you logged in to the SSO
and submit it on the curl requests the same way the UI is doing on all requests.
You should be able to open chrome dev tool tools while in the UI and look at
one of the request/responses and copy the value of the 'Authorization’ header
which should be in the form ‘Bearer ’.
Then send this on the curl command by specifying a header of -H 'Authorization:
Bearer '
On Sun, Dec 23, 2018 at 6:28 PM dan young
mailto:danoyo...@gmail.com>> wrote:
I forgot to mention that we're using the OpenId Connect SSO . Is there a way
to run these command via curl when we have the cluster configured this way? If
so would anyone be able to provide some insight/examples.
Happy Holidays!
Regards,
Dano
On Sun, Dec 23, 2018 at 3:53 PM dan young
mailto:danoyo...@gmail.com>> wrote:
This is what I'm seeing in the logs when I try to access the
nifi-api/flow/about for example...
2018-12-23 22:51:45,579 INFO [NiFi Web Server-24201]
o.a.n.w.s.NiFiAuthenticationFilter Authentication success for
d...@looker.com
2018-12-23 22:52:01,375 INFO [NiFi Web Server-24136]
o.a.n.w.a.c.AccessDeniedExceptionMapper identity[anonymous], groups[none] does
not have permission to access the requested resource. Unknown user with
identity 'anonymous'. Returning Unauthorized response.
On Sun, Dec 23, 2018 at 3:50 PM dan young
mailto:danoyo...@gmail.com>> wrote:
Hello Mark,
I have a queue again with a "stuck/phantom" flowfile again. When I try to call
the nifi-api/processors//diagnostics against a processor, in the
UI after I authenticate, I get a "Unknown user with identity 'anonymous'.
Contact the system administrator." We're running a secure 3x node cluster. I
tried this via the browser and also via the command line with curl on one of
the nodes. One clarification point, what processor id should I be trying to
gather the diagnostics on? the the queue is in between two processor groups.
Maybe the issue with the Unknown User has to do with some policy I don't have
set correctly?
Happy Holidays!
Regards,
Dano
On Wed, Dec 19, 2018 at 6:51 AM Mark Payne
mailto:marka...@hotmail.com>> wrote:
Hey Josef, Dano,
Firstly, let me assure you that while I may be the only one from the NiFi side
who's been engaging on debugging
this, I am far from the only one who cares about it! :) This is a pretty big
new feature that was added to the latest
release, so understandably there are probably not yet a lot of people who
understand the code well enough to
debug. I have tried replicating the issue, but have not been successful. I have
a 3-node cluster that ran for well over
a month without a restart, and i've also tried restarting it every few hours
for a couple of days. It has about 8 different
load-balanced connections, with varying data sizes and volumes. I've not been
able to get into this situation, though,
unfortunately.
But yes, I think that we've seen this issue arise from each of the two of you
and one other on the mailing list, so it
is certainly something that we need to nail down ASAP. Unfortunately, debugging
an issue that involves communication
between multiple nodes is often difficult to fully understand, so it may not be
a trivial task to debug.
Dano, if you are able to get to the diagnostics, as Josef mentioned, that is
likely to be
Re: InvokeHTTP failure to schedule fro CRON?
Could you try using an explicit path to the cacerts provided by your
JDK/JRE, instead of referring to $JAVA_HOME?
Tried without success...
Were you able to successfully start the SSLContextService after configuring
it?
Yes
InvokeHTTP needs to present a certificate that is signed by a CA that is in
the default cacerts
Not sure how to identify one that is supposed to be presented
On Sun, Dec 23, 2018 at 1:32 PM Jeff wrote:
> Could you try using an explicit path to the cacerts provided by your
> JDK/JRE, instead of referring to $JAVA_HOME? Andy gave an example of
> "/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts",
> which you would update with the path to the JDK you are using. Referencing
> an environment variable (without using EL) will not work for a NiFi
> property. It does not appear that EL is supported for the keystore and
> truststore properties, as that could lead to security issues. Those
> properties have validators that should also verify that the
> keystore/truststore exists and is readable. Were you able to successfully
> start the SSLContextService after configuring it?
>
> Also, as Andy mentioned, the URL you are using in InvokeHTTP needs to
> present a certificate that is signed by a CA that is in the default
> cacerts. Can you please verify this? You can get a list of what is
> contained in cacerts by using keytool, and specifying the path to cacerts,
> the password, and the list command. For example:
>
> keytool -storepass changeit -keystore
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/security/cacerts
> -list
>
> - Jeff
>
> On Fri, Dec 21, 2018 at 2:55 PM l vic wrote:
>
>> I put "default" parameters for trust-store:
>> Path: $JAVA_HOME/jre/lib/security/cacerts
>> Password: changeit (default)
>> Type: JKS
>> and got "invalid path" exception ( see below)
>> How does that missing cert file should look like?
>> Thanks again...
>>
>> 2018-12-21 14:46:00,021 ERROR [Timer-Driven Process Thread-1]
>> o.a.nifi.processors.standard.InvokeHTTP
>> InvokeHTTP[id=0929346d-d742-1fd9-e41a-8e4324b73349] Yielding processor due
>> to exception encountered as a source processor:
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target: {}
>>
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target
>>
>>at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>
>>at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964)
>>
>>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
>>
>>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>
>>at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
>>
>>at
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>
>>at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>
>>at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
>>
>>at
>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:267)
>>
>>at
>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:237)
>>
>>at
>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:148)
>>
>>at
>> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:186)
>>
>>at
>> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
>>
>>at
>> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
>>
>>at
>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
>>
>>at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>
>>at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>
>>at
>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
>>
>>at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterc
>> eptorChain.java:92)
>>
>>at
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>
>>at
>>
