Re: Insufficient permissions on initial start up (NiFi 2.0)

[Matt Gilman]

No problem. Glad you got it sorted.

Yet at the https URL in the browser, it still tells me it is insecure.
>

This is happening because the certificate is not issued by a trusted
Certificate Authority. The certificate from TinyCert is valid (the
connection works) but TinyCert is not a trusted CA from your browser's
perspective.

Matt

On Thu, Apr 25, 2024 at 4:44 PM James McMahon  wrote:

> When I learned that the initial user in authorizers.xml must match the
> certificate *exactly*, I figured it would be an easy matter to use openssl
> to inspect the cert. I wanted to be *absolutely* certain I matched it
> correctly.
> Here is the command:
> /opt/nifi/config_resources/keys$ openssl pkcs12 -info -in CN=admin2.p12
> -nodes
>
> Here is the output. The output informed me of what I thought was the
> subject info in the cert:
> ..
> subject=C = US, ST = Virginia, L = Reston, O = C4 Rampart, OU = NIFI, CN
> = admin2
> issuer=C = US, ST = Virginia, L = Reston, O = C4 Rampart, OU = Secure
> Digital Certificate Signing, CN = C4 Rampart CA
> -BEGIN CERTIFICATE-
> MIIExDCCA6ygAwIBAgICeYowDQYJKoZI..
>
> This is why I put that in the way that I did. I mean why on earth would it
> reverse the DN info and add extra spaces, right!? It's like having a
> separate knob for volume on the alarm clock in Seinfeld. Why separate knob,
> WHY?
>
> Anyway I reversed my entry and squashed the extra spacing. I updated
> authorizers.xml. I blew away authorizations.xml and users.xml so that nifi
> would recreate them at startup. I also fixed
> nifi.security.identity.mapping.pattern.dn and
> nifi.security.identity.mapping.value.dn in nifi.properties.I restarted
> nifi. And as Bryan, Matt, and Isha certainly already suspect, it worked.
>
> I still have one more thing to figure out. I've got the CA and user cert
> info in my browser. I've got a server cert. All have been generated by
> tinycert.org. Yet at the https URL in the browser, it still tells me it
> is insecure. I do not understand why.
> x Not secure  https://
> ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi/
>
> Anyway, thank you Bryan, Matt, and Isha for replying.
>
> On Wed, Apr 24, 2024 at 9:05 PM Matt Gilman 
> wrote:
>
>> What is this Access Token it cites at top?
>>>
>>
>> NiFi UI attempts to get the access token expiration. However, since
>> you're authenticating with a certificate the endpoint returns an
>> IllegalState because there was no token in the request.
>>
>> Looking at the logs and the supplied configuration it appears there are
>> spaces and the ordering is reversed in your initial admin identity when
>> compared with the value from the certificate.
>>
>> C = US, ST = Virginia, L = Reston, O = C4 Rampart, OU = NIFI, CN = admin2
>> CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virginia, C=US
>>
>> Hope this helps!
>>
>> Matt
>>
>> On Wed, Apr 24, 2024 at 8:41 PM James McMahon 
>> wrote:
>>
>>> Looking at the nifi-user.log, I find I am getting a Conflict response,
>>> Access Token not found.
>>>
>>>  more ./nifi-user.log
>>> 2024-04-25 00:23:49,329 INFO [main] o.a.n.a.FileUserGroupProvider
>>> Creating new users file at /opt/nifi/config_resources/users.xml
>>> 2024-04-25 00:23:49,352 INFO [main] o.a.n.a.FileAccessPolicyProvider
>>> Creating new authorizations file at
>>> /opt/nifi/config_resources/authorizations.xml
>>> 2024-04-25 00:23:49,573 INFO [main] o.a.n.a.FileAccessPolicyProvider
>>> Populating authorizations for Initial Admin: C = US, ST = Virginia, L =
>>> Reston, O = C4 Rampart, OU = NIFI,
>>> CN = admin2
>>> 2024-04-25 00:24:51,107 INFO [NiFi Web Server-100]
>>> o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 173.73.40.110
>>> [CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virgi
>>> nia, C=US] POST
>>> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/kerberos
>>> 2024-04-25 00:24:51,110 INFO [NiFi Web Server-100]
>>> o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=admin2,
>>> OU=NIFI, O=C4 Rampart, L=Reston, ST=Virginia, C=US] 173
>>> .73.40.110 POST
>>> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/kerberos
>>> 2024-04-25 00:24:51,166 INFO [NiFi Web Server-104]
>>> o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 173.73.40.110
>>> [CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virgi
>>> nia, C=US] GET
>>> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/token/expiration
>>> 

Re: Insufficient permissions on initial start up (NiFi 2.0)

[Matt Gilman]

>
> What is this Access Token it cites at top?
>

NiFi UI attempts to get the access token expiration. However, since you're
authenticating with a certificate the endpoint returns an IllegalState
because there was no token in the request.

Looking at the logs and the supplied configuration it appears there are
spaces and the ordering is reversed in your initial admin identity when
compared with the value from the certificate.

C = US, ST = Virginia, L = Reston, O = C4 Rampart, OU = NIFI, CN = admin2
CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virginia, C=US

Hope this helps!

Matt

On Wed, Apr 24, 2024 at 8:41 PM James McMahon  wrote:

> Looking at the nifi-user.log, I find I am getting a Conflict response,
> Access Token not found.
>
>  more ./nifi-user.log
> 2024-04-25 00:23:49,329 INFO [main] o.a.n.a.FileUserGroupProvider Creating
> new users file at /opt/nifi/config_resources/users.xml
> 2024-04-25 00:23:49,352 INFO [main] o.a.n.a.FileAccessPolicyProvider
> Creating new authorizations file at
> /opt/nifi/config_resources/authorizations.xml
> 2024-04-25 00:23:49,573 INFO [main] o.a.n.a.FileAccessPolicyProvider
> Populating authorizations for Initial Admin: C = US, ST = Virginia, L =
> Reston, O = C4 Rampart, OU = NIFI,
> CN = admin2
> 2024-04-25 00:24:51,107 INFO [NiFi Web Server-100]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 173.73.40.110
> [CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virgi
> nia, C=US] POST
> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/kerberos
> 2024-04-25 00:24:51,110 INFO [NiFi Web Server-100]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=admin2,
> OU=NIFI, O=C4 Rampart, L=Reston, ST=Virginia, C=US] 173
> .73.40.110 POST
> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/kerberos
> 2024-04-25 00:24:51,166 INFO [NiFi Web Server-104]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 173.73.40.110
> [CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virgi
> nia, C=US] GET
> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/token/expiration
> 2024-04-25 00:24:51,166 INFO [NiFi Web Server-104]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=admin2,
> OU=NIFI, O=C4 Rampart, L=Reston, ST=Virginia, C=US] 173
> .73.40.110 GET
> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/access/token/expiration
> 2024-04-25 00:24:51,172 WARN [NiFi Web Server-104]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> *Access Token not found. Returning Conflict
> response.java.lang.IllegalStateException: Access Token not found*
> at
> org.apache.nifi.web.api.AccessResource.getAccessTokenExpiration(AccessResource.java:463)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>
> followed by
>
> 2024-04-25 00:24:51,185 INFO [NiFi Web Server-100]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 173.73.40.110
> [CN=admin2, OU=NIFI, O=C4 Rampart, L=Reston, ST=Virgi
> nia, C=US] GET
> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/flow/current-user
> 2024-04-25 00:24:51,186 INFO [NiFi Web Server-100]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [CN=admin2,
> OU=NIFI, O=C4 Rampart, L=Reston, ST=Virginia, C=US] 173
> .73.40.110 GET
> https://ec2-44-219-227-80.compute-1.amazonaws.com:8443/nifi-api/flow/current-user
> 2024-04-25 00:24:51,192 INFO [NiFi Web Server-100]
> o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=admin2, OU=NIFI, O=C4
> Rampart, L=Reston, ST=Virginia, C=US], groups[]
> *does not have permission to access the requested resource. Unable to view
> the user interface. Returning Forbidden response.*
>
> What is this Access Token it cites at top?
>
> Based on what I have read in the documentation, nifi itself must be
> allowed to create the authorizations.xml file at initial startup. Is there
> a reason it would omit permission to view and use the UI?
>
> On Wed, Apr 24, 2024 at 5:18 PM Matt Gilman 
> wrote:
>
>> James,
>>
>> If you check the nifi-user.log in the logs directory, you should see
>> messages for the requests that are being rejected. In that log message you
>> should see the identity that you're authenticated with. Can you compare
>> that with the user that you've configured the policies for. Hopefully, that
>> will help point to where the issue is.
>>
>> Matt
>>
>> On Wed, Apr 24, 2024 at 5:03 PM James McMahon 
>> wrote:
>>
>>> I still cannot access my own NiFi 2.0 instance. I continue to get this
>>> r

Re: Insufficient permissions on initial start up (NiFi 2.0)

[Matt Gilman]

James,

If you check the nifi-user.log in the logs directory, you should see
messages for the requests that are being rejected. In that log message you
should see the identity that you're authenticated with. Can you compare
that with the user that you've configured the policies for. Hopefully, that
will help point to where the issue is.

Matt

On Wed, Apr 24, 2024 at 5:03 PM James McMahon  wrote:

> I still cannot access my own NiFi 2.0 instance. I continue to get this
> rejection:
>
> Insufficient Permissions
>
>- home
>
> Unable to view the user interface. Contact the system administrator.
>
>
> The canvas flashes for an instant when I try to hit my secure URL, but is
> immediately replaced with this rejection message.
>
> There is no error or warning in nifi-app.log
>
> Has anyone experienced a similar problem?
>
>
> Here is my authorizers.xml:
>
> 
> 
> file-user-group-provider
> org.apache.nifi.authorization.FileUserGroupProvider
> /opt/nifi/config_resources/users.xml
> 
> C = US, ST = Virginia, L
> = Reston, O = C4 Rampart, OU = NIFI, CN = admin2
> 
> 
> file-access-policy-provider
>
> org.apache.nifi.authorization.FileAccessPolicyProvider
> file-user-group-provider
> /opt/nifi/config_resources/authorizations.xml
> C = US, ST = Virginia, L =
> Reston, O = C4 Rampart, OU = NIFI, CN = admin2
> 
> 
> 
> 
> managed-authorizer
>
> org.apache.nifi.authorization.StandardManagedAuthorizer
> file-access-policy-provider
> 
> 
>
>
> Here is my authorizations.xml (nifi creates at first startup):
>
> 
> 
> 
>  resource="/flow" action="R">
> 
> 
>  resource="/data/process-groups/ca7090bc-018e-1000-6a92-e199f7d9e48e"
> action="R">
> 
> 
>  resource="/data/process-groups/ca7090bc-018e-1000-6a92-e199f7d9e48e"
> action="W">
> 
> 
>  resource="/process-groups/ca7090bc-018e-1000-6a92-e199f7d9e48e" action="R">
> 
> 
>  resource="/process-groups/ca7090bc-018e-1000-6a92-e199f7d9e48e" action="W">
> 
> 
>  resource="/restricted-components" action="W">
> 
> 
>  resource="/tenants" action="R">
> 
> 
>  resource="/tenants" action="W">
> 
> 
>  resource="/policies" action="R">
> 
> 
>  resource="/policies" action="W">
> 
> 
>  resource="/controller" action="R">
> 
> 
>  resource="/controller" action="W">
> 
> 
> 
> 
>
>
>
> Here is my users.xml (nifi creates at first startup):
>
> 
> 
> 
> 
>  identity="C = US, ST = Virginia, L = Reston, O = C4 Rampart, OU = NIFI, CN
> = admin2"/>
> 
> 
>
>
> On Wed, Apr 24, 2024 at 8:21 AM James McMahon 
> wrote:
>
>> I'll review this closely once again when I get back to this system
>> tonight - thanks very much for your reply, Isha.
>>
>> I also feel I need to look more closely in nifi.properties, at values I
>> have set for keys nifi.security.identity.mapping.[value, transform,
>> pattern].CN1
>>
>> I noticed some odd behavior and suspect it is a reflection of an issue I
>> have not set properly in my configuration:
>> The first time I started my 2.0 instance with my Initial Admin Identity
>> defined as shown, the UI in my browser actually presented me with a list
>> (of one) Personal cert to select from - the cert for admin2. I was in a
>> happy place: *finally*, nifi and the browser appeared to be in synch for
>> the Subject name in the cert.
>>
>> I selected this cert, but then was crushed by the rejection mentioned
>> above:
>>  Unable to view the user interface. Contact the system administrator.
>>  Insufficient Permissions home
>>
>> I restarted nifi so I could "tail -f" nifi-app.log.
>> After restart, I once again tried to hit my NiFi URL.
>> This time though, the browser failed to present the admin2 cert for
>> selection.  Shouldn't it have still presented that to me in the browser fro
>> my selection?
>> Do you have any thoughts why this behavior is occurring?
>>
>> Would you say it is it advisable to manually create by hand an
>> authorizations.xml file should I continue to experience Insufficient
>> Permissions problems? I recall reading that users.xml and
>> authorizations.xml - if absent at initial startup - should be created by
>> nifi from info in authorizers.xml. But this Insufficient Permissions makes
>> me suspect something is missing from authorizations.
>>
>> Jim
>>
>> On Wed, Apr 24, 2024 at 5:33 AM Isha Lamboo <
>> isha.lam...@virtualsciences.nl> wrote:
>>
>>> Hi James,
>>>
>>>
>>>
>>> Have you changed these settings in authorizers.xml since you first
>>> started NiFi? If so, you may need to delete users.xml and
>>> authorizations.xml.
>>>
>>> A new admin user will not be

Re: UI is not as responsive...

[Matt Gilman]

Josef,

There was a regression in Chrome 92.x that affects SVG heavy web
applications like NiFi. Here is the Chrome issues tracking this [1]. And
here is a Chrome Help thread discussing the matter [2].

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=1235045
[2]
https://support.google.com/chrome/thread/118284571/any-one-suffers-from-the-newest-ver-92-rendering-some-heavy-svg-jobs?hl=en

On Mon, Sep 6, 2021 at 10:12 AM  wrote:

> Hi guys
>
>
>
> We can confirm the slow browser behavior as well and it’s very annoying.
> We have single node NiFis as well multiple NiFi clusters with different
> sizes. It happens everywhere and is definitely browser specific. We’ve also
> tried to restart NiFi, but no change at all. It so slow that in 2-Node
> cluster and a PG with 300 processors it sometimes takes longer than the
> browser timeout to just ENTER the PG.
>
>
>
> It happens with Chrome 92.x and as well with Edge 93.x (both based on
> Chromium?). Firefox is way faster -> we switched over to Firefox. We don’t
> exactly know when the issue started, but we have definitely just slightly
> modified our workflows in the last 2-3 months and we were using NiFi 1.13.2
> and the same Java Version since multiple months. We are working with NiFi
> since 1.4.x, so we are not new into NiFi.
>
>
>
> We see that memory goes up fast when we try to open a PG with Chrome, but
> we don’t know what’s normal.
>
>
>
> To answer Marks questions:
>
>1. It’s faster when we zoom in/out in a way that NiFi stops rendering
>the stats
>2. GUI refresh for a single NiFi PG with 30 processors takes 2-3s
>while the logs shows at max  40-50milis for the GET.
>3. Network is fast as hell, no change there. As Firefox is way faster
>than Chrome/Edge I don’t think it’s a connectivity issue.
>- NiFi 1.13.2
>- Java 1.8.0_282
>- In “bigger” PGs with 300 processors it takes more than 10s to open
>the flow. Most of the time the browser windows crashes due to long timeout.
>
>
>
> Hope this helps.
>
>
>
> Could it be that Pierre referred to this issue/improvement for NiFi 1.15.0?
>
> https://issues.apache.org/jira/browse/NIFI-9061
>
>
>
> For us this is a major issue, but as we have a working alternative
> (Firefox) we didn’t raised a jira ticket yet.
>
>
>
> Cheers Josef
>
>
>
>
>
> *From: *Mark Payne 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Saturday, 4 September 2021 at 15:55
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: UI is not as responsive...
>
>
>
> OK so there are really 3 parts to consider here in order to understand
> what is making things sluggish:
>
>
>
> - Front-end rendering
>
> - Backend processing
>
> - Network between your browser and the back end
>
>
>
> So a few things to consider here:
>
>
>
> - If you’re seeing the sluggishness in a Process Group with only a few
> elements, that leads me to believe it’s probably NOT the browser rendering
> that’s an issue. But another thing to check, to help verify: zoom out using
> your mouse wheel to the point where NiFi no longer renders the stats on the
> processors. Once you reach this level of zoom, the rendering is much
> cheaper. Do you still the same lag, or is the lag less at this point?
>
>
>
> - To understand how long the backend is taking to process the request, you
> can add the following to your conf/logback.xml file:
>
>
>
> 
>
>This will cause nifi to log in the nifi-app.log file something like:
>
> GET /flow/1234 from localhost duration for Request ID 4567: 102 millis
>
>
>
>So watch the logs here. Are you seeing the request times in the logs
> are constantly very short while the UI takes a long time to render the
> request?
>
>
>
> - Do you have any idea what kind of latency and throughput you expect
> between the machine running the browser and the machine running nifi?
>
>
>
> Also, a few other things to understand:
>
> - What version of NiFi are you running?
>
> - What version of Java?
>
> - When you say the UI is not as responsive, what kind of delay are you
> seeing? 1 second to refresh the UI? 10 seconds?
>
>
>
> Thanks
>
> -Mark
>
>
>
>
>
>
>
> On Sep 3, 2021, at 1:42 PM, Jean-Sebastien Vachon 
> wrote:
>
>
>
> Hi Mark,
>
>
>
> thanks for the quick response. I am running a single stand-alone Nifi
> instance (1.13.2)
>
> I tried with a smaller group (1 input port and 8 processors), and I still
> experience slow downs.
>
>
>
> I've looked at the timing of the backend calls and everything seems in
> order.
>
>
>
> I am using Edge but some of my colleagues are using Firefox/Chrome and
> experienced the same.
>
>
>
> One of the flows we are dealing with is relatively complex and involves
> about 50 processors.
>
> I will try to split it into smaller groups and see how it goes.
>
>
>
> Thanks
>
>
>
> *Jean-Sébastien Vachon*
>
> Co-Founder & Architect
>
> *Brizo Data, Inc. www.brizodata.com
> *
> --
>
> *

Re: Indications in the UI of which cluster node hosts a “stuck” thread?

[Matt Gilman]

Hi Jim,

If you open the Summary page from the global menu you should see the active 
threads in parentheses next to the scheduled state. Find the row in question 
and click the cluster icon from the actions column. This will open a dialog 
with a node-wise breakdown. I believe that the thread count is one of the 
metrics that is broken down per node.

Hope this helps! Adding this breakdown to the main canvas would be a great 
addition. Maybe these breakdowns could be offered in a tooltip first each 
metric.

Matt

Sent from my iPhone

> On Jun 24, 2020, at 21:05, James McMahon  wrote:
> 
> 
> Our production nifi cluster is exhibiting repeated problems with threads that 
> do not end. It is happening with processors that have complex configurations 
> and dependencies (ConsumeAMQP), and - more troubling - it is also occurring 
> periodically for simple processors like ControlRate. I’ll have a Control 
> processor sitting in a running state with no active running thread,I select 
> Stop on that processor, get a thread I presume to be responsible for stopping 
> the processor, and that thread will never end. This renders my processor in a 
> useless state - not stopped, not really running, and not accessible to 
> reconfigure.
> 
> I read a blog by Pierre Villard on using nifi.sh for thread dumps. I’ll dig 
> into that. My questions:
> 
> 1. In a cluster, is there anything I can use in the UI to tell me which 
> cluster node hosts the bad thread? Digging through thread dumps from multiple 
> cluster nodes seems impractical, and I’m hoping there’s a way to zero in on a 
> node.
> 
> 2. What nifi system resources in my configuration influence the management 
> and well-being of these threads?
> 
> 3. Has anyone debugged such a thread issue in a clustered nifi environment, 
> and if so can you offer any tips based on your experience?
> 
> Thanks in advance for any help.
> Jim

Re: alternate web root directory?

[Matt Gilman]

Wyllys,

The guidance for the proxy configuration is to map it behind a single path.
In order to work with any number of context paths that NiFi supports the
proxying needs to be based on the root path. Let me know if I'm
misunderstanding something. Unfortunately, I'm not super familiar with
nginx ingress in k8s environments but it sounds like that session affinity
may not working right.

On Wed, Mar 18, 2020 at 5:37 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

> Yeah, I've read all of that and I have a semi-working configuration.  The
> problem is that when using OpenID tokens (oidc) in a clustered
> configuration, the node that requests the authentication is the only one
> that can validate it.  If a user authenticates to node-1, but then later
> node-2 gets a request with the token (because its clustered and the user
> has no control over which node handles the request), node-2 cannot verify
> the token and rejects it.   Even configuring sticky-sessions and cookie
> affinity in the nginx ingress proxy don't solve the problem as far as I can
> tell.
>
> I don't even know if having it all behind a single root path would make a
> difference for the authentication issue, it just makes setting up the
> reverse proxy configuration easier since you only need to worry about 1
> path instead of multiple.
>
>
>
> On Wed, Mar 18, 2020 at 2:46 PM Matt Gilman 
> wrote:
>
>> Wyllys,
>>
>> NiFi is comprised of any number of web applications. NiFi offers
>> extension points for custom processor configuration UIs and data type
>> viewers. These UIs can be bundled and discovered at runtime. These docs [1]
>> detail the steps necessary for proxying a NiFi instance.
>>
>> Thanks
>>
>> Matt
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration
>>
>> On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>>
>>>
>>> I'm surprised you haven't had lots of requests for this already.  As it
>>> stands now, I cannot figure out how to configure a secure cluster behind a
>>> reverse proxy (for example, in a kubernetes environment behind an nginx
>>> ingress) that also incorporates OpenID authentication from an external
>>> service. I was thinking that if the NiFi nodes were able to operate under a
>>> single root path, it might make it easier to reverse proxy all of the
>>> different paths that Nifi uses (/nifi, /nifi-api, for example) behind a
>>> single ingress.  I think having multiple ingress paths for the nifi service
>>> makes the reverse proxy configuration very complex when authentication
>>> tokens are involved.  Without authentication, it works fine.
>>>
>>> Thanks,
>>>   Wyllys Ingersoll
>>>
>>> On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto 
>>> wrote:
>>>
>>>> Hi Wyllys,
>>>>
>>>> As I started reading, I was going to suggest the proxy approach.
>>>> Unfortunately, at this time I am unaware of any way to change the paths
>>>> within NiFi itself - there would be substantial refactoring required to
>>>> make that an option. You can open a feature request Jira for that, or
>>>> perhaps the ability to inject a path prefix, but I expect it to be a high
>>>> level of effort to implement.
>>>>
>>>>
>>>> Andy LoPresto
>>>> alopre...@apache.org
>>>> *alopresto.apa...@gmail.com *
>>>> He/Him
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>
>>>> On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll <
>>>> wyllys.ingers...@keepertech.com> wrote:
>>>>
>>>>
>>>> Is there a way to configure nifi to use a different root directory for
>>>> web requests?
>>>>
>>>> We would like everything to be under a common root such as:
>>>> /XXX/nifi/...
>>>> /XXX/nifi-api/...
>>>>
>>>> Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult
>>>> to setup a reverse proxy that also can incorporate OpenID authentication
>>>> tokens to a secure backend cluster of nodes.
>>>>
>>>>
>>>>
>>>>
>>>>

Re: alternate web root directory?

[Matt Gilman]

Wyllys,

NiFi is comprised of any number of web applications. NiFi offers extension
points for custom processor configuration UIs and data type viewers. These
UIs can be bundled and discovered at runtime. These docs [1] detail the
steps necessary for proxying a NiFi instance.

Thanks

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration

On Wed, Mar 18, 2020 at 1:29 PM Wyllys Ingersoll <
wyllys.ingers...@keepertech.com> wrote:

>
> I'm surprised you haven't had lots of requests for this already.  As it
> stands now, I cannot figure out how to configure a secure cluster behind a
> reverse proxy (for example, in a kubernetes environment behind an nginx
> ingress) that also incorporates OpenID authentication from an external
> service. I was thinking that if the NiFi nodes were able to operate under a
> single root path, it might make it easier to reverse proxy all of the
> different paths that Nifi uses (/nifi, /nifi-api, for example) behind a
> single ingress.  I think having multiple ingress paths for the nifi service
> makes the reverse proxy configuration very complex when authentication
> tokens are involved.  Without authentication, it works fine.
>
> Thanks,
>   Wyllys Ingersoll
>
> On Wed, Mar 18, 2020 at 12:56 PM Andy LoPresto 
> wrote:
>
>> Hi Wyllys,
>>
>> As I started reading, I was going to suggest the proxy approach.
>> Unfortunately, at this time I am unaware of any way to change the paths
>> within NiFi itself - there would be substantial refactoring required to
>> make that an option. You can open a feature request Jira for that, or
>> perhaps the ability to inject a path prefix, but I expect it to be a high
>> level of effort to implement.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Mar 18, 2020, at 9:25 AM, Wyllys Ingersoll <
>> wyllys.ingers...@keepertech.com> wrote:
>>
>>
>> Is there a way to configure nifi to use a different root directory for
>> web requests?
>>
>> We would like everything to be under a common root such as:
>> /XXX/nifi/...
>> /XXX/nifi-api/...
>>
>> Having to proxy 2 (/nifi and /nifi-api) paths makes it very difficult to
>> setup a reverse proxy that also can incorporate OpenID authentication
>> tokens to a secure backend cluster of nodes.
>>
>>
>>
>>
>>

Re: Exception is showing in nifi UI users page

[Matt Gilman]

I would recommend upgrading. If I recall correctly, there were actually a
couple of JIRAs addressing different but similar issues dealing with users
or groups being removed from a directory server. Even if you could identify
and remove the problematic entries, you could possibly hit this again if
another user or group were removed later.

Thanks

On Fri, Mar 6, 2020 at 6:35 AM sanjeet rath  wrote:

> Hi ,
>
> The AD group names are present in Authoriser.xml file,
> In authorisation.xml file the group identifier (unique id) is present
> inside policy identifier tag.
> So could you please help me to understand where AD group name in
> authoriser.xml file and group identifier in authorisation.xml map together.
> So that i can delete it in authoriser.xml file.
>
> Thanks,
> Sanjeet
>
>
>
>
>
>
> On Fri, Mar 6, 2020 at 12:04 AM sanjeet rath 
> wrote:
>
>> Thanks Mat,
>>
>> For this quick response.
>>
>> Thanks a lot ,
>>
>> Sanjeet
>>
>> On Thu, 5 Mar 2020, 11:43 pm Matt Gilman, 
>> wrote:
>>
>>> I just responded to your StackOverflow post:
>>>
>>>
>>> https://stackoverflow.com/questions/60551242/nifi-user-addition-gives-u-null-pointer-exception/60551638#60551638
>>>
>>> I believe you'll need to upgrade to a version that addresses the BUG.
>>>
>>> Thanks!
>>>
>>> On Thu, Mar 5, 2020 at 1:10 PM sanjeet rath 
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> I am using nifi cluster from 1 month & i am able to add new user
>>>> policies everything.its a LDAP based user addition.
>>>> but suddenly from last 2 days , in nifi user addition page(after
>>>> clicking on users in nifi UI) i am getting Error message "An unexcepted
>>>> error has occure.please click logs for more details".
>>>> and in nifif-user.log i found the bellow log.
>>>>
>>>> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has
>>>> occurred: java.lang.NullPointerException. Returning Internal Server Error
>>>> response. java.lang.NullPointerException: null at
>>>> org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO.lambda$null$2(StandardPolicyBasedAuthorizerDAO.java:285)
>>>> at
>>>> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>>>> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553) at
>>>> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at
>>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>>>> at
>>>> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>>>> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
>>>> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>>>>
>>>> I am not able to figuring out where should i start looking out.
>>>> Could please someone help me to look a starting point where can check
>>>> and what need to be checked.
>>>>
>>>> Thanks,
>>>> Sanjeet
>>>>
>>>>
>>>> --
>>>> Sanjeet Kumar Rath,
>>>> mob- +91 8777577470
>>>>
>>>>
>
> --
> Sanjeet Kumar Rath,
> mob- +91 8777577470
>
>

Re: Exception is showing in nifi UI users page

[Matt Gilman]

I just responded to your StackOverflow post:

https://stackoverflow.com/questions/60551242/nifi-user-addition-gives-u-null-pointer-exception/60551638#60551638

I believe you'll need to upgrade to a version that addresses the BUG.

Thanks!

On Thu, Mar 5, 2020 at 1:10 PM sanjeet rath  wrote:

> Hi Team,
>
> I am using nifi cluster from 1 month & i am able to add new user policies
> everything.its a LDAP based user addition.
> but suddenly from last 2 days , in nifi user addition page(after clicking
> on users in nifi UI) i am getting Error message "An unexcepted error has
> occure.please click logs for more details".
> and in nifif-user.log i found the bellow log.
>
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException. Returning Internal Server Error response.
> java.lang.NullPointerException: null at
> org.apache.nifi.web.dao.impl.StandardPolicyBasedAuthorizerDAO.lambda$null$2(StandardPolicyBasedAuthorizerDAO.java:285)
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553) at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>
> I am not able to figuring out where should i start looking out.
> Could please someone help me to look a starting point where can check and
> what need to be checked.
>
> Thanks,
> Sanjeet
>
>
> --
> Sanjeet Kumar Rath,
> mob- +91 8777577470
>
>

Re: Unable to validate the access token.

[Matt Gilman]

Dweep,

Sorry you're having trouble with this. Can you please confirm the URL you
direct your browser to initially and compare that to the address you are
returned to following successful authentication? Would also be interested
in seeing if the value in the redirect_uri query parameter when logging in
with your identity provider.

On Wed, Sep 25, 2019 at 5:26 AM Dweep Sharma 
wrote:

> Hi All,
>
> I am receiving this error on my nifi cluster setup
>
> Error -ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService
> There was an error validating the JWT
> io.jsonwebtoken.JwtException: Unable to validate the access token.
> WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
> access to web api: Untrusted proxy CN=dp1.domain.com, OU=NIFI
>
> I think the request for access token is being served by secondary node
> instead of primary and due to this, there is a mismatch in authorizing the
> user to primary (1st node)
>
> I am using OpenID Connect for auth following the steps mentioned here ref:
> https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
> My cluster setup is 2nodes with open id connect hosted on AWS
>
> I have added open id SSO config in both  the nodes.
>
> The problem is if i run it in cluster mode I am getting this error,
> whereas if run as individual node, it is  working perfectly fine.
>
> NOTE: I am not using any load balancer in this case but I can see
> nifi-user logs in both the nodes even if I am trying to connect to one node.
>
> Logs from
> 2019-09-25 08:24:55,244 INFO [NiFi Web Server-20]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET
> https://:8443/nifi-api/flow/current-user (source ip:
> 10.5.1.182)
> 2019-09-25 08:24:55,260 INFO [NiFi Web Server-20]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication success for
> du...@domain.com
> 2019-09-25 08:24:55,372 INFO [NiFi Web Server-16]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<
> du...@domain.com>, OU=NIFI>) GET
> https://domain.com:8443/nifi-api/flow/current-user (source ip:
> xx.xx.xx.xx)
> 2019-09-25 08:24:55,375 WARN [NiFi Web Server-16]
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted
> proxy CN=, OU=NIFI
>
> 
> 2019-09-25 08:37:32,111 INFO [NiFi Web Server-16]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET
> https://dp1.domain.com:8443/nifi-api/flow/current-user
> 
> (source ip: 10.5.1.182)
> 2019-09-25 08:37:32,135 ERROR [NiFi Web Server-16]
> o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
> io.jsonwebtoken.JwtException: *Unable to validate the access token.*
> at
> org.apache.nifi.web.security.jwt.JwtService.parseTokenFromBase64EncodedString(JwtService.java:106)
> at
> org.apache.nifi.web.security.jwt.JwtService.getAuthenticationFromToken(JwtService.java:60)
> at
> org.apache.nifi.web.security.jwt.JwtAuthenticationProvider.authenticate(JwtAuthenticationProvider.java:48)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
> at
> org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:78)
> at
> org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:99)
> at
> org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
> at
> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
>
> Please suggest how to enable stickiness in this case. There was a similar
> issue mentioned here -
> https://community.cloudera.com/t5/Support-Questions/NIFI-Unable-to-validate-the-access-token-Accessing-GUI/m-p/214525
> 

Re: X-Forwarded-Context whitelisting not working

[Matt Gilman]

Curtis,

I haven't set this up recently but it was working the last time I tried it.
Just wanted to ensure you that were following the guidance in our admin
guide for standing up instances behind a proxy [1].

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration

On Tue, May 14, 2019 at 6:17 PM Curtis Ruck  wrote:

> I am attempting (unsuccessfully) to configure multiple numbered
> unclustered nifi instances behind a single reverse proxy vhost (external
> limitations on single vhost and no clustering)
>
> In my reverse proxy I have X-Forwarded-Context set and in debug logging
> both CatchAllFilter and SanitizeContextPathFilter see the passed context
> (/nifi1 ... /nifiN).  In debug logging though, CatchAllFilter isn't seeing
> any items from getWhitelistedContextPath() where as
> SanitizeContextPathFilter does show the items in the
> getWhitelistedContextPath().
>
> Since CatchAllFilter extends SanitizeContextPathFilter, it should work
> except CatchAllFilter isn't calling super.init() which means the
> private whitelistedContextPaths never gets initialized.
>
> Has anyone gotten Nifi working at a nested context path i.e. (/nifi1/nifi,
> /nifi1/nifi-api)?
>
> --
> Curtis Ruck
>

Re: How to Fetch Bulletins for Process Groups Recursivly

[Matt Gilman]

Bulletins for a Process Group should be available in the bulletin field in
each Process Group. For instance, if you query for a Process Group

/nifi-api/process-groups/{id}

it should contain the bulletins rolled up from all descendant components.

Matt

On Thu, May 2, 2019 at 9:33 AM Shawn Weeks 
wrote:

> How do I fetch all the bulletins for a Process Group including all of its
> children. When I provide a filter groupId= to the rest api
> /flow/bulletin-board it doesn’t include children.
>
>
>
> Thanks
>
> Shawn Weeks
>

Re: Empty "nifi users" page.

[Matt Gilman]

Yes, I believe the issue is that you have a user and a group with the same
name. The UUID's are determined deterministically based on the user
identity and group name. This is done so that we can ensure the same data
model across a cluster. While not ideal, this doesn't present any issue
managing them or the policies. However, the UI attempts to present them in
a single table using that UUID as an identifier. In this case, they are not
unique. We should likely do something to ensure that users and group have
unique identifiers regardless if there are name conflicts. I have filed a
JIRA for this [1]. If possible, you could work around this by ensuring that
your users and groups have unique names.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-6161

On Thu, Mar 28, 2019 at 12:38 PM DEHAY Aurelien 
wrote:

> Well, I can compare a lot of things I don’t really know where to search.
>
>
>
> What are to be unique? It’s the UUID (id field) present in
> users/userGroups endpoints? The identity (short names)?
>
> I may have some duplicated between names between user and groups, I’m
> going to check that point.
>
>
>
> Thanks.
>
>
>
> *From:* Matt Gilman 
> *Sent:* mercredi 27 mars 2019 19:32
> *To:* users@nifi.apache.org
> *Subject:* Re: Empty "nifi users" page.
>
>
>
> Is it possible that you have a user and a group with the same
> name/identity? Can you compare what is in your users.xml file and what is
> being loaded from your directory server? To see what's being loaded from
> your directory server, if your running 1.9.x you should be able to enable
> debug level logging on
>
> org.apache.nifi.ldap.tenants.LdapUserGroupProvider
>
> Thanks
>
>
>
> Matt
>
>
>
> On Wed, Mar 27, 2019 at 5:46 AM DEHAY Aurelien <
> aurelien.de...@faurecia.com> wrote:
>
> Thanks for you answer.
>
>
>
> Like Bryan and you made me investigate on the browser side, I’ve check
> cache and dev tools.
>
>
>
> Indeed the data are correctly sent to the browser. The users & groups api
> are correctly called and results are good.
>
>
>
> So I suppose it’s something on browser side (tested with FF and chrome
> with same issue).
>
>
>
> I got these errors in the console without guarantee that it’s linked to
> the issue:
>
>
>
> Content Security Policy: « x-frame-options » ignoré en raison de la
> directive « frame-ancestors ».
>
> CURRENT_SITE:  par01prdedge1.soft.fau:18443/nifi/users
> KeeperFill-var.js:30:935
>
> uncaught exception: Each data element must implement a unique 'id' property
>
>
>
> Thanks.
>
>
>
> *From:* Matt Gilman 
> *Sent:* mardi 26 mars 2019 17:49
> *To:* users@nifi.apache.org
> *Subject:* Re: Empty "nifi users" page.
>
>
>
> Hi.
>
>
>
> Could you open up your browsers Developer Tools and check the Console for
> any errors and look at the Network requests and ensure that they are
> returning correctly. The behavior your describing indicate one of two
> things. Either the request is not returning from the server or there is a
> front end code bug.
>
>
>
> We have addressed a couple of issues recently that are scheduled to land
> in 1.10.0 where users are removed (from LDAP externally) and they are still
> referenced in NiFi policies or groups. However, I don't think the behavior
> is exactly how you describe.
>
>
>
> Also, are you able to build current master and try that out? If your
> running into the same problem, it's possible its already addressed.
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
> On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende  wrote:
>
> What browser and browser version are you using?
>
> Have you tried clearing your browser cache just to make sure the page
> is loading properly?
>
> On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien
>  wrote:
> >
> > Hello.
> >
> > I expect to see at least the local users with associated rights (e.g.
> content of users.xml and authorizations.xml), I've got 3 local users and a
> bunch of associated rights with ldap users/groups.
> >
> > I'll try the search filter, but the list is ok when modifying the
> policies of objects, I can see groups &users.
> >
> > Thanks.
> >
> > -Original Message-
> > From: Kevin Doran 
> > Sent: samedi 23 mars 2019 18:52
> > To: users@nifi.apache.org
> > Subject: Re: Empty "nifi users" page.
> >
> > How many users and groups do you expect to sync? Are you able to test
> your search base and filter are correct using a command line tool such as
> ldapsearch? I'm not

Re: Empty "nifi users" page.

[Matt Gilman]

Is it possible that you have a user and a group with the same
name/identity? Can you compare what is in your users.xml file and what is
being loaded from your directory server? To see what's being loaded from
your directory server, if your running 1.9.x you should be able to enable
debug level logging on

org.apache.nifi.ldap.tenants.LdapUserGroupProvider

Thanks

Matt

On Wed, Mar 27, 2019 at 5:46 AM DEHAY Aurelien 
wrote:

> Thanks for you answer.
>
>
>
> Like Bryan and you made me investigate on the browser side, I’ve check
> cache and dev tools.
>
>
>
> Indeed the data are correctly sent to the browser. The users & groups api
> are correctly called and results are good.
>
>
>
> So I suppose it’s something on browser side (tested with FF and chrome
> with same issue).
>
>
>
> I got these errors in the console without guarantee that it’s linked to
> the issue:
>
>
>
> Content Security Policy: « x-frame-options » ignoré en raison de la
> directive « frame-ancestors ».
>
> CURRENT_SITE:  par01prdedge1.soft.fau:18443/nifi/users
> KeeperFill-var.js:30:935
>
> uncaught exception: Each data element must implement a unique 'id' property
>
>
>
> Thanks.
>
>
>
> *From:* Matt Gilman 
> *Sent:* mardi 26 mars 2019 17:49
> *To:* users@nifi.apache.org
> *Subject:* Re: Empty "nifi users" page.
>
>
>
> Hi.
>
>
>
> Could you open up your browsers Developer Tools and check the Console for
> any errors and look at the Network requests and ensure that they are
> returning correctly. The behavior your describing indicate one of two
> things. Either the request is not returning from the server or there is a
> front end code bug.
>
>
>
> We have addressed a couple of issues recently that are scheduled to land
> in 1.10.0 where users are removed (from LDAP externally) and they are still
> referenced in NiFi policies or groups. However, I don't think the behavior
> is exactly how you describe.
>
>
>
> Also, are you able to build current master and try that out? If your
> running into the same problem, it's possible its already addressed.
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
> On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende  wrote:
>
> What browser and browser version are you using?
>
> Have you tried clearing your browser cache just to make sure the page
> is loading properly?
>
> On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien
>  wrote:
> >
> > Hello.
> >
> > I expect to see at least the local users with associated rights (e.g.
> content of users.xml and authorizations.xml), I've got 3 local users and a
> bunch of associated rights with ldap users/groups.
> >
> > I'll try the search filter, but the list is ok when modifying the
> policies of objects, I can see groups &users.
> >
> > Thanks.
> >
> > -Original Message-
> > From: Kevin Doran 
> > Sent: samedi 23 mars 2019 18:52
> > To: users@nifi.apache.org
> > Subject: Re: Empty "nifi users" page.
> >
> > How many users and groups do you expect to sync? Are you able to test
> your search base and filter are correct using a command line tool such as
> ldapsearch? I'm not sure if a search filter is required (although it would
> probably fail with an error if it was), but maybe try setting a "match all"
> filter such as (uid=*).
> >
> > Hope this helps,
> > Kevin
> >
> > On Fri, Mar 22, 2019 at 11:08 AM DEHAY Aurelien <
> aurelien.de...@faurecia.com> wrote:
> > >
> > > Hello.
> > >
> > >
> > >
> > > My “nifi users” page is empty, with only the spinner spinning at the
> bottom of the page.
> > >
> > >
> > >
> > > I’ve tried a lot of conf, and for now everything work fine, except
> that. There is nothing in the log, even when I tried to put “debug” for
> authorization parts in logback.
> > >
> > >
> > >
> > > I had the same behavior with 1.7, 1.8, 1.9 and 1.9.1.
> > >
> > >
> > >
> > > My configuration authorizers.xml and the part of user mapping in the
> > > nifi.properties is
> > > https://gist.github.com/zorel/f76c3924f2b266e8a878397271a54f24
> > >
> > >
> > >
> > > I’ve read Pierre Villard blog post, nifi doc and a lot of post in
> hortonworks site, but I’ve must I missed something. Any pointer is
> appreciated.
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> > > This electronic transmission (and any attac

Re: Changing All Stopped Processors to Disabled

[Matt Gilman]

Ryan,

If you're running 1.7 or later you can enable or disable in bulk just like
you can start or stop. It's supported right in NiFi UI. You can use actions
out of the Operate palette or the context menu.

https://issues.apache.org/jira/browse/NIFI-5066

Matt

On Wed, Mar 27, 2019 at 2:07 PM Andy LoPresto  wrote:

> Ryan,
>
> Reading through your email, my immediate suggestion was NiPyAPI. I think
> Dan has wrapped some useful query methods there that could make this quite
> easy. Obviously you are aware of it, but it’s still my best recommendation
> for now.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 27, 2019, at 10:58 AM, Ryan H 
> wrote:
>
> Hi All,
>
> Is there a good way to change all processors on the canvas that are
> stopped to a disabled state instead? The problem is that we have a large
> amount of processors on our canvas that are in the stopped state which is
> killing the UI performance (wouldn't want to go to each of the 2,000
> stopped processors individually and mark as disabled). We just learned that
> this isn't an issue (with regard to UI performance) when processors are in
> the disabled state due to the way status checks are performed. I'm sure
> this could be scripted with NiPy or something else, but just wanted to
> throw the question out to the community first before delving into this.
>
> Cheers,
>
> Ryan H
>
>
>

Re: Empty "nifi users" page.

[Matt Gilman]

Hi.

Could you open up your browsers Developer Tools and check the Console for
any errors and look at the Network requests and ensure that they are
returning correctly. The behavior your describing indicate one of two
things. Either the request is not returning from the server or there is a
front end code bug.

We have addressed a couple of issues recently that are scheduled to land in
1.10.0 where users are removed (from LDAP externally) and they are still
referenced in NiFi policies or groups. However, I don't think the behavior
is exactly how you describe.

Also, are you able to build current master and try that out? If your
running into the same problem, it's possible its already addressed.

Thanks

Matt

On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende  wrote:

> What browser and browser version are you using?
>
> Have you tried clearing your browser cache just to make sure the page
> is loading properly?
>
> On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien
>  wrote:
> >
> > Hello.
> >
> > I expect to see at least the local users with associated rights (e.g.
> content of users.xml and authorizations.xml), I've got 3 local users and a
> bunch of associated rights with ldap users/groups.
> >
> > I'll try the search filter, but the list is ok when modifying the
> policies of objects, I can see groups &users.
> >
> > Thanks.
> >
> > -Original Message-
> > From: Kevin Doran 
> > Sent: samedi 23 mars 2019 18:52
> > To: users@nifi.apache.org
> > Subject: Re: Empty "nifi users" page.
> >
> > How many users and groups do you expect to sync? Are you able to test
> your search base and filter are correct using a command line tool such as
> ldapsearch? I'm not sure if a search filter is required (although it would
> probably fail with an error if it was), but maybe try setting a "match all"
> filter such as (uid=*).
> >
> > Hope this helps,
> > Kevin
> >
> > On Fri, Mar 22, 2019 at 11:08 AM DEHAY Aurelien <
> aurelien.de...@faurecia.com> wrote:
> > >
> > > Hello.
> > >
> > >
> > >
> > > My “nifi users” page is empty, with only the spinner spinning at the
> bottom of the page.
> > >
> > >
> > >
> > > I’ve tried a lot of conf, and for now everything work fine, except
> that. There is nothing in the log, even when I tried to put “debug” for
> authorization parts in logback.
> > >
> > >
> > >
> > > I had the same behavior with 1.7, 1.8, 1.9 and 1.9.1.
> > >
> > >
> > >
> > > My configuration authorizers.xml and the part of user mapping in the
> > > nifi.properties is
> > > https://gist.github.com/zorel/f76c3924f2b266e8a878397271a54f24
> > >
> > >
> > >
> > > I’ve read Pierre Villard blog post, nifi doc and a lot of post in
> hortonworks site, but I’ve must I missed something. Any pointer is
> appreciated.
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> > > This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it immediately and notify the sender. Any
> unauthorized use or disclosure of this message is strictly prohibited.
> Faurecia does not guarantee the integrity of this transmission and shall
> therefore never be liable if the message is altered or falsified nor for
> any virus, interception or damage to your system.
> >
> > This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it immediately and notify the sender. Any
> unauthorized use or disclosure of this message is strictly prohibited.
> Faurecia does not guarantee the integrity of this transmission and shall
> therefore never be liable if the message is altered or falsified nor for
> any virus, interception or damage to your system.
>

Re: Search scope for LDAP user authentication

[Matt Gilman]

That setting within the context of LDAP based authentication is not
currently configurable. If it is desirable, we could certainly expose a
configuration option for it.

Thanks

On Mon, Mar 25, 2019 at 9:14 AM  wrote:

> Hi,
>
>
>
> Thanks Matt for the quick reply. Is there is a way to set/change this
> value through the configurations? (I just want to make sure that I didn’t
> accidently change it somehow).
>
>
>
> Thanks,
>
>
>
>
>
> User
>
> *From:* Matt Gilman [mailto:matt.c.gil...@gmail.com]
> *Sent:* Monday, March 25, 2019 9:04 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Search scope for LDAP user authentication
>
>
>
> Hi,
>
>
>
> I just took a quick look at the code and the underlying library in use and
> it appears that searching SUBTREE should be enabled by default.
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
> On Mon, Mar 25, 2019 at 8:59 AM  wrote:
>
> Hi,
>
>
>
> I was wondering if there was a way to set the Search Scope to SUBTREE when
> using LDAP for user Authentication?
>
>
>
> Thanks,
>
>
>
> User
>
>
> If you are not the addressee, please inform us immediately that you have
> received this e-mail by mistake, and delete it. We thank you for your
> support.
>
>
>
>
> If you are not the addressee, please inform us immediately that you have
> received this e-mail by mistake, and delete it. We thank you for your
> support.
>
>

Re: Search scope for LDAP user authentication

[Matt Gilman]

Hi,

I just took a quick look at the code and the underlying library in use and
it appears that searching SUBTREE should be enabled by default.

Thanks

Matt

On Mon, Mar 25, 2019 at 8:59 AM  wrote:

> Hi,
>
>
>
> I was wondering if there was a way to set the Search Scope to SUBTREE when
> using LDAP for user Authentication?
>
>
>
> Thanks,
>
>
>
> User
>
> If you are not the addressee, please inform us immediately that you have
> received this e-mail by mistake, and delete it. We thank you for your
> support.
>
>

Re: Have localization been supported in NiFi Releases?

[Matt Gilman]

Jianan,

There have been a few attempts where community members submitted a PR. [1]
is another. Unfortunately, they all seemed to have stalled out. The latest
NiFi does not support localization. It's a really important feature that
we'd like to see soon. But to my knowledge, it is not currently being
worked on.

Matt

[1] https://github.com/apache/nifi/pull/2408

On Sun, Mar 3, 2019 at 10:47 PM Jianan Zhang 
wrote:

> Hi all,
>
> I need to localize NiFi, and I found some related works, such as #1913
> , but it seems not merged into
> released version. Does the NiFi latest release support localization.
>
> Thanks,
> Jianan
>

Re: issue with Nifi UI

[Matt Gilman]

How are you trying to access the UI?

Matt

On Wed, Oct 24, 2018 at 8:38 AM N, Vyshali  wrote:

> Hi,
>
>
>
> I have installed Nifi(1.7 version) in windows 10 and configured the port
> as 8081
>
> The logs don’t throw any error but the Nifi UI doesn’t come up.
>
> Attached is the nifi properties file and logs folder for your reference.
>
>
>
> Thanks in advance
>
>
>
> Regards
>
> vyshali
>

Re: NiFi Web UI in split-horizon DNS environments

[Matt Gilman]

Sounds good. In case you hadn't seen it, here's a link to our proxy
documentation [1].

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration

On Fri, Jun 8, 2018 at 2:24 PM, Matthew McDowell 
wrote:

> Hey Matt,
>
>
>
> I haven’t tried with other components, though I had assumed the problem
> would happen elsewhere. This was my first time observing the problem.
> You’re correct that I’m running behind a proxy, specifically an nginx
> reverse proxy, so modifying headers can be done pretty easily.
> Unfortunately (sigh) the way this proxy is set up, I’m actually setting the
> Host header to the unresolvable one on the other side of the proxy, which
> is presumably what’s screwing me. So the NiFi server doesn’t actually know
> I’m using split-horizon DNS if its considering that header. I’m doing this
> to fix a totally different problem with another product in the stack.
>
>
>
> Anyway, our proxy setup is very much an edge case, and if NiFi is using
> the Host header to generate that API response, that seems like a perfectly
> reasonable approach. We can just stick with the workaround we have.
>
>
>
> Thanks very much for the help!
>
> Matt
>
>
>
>
>
> *From: *Matt Gilman 
> *Reply-To: *"users@nifi.apache.org" 
> *Date: *Friday, June 8, 2018 at 2:10 PM
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi Web UI in split-horizon DNS environments
>
>
>
> Matt,
>
>
>
> Have you tried updating other components as well? All components
> (Processors, Ports, RPGs, etc) return their own URL in their object model
> (JSON blob). I would assume if the Reporting Tasks URLs are not formed
> correctly then the URLs for other components will also not be formed
> correctly. Just to confirm, are you running behind a proxy? If so there are
> some headers you can set to control this. If your not, then I believe the
> hostname comes from the incoming HttpServletRequest. In NiFi's case, we run
> embedded Jetty. If I recall correctly, their implementation will attempt to
> get the hostname from a couple different places before falling back to
> asking the server its hostname. One of those places (in fact I believe the
> first place considered) is the incoming HOST header.
>
>
>
> Hope that helps debug further. Let me know if you have any follow up
> questions.
>
>
>
> Matt
>
>
>
>
>
> On Fri, Jun 8, 2018 at 1:03 PM, Matthew McDowell <
> mmcdow...@hortonworks.com> wrote:
>
> Hi everyone,
>
>
>
> I’m using NiFi in an environment where the server running the NiFi UI is
> reached via different hostnames depending on the where the user is coming
> from. This is an unfortunate, but unavoidable, consequence of the security
> architecture in this environment. The vast majority of the UI works just
> fine, and appears to handle things using relative URLs, so we get away with
> it.
>
>
>
> The exception I’ve run into is when trying to stop or start reporting
> tasks. Clicking the button in the UI sends a PUT to a URL that is derived
> from the ‘url’ parameter in a JSON blob retrieved from
> nifi-api/flow/reporting-tasks. That URL is a full URL (i.e. http://
> :9090/nifi-api/reporting-tasks/) has the hostname as
> (presumably) the hostname that the server uses to identify itself. As a
> result this particular part of the UI doesn’t work unless you are able to
> reach the server using that hostname.
>
>
>
> We’ve established a workaround, so this is far from a show-stopper bug,
> and I realize something like split-horizon DNS is an edge case. With that
> said, I wanted to see if it’s possible to do this as a relative link
> instead, or if there are design considerations that make that unworkable.
>
>
>
> Thanks!
>
> Matt
>
>
>
>
>

Re: NiFi Web UI in split-horizon DNS environments

[Matt Gilman]

Matt,

Have you tried updating other components as well? All components
(Processors, Ports, RPGs, etc) return their own URL in their object model
(JSON blob). I would assume if the Reporting Tasks URLs are not formed
correctly then the URLs for other components will also not be formed
correctly. Just to confirm, are you running behind a proxy? If so there are
some headers you can set to control this. If your not, then I believe the
hostname comes from the incoming HttpServletRequest. In NiFi's case, we run
embedded Jetty. If I recall correctly, their implementation will attempt to
get the hostname from a couple different places before falling back to
asking the server its hostname. One of those places (in fact I believe the
first place considered) is the incoming HOST header.

Hope that helps debug further. Let me know if you have any follow up
questions.

Matt


On Fri, Jun 8, 2018 at 1:03 PM, Matthew McDowell 
wrote:

> Hi everyone,
>
>
>
> I’m using NiFi in an environment where the server running the NiFi UI is
> reached via different hostnames depending on the where the user is coming
> from. This is an unfortunate, but unavoidable, consequence of the security
> architecture in this environment. The vast majority of the UI works just
> fine, and appears to handle things using relative URLs, so we get away with
> it.
>
>
>
> The exception I’ve run into is when trying to stop or start reporting
> tasks. Clicking the button in the UI sends a PUT to a URL that is derived
> from the ‘url’ parameter in a JSON blob retrieved from
> nifi-api/flow/reporting-tasks. That URL is a full URL (i.e. http://
> :9090/nifi-api/reporting-tasks/) has the hostname as
> (presumably) the hostname that the server uses to identify itself. As a
> result this particular part of the UI doesn’t work unless you are able to
> reach the server using that hostname.
>
>
>
> We’ve established a workaround, so this is far from a show-stopper bug,
> and I realize something like split-horizon DNS is an edge case. With that
> said, I wanted to see if it’s possible to do this as a relative link
> instead, or if there are design considerations that make that unworkable.
>
>
>
> Thanks!
>
> Matt
>
>
>

Re: Authenticating via OIDC in NiFi 1.6.0

[Matt Gilman]

Paul,

Unfortunately, there is no workaround in NiFi currently. First the
OidcIdentityProvider attempts to use email scope if its supported. If it is
not, it attempts to use the email claim on the UserInfo endpoint. Until
NiFi is updated to support a configurable user identity claim (like upn in
the ad case) you may be able to update your Identity Provider to include
the email claim if that is a possibility.

Matt

On Wed, May 16, 2018 at 9:53 AM, Paul Riddle 
wrote:

> NiFi Team,
>
> We are trying to use the oidc settings in nifi.properties to authenticate,
> and it appears to work until we put in our credentials and are returned to
> the NiFi site.  We get an error page from NiFi stating the following:
>
> Unable to continue login sequence
>
>- home
>
> Unable to exchange authorization for ID token: Unable to extract email
> from the UserInfo token.
>
> After some research, we found that there was a related ticket
> (NIFI-4698).
>
> Is there currently a workaround for this?
>
> Regards,
> Paul
>
>
>

Re: processGroupId URL

[Matt Gilman]

Hey Scott,

Sorry, I misunderstood the original question. I thought you wanted to get
able to get the URL for a specific processor in the REST API. You actually
wanted the UI URL with the query parameters for the component in question.
This does require the parent Process Group ID and unfortunately, it is not
possible yet. I've filed a JIRA to address this in the future [1].

The red indicators you mentioned are bulletins. Bulletins are designed to
be a notification about a passing event. They are not meant to be a
persisted or acknowledged in any manner. The main factor there is the rate
at which they are generated. A bulletin will be created for every log
message with a severity matching or worse than the configured threshold. If
you're looking to do something more meaningful with the logs you may want
to consider introducing a tool built for that. We do offer a Reporting Task
that will send the Bulletins via Site to Site to another NiFi instance
which could be used to load/store them someplace.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-5072

On Fri, Apr 6, 2018 at 11:59 PM, scott  wrote:

> Hi Matt,
>
> No, I'm looking for the main UI URL, like so:
>
> http://localhost:8899/nifi/?processGroupId=9e338e2f-0162-
> 1000-969a-043d35e4c41d&componentIds=
>
> I'm using NiFi 1.5. What I'd really like is some way to know, using a
> variable or attribute, that the processor which caused the error resides
> within processGroupId 9e338e2f-0162-1000-969a-043d35e4c41d so that I can
> build the complete URL. The URL will then be written to my log table so
> that operational users can go to the problem area with one click to
> troubleshoot.
>
> This reminds me I have another problem with troubleshooting I wanted to
> ask about. You know the errors that are obvious with a red indicator in the
> upper right corner of a processor or process group you can hover over? They
> go away after what seems like a few minutes. I'd really love it they would
> stay persistent until some action is taken, like a user clears an alarm.
> Any way to do that? Or, at the very least, can I make them last a lot
> longer?
>
>
> Thanks,
>
> Scott
>
> On 04/06/2018 07:20 AM, Matt Gilman wrote:
>
> Scott,
>
> What version of NiFi are you running? If your running NiFi 1.x then the
> parent process group ID shouldn't be necessary. The fully qualified URL of
> a processor would look something like
>
> http://localhost:8080/nifi-api/processors/9b52d2bf-0162-
> 1000-d1ac-89e944aa1bfe
>
> Or are you trying to build a different URL?
>
> Thanks
>
> Matt
>
> On Fri, Apr 6, 2018 at 12:07 AM, scott  wrote:
>
>> Hello community,
>>
>> I'm trying to write a log when errors occur in my flow, and I want to
>> include the fully qualified URL to the location within my flow where the
>> error occurred. I know the component ID is an attribute I can use, but I
>> also need the processGroupId where the component is located within my flow,
>> so that I can build a fully qualified URL and put it into my log to make
>> operational troubleshooting easier. Does anyone have any advice on how I
>> can get the processGroupId as a variable/attribute I can include in my log?
>>
>>
>> Thanks,
>>
>> Scott
>>
>>
>
>

Re: PUT/flow/process-groups/{id} API

[Matt Gilman]

Vitaly,

That endpoint will start/stop every component the requesting user has access 
to. Is your instance secured? If so please ensure that your including necessary 
authentication details. For instance, if your using client certificate in your 
browser be sure you request in your Java program uses a similar SSL context.

Matt

Sent from my iPhone

> On Apr 6, 2018, at 11:15 AM, Vitaly Krivoy  wrote:
> 
> When I start a group from NiFi UI and I examine the request in Chrome 
> Developers Tools, it is identical to the one that I send from a Java program. 
> What gives?
> 
>  
> 
> From: Vitaly Krivoy [mailto:vitaly_kri...@jhancock.com] 
> Sent: Thursday, April 05, 2018 5:49 PM
> To: users@nifi.apache.org
> Subject: PUT/flow/process-groups/{id} API
>  
> 
> I am trying to use PUT/flow/process-groups/{id} to start a processing group 
> from a Java program, but nothing is happening.  I set state RUNNING in 
> ScheduleComponentsEntity and when an API call returns it shows the state in 
> the returned object as RUNNING. But I see on the canvas that the group was 
> not launched. Yet I can start a group in NiFi GUI by right clicking on the 
> group and selecting start. What am I missing? Thanks.
> 
>  
> STATEMENT OF CONFIDENTIALITY The information contained in this email message 
> and any attachments may be confidential and legally privileged and is 
> intended for the use of the addressee(s) only. If you are not an intended 
> recipient, please: (1) notify me immediately by replying to this message; (2) 
> do not use, disseminate, distribute or reproduce any part of the message or 
> any attachment; and (3) destroy all copies of this message and any 
> attachments.
>  
> STATEMENT OF CONFIDENTIALITY The information contained in this email message 
> and any attachments may be confidential and legally privileged and is 
> intended for the use of the addressee(s) only. If you are not an intended 
> recipient, please: (1) notify me immediately by replying to this message; (2) 
> do not use, disseminate, distribute or reproduce any part of the message or 
> any attachment; and (3) destroy all copies of this message and any 
> attachments.

Re: processGroupId URL

[Matt Gilman]

Scott,

What version of NiFi are you running? If your running NiFi 1.x then the
parent process group ID shouldn't be necessary. The fully qualified URL of
a processor would look something like

http://localhost:8080/nifi-api/processors/9b52d2bf-0162-1000-d1ac-89e944aa1bfe

Or are you trying to build a different URL?

Thanks

Matt

On Fri, Apr 6, 2018 at 12:07 AM, scott  wrote:

> Hello community,
>
> I'm trying to write a log when errors occur in my flow, and I want to
> include the fully qualified URL to the location within my flow where the
> error occurred. I know the component ID is an attribute I can use, but I
> also need the processGroupId where the component is located within my flow,
> so that I can build a fully qualified URL and put it into my log to make
> operational troubleshooting easier. Does anyone have any advice on how I
> can get the processGroupId as a variable/attribute I can include in my log?
>
>
> Thanks,
>
> Scott
>
>

Re: Permission to access the API

[Matt Gilman]

Just to follow up. For the most part, the permission required is the same
as the resource being requested. In this case, it is the /flow (view the
flow) resource. Status/Stats for a component are available for anyone with
permission to view the flow. 'View the component" will expose configuration
details for the corresponding components.

Matt

On Mon, Apr 2, 2018 at 1:16 PM, Pierre Villard 
wrote:

> Hi Mike,
>
> System Diagnostic is a completely different view in the UI when going into
> Summary and then System Diagnostics (bottom right of the summary view).
> In this case, your colleague wants to access all the information of the
> workflow. IIRC he'd need access to the flow + "view the component" read
> access on the root process group recursively.
>
> Pierre
>
>
>
> 2018-04-02 19:09 GMT+02:00 Mike Thomsen :
>
>> Colleague of mine is trying to poll this API:
>>
>> /nifi-api/flow/process-groups/root/status?recursive=true
>>
>> The instance is protected with SSL and LDAP auth. The user account I'm
>> trying has "view system diagnostics," but it still gets a 403. Any ideas?
>>
>> Thanks,
>>
>> Mike
>>
>
>

Re: Problem enabling/disabling NiFi controller services from behind proxy

[Matt Gilman]

Hi Tim,

Sorry for the inconvenience but this was a bug that was introduced in
Apache NiFi 1.5.0. This was addressed in this JIRA [1] and will be included
in Apache NiFi 1.6.0 which is currently under vote.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-4894

On Thu, Mar 22, 2018 at 3:13 PM, Tim Dean  wrote:

> Hello,
>
> I am working with NiFi 1.5.0 and I have created a customer service
> controller for my project. I am able to add my controller service to my
> NiFi flow configuration, but I am unable to enable that controller service
> without first encountering error messages that I need to dismiss. After
> that the controller services do appear to be enabled, but then I can’t
> disable them.
>
> This problem does not occur when I connect directly to my NiFi application
> from the browser. But it does not work when NiFi is deployed behind an
> Nginx reverse proxy that is how we deploy things in production. I have
> configured Nginx to set headers as described at https://community.
> hortonworks.com/articles/113240/running-apache-nifi-behind-a-proxy-2.html.
> Specifically, my Nginx configuration looks like this:
>
> location /workflow/ {
> proxy_set_header Host $host;
> proxy_set_header X-Real-IP $remote_addr;
> proxy_set_header X-ProxyScheme $scheme;
> proxy_set_header X-ProxyHost $host;
> proxy_set_header X-ProxyPort $server_port;
> proxy_set_header X-ProxyContextPath "/workflow/";
> proxy_pass http://127.0.0.1:8080/;
> }
>
>
> This appears to work for most operations: I can configure my processors,
> start and stop them, etc. But when I try to click the button to enable my
> service controller, the proxy settings set up in my Nginx configuration
> above don’t seem to be honored.
>
> As an example, here the HTTP request I see in developer tools for starting
> a processor looks like this:
>
> PUT http://vi-codex.eng.local/workflow/nifi-api/processors/
> 5d8deeca-015e-1000-be6e-151bd1c4b56a
>
>
> But when I click on the button to enable my controller service, the HTTP
> request looks like this:
>
> GET http://vi-codex.eng.local/nifi-api/flow/bulletin-board?
> sourceId=4e9f82cf-0162-1000-ee16-fb9ce9599ac6
>
>
> As you can see, the “/workflow” context path I set via Nginx is being used
> for the first (and every other) request, but not when I try to enable or
> disable a service controller. For enabling and disabling, I get an error
> because it attempts to access “nifi-api” without properly prefixing it with
> the correct context path.
>
> Have I done something wrong in my configuration that is causing this? Or
> is this a bug in NiFi 1.5?
>
> Thanks
>
> -Tim
>

Re: Connection between RPG and input port is failing via API

[Matt Gilman]

Ravi,

The available ports are independently retrieved on each node. When
clustered the response to the client should only include the ports that all
nodes have discovered. However, I believe that currently, it's returning
ports that any node has discovered. I've filed a JIRA [1] to address this
issue. Thanks for reporting.

[1] https://issues.apache.org/jira/browse/NIFI-4973

On Wed, Mar 14, 2018 at 1:38 AM, Ravi Papisetti (rpapiset) <
rpapi...@cisco.com> wrote:

> Hi,
>
>
>
> Nifi version 1.5
>
>
>
> I am trying to create connection between output port to input port at root
> RPG. Below is my json request:
>
>
>
> {"revision": {"clientId": "anonymous","version": 0},"component": {"name":
> "","source": {  "id": "22df69f1-0162-1000--6c01773e",  "groupId":
> "22df42c1-0162-1000--cd3f7d2b",  "type":
> "OUTPUT_PORT"},"destination": {  "id": "16628c20-0ed4-3372-a9cc-
> 9b38c49aaf29",  "groupId": "ad4a6524-0161-1000--f8293860",  "type":
> "REMOTE_INPUT_PORT"}}}
>
>
>
> This works fine if I put delay about 45 secs after creating destination
> port (remote input port). Otherwise it is failing with below error:
>
> Response to create connection from out to RPG: Node xxx is unable to
> fulfill this request due to: Unable to find the specified destination.
>
>
>
> This port will show up in the response from /nifi-api/
> remote-process-groups/" + rootRPGId. Still it complains saying unable to
> find specified destination.
>
>
>
> Are there any other parameters to be checked before issuing the connection
> request?
>
>
>
> Appreciate any help.
>
>
>
> Thanks,
>
> Ravi Papisetti
>
>
>
>
>
>
>

Re: Error when executing NiFi REST API PUT /processors/{id} call

[Matt Gilman]

Vitaly,

I believe that error is being generated by Jackson which handles the JSON
(de)serialization. Often times there is additional information that gets
logged to your logs/nifi-user.log. Check there to see if there is anything
helpful.

By any chance, are you attempting to pass the status back into the update
call? The entity object holds information about the component (the
configuration, the revision, status, permissions, etc). When performing an
update request you only need to pass in the revision (RevisionDTO) and the
configuration (ProcessorDTO). Check out the Developer Tools in your browser
to see various requests in action.

Hope this helps.

Matt

On Tue, Mar 13, 2018 at 2:45 PM, Vitaly Krivoy 
wrote:

>
>
>
>
>
>
> I am trying to update processors for a workflow, which was previously
> generated from a template by instantiating it. All work is done from a Java
> program through NiFi REST API. Template holds processors in a group. Since
> I know the template group name and strategy by which NiFi assigns a new
> group name when it instantiates a template (“Copy of “  in a template>), I find instantiated group by name, get its group id and
> get list of processors in the group a ProcessorsEntity object. I then
> step through the list of processors contained in ProcessorsEntity, and
> for each processor set desired properties contained in  ProcessorEntity and
> its linked ProcessorDTO and ProcessorConfigDTO classes. I then set ClientId
> in the Revison object like this:
> proc.getRevision().setClientId(restClient.getClientId());
>
> Here *proc* is ProcessorEntity and restClient is a custom class which
> contains all code necessary to communicate with NiFi REST API.
>
> At this point I am trying to update the processor through
> PUT/processors/{id} call, passing it modified data in ProcessorEntity, the
> same one that I got from ProcessorsEntity and update in place, rather
> than updating a copy. I figured that there is no need to do it for a DTO
> object.
> When I execute PUT/processors/{id} call, I get an exception, which I need
> the help with. Here is what I see in Eclipse.
>
> Exception when calling: ProcessorsApi#updateProcessor
>
> Response body: Text '01/01/1970 13:04:02 EST' could not be parsed at index
> 2 (through reference chain: org.apache.nifi.web.api.
> entity.ProcessorEntity["status"]->org.apache.nifi.web.api.dto.status.
> ProcessorStatusDTO["statsLastRefreshed"])
>
> io.swagger.client.ApiException: Bad Request
>
>at io.swagger.client.ApiClient.
> handleResponse(ApiClient.java:1058)
>
>at io.swagger.client.ApiClient.execute(ApiClient.java:981)
>
>at io.swagger.client.api.ProcessorsApi.
> updateProcessorWithHttpInfo(ProcessorsApi.java:707)
>
>at io.swagger.client.api.ProcessorsApi.updateProcessor(
> ProcessorsApi.java:692)
>
>at com.bdss.nifi.trickle.NiFiRestClient.updateProcessor(
> NiFiRestClient.java:149)
>
>at com.bdss.nifi.trickle.TemplateConfigurator.
> configureGroupProcessors(TemplateConfigurator.java:84)
>
>at com.bdss.nifi.trickle.MdmAzureConfigurator.
> configureGroup(MdmAzureConfigurator.java:89)
>
>at com.bdss.nifi.trickle.Deployer.deployTemplate(
> Deployer.java:52)
>
>at com.bdss.nifi.trickle.Trickle.main(Trickle.java:132)
>
>
>
> Notes:
>
> Everything that begins with com.bdss.nifi.trickle are classes in the Java
> application which I am implementing.
> To access NiFi REST API I am using this REST client: https://github.com/
> simplesteph/nifi-api-client-java, which itself relies on Swagger.
>
>
>
> Many thanks for any tips.
>
>
>
> STATEMENT OF CONFIDENTIALITY The information contained in this email
> message and any attachments may be confidential and legally privileged and
> is intended for the use of the addressee(s) only. If you are not an
> intended recipient, please: (1) notify me immediately by replying to this
> message; (2) do not use, disseminate, distribute or reproduce any part of
> the message or any attachment; and (3) destroy all copies of this message
> and any attachments.
>

Re: Disconnect nifi node cluster via API

[Matt Gilman]

Jorge,

A node must be disconnected before it can be removed. If you're looking to
invoke the REST endpoints, I would suggest looking at your browsers Dev
Tools to see the requests in action. Alternatively, you could use the
NodeManagerTool [1] that is bundled in the toolkit [2].

Matt

[1]
https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-admin/src/main/groovy/org/apache/nifi/toolkit/admin/nodemanager/NodeManagerTool.groovy
[2] https://nifi.apache.org/download.html.

On Thu, Feb 22, 2018 at 8:12 AM, Charlie Meyer <
charlie.me...@civitaslearning.com> wrote:

> On https://nifi.apache.org/docs/nifi-docs/rest-api/index.html take a look
> at DELETE /controller/cluster/nodes/{id}
>
> That might have what you need.
>
> -Charlie
>
> On Feb 22, 2018 02:19, "Jorge Machado"  wrote:
>
> Hi guys,
>
> is there an api to disconnect a node from the cluster ? This would be
> great to securely automate a rolling restart.
>
>
> Jorge
>
>
>
>
>
>
>

Re: Issues with service controller when using NiFi behind the proxy

[Matt Gilman]

I've gone ahead and issues a PR [1] for this issue. Feel free to check it
out and verify it in your environment if possible.

Thanks

Matt

[1] https://github.com/apache/nifi/pull/2482

On Tue, Feb 20, 2018 at 9:33 AM, Matt Gilman 
wrote:

> Thanks for the email. I just looked and this does appear to be a bug in
> the Javascript that was introduced in 1.5.0 when addressing this issue [1].
> I've filed a JIRA [2] to address this issue.
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-4518
> [2] https://issues.apache.org/jira/browse/NIFI-4894
>
> On Mon, Feb 19, 2018 at 4:28 PM, Alexander Lobunets <
> alexander.lobun...@gmail.com> wrote:
>
>> Hello all,
>>
>> I'm trying to use NiFi behind the proxy (tried both options: Nginx and
>> Apache2). In every case I get the NiFi UI loaded properly and I'm able to
>> compose a simple flow, define service controller. But when I try to
>> disable/edit service controller I receive an ERROR dialog with the
>> following content:
>>
>>  404 Not Found 
>> 404 Not Found nginx/1.10.3
>> (Ubuntu)  
>> When I look into the inspection page I see 404 for the following request:
>> https:///nifi-api/flow/bulletin-b
>> oard?sourceId=bf7ec936-d34e-3e0a-3624-33f457e2c7a2
>> which does not have a URI prefix (context path) my proxy is configured
>> with: "/admin". When I add the missing prefix manually I do receive a
>> proper HTTP response.
>>
>> As a result this prevents me from disabling/modifying/deleting service
>> controller.
>> The proxy configurations are provided below.
>>
>> Did anyone face the similar issue? Is this a bug or a possible
>> misconfiguration?
>>
>> Thank you in advance,
>> Alexander
>>
>>
>> == Nginx proxy config =
>> location ~ ^/admin(?.*) {
>> proxy_pass http://127.0.0.1:8080$section$is_args$args;
>> proxy_redirect off;
>> proxy_set_header  X-ProxyScheme "https";
>> proxy_set_header  X-ProxyHost $http_host;
>> proxy_set_header  X-ProxyPort 443;
>> proxy_set_header  X-ProxyContextPath "/admin";
>> proxy_connect_timeout   180;
>> proxy_send_timeout  180;
>> proxy_read_timeout  180;
>> }
>> 
>>
>> = Apache proxy config ===
>> 
>> RequestHeader add X-ProxyScheme "http"
>> RequestHeader add X-ProxyHost ""
>> RequestHeader add X-ProxyPort "80"
>> RequestHeader add X-ProxyContextPath "/admin"
>> ProxyPass http://127.0.0.1:8080
>> ProxyPassReverse http://127.0.0.1:8080
>> 
>>
>> 
>>
>> --
>> All the best,
>> Alexander.
>>
>
>

Re: Issues with service controller when using NiFi behind the proxy

[Matt Gilman]

Thanks for the email. I just looked and this does appear to be a bug in the
Javascript that was introduced in 1.5.0 when addressing this issue [1].
I've filed a JIRA [2] to address this issue.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-4518
[2] https://issues.apache.org/jira/browse/NIFI-4894

On Mon, Feb 19, 2018 at 4:28 PM, Alexander Lobunets <
alexander.lobun...@gmail.com> wrote:

> Hello all,
>
> I'm trying to use NiFi behind the proxy (tried both options: Nginx and
> Apache2). In every case I get the NiFi UI loaded properly and I'm able to
> compose a simple flow, define service controller. But when I try to
> disable/edit service controller I receive an ERROR dialog with the
> following content:
>
>  404 Not Found 
> 404 Not Found nginx/1.10.3
> (Ubuntu)  
> When I look into the inspection page I see 404 for the following request:
> https:///nifi-api/flow/bulletin-
> board?sourceId=bf7ec936-d34e-3e0a-3624-33f457e2c7a2
> which does not have a URI prefix (context path) my proxy is configured
> with: "/admin". When I add the missing prefix manually I do receive a
> proper HTTP response.
>
> As a result this prevents me from disabling/modifying/deleting service
> controller.
> The proxy configurations are provided below.
>
> Did anyone face the similar issue? Is this a bug or a possible
> misconfiguration?
>
> Thank you in advance,
> Alexander
>
>
> == Nginx proxy config =
> location ~ ^/admin(?.*) {
> proxy_pass http://127.0.0.1:8080$section$is_args$args;
> proxy_redirect off;
> proxy_set_header  X-ProxyScheme "https";
> proxy_set_header  X-ProxyHost $http_host;
> proxy_set_header  X-ProxyPort 443;
> proxy_set_header  X-ProxyContextPath "/admin";
> proxy_connect_timeout   180;
> proxy_send_timeout  180;
> proxy_read_timeout  180;
> }
> 
>
> = Apache proxy config ===
> 
> RequestHeader add X-ProxyScheme "http"
> RequestHeader add X-ProxyHost ""
> RequestHeader add X-ProxyPort "80"
> RequestHeader add X-ProxyContextPath "/admin"
> ProxyPass http://127.0.0.1:8080
> ProxyPassReverse http://127.0.0.1:8080
> 
>
> 
>
> --
> All the best,
> Alexander.
>

Re: Invalid header error

[Matt Gilman]

Robert, James, All,

NiFi has been updated to be a little more strict regarding incoming HTTP
requests. If the Host header does not comply with an expected value, the
request is rejected. Currently, the expected value comes from those .host
properties. What's happening is the proxy is likely passing through all
incoming header values. When NiFi sees the request, it appears as those it
was not meant for it so it's rejected. I believe there are two valid
options here:

1) Remove the Host header at the proxy. This should allow it to explicitly
set it to the NiFi Host when issuing the request instead of passing through
the incoming value.
2) Update NiFi to allow whitelisting of expected Host values like we did
for context paths. I've created a JIRA for this option [1].

We'll make sure these get appropriately documented for folks running behind
a proxy.

Thanks

Matt

[1] https://issues.apache.org/jira/browse/NIFI-4501

On Wed, Jan 10, 2018 at 5:00 AM, Robert R. Bruno  wrote:

> James,
>
> Funny enough I was thinking of the same hack, but as you said sounds a bit
> nasty.  Hopefully there is a better solution.  Also for me, I may not
> always have local admin rights on my client machine which I believe is
> required to change the hosts file.
>
> Thanks,
> Robert
>
> On Wed, Jan 10, 2018, 00:18 James Wing  wrote:
>
>> Robert,
>>
>> I had the same problem.  One workaround I have used was to add the DNS
>> name to the /etc/hosts file with a local IP address, so that I could
>> configure that name in nifi.web.http.host and NiFi would still bind to the
>> right IP.  It sounds like a nasty hack now that I describe it, but it
>> worked.
>>
>> Perhaps someone else knows a more elegant configuration?
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Jan 9, 2018 at 7:33 AM, Robert R. Bruno 
>> wrote:
>>
>>> I just ran into this as well while trying out 1.5.0-SNAPSHOT.
>>>
>>> What is the solution where you are running nifi behind a proxy?  I tried
>>> setting nifi.web.http.host to my proxy ip but then nifi attempted to bind
>>> to this ip address.
>>>
>>> Hopefully I am missing something.  If not any chance a config value for
>>> allowed proxies before the release?
>>>
>>>
>>>
>>> On Fri, Dec 15, 2017, 19:26 Mike Thomsen  wrote:
>>>
 Thanks. Is that documented?

 On Fri, Dec 15, 2017 at 7:02 PM, Andy LoPresto 
 wrote:

> Hi Mike,
>
> This is a recent change introduced in 1.5.0-SNAPSHOT (master). You can
> resolve this by setting nifi.web.http.host in nifi.properties to the value
> of SERVER_HERE.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Dec 15, 2017, at 3:32 PM, Mike Thomsen 
> wrote:
>
> I get this error after I installed a new build:
>
> The request contained an invalid host header [SERVER_IP:8080] in the
> request [/]. Check for request manipulation or third-party intercept.
>
> In the logs it says:
>
> 2017-12-15 18:34:59,937 WARN [NiFi Web Server-66] o.a.n.w.s.
> HostHeaderSanitizationCustomizer Request host header
> [SERVER_HERE:8080] different from web hostname [(:8080)]. Overriding to
> [:8080/nifi/]
> 2017-12-15 18:34:59,938 WARN [NiFi Web Server-66] 
> o.a.nifi.web.server.HostHeaderHandler
> Request host header [SERVER_HERE:8080] different from web hostname
> [localhost(:8080)]. Overriding to [localhost:8080/nifi/]
> 2017-12-15 18:35:00,059 WARN [NiFi Web Server-59] o.a.n.w.s.
> HostHeaderSanitizationCustomizer Request host header
> [SERVER_HERE:8080] different from web hostname [(:8080)]. Overriding to
> [:8080/favicon.ico]
> 2017-12-15 18:35:00,059 WARN [NiFi Web Server-59] 
> o.a.nifi.web.server.HostHeaderHandler
> Request host header [SERVER_HERE:8080] different from web hostname
> [localhost(:8080)]. Overriding to [localhost:8080/favicon.ico]
>
> Never saw this with 1.4 and earlier. Any ideas?
>
> Thanks,
>
> Mike
>
>
>

>>

Re: nifi webui in browser not responding / waiting

[Matt Gilman]

Amit,

The splash screen is shown while waiting for a number of requests to return
from the server. In your logs, I can see a number those requests coming in.
Can you please let me know what the Network tab is showing? In addition,
what the Console tab is showing?

Thanks

Matt

On Sun, Jan 7, 2018 at 12:23 AM, Amit Pandit  wrote:

> Here is the logs files from my machine.
>
> Also, I did not notice any issues when I opened browser developer tools on
> that site.
>
> Also like I said, all other HDF services seem to work, except this one.
>
> For this NiFi, the nice splash logo comes on screen and its waiting on
> something.
>
>
>
> Please let me know any resolutions.
>
>
>
>
>
>
>
>
>
> Sent from Mail  for
> Windows 10
>
>
> --
> *From:* Joe Witt 
> *Sent:* Saturday, January 6, 2018 8:05:46 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: nifi webui in browser not responding / waiting
>
> grab a thread dump with bin/nifi.sh amd share the logs folder in tar.gz pls
>
> On Jan 6, 2018 7:41 PM, "Amit Pandit"  wrote:
>
>> It shows the splash page, but it seems to be waiting on something.
>>
>>
>>
>> Logs show that the Jetty server is connected.
>>
>> What can be the problems.
>>
>> I am using the Hortonworks sandbox connecting from my windows 10 machine
>>
>>
>>
>>
>>
>> Sent from Mail  for
>> Windows 10
>>
>>
>>
>

Re: nifi webui in browser not responding / waiting

[Matt Gilman]

Amit,

Anything helpful in the user log? Or what do you see if you open the Developer 
Tools in your browser and check the Network tab?

Matt

Sent from my iPhone

> On Jan 6, 2018, at 7:41 PM, Amit Pandit  wrote:
> 
> It shows the splash page, but it seems to be waiting on something.
>  
> Logs show that the Jetty server is connected.
> What can be the problems.
> I am using the Hortonworks sandbox connecting from my windows 10 machine
>  
>  
> Sent from Mail for Windows 10
>  

Re: Buttons are greyed out when initial admin account logs in

[Matt Gilman]

Hey guys,

I just wanted to chime in here. The grayed out buttons on the canvas may be
due to the fact that there was no flow.xml.gz in your conf directory when
the users/authorizers were seeded. Due to the clustering model, a node may
not know what the root group will be. Because of this, the initial admin is
granted the necessary permissions to modify these policies through the UI
after starting up. If there is a flow.xml.gz in the conf directory, the
initial admin will be automatically given permissions for the root group.

>From the admin guide...

"For a brand new secure flow, providing the "Initial Admin Identity" gives
that user access to get into the UI and to manage users, groups and
policies. But if that user wants to start modifying the flow, they need to
grant themselves policies for the root process group. The system is unable
to do this automatically because in a new flow the UUID of the root process
group is not permanent until the flow.xml.gz is generated. If the NiFi
instance is an upgrade from an existing flow.xml.gz or a 1.x instance going
from unsecure to secure, then the "Initial Admin Identity" user is
automatically given the privileges to modify the flow."

Matt

On Mon, Dec 4, 2017 at 4:30 AM, Pierre Villard 
wrote:

> Hey guys,
>
> I'll try to write a new blog with all the new features coming with NiFi
> 1.4.0.
> All the new stuff to have LDAP sync is really nice.
>
> Pierre
>
> 2017-12-03 19:12 GMT+01:00 Kevin Doran :
>
>> Hi Mike,
>>
>> You also have to enable the LdapUserGroupProvider in authorizes xml by
>> uncommenting it, configuring the properties, and changing the
>> FileAccessPolicyProvider (also in authorizers.xml) to use the
>> ldap-user-group-provider instead of the default file-user-group-provider.
>>
>> Then delete users.xml and authorizations.xml and restart.
>>
>> This will disable any certificate-based identities you have configured,
>> so you will need to choose an ldap-based user to be your initial admin. Or
>> configure a CompositeUserGroupProvider so that you can use certificates and
>> only require ldap login in absence of a client certificate.
>>
>> -Kevin
>>
>> --
>> *From:* Mike Thomsen 
>> *Sent:* Sunday, December 3, 2017 9:45:18 AM
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Buttons are greyed out when initial admin account logs in
>>
>> I added the ldap-provider to the identity provider line in
>> nifi.properties, but I don't see any users from LDAP. I tried deleting
>> users.xml and authorizations.xml and restarting, but the user listing
>> doesn't show any of the users from LDAP. Any ideas on how to troubleshoot?
>>
>> Thanks,
>>
>> Mike
>>
>> On Fri, Dec 1, 2017 at 7:05 PM, Kevin Doran 
>> wrote:
>>
>>> Mike,
>>>
>>>
>>>
>>> I should also mention that since the time of Pierre's inital blog post
>>> on LDAP integration, support for user & group syncing with LDAP has been
>>> added to NiFi. See the instructions for the "LdapUserGroupProvider" in
>>> Authorizers.xml section of the the Admin Guide [1].
>>>
>>>
>>>
>>> You will still need to set per-group or per-user policies as the initial
>>> admin, but you do not need to manually add users and groups in order to set
>>> policies. Also, your initial admin can use an identity from LDAP rather
>>> than a certificate (if that is preferred, otherwise, you can still use
>>> certificates alongside LDAP by using a CompositeUserGroupProvider as
>>> described in the Admin Guide).
>>>
>>>
>>>
>>> [1] https://nifi.apache.org/docs/nifi-docs/html/administration-g
>>> uide.html#authorizers-setup
>>>
>>>
>>>
>>> -Kevin
>>>
>>>
>>>
>>> *From: *Kevin Doran 
>>> *Date: *Friday, December 1, 2017 at 18:43
>>> *To: *
>>> *Subject: *Re: Buttons are greyed out when initial admin account logs in
>>>
>>>
>>>
>>> Hi Mike,
>>>
>>>
>>>
>>> Your authorizers.xml and nifi.properties look correct to me to establish
>>> the certificate "CN=admin, OU=NIFI" as an admin user.
>>>
>>>
>>>
>>> Here's one idea that you may have already thought of... the initial
>>> admin is only granted admin policies if users/policies are empty on
>>> startup. Try deleting conf/users.xml and conf/authorizations.xml and
>>> restarting NiFi.
>>>
>>>
>>>
>>> Hope this helps! If you have any other questions about configuring LDAP
>>> or authorizers, let me know.
>>>
>>>
>>>
>>> Kevin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *Mike Thomsen 
>>> *Reply-To: *
>>> *Date: *Friday, December 1, 2017 at 18:27
>>> *To: *
>>> *Subject: *Buttons are greyed out when initial admin account logs in
>>>
>>>
>>>
>>> I'm following Pierre's blog post that shows how to set up LDAP w/
>>> ApacheDS:
>>>
>>> https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap
>>>
>>> I've tried this with 1.4.0 and 1.5.0-SNAPSHOT (toolkits built for each
>>> too) for what it's worth.
>>>
>>> Built the certs with this command:
>>>
>>> bin/tls-toolkit.sh standalone -n localhost -C "CN=admin,OU=NIFI" -O -o
>>> ../security_output
>>>
>>> Copied security_outpu

Re: observing strange behavior nifi UI flow on a system from a recent nifi 0.6.x to nifi 1.3.x upgrade

[Matt Gilman]

Jeremy,

The behavior your describing happens when you do not have permissions to
the root group. This is the behavior when the policies for the initial
admin (configured either through the property or through a legacy
authorized-users.xml file) are created but there is no existing flow.xml.gz
in the conf directory. In this case, the admin can log in and grant
themselves the necessary permissions to the root group. From the admin
guide...

For a brand new secure flow, providing the "Initial Admin Identity" gives
that user access to get into the UI and to manage users, groups and
policies. But if that user wants to start modifying the flow, they need to
grant themselves policies for the root process group. The system is unable
to do this automatically because in a new flow the UUID of the root process
group is not permanent until the flow.xml.gz is generated. If the NiFi
instance is an upgrade from an existing flow.xml.gz or a 1.x instance going
from unsecure to secure, then the "Initial Admin Identity" user is
automatically given the privileges to modify the flow.

Hope this helps.

Matt

On Wed, Nov 22, 2017 at 2:13 PM, Jeremy Taylor 
wrote:

> Greetings,
>
> I’d like to describe some observations we are witnessing and to see if
> anyone has seen this before or has any thoughts on the matter.  Any helpful
> advice would be appreciated. I am remotely trying to help some folks on a
> system that has just been upgraded from nifi 0.6.x to 1.3.x.  They are
> simply running nifi via the “nifi.sh start” technique now and not as a
> service and have nuked their nifi repos to help avoid issues when
> upgrading.  On our development baseline we’ve done the upgrade smoothly to
> nifi 1.3.0 months ago and have never seen what is being described.  Thus, a
> nifi flow is being used that has already been updated months ago for nifi
> 1.3.0 and thus is already complaint.  I’ve also ensured our custom nars are
> in the nifi-lib dir, etc.  And, those custom nars were also upgraded months
> ago to use 1.3.0 compliant nifi processor related classes.
>
>
>
> The observations occurring in NiFi UI:
>
> 1)   the nifi processors are coming up unlabeled and their outlines
> are skeleton-dotted-lines.
>
> 2)   all connections to each processor are each red dotted lines
>
> 3)   in the top left menu of the nifi where all the icons are for
> adding processors, input ports, etc., when a mouseover occurs, instead of
> seeing a hand, we see a the equivalent of a “do NOT symbol”.  (i.e. the
> symbol in the do not smoking sign)
>
> I’ve had someone lightly look at the logs for anything special, especially
> the word “validation” in case there were any bizarre XML validation errors
> on the nifi flow file.  Again, any thoughts or advice would be
> appreciated.  Thanks.
>
>
>
> Regards,
>
>
>
> --
>
> Jeremy H. Taylor
>
> Software Developer
>
> ACES, Incorporated
>
> http://acesinc.net
>

Re: Configuration for a Web Load Balancer for a secured NiFi cluster and client certificate authentication

[Matt Gilman]

Fabian,

I have not personally set up a load balancer. However, I can offer that the
LDAP authentication utilizes signed JWT bearer tokens. The key used to sign
each token is unique to a given user. The signing key is persisted on a
given node and is currently not shared across the cluster. Because of this,
the load balancer would need to be able to associate a given client with a
specific node.

Thanks

Matt

On Thu, Oct 26, 2017 at 11:00 AM, Necci, Fabian <
fabian.ne...@thinkbiganalytics.com> wrote:

> Hi all,
>
> I have a secured NiFi cluster which supports Client Certificate
> Authentication and LDAP Authentication.
>
>
>
> We want to put a web load balancer to access the UI and the Rest API in
> front of the cluster. I have concerns about how this web load balancer
> should be configured to still support client certificate authentication
> when NiFi is accessed through the load balancer. In the researches I did,
> one possible configuration is to have a PassThrough web load balancer which
> basically sends the encrypted data to the backend servers without knowing
> anything about the data.
>
>
>
> Anybody has done this before?
>
>
>
> Also, my concern is about session stickiness, as with this type of
> configuration the load balancer is not able to perform any type of
> stickiness as it cannot read the network that is passing through.
>
>
>
> Is the lack of stickiness in the load balancer a problem?
>
>
>
> Thanks for any hints you can give me,
>
> Fabian Necci
>
>
>

Re: Controller settings and process groups

[Matt Gilman]

Uwe,

Services defined in the Controller Settings are available to Reporting
Tasks and other services defined there. Services defined within a Process
Group are available to descendent components. To create a service that is
available to all components in the flow, define it in the root Process
Group.

https://community.hortonworks.com/content/kbentry/90259/understanding-controller-service-availability-in-a.html

Matt

On Thu, Oct 12, 2017 at 2:36 PM, Uwe Geercken  wrote:

> Hello,
>
> I am using Nifi for quite a while, but I have never understood how the
> inheritance of the controller settings works.
>
> If I have a blank flow and add controller settings - e.g. a DHCPConnection
> Pool - and then add a process group, then the controller settings are not
> visible. Why is that? I thought the controller settings for the parent flow
> are visible for all descendants?
>
> rgds,
>
> Uwe
>
>
>

Re: Deep linking doesn’t work in secured NiFi

[Matt Gilman]

I've responded to your question on SO [1].

Matt

[1] https://stackoverflow.com/a/46648603/5969130

On Mon, Oct 9, 2017 at 1:01 AM, mathes waran 
wrote:

> Hi Jeff,
>
>
> Thank you for your response.
>
>
> I didn’t configure *SPNEGO*. Even LDAP authentication won’t work for deep
> link access. It navigates to login page and enter root URL after successful
> login.
>
>
> Deep link works fine when no authentication enabled. But it fails when it
> navigates to login page for authentication and then enter to root URL
> instead of actual URL (Selected Processor link).
>
>
> Consider this case and add support for the same if it is valid.
>
>
>
> Regards,
>
> Matheswaran
>
> On Sat, Sep 30, 2017 at 12:41 AM, Jeff  wrote:
>
>> Hello Matheswaran,
>>
>> Sorry you're having trouble accessing deep links with secured NiFi.  Have
>> you configured SPNEGO in your browser in order to access the NiFi UI?  I
>> believe, if you've acquired a Kerberos TGT through kinit, and your browser
>> is configured properly for Kerberos/SNPEGO, you should not see a login page
>> when you go to the NiFi UI.
>>
>> If you try to access NiFi via https://host:port/nifi/, are you able to
>> see the root group, or are you redirected to a login page?  If that is the
>> case, you should be able to log in with your Kerberos credentials via the
>> login page, and I think you'll be redirected to the original URL you were
>> attempting to access (even if it's a deep linking URL).
>>
>> If your browser isn't set up for SPNEGO access to the KDC used by NiFi,
>> you will be redirected to the login page.  Once you have configured your
>> browser for SNPEGO and have done a kinit locally, you won't have to log in
>> through the NiFi UI via the login page.
>>
>> Here's a link I found that might help you haven't configured your browser
>> for SPNEGO access to the NiFi UI:  https://ping.force.com/Suppor
>> t/PingFederate/Integrations/How-to-configure-supported-
>> browsers-for-Kerberos-NTLM
>>
>> On Thu, Sep 28, 2017 at 3:04 AM mathes waran 
>> wrote:
>>
>>>
>>> Hi,
>>>
>>>
>>> Sub: Deep linking doesn’t work in secured NiFi
>>>
>>>
>>>
>>> I have configured NiFi with Kerberos authentication. When I open deep
>>> link to access a processor group, it redirects to login page for
>>> authentication but after successful login, it navigate to root url instead
>>> of that particular processor group.
>>>
>>>
>>>
>>> Jira link found for this support - https://issues.apache.org/jira
>>> /browse/NIFI-3035
>>>
>>>
>>>
>>> But it is not working in secure mode. Is this support added only for
>>> normal mode? How can I achieve this in Kerberos mode?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Matheswaran.S
>>>
>>
>

Re: Bug in controller service view HDF 2.1.3

[Matt Gilman]

Unfortunately, no. An upgrade is required.

Matt

On Tue, Sep 12, 2017 at 12:32 PM, Papa Samba DIOP <
papasamba.d...@orange-sonatel.com> wrote:

> ***
>
> Sonatel:Scan antiviral effectue (3)
>
> *
>
> Thanks your help.
>
>
>
> Does exist a quick solution to fix it . ;) a little a patch for example,
> if it already exists
>
>
>
> Regards
>
>
>
> *De :* Matt Gilman [mailto:matt.c.gil...@gmail.com]
> *Envoyé :* mardi 12 septembre 2017 16:14
>
> *À :* users@nifi.apache.org
> *Objet :* Re: Bug in controller service view HDF 2.1.3
>
>
>
> ***
>
>
>
> Sonatel:Scan antiviral effectue (3)
>
>
>
> *
>
> Ok. There is a known bug in that version. Upgrading to the next version
> addresses this issue.
>
>
>
> This mailing list is for Apache NiFi. The most recent version of Apache
> NiFi can be downloaded here [1].
>
>
>
> Matt
>
>
>
> [1] https://nifi.apache.org/download.html
>
>
>
> On Tue, Sep 12, 2017 at 11:13 AM, Papa Samba DIOP  sonatel.com> wrote:
>
> 
>
>
>
> Sonatel:Scan antiviral effectue (1)
>
>
>
> 
>
> I’m using chrome, I have in console :
>
> nf-canvas-all.js?1.1.0.2.1.3.0-6:14 Uncaught ReferenceError:
> nfCanvasUtils is not defined
>
> at f (nf-canvas-all.js?1.1.0.2.1.3.0-6:14)
>
> at appendCellHtml (slick.grid.js:1)
>
> at appendRowHtml (slick.grid.js:1)
>
> at renderRows (slick.grid.js:1)
>
> at render (slick.grid.js:1)
>
> at handleScroll (slick.grid.js:1)
>
> at SlickGrid.resizeCanvas (slick.grid.js:1)
>
> at Object.resetTableSize (nf-canvas-all.js?1.1.0.2.1.3.0-6:14)
>
> at Object.resetTableSize (nf-canvas-all.js?1.1.0.2.1.3.0-6:18)
>
> at HTMLLIElement.select (nf-canvas-all.js?1.1.0.2.1.3.0-6:18)
>
>
>
>
>
>
>
> *De :* Matt Gilman [mailto:matt.c.gil...@gmail.com]
> *Envoyé :* mardi 12 septembre 2017 13:42
> *À :* users@nifi.apache.org
> *Objet :* Re: Bug in controller service view HDF 2.1.3
>
>
>
> ***
>
>
>
> Sonatel:Scan antiviral effectue (3)
>
>
>
> *
>
> What browser are you using? Are there any errors in the Console of the
> Developer Tools in your browser?
>
>
>
> Matt
>
>
>
> On Tue, Sep 12, 2017 at 4:22 AM, Papa Samba DIOP  sonatel.com> wrote:
>
> ***
>
>
>
> Sonatel:Scan antiviral effectue (3)
>
>
>
> *
>
> Hi all,
>
>
>
> Anyone has encountered this bug with HDF v2.1.3.0 in controller service
> UI. If someone may help on this. T
>
>
>
> Thanks in advance
>
>
>
>
>
>
>

Re: Bug in controller service view HDF 2.1.3

[Matt Gilman]

Ok. There is a known bug in that version. Upgrading to the next version
addresses this issue.

This mailing list is for Apache NiFi. The most recent version of Apache
NiFi can be downloaded here [1].

Matt

[1] https://nifi.apache.org/download.html

On Tue, Sep 12, 2017 at 11:13 AM, Papa Samba DIOP <
papasamba.d...@orange-sonatel.com> wrote:

> 
>
> Sonatel:Scan antiviral effectue (1)
>
> 
>
> I’m using chrome, I have in console :
>
> nf-canvas-all.js?1.1.0.2.1.3.0-6:14 Uncaught ReferenceError:
> nfCanvasUtils is not defined
>
> at f (nf-canvas-all.js?1.1.0.2.1.3.0-6:14)
>
> at appendCellHtml (slick.grid.js:1)
>
> at appendRowHtml (slick.grid.js:1)
>
> at renderRows (slick.grid.js:1)
>
> at render (slick.grid.js:1)
>
> at handleScroll (slick.grid.js:1)
>
> at SlickGrid.resizeCanvas (slick.grid.js:1)
>
> at Object.resetTableSize (nf-canvas-all.js?1.1.0.2.1.3.0-6:14)
>
> at Object.resetTableSize (nf-canvas-all.js?1.1.0.2.1.3.0-6:18)
>
> at HTMLLIElement.select (nf-canvas-all.js?1.1.0.2.1.3.0-6:18)
>
>
>
>
>
>
>
> *De :* Matt Gilman [mailto:matt.c.gil...@gmail.com]
> *Envoyé :* mardi 12 septembre 2017 13:42
> *À :* users@nifi.apache.org
> *Objet :* Re: Bug in controller service view HDF 2.1.3
>
>
>
> ***
>
>
>
> Sonatel:Scan antiviral effectue (3)
>
>
>
> *
>
> What browser are you using? Are there any errors in the Console of the
> Developer Tools in your browser?
>
>
>
> Matt
>
>
>
> On Tue, Sep 12, 2017 at 4:22 AM, Papa Samba DIOP  sonatel.com> wrote:
>
> ***
>
>
>
> Sonatel:Scan antiviral effectue (3)
>
>
>
> *
>
> Hi all,
>
>
>
> Anyone has encountered this bug with HDF v2.1.3.0 in controller service
> UI. If someone may help on this. T
>
>
>
> Thanks in advance
>
>
>
>
>

Re: Bug in controller service view HDF 2.1.3

[Matt Gilman]

What browser are you using? Are there any errors in the Console of the
Developer Tools in your browser?

Matt

On Tue, Sep 12, 2017 at 4:22 AM, Papa Samba DIOP <
papasamba.d...@orange-sonatel.com> wrote:

> ***
>
> Sonatel:Scan antiviral effectue (3)
>
> *
>
> Hi all,
>
>
>
> Anyone has encountered this bug with HDF v2.1.3.0 in controller service
> UI. If someone may help on this. T
>
>
>
> Thanks in advance
>
>
>
>

Re: Nifi doesn't work behind reverse proxy

[Matt Gilman]

Test 1 is the correct mapping. NiFi is comprised of a number of web
applications so you'll need to just map to the root context path and then
access the UI at public-ip/pipeline/nifi. The number of web applications is
technically unknown until startup as there are defined extension points for
custom UIs.

Matt

On Mon, Aug 7, 2017 at 6:36 AM, Raviteja Lokineni <
raviteja.lokin...@gmail.com> wrote:

> Hi all,
>
> Need your help one more time, I am trying to reverse proxy to nifi
> instance like below:
>
> Test 1:
> public-ip/pipeline -> http://nifi-host:8080
>
> This gives me a error to use /nifi when accessing public-ip/pipeline
>
> Test 2:
> public-ip/pipeline -> http://nifi-host:8080/nifi
> public-ip/nifi-docs -> http://nifi-host:8080/nifi-docs
> public-ip/nifi-api -> http://nifi-host:8080/nifi-api
>
> This redirects public-ip/pipeline to public-ip/nifi
>
> I am stuck trying to figure out all the proxies that nifi needs. Someone
> please help me out.
>
> Thanks,
>
> On Wed, Jul 19, 2017 at 6:28 PM, Matt Gilman 
> wrote:
>
>> Awesome. Glad you got everything working. I've filed a JIRA for updating
>> the documentation accordingly [1].
>>
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-4206
>>
>> On Wed, Jul 19, 2017 at 8:31 AM, Raviteja Lokineni <
>> raviteja.lokin...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> This is what I used in httpd/conf.d/nifi.conf to resolve the problem:
>>>
>>> ProxyPass/ http://:8080/
>>>> ProxyPassReverse / http://:8080/
>>>
>>>
>>>> RequestHeader add X-ProxyScheme "http"
>>>> RequestHeader add X-ProxyHost ""
>>>> RequestHeader add X-ProxyPort "80"
>>>> RequestHeader add X-ProxyContextPath "/"
>>>
>>>
>>> I wasted sometime searching for a fix. Honestly this should be part of
>>> deployment guide in the documentation
>>>
>>> Thanks,
>>>
>>> On Wed, Jul 19, 2017 at 5:30 PM, Raviteja Lokineni <
>>> raviteja.lokin...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I was trying to configure a Nifi instance under Apache HTTPD reverse
>>>> proxy. So I have set the following property: 
>>>> nifi.web.http.port.forwarding=80
>>>> but the nifi api still tries to communicate over 8080. Am I missing
>>>> something?
>>>>
>>>> Version: 1.3.0
>>>>
>>>> Thanks,
>>>> --
>>>> *Raviteja Lokineni* | Business Intelligence Developer
>>>> TD Ameritrade
>>>>
>>>> E: raviteja.lokin...@gmail.com
>>>>
>>>> [image: View Raviteja Lokineni's profile on LinkedIn]
>>>> <http://in.linkedin.com/in/ravitejalokineni>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Raviteja Lokineni* | Business Intelligence Developer
>>> TD Ameritrade
>>>
>>> E: raviteja.lokin...@gmail.com
>>>
>>> [image: View Raviteja Lokineni's profile on LinkedIn]
>>> <http://in.linkedin.com/in/ravitejalokineni>
>>>
>>>
>>
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokin...@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>

Re: New "hierarchical" controller service system confusing

[Matt Gilman]

Russell,

Thanks for the suggestion on the improved Controller Service UX. Would you
mind filing a JIRA for this improvement?

In 1.x, the user can see the components referencing a Controller Service in
1 of 3 places. The references are shown in the read-only details dialog,
the service configuration dialog, and the enable/disable service dialog.
The references will include Processors, Reporting Tasks, and Controller
Services (and components that reference those services and so on).

Thanks

Matt


On Mon, Jul 31, 2017 at 12:45 PM, Russell Bateman 
wrote:

> Friends,
>
> I find the new (well, since 0.7.x -> 1.x) way of associating controller
> services based on the process group a bit disorienting. When one follows
> the right arrow from a consuming processor and lands on the Process Group
> Configuration page where the services are configured, one is obliged to
> click on the General tab to figure out whether the controller service is
> at the root (the case for services whose flows were upgraded from the 0.7.x
> world) or for a specific process group (the case for configuration
> accomplished post-upgrade/native 1.x).
>
> I realize that this will be a dwindling problem, but there's enough white
> space in this dialog to justify putting the Process Group Name to the right
> of the dialog title (Process Group Configuration) while on the Controller
> Services tab, even without the momentary confusion of origin (that is, a
> flow from 0.7.x vs. a "native" 1.x flow). So, I consider that this
> enhancement holds regardless of whether upgraded flows are present or not.
>
> At the same time, I'm tempted to ask how, when looking on the Controller
> Services tab/page, one figures out which processor instance(s) is (are)
> consuming a particular controller service I'm looking at?
>
> Russ
>

Re: Reload authorizers.xml file at runtime

[Matt Gilman]

Kumar,

There is no alternative solution. During your initial configuration, you
need to define the admin user. See your SO question for answers to those
questions.

Thanks

Matt

On Thu, Jul 27, 2017 at 5:42 AM, kumar r  wrote:

> Hi Matt,
>
> Thank you for the response.
>
> Is there any alternate solution to achieve this?
>
> Also please look at my another stack overflow question
> https://stackoverflow.com/questions/45346902/multiple-
> initial-admin-or-initial-admin-group
>
> Thanks,
> Kumar
>
> On Wed, Jul 26, 2017 at 6:57 PM, Matt Gilman 
> wrote:
>
>> Kumar,
>>
>> The seeding for the Initial Admin and Node Identities are performed at
>> startup and only when there are no existing users/policies in place. You'll
>> need to provide an Initial Admin when securing the instance so there is a
>> user that can manage users/policies [1]. The seeding only happens during
>> the initial configuration. Following that, the users/policies are managed
>> through the application via the UI or REST API. Re-seeding on subsequent
>> start ups does not happen as Initial Admin during the initial configuration
>> may not be the same that is actively managed through the application.
>>
>> Matt
>>
>> [1] https://nifi.apache.org/docs/nifi-docs/html/administrati
>> on-guide.html#authorizers-setup
>>
>> On Wed, Jul 26, 2017 at 1:30 AM, kumar r  wrote:
>>
>>> Hi,
>>>
>>> I want to reload initial admin identity configuration at runtime. Is
>>> there any option to achieve this.
>>>
>>> Consider the case,
>>> If i have configured NiFi in secure mode using Kerberos and doesn't
>>> provided "Initial Admin". So i cannot login into NiFi with any user. I want
>>> to update the admin configuration in authorizers.xml after NiFi started and
>>> reload the file in runtime.
>>> How can i achieve this?
>>>
>>> Give me a solution or idea to make it succeeded.
>>>
>>> Thanks,
>>> Kumar
>>>
>>>
>>>
>>>
>>
>

Re: Reload authorizers.xml file at runtime

[Matt Gilman]

Kumar,

The seeding for the Initial Admin and Node Identities are performed at
startup and only when there are no existing users/policies in place. You'll
need to provide an Initial Admin when securing the instance so there is a
user that can manage users/policies [1]. The seeding only happens during
the initial configuration. Following that, the users/policies are managed
through the application via the UI or REST API. Re-seeding on subsequent
start ups does not happen as Initial Admin during the initial configuration
may not be the same that is actively managed through the application.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup

On Wed, Jul 26, 2017 at 1:30 AM, kumar r  wrote:

> Hi,
>
> I want to reload initial admin identity configuration at runtime. Is there
> any option to achieve this.
>
> Consider the case,
> If i have configured NiFi in secure mode using Kerberos and doesn't
> provided "Initial Admin". So i cannot login into NiFi with any user. I want
> to update the admin configuration in authorizers.xml after NiFi started and
> reload the file in runtime.
> How can i achieve this?
>
> Give me a solution or idea to make it succeeded.
>
> Thanks,
> Kumar
>
>
>
>

Re: Securing NiFi instance

[Matt Gilman]

Jennifer,

I just wanted to clarify that you can set up username/password
authentication with any directory server. It does not have to be Active
Directory. Many folks use OpenLDAP or ApacheDS depending on the
environment. As James mentioned Kerberos is also an option.

Let us know if you have any follow-up questions.

Thanks

Matt

On Tue, Jul 25, 2017 at 9:45 AM, James Wing  wrote:

> The NiFi distribution does not ship with support for username/password
> authentication outside LDAP and Kerberos.  But NiFi supports a pluggable
> model for login identity providers, so you can provide your own
> implementation.
>
> https://nifi.apache.org/docs/nifi-docs/html/administration-
> guide.html#user_authentication
>
> Thanks,
>
> James
>
> On Tue, Jul 25, 2017 at 4:59 AM, Tress, Jennifer L. 
> wrote:
>
>> I am hoping someone will be able to confirm what is and is not possible
>> in securing an instance of NiFi. I read some of this in previous posts but
>> wanted to be sure nothing was outdated. I am running an instance of NiFi
>> over https. Is there any way to set up username and password authentication
>> (without Active Directory) instead of certificates?
>>
>>
>>
>> Thanks!
>>
>
>

Re: Nifi doesn't work behind reverse proxy

[Matt Gilman]

Awesome. Glad you got everything working. I've filed a JIRA for updating
the documentation accordingly [1].

Matt

[1] https://issues.apache.org/jira/browse/NIFI-4206

On Wed, Jul 19, 2017 at 8:31 AM, Raviteja Lokineni <
raviteja.lokin...@gmail.com> wrote:

> Hi all,
>
> This is what I used in httpd/conf.d/nifi.conf to resolve the problem:
>
> ProxyPass/ http://:8080/
>> ProxyPassReverse / http://:8080/
>
>
>> RequestHeader add X-ProxyScheme "http"
>> RequestHeader add X-ProxyHost ""
>> RequestHeader add X-ProxyPort "80"
>> RequestHeader add X-ProxyContextPath "/"
>
>
> I wasted sometime searching for a fix. Honestly this should be part of
> deployment guide in the documentation
>
> Thanks,
>
> On Wed, Jul 19, 2017 at 5:30 PM, Raviteja Lokineni <
> raviteja.lokin...@gmail.com> wrote:
>
>> Hi all,
>>
>> I was trying to configure a Nifi instance under Apache HTTPD reverse
>> proxy. So I have set the following property: nifi.web.http.port.forwarding=80
>> but the nifi api still tries to communicate over 8080. Am I missing
>> something?
>>
>> Version: 1.3.0
>>
>> Thanks,
>> --
>> *Raviteja Lokineni* | Business Intelligence Developer
>> TD Ameritrade
>>
>> E: raviteja.lokin...@gmail.com
>>
>> [image: View Raviteja Lokineni's profile on LinkedIn]
>> 
>>
>>
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokin...@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> 
>
>

Re: NIFI FTP/SFTP modifications

[Matt Gilman]

Vijaya,

Here's a link to the contribution guide [1]. It's more oriented around
submitting a PR but should be helpful. The gist is that you'll want to run

mvn clean install

from the root directory where the NiFi source is cloned. Once that
completes the target directory of the nifi-assembly will contain the built
artifacts which you can run directly in that directory or you can copy to
another location for deployment.

Let us know if you have any follow-up questions.

Matt

[1]
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-Testyourchanges


On Wed, Jul 12, 2017 at 10:22 AM, Vijaya Maddela 
wrote:

>
>
>
>
> Hi All,
>
> We are looking to have dynamic behavior for FTP/SFTP processors.
>
>1. GETFTP processor should be used only as starting point in a flow
>(Not supported to use it in middle of the flow)
>2. FetchFTP: This supports using in middle of the flow, but doesn’t
>support to pass Dynamic properties from previous processor or step
>
>
>
> When we look into code we identified reasons for not supporting
>
>
>
> In
>
> GetFTP/GetSFTP:
>
> @InputRequirement(Requirement.INPUT_FORBIDDEN) instead of @
> InputRequirement(Requirement.INPUT_REQUIRED)
>
>
>
> FetchFTP/FetchSFTO:
>
> Doesn’t contain @DynamicProperties() in the code for supporting Dynamic
> properties
>
>
>
>
>
> Can you please some help someone how can we build the code and deploy by
> our self. Appreciate a link or tutorial for building code after modification
>
>
>
>
>
> Thank You
>
> Much Respect
>
> Vijaya Kumar Reddy.Maddela
>
> vijay.re...@eaiesb.com
>
> 91-40-40119993|9704179993 <(970)%20417-9993>
>
> USA:978-306-3751 <(978)%20306-3751>
>
> [image: O_GoldPartner]
>
>
>

Re: Getting untrusted proxy message while trying to setup secure NIFI cluster

[Matt Gilman]

Matheswaran,

The Node Identities in the authorizers.xml file need to correspond with the
DN from the server certificates each node is configured with. I would
recommend using keytool to print the certificate and then copy/paste that
DN into the authorizers.xml on each node.

NOTE: A authorizations.xml and users.xml may have already been created
during your initial attempts. You can either just edit the identities of
the nodes in the users.xml on each node or update the configuration in
authorizers.xml. If you update the authorizers.xml you'll want to remove
the authorizations.xml and users.xml as they are only generated during
startup if they do not exist currently.

Matt

On Thu, Jul 13, 2017 at 9:16 AM, James Srinivasan <
james.sriniva...@gmail.com> wrote:

> Hi,
>
> I found I had to add this to authorizations.xml for R & W, with
> corresponding users.xml entries:
>
>  action="W">
> 
> 
> 
> 
>
> Still not entirely sure my secured cluster is fully set up correctly -
> planning on writing up how we did it tho.
>
> James
>
>
> On 13 July 2017 at 13:47, mathes waran  wrote:
> > I am using nifi V-1.3, and trying to setup 3 node secure NIFI cluster.
> >
> > I have added all the required properties, I can see nodes sending
> heartbeats
> > in logs in all the nodes but on screen I'm getting Untrusted proxy
> message
> > for all nodes. error screen shot attached.
> >
> > Error log getting as NiFiAuthenticationFilter Rejecting access to web
> api:
> > Untrusted proxy CN=hostname
> >
> >  Find the nifi properties below:
> > 
> > file-provider
> > org.apache.nifi.authorization.FileAuthorizer
> >  > name="AuthorizationsFile">./conf/authorizations.xml
> > ./conf/users.xml
> > mat...@example.com
> > 
> > CN=no...@example.com,
> > OU=NIFI
> > CN=CN=no...@example.com,
> > OU=NIFI
> > CN=CN=no...@example.com,
> > OU=NIFI
> > 
> >
> >
> > could you please tell if anybody overcomes it.
> >
> > Thanks,
> > Matheswaran
>

Re: upgrading cluster from 1.0.0 to 1.3.0

[Matt Gilman]

Thanks for the great summarization Andy. Just wanted to clarify one small
detail. The default file-based authorizer will use
conf/authorizations.xml though this value is configurable in
conf/authorizers.xml. In the 0.x, when NiFi used role-based authorization
the file was called conf/authorized-users.xml.

Matt

On Thu, Jun 29, 2017 at 11:28 PM, Jeremy Farbota 
wrote:

> Andy,
>
> Thanks a ton. Huge help. We will report back.
>
> [image: Payoff, Inc.]
> *Jeremy Farbota*
> Software Engineer, Data
> Payoff, Inc.
>
> jfarb...@payoff.com
> (217) 898-8110 <+2178988110>
>
> On Thu, Jun 29, 2017 at 5:20 PM, Andy LoPresto 
> wrote:
>
>> I believe I missed a crucial file — if you are using the default
>> file-based authorizer provided by NiFi, you will also want to copy
>> conf/authorized-users.xml which defines the various users and their access
>> control policies. Sorry about that, the instance I referenced when typing
>> wasn’t using it so I forgot.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jun 29, 2017, at 5:17 PM, Andy LoPresto  wrote:
>>
>> Jeremy,
>>
>> Sorry to hear you are having difficulties upgrading. I’ll try to answer
>> your questions as best I can, but welcome others joining the thread.
>>
>> It sounds like this is less of an in-place upgrade and more of a cluster
>> migration since you are installing the software on new nodes and moving the
>> flow there. If you do not also move the various repositories, you
>> effectively have a new fresh install and have just copied the flow
>> definition.
>>
>> The Apache NiFi wiki does provide Migration Guidance [1] and this
>> document is updated with best practices/standard operating procedure with
>> each release. It is prepared by the developers who have tested the upgrade
>> process, and hopefully is helpful to end users doing the same. Perhaps even
>> more relevant is the Upgrade Guide [2], which has step-by-step instructions
>> for configuring an instance to be easily upgradable, and how to perform the
>> upgrades in-place quickly and repeatably.
>>
>> To “upgrade” to a new node and maintain the existing functionality and
>> configurations, you would want to copy additional files:
>>
>> * conf/authorizers.xml — contains the user/group providers and access
>> policy providers (separated during 1.3.0) which control permissions in the
>> application
>> * conf/bootstrap.conf — contains the master key for encryption and other
>> NiFi runtime arguments
>> * conf/logback.xml — defines log levels
>> * conf/login-identity-providers.xml — defines LDAP or Kerberos
>> connection details
>> * conf/nifi.properties — global property definition for the application
>> * conf/state-management.xml — defines state management configuration
>> (especially important for a cluster)
>> * conf/zookeeper.properties — defines ZK configurations and addresses
>> (especially important for a cluster)
>>
>> [Optional]
>>
>> * flowfile_repository/ — copy this to allow flowfile replay and history
>> * content_repository/ — copy this to allow flowfile replay and history
>> * provenance_repository/ — copy this to maintain the history of the
>> provenance records
>>
>> I’m not quite sure what you mean by “keeping the state of my consumers” —
>> is this specifically a ConsumeKafka processor or just source processors in
>> general? If you upgrade in-place or copy the various configs and
>> repositories mentioned above, when the app starts again, the component
>> state should be as you left it (barring code changes to the component in
>> the new release).
>>
>> Hopefully those instructions and the documents linked below will help
>> you. We definitely want to make upgrading a seamless experience because we
>> are constantly working to bring new features and optimizations into the
>> app, and supporting legacy versions is an added cost. We are not looking
>> for version fragmentation here. Any suggestions you have for improving the
>> upgrade process and our communication around those instructions is welcome.
>> Thanks.
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>> [2] https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Jun 29, 2017, at 4:15 PM, Jeremy Farbota  wrote:
>>
>> We are attempting to upgrade and we're having issues:
>>
>> We created new nodes with the new version (1.3.0). We stopped all flowed
>> and ensured nothing is in queue. We copied the flow.xml.gz file to the new
>> node.
>>
>> Are following the right protocol? Should we be copying the users.xml file
>> as well?
>>
>> Is there a way to migrate to the new version while keeping the state of
>> my consumers?
>>
>> It seems like I have to build a cluster and recreate the user permissions
>> and everything from scrat

Re: Why NiFi shouldn't restrict the policy removal for admin user?

[Matt Gilman]

Matheswaran,

Here is the JIRA [1] for the discussed improvement.

Thanks

Matt

[1] https://issues.apache.org/jira/browse/NIFI-4134

On Tue, Jun 27, 2017 at 8:31 AM, Matt Gilman 
wrote:

> Matheswaran,
>
> The Initial Admin grants the user with that identity the required
> permissions to administer that instance including adding/updating/removing
> users, groups, and policies. The policies are granted at a resource level.
> This means that you can introduce new administers at a Process Group level
> if you desire. The Initial Admin is not considered a special user. If they
> add another user and assign that user to the same policies, the new user
> will have equivalent permissions.
>
> That said, I think it may make sense to prevent a user from removing
> themselves from the global/top level admin policies. I'll file a JIRA to
> this effect later today.
>
> I saw your other email and SO post. If you don't have
> users/groups/policies that you had previously set up, you can just delete
> your /conf/authorizations.xml and restart. The Initial Admin
> policies will be restored. If you do have other users/groups/policies that
> you don't want to lose, I can help you restore the lost permissions by hand
> editing the authorizations.xml. Just let me know.
>
> Thanks.
>
> Matt
>
> On Tue, Jun 27, 2017 at 6:53 AM, mathes waran 
> wrote:
>
>> Hi,
>>
>> I am using nifi -1.2.0, enabled Kerberos authentication. I set the admin
>> user in initial admin Identity property of authorizers.xml file. By
>> Default, admin have full permission in NiFi. But admin can able to delete
>> his own permission.
>>
>> Once access policy removed for the admin user, then policies cannot be
>> set to any other users by admin. This behaviour looks odd. Policy for admin
>> should not be removed in any cases as we set admin user in authorizers.xml
>> file.
>>
>> Why nifi shouldn't restricted policy removal for admin user? Is there any
>> need to delete permission for admin user itself.
>>
>> Help me to understand the security flow.
>>
>> Please let me know if you have any queries,
>>
>> Thanks,
>> Matheswaran. S
>>
>
>

Re: Why NiFi shouldn't restrict the policy removal for admin user?

[Matt Gilman]

Matheswaran,

The Initial Admin grants the user with that identity the required
permissions to administer that instance including adding/updating/removing
users, groups, and policies. The policies are granted at a resource level.
This means that you can introduce new administers at a Process Group level
if you desire. The Initial Admin is not considered a special user. If they
add another user and assign that user to the same policies, the new user
will have equivalent permissions.

That said, I think it may make sense to prevent a user from removing
themselves from the global/top level admin policies. I'll file a JIRA to
this effect later today.

I saw your other email and SO post. If you don't have users/groups/policies
that you had previously set up, you can just delete your
/conf/authorizations.xml and restart. The Initial Admin policies
will be restored. If you do have other users/groups/policies that you don't
want to lose, I can help you restore the lost permissions by hand editing
the authorizations.xml. Just let me know.

Thanks.

Matt

On Tue, Jun 27, 2017 at 6:53 AM, mathes waran 
wrote:

> Hi,
>
> I am using nifi -1.2.0, enabled Kerberos authentication. I set the admin
> user in initial admin Identity property of authorizers.xml file. By
> Default, admin have full permission in NiFi. But admin can able to delete
> his own permission.
>
> Once access policy removed for the admin user, then policies cannot be set
> to any other users by admin. This behaviour looks odd. Policy for admin
> should not be removed in any cases as we set admin user in authorizers.xml
> file.
>
> Why nifi shouldn't restricted policy removal for admin user? Is there any
> need to delete permission for admin user itself.
>
> Help me to understand the security flow.
>
> Please let me know if you have any queries,
>
> Thanks,
> Matheswaran. S
>

Re: Migration - 1.2 to 1.3

[Matt Gilman]

Hey Neil,

The issue you're seeing is because some of your nodes have not been
upgraded yet. In this particular scenario, a node that hasn't been upgraded
is attempting to replicate a request to a node that has been upgraded. We
could better handle rolling upgrades in the 0.x because we could choose to
upgrade the NCM first. However, with HA clustering we don't have that
option anymore. The active coordinator is elected and assigned with ZK.
There are additional rolling upgrade concerns with the addition of the
versioned components introduced in 1.2.0.

I've created this JIRA to capture these concerns [1]. Sorry for the
inconvenience, but if you upgrade all nodes at the same time these errors
when you're using the UI should go away.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-4068

On Tue, Jun 13, 2017 at 6:19 PM, Neil Derraugh <
neil.derra...@intellifylearning.com> wrote:

> We're trying to migrate a cluster from 1.2.0 to 1.3.0 and I'm running into
> the following message after the 1.3.0 node runs for a few minutes or less.
> I have most of the 1.2.0 NARs in a custom lib directory.
>
> 2017-06-13 22:13:30,018 INFO [NiFi Web Server-694]
> org.apache.nifi.web.filter.RequestLogger Attempting request for ( found>) POST http://w.x.y.z:8080/nifi-api/access/kerberos (source ip:
> w.x.a.b)
> 2017-06-13 22:13:30,019 INFO [NiFi Web Server-694] 
> o.a.n.w.a.c.IllegalStateExceptionMapper
> java.lang.IllegalStateException: Access tokens are only issued over
> HTTPS.. Returning Conflict response.
> 2017-06-13 22:13:30,126 INFO [NiFi Web Server-691]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/current-user (source
> ip: w.x.a.b)
> 2017-06-13 22:13:30,128 INFO [NiFi Web Server-677]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/current-user (source
> ip: w.x.y.z)
> 2017-06-13 22:13:30,231 INFO [NiFi Web Server-680]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/client-id (source ip:
> w.x.a.b)
> 2017-06-13 22:13:30,309 INFO [NiFi Web Server-692]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/config (source ip:
> w.x.a.b)
> 2017-06-13 22:13:30,313 INFO [NiFi Web Server-694]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/cluster/summary (source
> ip: w.x.a.b)
> 2017-06-13 22:13:30,315 INFO [NiFi Web Server-680]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/config (source ip:
> w.x.y.z)
> 2017-06-13 22:13:30,418 INFO [NiFi Web Server-694]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/banners (source ip:
> w.x.a.b)
> 2017-06-13 22:13:30,491 INFO [NiFi Web Server-677]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/processor-types (source
> ip: w.x.a.b)
> 2017-06-13 22:13:30,493 INFO [NiFi Web Server-647]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/processor-types (source
> ip: w.x.y.z)
> 2017-06-13 22:13:30,499 INFO [NiFi Web Server-680]
> org.apache.nifi.web.filter.RequestLogger Attempting request for
> (anonymous) GET http://w.x.y.z:8080/nifi-api/flow/about (source ip:
> w.x.a.b)
> 2017-06-13 22:13:30,509 ERROR [NiFi Web Server-677]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> com.sun.jersey.api.client.ClientHandlerException:
> org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized
> field "deprecationReason" (Class 
> org.apache.nifi.web.api.dto.DocumentedTypeDTO),
> not marked as ignorable
>  at [Source: java.util.zip.GZIPInputStream@398a351; line: 1, column:
> 106647] (through reference chain: org.apache.nifi.web.api.
> entity.ProcessorTypesEntity["processorTypes"]->org.apache.
> nifi.web.api.dto.DocumentedTypeDTO["deprecationReason"]). Returning
> Internal Server Error response.
> com.sun.jersey.api.client.ClientHandlerException:
> org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized
> field "deprecationReason" (Class 
> org.apache.nifi.web.api.dto.DocumentedTypeDTO),
> not marked as ignorable
>  at [Source: java.util.zip.GZIPInputStream@398a351; line: 1, column:
> 106647] (through reference chain: org.apache.nifi.web.api.
> entity.ProcessorTypesEntity["processorTypes"]->org.apache.
> nifi.web.api.dto.DocumentedTypeDTO["deprecationReason"])
> at com.sun.jersey.api.client.ClientResponse.getEntity(
> ClientResponse.java:644)
> at com.sun.jersey.api.client.ClientResponse.getEntity(
> ClientResponse.java:586)
> at org.apache.nifi.cluster.coordination.http.endpoints.
> AbstractSingleDTOEndpoint.merge(AbstractSin

Re: NiFi web interface not starting

[Matt Gilman]

Georg,

Thanks for providing the logs. They look very normal and I'm not sure why
it's not working without knowing more. Other than port 9090, any other
customized properties/configuration? Any firewalls that could be preventing
access to that port?

Thanks

Matt

On Mon, Jun 12, 2017 at 11:19 AM, Georg Heiler 
wrote:

> Hi Matt,
>
> started vis bin/nifi.sh start
>
> Logs do not show anything interesting. Indeed, I see the mentioned lines,
> but a curl on http://localhost:9090/nifi returns curl: (52) Empty reply
> from server
>
> Bootstrap logs shows:
> 2017-06-12 18:01:50,459 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-06-12 18:01:50,462 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-06-12 18:01:50,462 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-06-12 18:01:50,462 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-06-12 18:01:50,487 INFO [main] org.apache.nifi.bootstrap.Command
> Starting Apache NiFi...
> 2017-06-12 18:01:50,488 INFO [main] org.apache.nifi.bootstrap.Command
> Working Directory: /opt/nifi/nifi-1.2.0-SNAPSHOT
> 2017-06-12 18:01:50,488 INFO [main] org.apache.nifi.bootstrap.Command
> Command: /lib/jvm/java-openjdk/bin/java -classpath
> /opt/nifi/nifi-1.2.0-SNAPSHOT/./conf:/opt/nifi/nifi-1.2.0-
> SNAPSHOT/./lib/nifi-properties-1.2.0-SNAPSHOT.jar:
> /opt/nifi/nifi-1.2.0-SNAPSHOT/./lib/nifi-nar-utils-1.2.0-
> SNAPSHOT.jar:/opt/nifi/nifi-1.2.0-SNAPSHOT/./lib/nifi-
> runtime-1.2.0-SNAPSHOT.jar:/opt/nifi/nifi-1.2.0-SNAPSHOT/.
> /lib/nifi-framework-api-1.2.0-SNAPSHOT.jar:/opt/nifi/nifi-1.
> 2.0-SNAPSHOT/./lib/log4j-over-slf4j-1.7.12.jar:/opt/nifi/
> nifi-1.2.0-SNAPSHOT/./lib/jcl-over-slf4j-1.7.12.jar:/opt/
> nifi/nifi-1.2.0-SNAPSHOT/./lib/ojdbc6.jar:/opt/nifi/nifi-
> 1.2.0-SNAPSHOT/./lib/jul-to-slf4j-1.7.12.jar:/opt/nifi/
> nifi-1.2.0-SNAPSHOT/./lib/nifi-api-1.2.0-SNAPSHOT.jar:/
> opt/nifi/nifi-1.2.0-SNAPSHOT/./lib/logback-classic-1.1.3.
> jar:/opt/nifi/nifi-1.2.0-SNAPSHOT/./lib/slf4j-api-1.7.
> 12.jar:/opt/nifi/nifi-1.2.0-SNAPSHOT/./lib/logback-core-1.1.3.jar
> -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m
> -Djava.library.path=/usr/hdp/2.6.0.3-8/hadoop/lib
> -Djava.security.egd=file:/dev/urandom 
> -Dsun.net.http.allowRestrictedHeaders=true
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC
> -Djava.protocol.handler.pkgs=sun.net.www.protocol
> -Dnifi.properties.file.path=/opt/nifi/nifi-1.2.0-SNAPSHOT/./conf/nifi.properties
> -Dnifi.bootstrap.listen.port=37055 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.
> config.log.dir=/opt/nifi/nifi-1.2.0-SNAPSHOT/logs org.apache.nifi.NiFi
> 2017-06-12 18:01:50,506 INFO [main] org.apache.nifi.bootstrap.Command
> Launched Apache NiFi with Process ID 20304
> 2017-06-12 18:01:51,063 INFO [NiFi Bootstrap Command Listener]
> org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening
> for Bootstrap requests on port 44585
>
>
> App log shows
> https://gist.github.com/geoHeil/9df1947ab7ddb5027c67669aa0f83d0e
>
> Regards,
> Georg
>
> Matt Gilman  schrieb am Mo., 12. Juni 2017 um
> 15:13 Uhr:
>
>> Georg,
>>
>> The UI will be available once you see these log messages:
>>
>> 2017-06-09 09:02:37,208 INFO [main] org.apache.nifi.web.server.JettyServer
>> NiFi has started. The UI is available at the following URLs:
>> 2017-06-09 09:02:37,208 INFO [main] org.apache.nifi.web.server.JettyServer
>> http://:/nifi
>> 2017-06-09 09:02:37,208 INFO [main] org.apache.nifi.web.server.JettyServer
>> http://:/nifi
>>
>> If you search through the logs looking for the port you're configured to
>> listen on, do you find anything? If these haven't printed yet, the UI will
>> not be available. If they have, then the UI should be available.
>>
>> Also, anything interesting in the bootstrap log?
>>
>> Thanks
>>
>> Matt
>>
>>
>> On Mon, Jun 12, 2017 at 9:09 AM, Georg Heiler 
>> wrote:
>>
>>> After restarting my NiFi host no longer the NiFi web interface will
>>> start. From the logs it looks like everything lese of NiFi is working fine.
>>>
>>> a curl to localhost: will return an empty response.
>>>
>>> What else can I try?
>>> The logs do not show any errors.
>>>
>>> Regards,
>>> Georg
>>>
>>
>>

Re: NiFi web interface not starting

[Matt Gilman]

Georg,

The UI will be available once you see these log messages:

2017-06-09 09:02:37,208 INFO [main] org.apache.nifi.web.server.JettyServer
NiFi has started. The UI is available at the following URLs:
2017-06-09 09:02:37,208 INFO [main] org.apache.nifi.web.server.JettyServer
http://:/nifi
2017-06-09 09:02:37,208 INFO [main] org.apache.nifi.web.server.JettyServer
http://:/nifi

If you search through the logs looking for the port you're configured to
listen on, do you find anything? If these haven't printed yet, the UI will
not be available. If they have, then the UI should be available.

Also, anything interesting in the bootstrap log?

Thanks

Matt


On Mon, Jun 12, 2017 at 9:09 AM, Georg Heiler 
wrote:

> After restarting my NiFi host no longer the NiFi web interface will start.
> From the logs it looks like everything lese of NiFi is working fine.
>
> a curl to localhost: will return an empty response.
>
> What else can I try?
> The logs do not show any errors.
>
> Regards,
> Georg
>

[ANNOUNCE] Apache NiFi CVE-2017-7667 and CVE-2017-7665

[Matt Gilman]

Apache NiFi PMC would like to announce the discovery and resolution of
CVE-2017-7667 and CVE-2017-7665. These issues have been resolved and new
versions of the Apache NiFi project were released in accordance with the
Apache Release Process.

Fixed in Apache NiFi 0.7.4 and 1.3.0

CVE-2017-7667: Apache NiFi XFS issue due to insufficient response headers

Severity: Important

Versions Affected:

Apache NiFi 0.0.1 - 0.7.3
Apache NiFi 1.0.0 - 1.2.0

Description: Apache NiFi needs to establish the response header telling
browsers to only allow framing with the same origin.

Mitigation: The fix to set this response header will be applied on Apache
NiFi 0.7.4 and Apache NiFi 1.3.0 releases.  Users running a prior 0.x or
1.x release should upgrade to the appropriate release.

Credit: This issue was discovered by Matt Gilman.

CVE-2017-7665: Apache NiFi XSS issue on certain user input components

Severity: Important

Versions Affected:

Apache NiFi 0.0.1 - 0.7.3
Apache NiFi 1.0.0 - 1.2.0

Description: There are certain user input components in the Apache NiFi UI
which had been guarding for some forms of XSS issues but were insufficient.

Mitigation: The fix for more complete user input sanitization will be
applied on Apache NiFi 0.7.4 and Apache NiFi 1.3.0 releases.  Users running
a prior 0.x or 1.x release should upgrade to the appropriate release.

Credit: This issue was discovered by Matt Gilman.

Re: NiFi 1.2.0 REST API problem

[Matt Gilman]

Raymond,

If you enable debug level logging, I believe that InvokeHTTP will log the
request and response. It may be helpful in diagnosing this issue. I think
you could just set the bulletin level to DEBUG to see these as messages as
bulletins. Additionally, you can update your conf/logback.xml to enable
DEBUG messages for org.apache.nifi.processors.standard.InvokeHTTP to see
these messages in your logs/nifi-app.log.

Thanks

Matt

On Thu, Jun 8, 2017 at 1:01 PM, Raymond Rogers 
wrote:

> No bulletins on any of the processors.  All of the output flow-files have
> 0 bytes and the error 401 in the attributes.
> All of the properties look correct and I can copy the values from the
> non-working to the manually created processor and it works fine.
> When you export the SSL context service and re-import it you have to reset
> the password on the trust store and that is the only change I am making.
>
> I will need to dig into the nifi logs to check for any errors there.
>
> On Thu, Jun 8, 2017 at 11:24 AM, Matt Gilman 
> wrote:
>
>> Raymond,
>>
>> When it's in a state that is not working are there any bulletins on the
>> second processor? When it's in that state and you view the configuration
>> details for that processor, do the properties look correct and the same as
>> when you manually re-add the processor through the UI? Specifically, I'm
>> wondering about the SSL Context Service since you mentioned fixing that
>> after an export/import process resolves the issue.
>>
>> Any other issues in the logs/nifi-app.log or the logs/nifi-user.log?
>>
>> Thanks
>>
>> Matt
>>
>> On Thu, Jun 8, 2017 at 11:59 AM, Raymond Rogers > > wrote:
>>
>>> We have a node.js service that automatically creates & manages NiFi
>>> groups using the REST API which works great in NiFi 1.1.1.  We are
>>> upgrading our NiFi instances to 1.2.0 and I have found that some of the
>>> processors are exhibiting odd behavior.
>>>
>>> We have a flow the connects to the Outlook 365 OWA service generates an
>>> access token and then uses that token in two different InvokeHTTP
>>> processors.  The first processor always works and the second always returns
>>> an HTTP error 401.
>>>
>>> If I delete and manually re-add the InvokeHTTP processor with the same
>>> configuration it always works.
>>>
>>> If I export this flow from the NiFi web interface and then re-import it,
>>> only fixing the SSL context service, it works every time.
>>>
>>> Using our node.js service to create the exact same flow in NiFi 1.1.1 it
>>> always works.
>>>
>>> Thanks,
>>> Raymond
>>>
>>
>>
>

Re: Saving controller services w/ templates?

[Matt Gilman]

Mike,

Currently, the services are saved if they are referenced by processors in
your data flow. There is an existing JIRA [1] to always include them.

Thanks

Matt

[1] https://issues.apache.org/jira/browse/NIFI-2895

On Thu, Jun 8, 2017 at 12:59 PM, Mike Thomsen 
wrote:

> Is it possible to save the controller services w/ a template?
>
> Thanks,
>
> Mike
>

Re: NiFi 1.2.0 REST API problem

[Matt Gilman]

Raymond,

When it's in a state that is not working are there any bulletins on the
second processor? When it's in that state and you view the configuration
details for that processor, do the properties look correct and the same as
when you manually re-add the processor through the UI? Specifically, I'm
wondering about the SSL Context Service since you mentioned fixing that
after an export/import process resolves the issue.

Any other issues in the logs/nifi-app.log or the logs/nifi-user.log?

Thanks

Matt

On Thu, Jun 8, 2017 at 11:59 AM, Raymond Rogers 
wrote:

> We have a node.js service that automatically creates & manages NiFi groups
> using the REST API which works great in NiFi 1.1.1.  We are upgrading our
> NiFi instances to 1.2.0 and I have found that some of the processors are
> exhibiting odd behavior.
>
> We have a flow the connects to the Outlook 365 OWA service generates an
> access token and then uses that token in two different InvokeHTTP
> processors.  The first processor always works and the second always returns
> an HTTP error 401.
>
> If I delete and manually re-add the InvokeHTTP processor with the same
> configuration it always works.
>
> If I export this flow from the NiFi web interface and then re-import it,
> only fixing the SSL context service, it works every time.
>
> Using our node.js service to create the exact same flow in NiFi 1.1.1 it
> always works.
>
> Thanks,
> Raymond
>

Re: Manage NiFi Users via Rest Api

[Matt Gilman]

Matt,

When a request fails, the reason for the failure will is typically included
in the response body as text/plain. This should provide some insight about
why the requests aren't working. These same details should be in the
logs/nifi-user.log of your NiFi instance.

Also, I would suggest opening the Developer Tools of your web browser while
using the UI to see examples of the requests. The UI using the REST API
exclusively.

If you're attempting to create a new user based off of an existing user
you're probably having issues with the revision and/or identifier. NiFi
will generate the identifier and the revision must start at zero.

Thanks

Matt

On Thu, Jun 1, 2017 at 3:47 PM, Matt Gibboney 
wrote:

> I am using Apache Nifi 1.1.1.  I want to be able to create, update, and
> remove users without having to log on to the NiFi web page.
>
> I have been trying to use the NiFi processors to do this, and have only
> been
> successful in using the GetHTTP processor to retrieve user information (GET
> /tenants/users/{id}).  I used that user information as a template and
> modified the "id" with a new UUID, and the "identity".  With this new flow
> file I used the PostHTTP (POST /tenants/users) to try and add the new user,
> but ended up with a 400 status code.
>
> Any help in the right direction would be much appreciated.
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.
> 2361937.n4.nabble.com/Manage-NiFi-Users-via-Rest-Api-tp2075.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>

Re: Nifi Authentication

[Matt Gilman]

No worries. Check the download page that I linked to. The nifi-toolkit is
separate. The downloads page contains the toolkit for 1.2.0. I just double
checked and it looks like support for encrypted configs was introduce in
1.1.0 so you should be good.

Thanks

Matt

On Fri, May 26, 2017 at 2:55 PM, Shrey Kapoor 
wrote:

> Hi Matt,
>
>
>
> I have nifi-1.1.1-bin.tar.gz installed. But I do not find encrypt-config
> tool in my bin directory. Is encrypt-config tool a part of nifi-toolkit
> only or Is it not a part version 1.1.1 ? Apologies for my ignorance. I am
> now to nifi.
>
>
>
> Regards,
>
> Shrey
>
>
>
> *From:* Matt Gilman [mailto:matt.c.gil...@gmail.com]
> *Sent:* Friday, May 26, 2017 11:38 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Nifi Authentication
>
>
>
> Shrey,
>
>
>
> 1) I believe that you can use the encrypt-config tool that is part of the
> nifi-toolkit to encrypt the sensitive parts of the
> login-identity-providers.xml file. The documentation for that is
> available in the admin guide [1]. The nifi-toolkit is available on the
> downloads page [2].
>
>
>
> 2) NiFi does not support AD groups yet. There is work currently underway
> that will facilitate groups being loaded from an external source.
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> encrypted-passwords-in-configuration-files
>
> [2] https://nifi.apache.org/download.html
>
>
>
> On Fri, May 26, 2017 at 2:15 PM, Shrey Kapoor 
> wrote:
>
> Hi,
>
>
>
> I have a couple of questions.
>
> 1. Is it possible to NOT put the Manager Password, Truststore Password and
> Keystore password in clear text in login-identity-providers.xml file?
>
> 2. Does Nifi support adding AD groups yet?
>
>
>
> Regards,
>
> Shrey
>
>
>
>
>

Re: Nifi Authentication

[Matt Gilman]

Shrey,

1) I believe that you can use the encrypt-config tool that is part of the
nifi-toolkit to encrypt the sensitive parts of the
login-identity-providers.xml file. The documentation for that is available
in the admin guide [1]. The nifi-toolkit is available on the downloads page
[2].

2) NiFi does not support AD groups yet. There is work currently underway
that will facilitate groups being loaded from an external source.

Thanks

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypted-passwords-in-configuration-files
[2] https://nifi.apache.org/download.html

On Fri, May 26, 2017 at 2:15 PM, Shrey Kapoor 
wrote:

> Hi,
>
>
>
> I have a couple of questions.
>
> 1. Is it possible to NOT put the Manager Password, Truststore Password and
> Keystore password in clear text in login-identity-providers.xml file?
>
> 2. Does Nifi support adding AD groups yet?
>
>
>
> Regards,
>
> Shrey
>
>
>

Re: using rest api to enable transmission in remote process group

[Matt Gilman]

Once the RPG is created it starts communicating with the target instance.
This process is done asynchronously so you will need to wait for that to
complete. The different options are really just the condition that
identifies when the connection can be created. An alternative would be to
loop and GET the RPG with each iteration and break out of your loop once
the desired port is available. At that point, your request to create the
connection should work on the first attempt.

Thanks

Matt

On Fri, May 26, 2017 at 9:08 AM, mohammed shambakey 
wrote:

> Thanks Matt
>
> I used the "/remote-processor-groups//input-ports/"
> with a JSON file that has only "revision", and under
> "remoteProcessGroupPort", it has "id", "groupId" and "transmitting", and it
> worked. (Currently, I'm using NIFI 1.2.0 and the docs says: "This
> endpoint is subject to change as NiFi and it's REST API evolve").
>
> One more thing please, I had to make a loop to make a connection between a
> "processor" and a created "remote process group". In each iteration, I
> check the status of the rest api to find if the connection is created
> successfully or not, otherwise, the loop sleeps for some time then begins
> the next iteration. I just wonder if there is something I should do in NIFI
> configuration to the need for this loop?
>
> Regards
>
> On Thu, May 25, 2017 at 11:06 AM, Matt Gilman 
> wrote:
>
>> Mohammed,
>>
>> I would guess that the request failed and the response body contains an
>> error message and you're attempting to parse that as JSON. I'd suggest
>> inspecting the response codes from the request before attempting to
>> interpret the response body. You can also look at your logs/nifi-user.log
>> for any errors from your request.
>>
>> Thanks
>>
>> Matt
>>
>> On Thu, May 25, 2017 at 10:04 AM, mohammed shambakey <
>> shambak...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I'm trying to call nifi rest api from Python "request" module (
>>> http://docs.python-requests.org/en/master/) to enable transmission in a
>>> remote process group (I think this is done by changing the "transmitting"
>>> element in the required input port to a boolean "true").
>>>
>>> So, I get the JSON file of the remote process group using this Python
>>> "requests" module commad:
>>>
>>> rpg=requests.get("http://localhost:8085/nifi-api/remote-process-groups/>> of remote process group>").json()
>>> The output json file is attached
>>>
>>> I modified the "transmitting" element of the the input port to boolean
>>> "True", then, I tried to pass the modified JSON file to the following
>>> command
>>>
>>> rpg_new=requests.put("http://localhost:8085/nifi-api/remote-
>>> process-groups/", json=rpg).json()
>>>
>>> But I receive the following error:
>>> "
>>> Traceback (most recent call last):
>>>   File "rpg_nifi_create.py", line 116, in 
>>> rpg_update=requests.put(url_rpg_update+rpg['id'],json=rpg,he
>>> aders=token_header).json()
>>>   File 
>>> "/home/shambakey1/miniconda2/lib/python2.7/site-packages/requests/models.py",
>>> line 826, in json
>>> return complexjson.loads(self.text, **kwargs)
>>>   File "/home/shambakey1/miniconda2/lib/python2.7/json/__init__.py",
>>> line 339, in loads
>>> return _default_decoder.decode(s)
>>>   File "/home/shambakey1/miniconda2/lib/python2.7/json/decoder.py",
>>> line 367, in decode
>>> raise ValueError(errmsg("Extra data", s, end, len(s)))
>>> ValueError: Extra data: line 1 column 2 - line 1 column 63 (char 1 - 62)
>>> "
>>>
>>> I tried to truncate the passed JSON file such that it contains
>>> information only about "revision" and the modified "inputPorts" list, but
>>> still the same problem.
>>>
>>> I can the JSON file (before and after modification) using jsonview, so I
>>> think there is nothing wrong the JSON structure (e.g., no independent
>>> dictionaries). I don't know what is the problem?
>>>
>>> Regards
>>>
>>> --
>>> Mohammed
>>>
>>
>>
>
>
> --
> Mohammed
>

Re: NIFI startup error

[Matt Gilman]

Ben,

The warning in your log is simply letting the developer of those components
know that the bundle is not following the suggested best practices [1].
NiFi has detected this but will not have any ill effects based on other
isolated classpath configurations and referencing components. If the
configuration would cause the component to not functional correctly, an
error would have been logged and the component would have been skipped.
This has been described more here [2] (specifically the last paragraph in
that section).

Thanks

Matt

[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
[2]
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#per-instance-classloading

On Fri, May 26, 2017 at 3:34 AM, 尹文才  wrote:

> Sorry for that the reason why my NIFI didn't get up is not due to the
> warning listed above, but I still wonder why there're such warnings.
>
> Regards,
> Ben
>
> 2017-05-26 15:06 GMT+08:00 尹文才 :
>
>> Hi guys, I'm currently writing a customized NIFI processor and after
>> putting the nar into the lib folder
>> and tried to relaunch NIFI, it was giving a bunch of following errors:
>>
>> 2017-05-26 14:52:33,183 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Controller Service org.apache.nifi.couchbase.CouchbaseClusterService is
>> bundled with its supporting APIs 
>> org.apache.nifi.couchbase.CouchbaseClusterControllerService.
>> The service APIs should not be bundled with the implementations.
>> 2017-05-26 14:52:33,198 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.couchbase.GetCouchbaseKey is
>> bundled with its referenced Controller Service APIs
>> org.apache.nifi.couchbase.CouchbaseClusterControllerService. The service
>> APIs should not be bundled with component implementations that reference it.
>> 2017-05-26 14:52:33,214 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.couchbase.PutCouchbaseKey is
>> bundled with its referenced Controller Service APIs
>> org.apache.nifi.couchbase.CouchbaseClusterControllerService. The service
>> APIs should not be bundled with component implementations that reference it.
>> 2017-05-26 14:52:33,355 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Controller Service org.apache.nifi.processors.gcp
>> .credentials.service.GCPCredentialsControllerService is bundled with its
>> supporting APIs org.apache.nifi.processors.gcp
>> .credentials.service.GCPCredentialsService. The service APIs should not
>> be bundled with the implementations.
>> 2017-05-26 14:52:33,417 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.gcp.storage.PutGCSObject is bundled
>> with its referenced Controller Service APIs org.apache.nifi.processors.gcp
>> .credentials.service.GCPCredentialsService. The service APIs should not
>> be bundled with component implementations that reference it.
>> 2017-05-26 14:52:33,417 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.gcp.storage.FetchGCSObject is
>> bundled with its referenced Controller Service APIs
>> org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService.
>> The service APIs should not be bundled with component implementations that
>> reference it.
>> 2017-05-26 14:52:33,417 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.gcp.storage.DeleteGCSObject is
>> bundled with its referenced Controller Service APIs
>> org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService.
>> The service APIs should not be bundled with component implementations that
>> reference it.
>> 2017-05-26 14:52:33,433 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.gcp.storage.ListGCSBucket is
>> bundled with its referenced Controller Service APIs
>> org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsService.
>> The service APIs should not be bundled with component implementations that
>> reference it.
>> 2017-05-26 14:52:34,511 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Controller Service org.apache.nifi.processors.aws
>> .credentials.provider.service.AWSCredentialsProviderControllerService is
>> bundled with its supporting APIs org.apache.nifi.processors.aws
>> .credentials.provider.service.AWSCredentialsProviderService. The service
>> APIs should not be bundled with the implementations.
>> 2017-05-26 14:52:34,543 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.aws.s3.FetchS3Object is bundled
>> with its referenced Controller Service APIs org.apache.nifi.processors.aws
>> .credentials.provider.service.AWSCredentialsProviderService. The service
>> APIs should not be bundled with component implementations that reference it.
>> 2017-05-26 14:52:34,558 WARN [main] org.apache.nifi.nar.ExtensionManager
>> Component org.apache.nifi.processors.aws.s3.PutS3Object is bundled with
>> its referenced Contro

Re: using rest api to enable transmission in remote process group

[Matt Gilman]

Mohammed,

I would guess that the request failed and the response body contains an
error message and you're attempting to parse that as JSON. I'd suggest
inspecting the response codes from the request before attempting to
interpret the response body. You can also look at your logs/nifi-user.log
for any errors from your request.

Thanks

Matt

On Thu, May 25, 2017 at 10:04 AM, mohammed shambakey 
wrote:

> Hi
>
> I'm trying to call nifi rest api from Python "request" module (
> http://docs.python-requests.org/en/master/) to enable transmission in a
> remote process group (I think this is done by changing the "transmitting"
> element in the required input port to a boolean "true").
>
> So, I get the JSON file of the remote process group using this Python
> "requests" module commad:
>
> rpg=requests.get("http://localhost:8085/nifi-api/remote-process-groups/ of remote process group>").json()
> The output json file is attached
>
> I modified the "transmitting" element of the the input port to boolean
> "True", then, I tried to pass the modified JSON file to the following
> command
>
> rpg_new=requests.put("http://localhost:8085/nifi-api/
> remote-process-groups/", json=rpg).json()
>
> But I receive the following error:
> "
> Traceback (most recent call last):
>   File "rpg_nifi_create.py", line 116, in 
> rpg_update=requests.put(url_rpg_update+rpg['id'],json=rpg,
> headers=token_header).json()
>   File 
> "/home/shambakey1/miniconda2/lib/python2.7/site-packages/requests/models.py",
> line 826, in json
> return complexjson.loads(self.text, **kwargs)
>   File "/home/shambakey1/miniconda2/lib/python2.7/json/__init__.py", line
> 339, in loads
> return _default_decoder.decode(s)
>   File "/home/shambakey1/miniconda2/lib/python2.7/json/decoder.py", line
> 367, in decode
> raise ValueError(errmsg("Extra data", s, end, len(s)))
> ValueError: Extra data: line 1 column 2 - line 1 column 63 (char 1 - 62)
> "
>
> I tried to truncate the passed JSON file such that it contains information
> only about "revision" and the modified "inputPorts" list, but still the
> same problem.
>
> I can the JSON file (before and after modification) using jsonview, so I
> think there is nothing wrong the JSON structure (e.g., no independent
> dictionaries). I don't know what is the problem?
>
> Regards
>
> --
> Mohammed
>

Re: Content Repository Retention

[Matt Gilman]

Jim,

Yes, the contain can be archived once the flowfile has exited NiFi. This
allows the content to be downloaded/viewed from provenance events for that
flowfile. This behavior can be controlled using
the nifi.content.repository.archive.* properties [1].

Thanks

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-system-content-repository-properties

On Wed, May 24, 2017 at 2:46 PM, James McMahon  wrote:

> Good afternoon. I am running Apache NiFi 0.7.x. This morning I ran a large
> body of content through my workflow, and my content repo usage as reported
> by 'df -kh' spiked from less than one gigabyte of used capacity to 40G.
>
> My files fully processed without error. My processors and queues now show
> 0 bytes across the board, including all processor groups.
>
> My content repository is still showing 22G usage. My expectation was that
> it would reduce back down to minimal capacity usage when my workflow
> completed processing all these files, but it has not.
>
> I run nifi on my Linux box as a service. I gracefully stopped the service.
> Let it sit for a minute or two. Checked my logs. No problems noted. I
> started nifi again as a service. No errors in the logs. I still find that
> the content repo disk device shows 22G used capacity.
>
> Does the content repository retain content history for a period of time
> after a flowfile has exited from nifi? Can that be regulated?
>
> Thanks in advance for your insights.
>
> Jim
>

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Neil,

Disconnecting a node prior to removal is the correct process. It appears
that the check was lost going from 0.x to 1.x. Folks reported this JIRA [1]
indicating that deleting a connected node did not work. This process does
not work because the node needs to be disconnected first. The JIRA was
addressed by restoring the check that a node is disconnected prior to
deletion.

Hopefully the JIRA I filed earlier today [2] will address the phantom node
you were seeing. Until then, can you update your workaround to disconnect
the node in question prior to deletion?

Thanks

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3295
[2] https://issues.apache.org/jira/browse/NIFI-3933

On Thu, May 18, 2017 at 12:29 PM, Neil Derraugh <
neil.derra...@intellifylearning.com> wrote:

> Pretty sure this is the problem I was describing in the "Phantom Node"
> thread recently.
>
> If I kill non-primary nodes the cluster remains healthy despite the lost
> nodes.  The terminated nodes end up with a DISCONNECTED status.
>
> If I kill the primary it winds up with a CONNECTED status, but a new
> primary/cluster coordinator gets elected too.
>
> Additionally it seems in 1.2.0 that the REST API no longer support
> deleting a node in a CONNECTED state (Cannot remove Node with ID
> 1780fde7-c2f4-469c-9884-fe843eac5b73 because it is not disconnected,
> current state = CONNECTED).  So right now I don't have a workaround and
> have to kill all the nodes and start over.
>
> On Thu, May 18, 2017 at 11:20 AM, Mark Payne  wrote:
>
>> Hello,
>>
>> Just looking through this thread now. I believe that I understand the
>> problem. I have updated the JIRA with details about what I think is the
>> problem and a potential remedy for the problem.
>>
>> Thanks
>> -Mark
>>
>> > On May 18, 2017, at 9:49 AM, Matt Gilman 
>> wrote:
>> >
>> > Thanks for the additional details. They will be helpful when working
>> the JIRA. All nodes, including the coordinator, heartbeat to the active
>> coordinator. This means that the coordinator effectively heartbeats to
>> itself. It appears, based on your log messages, that this is not happening.
>> Because no heartbeats were receive from any node, the lack of heartbeats
>> from the terminated node is not considered.
>> >
>> > Matt
>> >
>> > Sent from my iPhone
>> >
>> >> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
>> >>
>> >> Found something interesting in the centos-b debug logging
>> >>
>> >> after centos-a (the coordinator) is killed centos-b takes over. Notice
>> how
>> >> it "Will not disconnect any nodes due to lack of heartbeat" and how it
>> still
>> >> sees centos-a as connected despite the fact that there are no
>> heartbeats
>> >> anymore.
>> >>
>> >> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
>> >> o.apache.nifi.controller.FlowController This node elected Active
>> Cluster
>> >> Coordinator
>> >> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
>> >> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
>> >> o.apache.nifi.controller.FlowController This node has been elected
>> Primary
>> >> Node
>> >> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
>> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will
>> not
>> >> disconnect any nodes due to lack of heartbeat
>> >> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat
>> from
>> >> centos-b:8080
>> >> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
>> >>
>> >> Calculated diff between current cluster status and node cluster status
>> as
>> >> follows:
>> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
>> state=CONNECTED,
>> >> updateId=42]]
>> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
>> state=CONNECTED,
>> >> updateId=42]]
>> >> Difference: []
>> >>
>> >>
>> >> 2017-05-18 12:41:41,337 INFO [Proces

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Thanks for the additional details. They will be helpful when working the JIRA. 
All nodes, including the coordinator, heartbeat to the active coordinator. This 
means that the coordinator effectively heartbeats to itself. It appears, based 
on your log messages, that this is not happening. Because no heartbeats were 
receive from any node, the lack of heartbeats from the terminated node is not 
considered.

Matt

Sent from my iPhone

> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
> 
> Found something interesting in the centos-b debug logging 
> 
> after centos-a (the coordinator) is killed centos-b takes over. Notice how
> it "Will not disconnect any nodes due to lack of heartbeat" and how it still
> sees centos-a as connected despite the fact that there are no heartbeats
> anymore.
> 
> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
> o.apache.nifi.controller.FlowController This node elected Active Cluster
> Coordinator
> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
> o.apache.nifi.controller.FlowController This node has been elected Primary
> Node
> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will not
> disconnect any nodes due to lack of heartbeat
> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
> centos-b:8080
> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 
> 
> Calculated diff between current cluster status and node cluster status as
> follows:
> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Difference: []
> 
> 
> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341 bytes)
> from centos-b:8080 in 3 millis
> 2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-18
> 12:41:41,330 and sent to centos-b:10001 at 2017-05-18 12:41:41,339; send
> took 8 millis
> 2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in
> 93276 nanos
> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
> centos-b:8080
> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 
> 
> Calculated diff between current cluster status and node cluster status as
> follows:
> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Difference: []
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1950.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Thank you for the confirmation. I've filed this JIRA [1] to track the issue.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3933

On Thu, May 18, 2017 at 8:21 AM, ddewaele  wrote:

> Thanks for the response.
>
> When killing a non-coordinator node, it does take 8 * 5 seconds before I
> see
> this :
>
> nifi-app.log:2017-05-18 12:04:29,644 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.node.NodeClusterCoordinator Status of centos-b:8080 changed from
> NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED, updateId=26]
> to
> NodeConnectionStatus[nodeId=centos-b:8080, state=DISCONNECTED, Disconnect
> Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat
> from
> node in 43 seconds, updateId=27]
>
> When killing the coordinator node, the newly appointed coordinator doesn't
> seem to detect the heartbeat timeout.
>
> I'll see if I can enable the debug logging.
>
> My Nifi runs inside a KVM. KVM includes 3 seperate VMs. External zookeeper
> (replicated mode) running on the 3 VMs, and 2 VMs used for NiFi nodes.
>
> I have the same issue in a dockerized environment
>
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.
> 2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-
> node-when-node-was-killed-tp1942p1948.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Hi,

Once the new coordinator was elected, it is responsible for disconnecting
nodes due to lack of heartbeat. It will wait 8 times the
configured nifi.cluster.protocol.heartbeat.interval before the node
is disconnected. Can you confirm that this amount of time has elapsed?

Did you see any messages containing "Have not received a heartbeat from
node in" or "Failed to remove heartbeat for" during this time? Can you
describe your environment a little more? Are you running an external or
embedded zookeeper?

Can you enabled debug level logging for this package?
org.apache.nifi.cluster.coordination.heartbeat

Thanks

Matt

On Thu, May 18, 2017 at 7:30 AM, ddewaele  wrote:

> Hi,
>
> I have a NiFi cluster up and running and I'm testing various failover
> scenarios.
>
> I have 2 nodes in the cluster :
>
> - centos-a : Coordinator node / primary
> - centos-b : Cluster node
>
> I noticed in 1 of the scenarios when I killed the Cluster Coordinator node,
> that the following happened :
>
> centos-b couldn't contact the coordinator anymore and became the new
> coordinator / primary node. (as expected) :
>
> Failed to send heartbeat due to:
> org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message
> to Cluster Coordinator due to: java.net.ConnectException: Connection
> refused
> (Connection refused)
> This node has been elected Leader for Role 'Primary Node'
> This node has been elected Leader for Role 'Cluster Coordinator'
>
> When attempting to access the UI on centos-b, I got the following error :
>
> 2017-05-18 11:18:49,368 WARN [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to centos-a:8080 due to {}
>
> If my understanding is correct, NiFi will try to replicate to connected
> nodes in the cluster. Here, centos-a was killed a while back and should
> have
> been disconnected, but as far as NiFi was concerned it was still connected.
>
> As a result I cannot access the UI anymore (due to the replication error),
> but I can lookup the cluster info via the REST API. And sure enough, it
> still sees centos-a as being CONNECTED.
>
> {
> "cluster": {
> "generated": "11:20:13 UTC",
> "nodes": [
> {
> "activeThreadCount": 0,
> "address": "centos-b",
> "apiPort": 8080,
> "events": [
> {
> "category": "INFO",
> "message": "Node Status changed from CONNECTING to
> CONNECTED",
> "timestamp": "05/18/2017 11:17:31 UTC"
> },
> {
> "category": "INFO",
> "message": "Node Status changed from [Unknown Node]
> to CONNECTING",
> "timestamp": "05/18/2017 11:17:27 UTC"
> }
> ],
> "heartbeat": "05/18/2017 11:20:09 UTC",
> "nodeId": "a5bce78d-23ea-4435-a0dd-4b731459f1b9",
> "nodeStartTime": "05/18/2017 11:17:25 UTC",
> "queued": "8,492 / 13.22 MB",
> "roles": [
> "Primary Node",
> "Cluster Coordinator"
> ],
> "status": "CONNECTED"
> },
> {
> "address": "centos-a",
> "apiPort": 8080,
> "events": [],
> "nodeId": "b89e8418-4b7f-4743-bdf4-4a08a92f3892",
> "roles": [],
> "status": "CONNECTED"
> }
> ]
> }
> }
>
> When centos-a was brought back online, i noticed the following status
> change
> :
>
> Status of centos-a:8080 changed from
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=15]
> to
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTING, updateId=19]
>
> So it went from connected -> connecting.
>
> It clearly missed the disconnected step here.
>
> When shutting down the centos-a node using nifi.sh stop, it goes into the
> DISCONNECTED state :
>
> Status of centos-a:8080 changed from
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=12]
> to
> NodeConnectionStatus[nodeId=centos-a:8
> 080, state=DISCONNECTED, Disconnect Code=Node was Shutdown, Disconnect
> Reason=Node was Shutdown, updateId=13]
>
> How can I debug this further, and can somebody provide some additional
> insights ? I have seen nodes getting disconnected due to missing heartbeats
>
> tatus of centos-a:8080 changed from
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=10]
> to
> NodeConnectionStatus[nodeId=centos-a:8080, state=DISCONNECTED, Disconnect
> Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat
> from
> node in 41 seconds, updateId=11]
>
> But sometimes it doesn't seem to detect this, and NiFi keeps on thinking it
> is CONNECTED, despite not having 

Re: Queue incoherent state

[Matt Gilman]

Thanks for following up with these details. I'm going to add these
observations to the corresponding JIRA [1].

Thanks and sorry again for the inconvenience.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3897

On Tue, May 16, 2017 at 3:34 AM, Arnaud G  wrote:

> Hi Matt,
>
> Thanks for your reply!
>
> I finally solved the problem by deleting all the content in the flowfile
> directory, but here are my observations:
>
> 1) The problem was coming from one of the cluster node, when this node was
> out of the cluster, the queue were reporting 0 flowfile.
> 2) The first time I restarted this node, about 20'000 flowfile reappeared
> and were treated, every time I subsequently restarted this node about
> 20-30k flowfiles were again treated (I was only specifically monitoring one
> queue, but it happened for multiple other queues)
> 3) After 3-4 reboots of this node the queue reported 90K elements and
> remained in this state despite multiple other restart.
> 4) The flowfile directory on this node contained 200 MB of data
> 5) I tried to setup the flowfile expiration but it didn't do anything to
> the queue status
> 6) I tried to change the backpressure threshold without any effect.
> 7) During the problem the queue was operating normally on the cluster, and
> flowfiles were flowing through it without any issue.
>
> Arnaud
>
>
>
> On Mon, May 15, 2017 at 10:39 PM, Matt Gilman 
> wrote:
>
>> Sorry for the delayed response. Similar behavior has been reported by
>> some other users [1]. Does the connection have any back pressure threshold
>> configured? Can new flowfiles be enqueued? Do the expiration settings have
>> any affect?
>>
>> Lastly, if you restart the cluster does it claim the connection still has
>> flowfiles enqueued?
>>
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3897
>>
>> On Fri, May 12, 2017 at 5:47 AM, Arnaud G  wrote:
>>
>>> Hi again!
>>>
>>> I currently have  another issue with incoherent queue status.
>>>
>>> Following the upgrade to 1.2 of a cluster, I have a couple of queues
>>> that display through the GUI a high number of flowfiles.
>>>
>>> As the queue were no emptying despite tuning, I tried to list the
>>> content of the queue. This action returns that the queue contains no
>>> flowfile, which is not the expected as the GUI displays another value.
>>>
>>> If I try to empty the queue, I receive a message: 0 FlowFiles (0 bytes)
>>> out of 210'000 (92.71MB) were removed from the queue.
>>>
>>> And of course I cannot delete the queue as this action reports me that
>>> the queue is not empty.
>>>
>>> So somehow it seems that the queue are empty but that the current
>>> display of the queue don't reflect it (it is very likely that some data
>>> were lost during the upgrade procedure as we had to reboot a few node to
>>> change the heap property)
>>>
>>> What will be the best method to restore a proper state and be able to
>>> edit the flow file again?
>>>
>>> Thank you!
>>>
>>> Arnaud
>>>
>>>
>>>
>>
>

Re: Queue incoherent state

[Matt Gilman]

Sorry for the delayed response. Similar behavior has been reported by some
other users [1]. Does the connection have any back pressure threshold
configured? Can new flowfiles be enqueued? Do the expiration settings have
any affect?

Lastly, if you restart the cluster does it claim the connection still has
flowfiles enqueued?

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3897

On Fri, May 12, 2017 at 5:47 AM, Arnaud G  wrote:

> Hi again!
>
> I currently have  another issue with incoherent queue status.
>
> Following the upgrade to 1.2 of a cluster, I have a couple of queues that
> display through the GUI a high number of flowfiles.
>
> As the queue were no emptying despite tuning, I tried to list the content
> of the queue. This action returns that the queue contains no flowfile,
> which is not the expected as the GUI displays another value.
>
> If I try to empty the queue, I receive a message: 0 FlowFiles (0 bytes)
> out of 210'000 (92.71MB) were removed from the queue.
>
> And of course I cannot delete the queue as this action reports me that the
> queue is not empty.
>
> So somehow it seems that the queue are empty but that the current display
> of the queue don't reflect it (it is very likely that some data were lost
> during the upgrade procedure as we had to reboot a few node to change the
> heap property)
>
> What will be the best method to restore a proper state and be able to edit
> the flow file again?
>
> Thank you!
>
> Arnaud
>
>
>

Re: Importing template into 1.2.0

[Matt Gilman]

Tim,

Just wanted to provide a few more details. Prior to 1.2.0 the flow and
templates did not contain any details about the NAR (bundle) it was
packaged in. In 1.2.0, since your template did not contain any bundle
details NiFi attempted to find a compatible Processor (matched by class
name only). If the instance detects that more than one Processor exists
with that class name you will receive the error you described.

Re-exporting the template in 1.2.0 will include the relevant bundle details
for future use.

Hope this helps

Matt

On Thu, May 11, 2017 at 7:52 PM, Andy LoPresto  wrote:

> Hi Tim.
>
> What processor has multiple versions and which is the desired target? As
> your 1.2.0 installation is a clean install, I would actually recommend just
> exporting the complete flow.xml.gz file from your existing 1.1.2 instance
> and placing it in the $NIFI_HOME/conf directory for the 1.2.0 instance.
>
> Bryan Bende also wrote a good article describing the feature and some of
> the background if you haven’t seen it yet [1].
>
> [1] http://bryanbende.com/development/2017/05/10/apache-
> nifi-1-2-0-component-versioning
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 11, 2017, at 4:47 PM, Tim Zimmerman  wrote:
>
> First thanks for the 1.2.0 release. There are a few things I have been
> looking forward to (wait/notify).
>
> Now, I have a question. I was running 1.1.2 locally and experimenting with
> a few ideas. After creating a template and exporting one of those ideas,
> I unpacked 1.2.0 and started it up.
>
> I am able to import the template fine but when I try to place it on the
> canvas I get dialog that there is more than one version of the processor on
> the path and the template does not appear.
>
> I understand this is due to the new versioning and I thought maybe if I
> just hacked in the  element I could get this template working but I
> did that and tried again but it still does not seem to be working.
>
> How should I be migrating 1.1.2 work into 1.2.0?
>
>
>

Re: Issues with creating and connecting RPG programmatically

[Matt Gilman]

I've responded to the question on StackOverflow [1].

Thanks

Matt

[1]
http://stackoverflow.com/questions/43788780/nifi-issues-with-creating-and-connecting-rpg-programmatically/43789310#43789310

On Thu, May 4, 2017 at 12:49 PM, Pushkara R  wrote:

> Hi,
>
> I am trying to write a function that takes two parameters, a processor ID
> in machine A and an input port in machine B and creates an RPG in machine B
> which connects the above to.
>
> Now, how I am doing that is
> 1. A POST to the
> */nifi-api/process-groups//remote-process-groups*
> endpoint to create an RPG and retrieve the ID of that RPG
> 2. A POST to the */nifi-api/process-groups//connections
> *endpoint to create a connection between the processor and the input
> port. the processorID and the ID of the input port are being provided along
> with the list of the relationships.
> 3. A final PUT to */nifi-api/remote-process-groups/ *to enable the
> transmission between the machines*.*
>
> Now, the function always throws errors in step 2. A 409 is thrown for the
> POST request with the error being 'Unable to find specified destination'.
> (though refreshing the canvas on machine 1 shows the rpg having been
> created)
> However, when I manually run the steps 2 and 3 afterwards, with the same
> rpgid, the connection happens.
>
> Now, I'm not sure if this is a synchronization issue or not, but I want to
> figure it out because I would not want to separate out steps 1 2 and 3.
> Could somebody point out what could be the issue here?
>
> Pushkar
>
> PS - the post messages for steps 2 are the same when the api is called
> from within the function and manually.
>

Re: ERRORs when trying to make new RPGs

[Matt Gilman]

Yes, a position of (0, 0) will work. After 1.2.0 is released you won't need to 
specify it anymore and will default to (0, 0).

Matt

Sent from my iPhone

> On Apr 19, 2017, at 1:13 PM, Pushkara R  wrote:
> 
> Thanks a lot matt. I was not concerned with position which is why I did not 
> provide it in the payload. Post requests to create processors and ports 
> worked without the position so I assumed it would do so for RPGs as well.
> 
> Can I use 0,0 as a default position? 
> 
> Thanks
> Pushkar
> 
> On 19-Apr-2017 21:42, "Matt Gilman"  wrote:
> This is a known issue [1] that has been addressed in the upcoming 1.2.0 
> release. If you include a position then you should be able to creation the 
> component successfully. Sorry for the inconvenience. 
> 
> Matt
> 
> [1] https://issues.apache.org/jira/browse/NIFI-2994
> 
>> On Wed, Apr 19, 2017 at 12:00 PM, Bryan Rosander  
>> wrote:
>> One way to see what the payload should look like would be to open up the 
>> network tab in your browser and perform the action you're trying to 
>> automate.  You should be able to see what requests the NiFi UI performs and 
>> replicate those.
>> 
>> Thanks,
>> Bryan
>> 
>>> On Wed, Apr 19, 2017 at 11:49 AM, Joe Witt  wrote:
>>> Hello
>>> 
>>> It appears to be throwing an NPE when establishing the position of the
>>> component you're trying to add.  While clearly an unfriendly error
>>> message as-is did you specify the position in your request?
>>> 
>>> Thanks
>>> JOe
>>> 
>>> On Wed, Apr 19, 2017 at 10:36 AM, Pushkara R  wrote:
>>> > Hi,
>>> >
>>> > I'm writing a custom REST client for NiFi. I am having troubles with the
>>> > POST method on the /process-groups//remote-process-groups endpoint.
>>> >
>>> > The body of the POST method is as follows :
>>> >
>>> > {
>>> > "component": {
>>> > "targetUri": target_uri,
>>> > "communicationsTimeout": "30 sec",
>>> > "yieldDuration": "10 sec",
>>> > "transportProtocol": "HTTP",
>>> > "proxyHost": "",
>>> > "proxyPort": "",
>>> > "proxyUser": "",
>>> > "proxyPassword": ""
>>> > }, "revision": {
>>> > "clientId": str(uuid.uuid4()),
>>> > "version": 0
>>> > }
>>> > }
>>> >
>>> > Whenever I make the call with the target_uri as
>>> > "http://10.24.24.221:8080/nifi";, I get back a status of 500 and a response
>>> > that says "An unexpected error has occured. Please check the logs for
>>> > additional details."
>>> >
>>> > I have pasted the error logs from nifi-app.log and then below that logs 
>>> > from
>>> > nifi-user.log. Please let me know what could be causing this issue. It 
>>> > would
>>> > be really helpful.
>>> >
>>> > Thanks
>>> > Pushkar
>>> >
>>> > 
>>> > I 2017-04-19 19:54:35,663 ERROR [Flow Service Tasks Thread-2]
>>> > o.a.nifi.controller.StandardFlowService Unable to save flow controller
>>> > configuration due to: java.lang.NullPointerException
>>> > java.lang.NullPointerException: null
>>> > at
>>> > org.apache.nifi.controller.serialization.StandardFlowSerializer.addPosition(StandardFlowSerializer.java:141)
>>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>>> > at
>>> > org.apache.nifi.controller.serialization.StandardFlowSerializer.addPosition(StandardFlowSerializer.java:136)
>>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>>> > at
>>> > org.apache.nifi.controller.serialization.StandardFlowSerializer.addRemoteProcessGroup(StandardFlowSerializer.java:247)
>>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>>> > at
>>> > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:190)
>>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>>> > at
>>> > org.apache.nifi.controller.serialization.StandardFlowSerializer.serialize(StandardFlowSerializer.java:96)
>>> > ~[nifi-framework-core-1.1.2.

Re: ERRORs when trying to make new RPGs

[Matt Gilman]

This is a known issue [1] that has been addressed in the upcoming 1.2.0
release. If you include a position then you should be able to creation the
component successfully. Sorry for the inconvenience.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-2994

On Wed, Apr 19, 2017 at 12:00 PM, Bryan Rosander 
wrote:

> One way to see what the payload should look like would be to open up the
> network tab in your browser and perform the action you're trying to
> automate.  You should be able to see what requests the NiFi UI performs and
> replicate those.
>
> Thanks,
> Bryan
>
> On Wed, Apr 19, 2017 at 11:49 AM, Joe Witt  wrote:
>
>> Hello
>>
>> It appears to be throwing an NPE when establishing the position of the
>> component you're trying to add.  While clearly an unfriendly error
>> message as-is did you specify the position in your request?
>>
>> Thanks
>> JOe
>>
>> On Wed, Apr 19, 2017 at 10:36 AM, Pushkara R 
>> wrote:
>> > Hi,
>> >
>> > I'm writing a custom REST client for NiFi. I am having troubles with the
>> > POST method on the /process-groups//remote-process-groups endpoint.
>> >
>> > The body of the POST method is as follows :
>> >
>> > {
>> > "component": {
>> > "targetUri": target_uri,
>> > "communicationsTimeout": "30 sec",
>> > "yieldDuration": "10 sec",
>> > "transportProtocol": "HTTP",
>> > "proxyHost": "",
>> > "proxyPort": "",
>> > "proxyUser": "",
>> > "proxyPassword": ""
>> > }, "revision": {
>> > "clientId": str(uuid.uuid4()),
>> > "version": 0
>> > }
>> > }
>> >
>> > Whenever I make the call with the target_uri as
>> > "http://10.24.24.221:8080/nifi";, I get back a status of 500 and a
>> response
>> > that says "An unexpected error has occured. Please check the logs for
>> > additional details."
>> >
>> > I have pasted the error logs from nifi-app.log and then below that logs
>> from
>> > nifi-user.log. Please let me know what could be causing this issue. It
>> would
>> > be really helpful.
>> >
>> > Thanks
>> > Pushkar
>> >
>> > 
>> > I 2017-04-19 19:54:35,663 ERROR [Flow Service Tasks Thread-2]
>> > o.a.nifi.controller.StandardFlowService Unable to save flow controller
>> > configuration due to: java.lang.NullPointerException
>> > java.lang.NullPointerException: null
>> > at
>> > org.apache.nifi.controller.serialization.StandardFlowSeriali
>> zer.addPosition(StandardFlowSerializer.java:141)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.controller.serialization.StandardFlowSeriali
>> zer.addPosition(StandardFlowSerializer.java:136)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.controller.serialization.StandardFlowSeriali
>> zer.addRemoteProcessGroup(StandardFlowSerializer.java:247)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.controller.serialization.StandardFlowSeriali
>> zer.addProcessGroup(StandardFlowSerializer.java:190)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.controller.serialization.StandardFlowSeriali
>> zer.serialize(StandardFlowSerializer.java:96)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.controller.FlowController.serialize(FlowCont
>> roller.java:1431)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.
>> save(StandardXMLFlowConfigurationDAO.java:162)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > org.apache.nifi.controller.StandardFlowService$SaveReporting
>> Task.run(StandardFlowService.java:963)
>> > ~[nifi-framework-core-1.1.2.jar:1.1.2]
>> > at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > [na:1.8.0_121]
>> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
>> 308)
>> > [na:1.8.0_121]
>> > at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > [na:1.8.0_121]
>> > at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > [na:1.8.0_121]
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> > [na:1.8.0_121]
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> > [na:1.8.0_121]
>> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
>> >
>> >
>> > 
>> > 2017-04-19 19:54:35,219 INFO [NiFi Web Server-75]
>> > org.apache.nifi.web.filter.RequestLogger Attempting request for
>> (anonymous)
>> > POST
>> > http://10.24.24.222:8080/nifi-api/process-groups/8f5df7f8-01
>> 5a-1000-7fd2-ef2d19738677/remote-process-groups
>> > (source ip: 10.24.24.222)
>> > 2017-04-19 19:54:35,225 ERROR [NiFi Web Server-75]
>> > o.a.nifi.web.api.config.ThrowableMapper An unexpected

Re: Unable to access NIFI Rest api of another server in secured mode.

[Matt Gilman]

Nijandan,

I can't tell how you're creating your request based off your message. I
would say to make sure you are setting the Authorization as an HTTP header
and not part of the body of the request.

Matt

On Tue, Apr 4, 2017 at 3:11 AM, Nijandan D  wrote:

> Hi,
>
> I am trying to access NIFI rest api of another server from my machine
> using ajax like below,
>
> url:"https://serverip:port/api/";,
> {Authorization : 'Bearer ' + 'access token here'}
>
> But unable to access Nifi rest api of another server in secured mode. Any
> help appreciated.
>
>
> Thanks,
>
> Nijandan D
>

Re: REST API queries

[Matt Gilman]

Pushkar,

NiFi employs an optimistic locking scheme that requires clients to pass in
a revision when making a mutable request. The revision is comprised of a
version (a number that increments with each modification) and a client id.
The client id can be any string. You should continue to use this client id
with all subsequent requests. In order for a mutable request to be
accepted, a client must either have the current version or the client id
that last modified the flow. The second check allows clients to submit
requests asynchronous without having to wait for preceding requests to
complete. There is a good sequence diagram showing this at the bottom of
this article [1].

Each component will have its own revision. You should be able to obtain the
current revision from /connections/{id} or when obtaining the flow for a
given Process Group (/flow/process-groups/{id}).

Matt

[1]
https://community.hortonworks.com/content/kbentry/3160/update-nifi-flow-on-the-fly-via-api.html

On Fri, Mar 24, 2017 at 1:11 PM, Pushkara R  wrote:

> Hi,
>
> I am writing a client for NiFi in python that will be used as a library to
> dynamically change the processor arrangement and attributes.
>
> I have a question regarding the DELETE method for the /connections/{id}
> resource. The method expects query string fields for version and clientid.
> Could somebody elaborate on what these fields mean and where can my client
> retrieve them before doing a DELETE method call?
>
> Thanks
> Pushkar
>

Re: How to access Controller Service created in UI into Root processors in NiFi-1.1.1?

[Matt Gilman]

Prabhu,

Controller Service scoping/availability is unfortunately not as clear as it
needs to be. There have been some recent improvements to help with this
that will be part of an upcoming release. Those improvements are detailed
here [1].

Matt

[1]
https://community.hortonworks.com/articles/90259/understanding-controller-service-availability-in-a.html

On Thu, Mar 23, 2017 at 4:13 AM, Pierre Villard  wrote:

> Hi,
>
> I believe you created your controller service by going into "controller
> settings" in the main menu. This should not be done this way as controller
> services created here are only to be used by reporting tasks. To create
> controller services for processors, you need to create it at Process Group
> level using the operate palette and clicking on the configuration button,
> then you can go in the "controller services" tab. To access the root
> process group, you need to be at the root level and to not select anything
> on the canvas, this way the operate panel will focus on the root process
> group.
>
> Hope this helps,
> Pierre.
>
> 2017-03-23 5:39 GMT+01:00 prabhu Mahendran :
>
>> Hi Andy,
>>
>> Since my query is 'i have created controller service in NiFi UI'.I just
>> drag the putSQL which needs that controller service in which created.
>>
>> You said anonymous in the sense it could be accessible for every
>> processors in root right?
>>
>> But in JDBC Connection pool of PutSQL doesn't shown the created
>> controller service.
>>
>> So how to access Controller service created in root for all processors
>> which is in root canvas?
>>
>> On Thu, Mar 23, 2017 at 10:03 AM, Andy LoPresto <
>> alopresto.apa...@gmail.com> wrote:
>>
>>> If you are not using a secure instance of NiFi, the access policies and
>>> permissions are not available, as all users are unauthenticated and assumed
>>> to be anonymous.
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> alopresto.apa...@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Mar 22, 2017, at 20:52, prabhu Mahendran 
>>> wrote:
>>>
>>> No i am not running secure NiFi instance.Only viewed the url 
>>> (http://:/nifi)
>>> in windows 8.1
>>>
>>> On Wed, Mar 22, 2017 at 8:43 PM, Bryan Bende  wrote:
>>>
>>>> Are you running a secure NiFi instance with certificates over https?
>>>>
>>>> On Wed, Mar 22, 2017 at 11:02 AM, prabhu Mahendran <
>>>> prabhuu161...@gmail.com> wrote:
>>>>
>>>>> I am always use google chrome in windows to view the Nifi UI.
>>>>>
>>>>> And i cannot able to see the access policies in chrome also.
>>>>> On 22-Mar-2017 5:56 PM, "Matt Gilman"  wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What browser are you using? We had some issues the action icons in
>>>>>> Firefox in NiFi 1.1.0 wrapping to the next line that is addressed in this
>>>>>> JIRA [1]. If you are using Firefox, can you try Chrome to help narrow 
>>>>>> down
>>>>>> the issue? Does the policy links show up in the Controller Services tab?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/NIFI-3167
>>>>>>
>>>>>> On Wed, Mar 22, 2017 at 2:16 AM, prabhu Mahendran <
>>>>>> prabhuu161...@gmail.com> wrote:
>>>>>>
>>>>>>> I haven't see the access policies in "Reporting Tasks".Look at below
>>>>>>> image.
>>>>>>>
>>>>>>> ​
>>>>>>> ​But in your referral link i can see access policies in controller
>>>>>>> services tab.https://nifi.apache.org/do
>>>>>>> cs/nifi-docs/html/user-guide.html#Controller_Services_for_Re
>>>>>>> porting_Tasks
>>>>>>>
>>>>>>> On Sat, Mar 18, 2017 at 12:08 AM, Bryan Bende 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I responded on stackoverflow:
>>>>>>>>
>>>>>>>> https://stackoverflow.com/questions/42853055/how-to-access-c
>>>>>>>> ontroller-service-creat

Re: Admin policies not set by default

[Matt Gilman]

There is no option to add users with full privilege. Setting the initial
admin identity will give that user access to the UI and permissions to
manage users, groups, and policies. Additionally, if the NiFi instance has
an existing flow it will grant permissions to the root group. Otherwise,
the admin will need to grant those permissions explicitly.

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup

Matt

On Thu, Mar 23, 2017 at 3:24 AM, kumar r  wrote:

> Yes i have set Initial Admin Identity before first time itself. Is this a
> behavior or issue? I doubt that missed anything.
>
> On Thu, Mar 23, 2017 at 12:32 PM, Andy LoPresto <
> alopresto.apa...@gmail.com> wrote:
>
>> Did you add that user as the Initial Admin Identity in authorizers.xml
>> before the first time you tried to access the UI? If so, please file a Jira
>> issue, as all default permissions should be applied to that user.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Mar 22, 2017, at 23:54, kumar r  wrote:
>>
>> Hi,
>>
>> I have configured NiFi-1.1.1 with authentication. When accessing NiFi web
>> UI with admin user, i can't able to do some operations and it shows like
>> "user not allowed to perform this operation".
>>
>> When i go to policies and set policy for admin user, everything works
>> fine.
>>
>> For example, Admin couldn't view system diagnostic without policy
>> settings.
>>
>> By default, admin should have full privilege to access NiFi right? But
>> why should i add policies to admin manually.
>>
>> Is there any option to add users with full privilege?
>>
>> Thanks in advance.
>>
>>
>

Re: How to access Controller Service created in UI into Root processors in NiFi-1.1.1?

[Matt Gilman]

Hi,

What browser are you using? We had some issues the action icons in Firefox
in NiFi 1.1.0 wrapping to the next line that is addressed in this JIRA [1].
If you are using Firefox, can you try Chrome to help narrow down the issue?
Does the policy links show up in the Controller Services tab?

Thanks

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3167

On Wed, Mar 22, 2017 at 2:16 AM, prabhu Mahendran 
wrote:

> I haven't see the access policies in "Reporting Tasks".Look at below image.
>
> ​
> ​But in your referral link i can see access policies in controller
> services tab.https://nifi.apache.org/docs/nifi-docs/html/user-
> guide.html#Controller_Services_for_Reporting_Tasks
>
> On Sat, Mar 18, 2017 at 12:08 AM, Bryan Bende  wrote:
>
>> Hello,
>>
>> I responded on stackoverflow:
>>
>> https://stackoverflow.com/questions/42853055/how-to-access-
>> controller-service-created-in-ui-into-root-processors-in-nifi-1-1
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Fri, Mar 17, 2017 at 5:09 AM, prabhu Mahendran
>>  wrote:
>> >
>> > Nifi-1.1.0:
>> >
>> > In nifi-1.1.0 i have attached created controller services in
>> (Image1.png)
>> > Root canvas.
>> >
>> > After creating Controller services i have drag and drop the putSQL in
>> > UI.(Image2.png)
>> >
>> > In JDBC Connection Pool,it can show created controllerservice.
>> >
>> >
>> > But in NiFi-1.1.1 i do same procedure but ConnectionPool doesn't shown.
>> >
>> > Nifi-1.1.1:
>> >
>> > 1.i have created controller service attached image (Image3.png)
>> >
>> > 2.Drag the PutSQL in Root canvas.It couldn't show already created
>> service.
>> > (Image4.png)
>> >
>> > Note: I have attached all images in zip file.
>> >
>> > i don't know exactly is this correct behaviour for NiFi-1.1.1?
>> >
>> > How can i access the created controller service in Root into Root
>> > Processors?
>> >
>> > can anyone explain concepts behind controller service creation in
>> Nifi-1.1.1
>> > root canvas?
>>
>
>

Re: How to pass credential to NiFi rest api

[Matt Gilman]

Kumar,

Check out my response on Stack Overflow [1].

Thanks

Matt

[1]
http://stackoverflow.com/questions/42827820/how-to-pass-credential-to-nifi-rest-api

On Thu, Mar 16, 2017 at 3:33 AM, kumar r  wrote:

> Hi,
>
> I have enabled LDAP authentication for Apache NiFi-1.1.1.
>
> I can able to access NiFi web ui after logged in with ldap user.
>
> When i try to access REST API with basic authentication
> /process-groups/root, it shows
>
> Unable to perform the desired action due to insufficient permissions.
> Contact the system administrator.
>
> How to pass credential to NiFi REST API? How to access NiFi REST API with
> token?
>
> Thanks in advance.
>

Re: Nifi 1.1.2 - Ldap user group

[Matt Gilman]

Chakri,

Unfortunately, LDAP user groups are not currently supported. Currently,
user authentication through LDAP is supported but loading/interpreting
their groups are not.

Adding the concept of a group provider which would complement the existing
identity provider seems like a good fit. Also, introducing an LDAP
implementation of the user and group portion of the authorizer would allow
a user to define policies with the existing user/groups from LDAP.

I'll type something up and add a feature proposal to the wiki for this.

Thanks

Matt

On Wed, Mar 15, 2017 at 7:32 PM, Chakrader Dewaragatla <
chakrader.dewaraga...@lifelock.com> wrote:

> Hello All,
>
>
>
> Does Nifi 1.1.2 support ldap user groups? I was looking into the nifi
> documentation and it do not have any information how to setup. Let us know
> if anyone has already configured?
>
>
>
> Thanks,
>
> -Chakri
> --
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> --
>

Re: Process Groups and StandardClassificationService

[Matt Gilman]

In general, if the services share a common configuration and are accessible
to user's configuring the components in question they can be shared. In 1.x
we support Process Group scoped services because when the usage becomes so
broad users may not be permitted to modify all referencing components. This
flexibility supports the 'global' case where services are accessible to all
components on the canvas, and the fine-grained case where services are
defined within nested Process Groups.

Matt

On Mon, Mar 13, 2017 at 9:21 AM, James McMahon  wrote:

> Thank you Matt. Can you tell me this: what is considered best practice for
> StandardClassificationService, StandardHttpContextMap, and
> StandardSSLContextService in different process groups? Should a new
> instance of each be created that is specific to each and every process
> group? Or are common instances of one or more of these services typically
> shared across process groups?
>
> Regards,
> Jim
>
> On Mon, Mar 13, 2017 at 9:03 AM, Matt Gilman 
> wrote:
>
>> Jim, Joe,
>>
>> Just wanted to clarify. It sounds as though your running the 0.x
>> baseline. In 0.x, a single revision is maintained for the entire flow. If
>> another user makes a change, you will get a notification that you need to
>> Refresh to realize their changes before you can make any changes.
>>
>> In 1.x the Refresh link your referring to has been removed. With a focus
>> on multi-tenant usage, each component now maintains it's own revision.
>> Additionally, as other users make changes, your UI will update and realize
>> those changes automatically.
>>
>> Matt
>>
>> On Mon, Mar 13, 2017 at 8:04 AM, James McMahon 
>> wrote:
>>
>>> Joe what is considered best practice for StandardClassificationService,
>>> StandardHttpContextMap, and StandardSSLContextService in different process
>>> groups? Should a new instance of each be created that is specific for each
>>> and every process group? Or are common instances of one or more of these
>>> services typically shared across process groups?
>>>
>>> Regards,
>>> Jim
>>>
>>> On Mon, Mar 13, 2017 at 7:47 AM, Joe Skora  wrote:
>>>
>>>> James,
>>>>
>>>> I believe that even with multi-tenant isolation the graph is managed as
>>>> a whole so any change bumps the version for the whole thing.
>>>>
>>>> Regards,
>>>> Joe
>>>>
>>>> On Mon, Mar 13, 2017 at 7:00 AM, James McMahon 
>>>> wrote:
>>>>
>>>>> Good morning. My co-worker and I have independent Process Groups, PGA
>>>>> and PGB. Both exist at the workflow level immediately under NiFi Flow. 
>>>>> Each
>>>>> process group does indeed appear to have its own unique identifier. 
>>>>> However
>>>>> as we each run our workflows we find that we are periodically being told 
>>>>> by
>>>>> NiFi to refresh because the other has made updates. I noticed that we have
>>>>> each defined instances of StandardClassificationService. Could that be the
>>>>> reason we were getting these refresh messages?
>>>>>
>>>>> In general how should StandardClassificationService,
>>>>> StandardHttpContextMap, and StandardSSLContextService be instantiated for
>>>>> process groups? Should there only be one of each service and mapfor NiFi
>>>>> Flow, shared by all process groups and processors under NiFi flow? Or
>>>>> should each process group have its own instance, and - if so - why are we
>>>>> experiencing these refresh messages that appear to be driven by changes in
>>>>> the other independent process group?
>>>>>
>>>>> Thank you in advance for your help. -Jim
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Process Groups and StandardClassificationService

[Matt Gilman]

Jim, Joe,

Just wanted to clarify. It sounds as though your running the 0.x baseline.
In 0.x, a single revision is maintained for the entire flow. If another
user makes a change, you will get a notification that you need to Refresh
to realize their changes before you can make any changes.

In 1.x the Refresh link your referring to has been removed. With a focus on
multi-tenant usage, each component now maintains it's own revision.
Additionally, as other users make changes, your UI will update and realize
those changes automatically.

Matt

On Mon, Mar 13, 2017 at 8:04 AM, James McMahon  wrote:

> Joe what is considered best practice for StandardClassificationService,
> StandardHttpContextMap, and StandardSSLContextService in different process
> groups? Should a new instance of each be created that is specific for each
> and every process group? Or are common instances of one or more of these
> services typically shared across process groups?
>
> Regards,
> Jim
>
> On Mon, Mar 13, 2017 at 7:47 AM, Joe Skora  wrote:
>
>> James,
>>
>> I believe that even with multi-tenant isolation the graph is managed as a
>> whole so any change bumps the version for the whole thing.
>>
>> Regards,
>> Joe
>>
>> On Mon, Mar 13, 2017 at 7:00 AM, James McMahon 
>> wrote:
>>
>>> Good morning. My co-worker and I have independent Process Groups, PGA
>>> and PGB. Both exist at the workflow level immediately under NiFi Flow. Each
>>> process group does indeed appear to have its own unique identifier. However
>>> as we each run our workflows we find that we are periodically being told by
>>> NiFi to refresh because the other has made updates. I noticed that we have
>>> each defined instances of StandardClassificationService. Could that be the
>>> reason we were getting these refresh messages?
>>>
>>> In general how should StandardClassificationService,
>>> StandardHttpContextMap, and StandardSSLContextService be instantiated for
>>> process groups? Should there only be one of each service and mapfor NiFi
>>> Flow, shared by all process groups and processors under NiFi flow? Or
>>> should each process group have its own instance, and - if so - why are we
>>> experiencing these refresh messages that appear to be driven by changes in
>>> the other independent process group?
>>>
>>> Thank you in advance for your help. -Jim
>>>
>>
>>
>

Re: Kerberos authentication

[Matt Gilman]

Without knowing more of your setup, I would guess that your browser is 
prompting you based on the available certificates. What happens if you 
uninstall the certificate from your browser?

Matt

Sent from my iPhone

> On Mar 9, 2017, at 7:37 PM, Joe Skora  wrote:
> 
> I have an instance running Kerberos authentication that still asks for client 
> certificates.
> 
> The authentication seems to work ok, but the certificate issue is quite 
> annoying.  What will fix this?
> 
> Thanks,
> Joe

Re: Move flow into process group

[Matt Gilman]

BH,

You should be able to select the components in question and dragging and
dropping them into the desired Process Group. You can select the components
by holding down Shift and clicking on each one. Another option is to use a
selection box by clicking and dragging on the canvas. You should see a gray
dotted box drawn. Once you let go, it will automatically select all
components within your box. At that point, you should be able to click on
one of the selected components and drag them all into a Process Group. As
you hover over the Process Group you should see a blue border indicating a
valid drop target.

Let me know if you have any other questions.

Matt

On Mon, Feb 6, 2017 at 12:08 PM, Banias H  wrote:

> Hi,
>
> I am still fairly new to Nifi and as my Nifi flow grows, I would like to
> create process groups and place some of the flows into them.
>
> Is it possible to move flow into process group without losing its state?
>
> My flow has gone through so many files already. If I save the flow as
> template and deploy the flow inside the process group, it will start from
> the beginning. I would appreciate if there is a good way to move the flow
> and its state into a process group.
>
> Thanks,
> BH
>

Re: LDAP Group Authorization

[Matt Gilman]

Just wanted to add to Bryan's response that there is a JIRA [1] that
captures the ability to synchronize LDAP user/groups. This is probably a
good place to capture any additional use cases.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3409

On Sun, Jan 29, 2017 at 6:25 PM, Bryan Bende  wrote:

> Hi Tim,
>
> Currently NiFi only supports LDAP for authentication (i.e. login identity
> provider). In this case it is only using the LDAP to determine if the
> provided username and password are valid.
>
> For authorization there are currently two options - NiFi's policy-based
> authorizer and the Apache Ranger authorizer. With NiFi's policy-based
> authorizer, all of the users and groups have to be defined in NiFi, and
> then policies are defined for those users and groups. All of this is stored
> in the users.xml and authorizations.xml you referenced.
>
> I think there are two different improvements that might help with what you
> are looking for...
>
> The first would be a utility that could sync users/groups from LDAP into
> NiFi's policy-based authorizer. In this case you would still define all the
> policies in NiFi, but you wouldn't have to manually redefine the users and
> groups. You could probably create a simple tool that read from your LDAP
> and pushed users/group into NiFi using NiFi's REST API. Of course it gets a
> little tricky to keep this in sync when users are deleted, or their groups
> are changed, but it could be done.
>
> The second would be an Authorizer implementation that used LDAP. The
> Authorizer interface is an extension point and you could create a custom
> NAR with an authorizer implementation and deploy it to the lib directory.
> Its just that no one has gotten around to creating one that used LDAP yet.
> Either of the existing authorizers would be good examples to look at.
>
> Thanks,
>
> Bryan
>
> On Sun, Jan 29, 2017 at 4:56 PM, Timothy Findlay <
> timothyfind...@nbnco.com.au> wrote:
>
>> Hello Nifi Users,
>>
>>
>>
>> I have been getting Nifi up and running with LDAP, and this seems to work
>> for authentication, however it looks like I have to do the authorization
>> part manually.
>>
>>
>>
>> So far, I have made the “login-identity-providers.xml” and set:
>>
>> nifi.login.identity.provider.configuration.file=./conf/login
>> -identity-providers.xml
>>
>> nifi.security.user.login.identity.provider=ldap-provider
>>
>>
>>
>> I know I can configure my admin user/group in “users.xml” and the
>> authorizations in “authorizations.xml”.
>>
>>
>>
>> But I cant work out how to configure it to use LDAP groups. I suspect I
>> need to put something in authorizers.xml, but cant find much documentation
>> on this. Any tips/examples of how I can map the groups ?
>>
>>
>>
>> Thanks,
>>
>> Tim.
>>
>>
>>
>
>

Re: NiFi 1.1.1 behind Apache HTTPd Proxy

[Matt Gilman]

To interact with a secured REST API, either need to use client certificates
with mutual authentication or use a JWT token. If you log in via LDAP,
Kerberos, or Kerberos SPNEGO then a successful authentication will result
in a JWT token that can be used for subsequent requests.

I believe you have a couple options...

Use a server certificate with HTTPd and authorize it as a proxy in NiFi.
Additionally, you'll need to relay the end user identity in the
X-ProxiedEntitiesChain HTTP header.

Or...

Relay the Authorization HTTP header which the end user sent to the proxy.

Matt

On Mon, Jan 16, 2017 at 1:04 PM, Wayna Runa  wrote:

> Thanks Matt for your support.
> Yes, an unique context path for NiFi makes sense. I will work on that
> after solving the Proxy config.
>
> I don't have configured JWT in Proxy or NiFi. I don't understand why NiFi
> is asking for JWT.
> Please, Could anyone share your Apache HTTPd Proxy configuration ?
>
> Regards.
>
> - wr
>
>
>
> On 16 January 2017 at 14:38, Matt Gilman  wrote:
>
>> A couple comments. Apache NiFi is comprised of a number of web
>> applications. You've mapped to three of them. However, we also support UI
>> extensions for Custom Processor configuration (like UpdateAttribute and
>> JoltTransformJson) and custom data viewers (based on content type) when
>> looking at data through queues and data provenance.
>>
>> I would suggest mapping a single Location that does not point to any one
>> specific context path. Otherwise, a lot of maintenance would be required to
>> keep your Location's up to date.
>>
>> I've never tried using token based access from behind a proxy. Can you
>> ensure that Bearer token is being included by ensuring the Authorization
>> header is passed along?
>>
>> Matt
>>
>>

Re: NiFi 1.1.1 behind Apache HTTPd Proxy

[Matt Gilman]

A couple comments. Apache NiFi is comprised of a number of web
applications. You've mapped to three of them. However, we also support UI
extensions for Custom Processor configuration (like UpdateAttribute and
JoltTransformJson) and custom data viewers (based on content type) when
looking at data through queues and data provenance.

I would suggest mapping a single Location that does not point to any one
specific context path. Otherwise, a lot of maintenance would be required to
keep your Location's up to date.

I've never tried using token based access from behind a proxy. Can you
ensure that Bearer token is being included by ensuring the Authorization
header is passed along?

Matt

On Mon, Jan 16, 2017 at 8:41 AM, Wayna Runa  wrote:

> Hi there!
>
> I've configured Nifi 1.1.1 *Standalone* with Kerberos (FreeIPA) to do
> AuthN and AuthZ. Seems everything is OK because:
>
> 1.- Initial admin login through Kerberos works.
> 2.- SSL enabled: Nifi asks for User Cert, just cancel and browser is
> redirected to Nifi Login Page where I can introduce a Kerberos user. If I
> choose a valid User Cert, browser is redirected to a Nifi authorized canvas.
> 3.- I can add more users (by using Kerberos configuration) through Nifi UI
>
> Now, I have installed a Apache HTTPd Proxy in front of Nifi by using this
> config:
>
> Listen 443 https
> 
> 
> ...
> 
> Header always unset Strict-Transport-Security
> RequestHeader add X-ProxyScheme "https"
> RequestHeader add X-ProxyHost "my-proxy"
> RequestHeader add X-ProxyPort "443"
> RequestHeader add X-ProxyContextPath "/nifi"
> SSLRequireSSL
> SSLRequire %{SSL_CLIENT_S_DN_O} eq "FooBar"
> ProxyPass https://my-nifi-standalone:8443/nifi
> ProxyPassReverse https://my-nifi-standalone:8443/nifi
> 
> deny from all
> 
> 
>
> 
> Header always unset Strict-Transport-Security
> RequestHeader add X-ProxyScheme "https"
> RequestHeader add X-ProxyHost "my-proxy"
> RequestHeader add X-ProxyPort "443"
> RequestHeader add X-ProxyContextPath "/nifi-api"
> SSLRequireSSL
> SSLRequire %{SSL_CLIENT_S_DN_O} eq "FooBar"
> ProxyPass https://my-nifi-standalone:8443/nifi-api
> ProxyPassReverse https://my-nifi-standalone:8443/nifi-api
> 
> deny from all
> 
> 
>
> 
> Header always unset Strict-Transport-Security
> RequestHeader add X-ProxyScheme "https"
> RequestHeader add X-ProxyHost "my-proxy"
> RequestHeader add X-ProxyPort "443"
> RequestHeader add X-ProxyContextPath "/nifi-docs"
> SSLRequireSSL
> SSLRequire %{SSL_CLIENT_S_DN_O} eq "FooBar"
> ProxyPass https://my-nifi-standalone:8443/nifi-docs
> ProxyPassReverse https://my-nifi-standalone:8443/nifi-docs
> 
> deny from all
> 
> 
> 
>
> Then, when go to https://my-proxy-nifi/nifi and choose the same valid
> User Cert, I get a Nifi error page with this message:
>
> Unable to check Access Status
> Unable to validate the access token
>
>
> In Nifi server (nifi-user.log) error log is:
>
> INFO [NiFi Web Server-17] o.a.n.w.a.c.IllegalStateExceptionMapper
> java.lang.IllegalStateException: Kerberos ticket login not supported by
> this NiFi.. Returning Conflict response.
> INFO [NiFi Web Server-17] o.a.n.w.a.c.AccessDeniedExceptionMapper
> anonymous does not have permission to access the requested resource.
> Returning Unauthorized response.
> INFO [NiFi Web Server-17] o.a.nifi.web.security.jwt.JwtService There was
> an error validating the JWT
> io.jasonwebtoken.JwtException: Unable to validate the access token.
> 
> Caused by: io.jasonwebtoken.MalformedJwtException: JWT strings must
> contain exactly 2 period characters. Found: 0
> 
>
>
> And in the Browser I can see this error through Firefox Developer Plugin:
>
> GET https://my-proxy-nifi/nifi-api/flow/current-user
> Status code: 401 Unauthorized
>
>
> I've used this thread to configure the Apache HTTPd Proxy.
> https://mail-archives.apache.org/mod_mbox/nifi-dev/201509.mbox/%
> 3CCAFddr26dJFm4droVjVPOm-swUn0aR-=_vpqpxoixa1mo5mj...@mail.gmail.com%3E
>
> Any help is welcome !!.
>
> Regards.
>
> - wr
>
>
>

Re: How to use Rest API ressource data-transfer?

[Matt Gilman]

I'm not super familiar with this endpoint so hopefully someone else can
chime in here if necessary. But I believe the transactionId is already
composed in the Location header of this endpoint

POST
/data-transfer/input-ports/{portId}/transactions

There is a Java class [1] available in NiFi commons site to site client
that can be used to interact with these endpoints. Hopefully, this can be
helpful.

Matt

[1]
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/util/SiteToSiteRestApiClient.java

On Mon, Jan 16, 2017 at 8:35 AM, iboumedien  wrote:

> Hi guys,
>
> I want to use the REST API of Nifi in particularly this one:
>
> POST
> /data-transfer/input-ports/{portId}/transactions/{
> transactionId}/flow-files
>
> But I'm not to found how I ca get the transactionId?
>
> Can anyone help me?
>
> Best regards
>
> Ismael
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.
> 2361937.n4.nabble.com/How-to-use-Rest-API-ressource-data-
> transfer-tp666.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>

Re: Migrating NiFi templates/canvas

[Matt Gilman]

Panos,

I'm not sure what version of NiFi you're using but in 1.1.1 another issue
[1] was addressed that may be what you're seeing now. If you're still on
1.0.0 or 1.1.0 maybe try upgrading to 1.1.1.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3173

On Fri, Jan 13, 2017 at 1:51 PM, Panos Geo 
wrote:

> Hello Matt,
>
>
> Thanks for your reply.
>
>
> I fully get that, but my requirement is in addition to this. The
> (top) controller service is being referenced by multiple processors in my
> template, which however belong in different sub process groups. When I
> import this template in another NiFi instance this controller service is
> present, but it is being "split" to different process group controller
> services, which I cannot configure centrally (as I would have done if that
> was a top level controller service as it was in the original NiFi instance
> before exporting the template).
>
>
> Many thanks,
>
> Panos
>
>
> --
> *From:* Matt Gilman 
> *Sent:* Friday, January 13, 2017 6:27 PM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: Migrating NiFi templates/canvas
>
> Panos,
>
> The current behavior is that Controller Services are only included if they
> are referenced by a component in the template. While this was initially by
> designed, we have received comments about it from folks looking to export
> their entire configuration. There is an outstanding JIRA [1] to update this
> behavior.
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-2895
>
> On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo 
> wrote:
>
>> Having just checked the idea in my PS, it doesn't seem to work as the
>> transferred controller services are at a process group level and hence are "
>> compartmentalised" after the migration...
>>
>>
>> Any thoughts on this would be highly appreciated.
>>
>>
>> Many thanks,
>>
>> Panos
>>
>>
>> --
>> *From:* Panos Geo 
>> *Sent:* Friday, January 13, 2017 4:47 PM
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Migrating NiFi templates/canvas
>>
>>
>> Hello Oleg,
>>
>>
>> Many thanks for your reply!
>>
>>
>> I remembered having diff'ed NiFi templates in the past and that was a
>> bit messy. However, this was certainly before NiFi 1.0, so from a quick
>> look it appears that things are better now in that regard. Thanks for
>> the hint!
>>
>>
>> However, the situation with moving the controller services associated
>> with a processor seems still problematic in my humble opinion. Let's say
>> that in my source VM, I have created a Database and an HTTPContext map
>> service at the *top level* of my canvas. These are being used inside
>> many different processors that belong to different process groups. Now,
>> if I export my canvas as a template and import it on another VM, I don't
>> see these services at the *top level* of my canvas on my target VM. The
>> services are being "compartmentalised" into the process groups and sub
>> process groups of my template. Therefore, if I have to do a change (say
>> the port of the database is different) I need to go through all of them
>> manually as a change in a "compartmentalised" service is not reflected
>> on another process group.
>>
>>
>> An alternative, which I have been using, is to recreate the service at
>> the top level of the canvas on the target VM, but this means that I need to
>> go through all the processors of the template I moved and associate them
>> with the new top level service of my target VM, which is very error-prone
>> in an CI/CD pipeline.
>>
>>
>> Isn't there an easier way to migrate templates with their associated
>> services?
>>
>>
>> Many thanks again,
>>
>> Panos
>>
>>
>>
>> Ps. writing all this, it came to  mind that I could include all my canvas
>> in one process group in my source VM, generate my service inside that and
>> then move all of that in my target VM. Not ideal, but should work [image:
>> 😉]
>>
>>
>> --
>> *From:* Oleg Zhurakousky 
>> *Sent:* Friday, January 13, 2017 1:18 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: Migrating NiFi templates/canvas
>>
>> Panos
>>
>> What version of NiFi are you using?
>> The issue with the NiFi templates diffs has actually been addressed (see
>> https://issues.apache.org/jira/browse/NI

Re: Migrating NiFi templates/canvas

[Matt Gilman]

Panos,

The current behavior is that Controller Services are only included if they
are referenced by a component in the template. While this was initially by
designed, we have received comments about it from folks looking to export
their entire configuration. There is an outstanding JIRA [1] to update this
behavior.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-2895

On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo 
wrote:

> Having just checked the idea in my PS, it doesn't seem to work as the
> transferred controller services are at a process group level and hence are "
> compartmentalised" after the migration...
>
>
> Any thoughts on this would be highly appreciated.
>
>
> Many thanks,
>
> Panos
>
>
> --
> *From:* Panos Geo 
> *Sent:* Friday, January 13, 2017 4:47 PM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: Migrating NiFi templates/canvas
>
>
> Hello Oleg,
>
>
> Many thanks for your reply!
>
>
> I remembered having diff'ed NiFi templates in the past and that was a bit
> messy. However, this was certainly before NiFi 1.0, so from a quick look
> it appears that things are better now in that regard. Thanks for the hint!
>
>
> However, the situation with moving the controller services associated with
> a processor seems still problematic in my humble opinion. Let's say that
> in my source VM, I have created a Database and an HTTPContext map service
> at the *top level* of my canvas. These are being used inside many
> different processors that belong to different process groups. Now, if I
> export my canvas as a template and import it on another VM, I don't see
> these services at the *top level* of my canvas on my target VM. The
> services are being "compartmentalised" into the process groups and sub
> process groups of my template. Therefore, if I have to do a change (say
> the port of the database is different) I need to go through all of them
> manually as a change in a "compartmentalised" service is not reflected on
> another process group.
>
>
> An alternative, which I have been using, is to recreate the service at
> the top level of the canvas on the target VM, but this means that I need to
> go through all the processors of the template I moved and associate them
> with the new top level service of my target VM, which is very error-prone
> in an CI/CD pipeline.
>
>
> Isn't there an easier way to migrate templates with their associated
> services?
>
>
> Many thanks again,
>
> Panos
>
>
>
> Ps. writing all this, it came to  mind that I could include all my canvas
> in one process group in my source VM, generate my service inside that and
> then move all of that in my target VM. Not ideal, but should work [image:
> 😉]
>
>
> --
> *From:* Oleg Zhurakousky 
> *Sent:* Friday, January 13, 2017 1:18 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Migrating NiFi templates/canvas
>
> Panos
>
> What version of NiFi are you using?
> The issue with the NiFi templates diffs has actually been addressed (see
> https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you
> can export templates in a deterministic way and then execute diff on them
> to see only what was changed. It was addressed specifically for the SDLC
> issue you are describing. Also, when processor is associated with
> controller service and you created a template that includes such processor,
> its dependent controller service will end up on the template and back in
> your environment once such template is imported.
> [NIFI-826] Export templates in a deterministic way - ASF JIRA
> 
> issues.apache.org
> Templates should be exported in a deterministic way so that they can be
> compared or diff'ed with another. Items to consider... The ordering of
> components The id's ...
>
>
> Anyway, give it a shot and let us know.
> Cheers
> Oleg
>
> On Jan 13, 2017, at 4:49 AM, Panos Geo  wrote:
>
> Hello all,
>
> We have a Continuous Integration/Continuous Development pipeline that
> involves for each stage (development, testing, customer deployment) a
> dedicated virtual machine running a NiFi instance. As such, for each stage
> of the pipeline we have to migrate the changes of our NiFi canvas from one
> virtual machine to another.
>
> For this we encounter two big problems :
>
>1. As far as I know, there is no easy way to diff NiFi canvases (or
>templates) so that we can recognize the changes and don’t have to
>export and import the full canvas each time (as there are configuration
>changes in the target VM that we wouldn’t like to overwrite).
>2. If we have to export the full canvas, is there a way to reassociate
>processors with the required services on the target virtual machine (e.g.
>DBCPConnectionPool or HTTPContextMap) without having to go through each
>processor explicitly and do the reassignment?
>
>
> Many thanks in advance,
> Panos
>
>
>

Re: Write-lock processors

[Matt Gilman]

Alessio,

There is currently no way to lock the canvas. If the current user has
permissions to modify a given component it will be modifiable if the
component is in a valid state for the request action. For instance, if the
component is scheduled to execute it cannot be deleted.

There has been some discussion in the past on the mailing list to support
this use case. I've just filed a JIRA to capture it [1].

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3342

On Fri, Jan 13, 2017 at 10:59 AM, Alessio Palma <
alessio.pa...@docomodigital.com> wrote:

> Ups! What a mistake! This is the correct text:
>
> Hello all,
> I wonder if there is a fast way to set processor to READ only in order to
> stick it on the desktop making it impossible  move or delete it unless
> unlocked.
>
> It's really easy to deleted/move/change them on the workbench and I see no
> undo option.
>
> Happy Weekend!
>
> --
> *From:* Alessio Palma 
> *Sent:* Friday, January 13, 2017 4:26:44 PM
> *To:* NIFI User mailing
> *Subject:* Write-lock processors
>
>
> Hello all,
> I wonder if there is a fast way to set processor to write only.
> It's easy to deleted/move/change them on the workbench and I see no undo
> option.
>
>
>

Re: nifi 1.1.1 processor details don't show from summary view

[Matt Gilman]

Naz,

Thanks for feedback. This issue has been captured in the following JIRA
[1]. I apologize for any inconvenience.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3306

On Thu, Jan 12, 2017 at 10:22 AM, Irizarry Jr., Nazario 
wrote:

> In NiFi 1.1.1 (and 1.1.0) if you pull up the summary view then click on
> the “i” to the left of a processor nothing shows.  At the bottom of the
> page there is a spinning icon that never stops spinning.  However, the
> processor details do work from the Data Provenance page.
>
> This works as expected in NiFi 1.0.
>
> Cheers,
>
> Naz Irizarry
> MITRE Corp.
> 617-893-0074
>
>
>
>

Re: nifi 1.1.1 : Unable to empty queues

[Matt Gilman]

There were many updates between 1.0.0 and 1.1.1. You can check out the
Release Notes page here [1]. From there you can get access to the listing
of issues addressed in each release.

If the source of the connection happens to be a Remote Process Group then
you likely hit the underlying issue addressed here [2].

Thanks

Matt

[1] https://cwiki.apache.org/confluence/display/NIFI/
Release+Notes#ReleaseNotes-Version1.1.1
[2] https://issues.apache.org/jira/browse/NIFI-3133

On Tue, Jan 10, 2017 at 11:12 AM, Alessio Palma <
alessio.pa...@docomodigital.com> wrote:

> Hello Gilman,
>
> I found a way to have it working. I give view data and modify data
> policies to my user too.
>
> Nifi 1.0.0  was working without this setting and using nifi-1.1.1 need it.
>
>
> What are the differences between 1.0.0 and 1.1.1 about this point ?
> ------
> *From:* Matt Gilman 
> *Sent:* Tuesday, January 10, 2017 3:43:46 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: nifi 1.1.1 : Unable to empty queues
>
> If your instance is clustered, you'll need to authorize the nodes with the
> data policies as well. Any 'data-plane' endpoint (where data or meta-data
> is returned or modified) will require the nodes in the request chain to
> also be approved for access as the data will traverse some of them when the
> request is replicated. 'Control-plane' endpoints (where data flow
> configuration is retrieved/modified) are not subject to the additional
> check.
>
> FYI, there was a recent improvement committed the master branch that
> provides some level of details regarding the insufficient privileges. This
> should make it into the next release (1.2.0).
>
> Matt
>
> On Tue, Jan 10, 2017 at 9:36 AM, Alessio Palma <
> alessio.pa...@docomodigital.com> wrote:
>
>> Hello all,
>> I'm checking nifi 1.1.1 before moving it to production and something
>> weird is happening when I try to empty a queue.
>> System replies telling I'm unable to perform the desired action due to
>> insufficient permissions. My user has any available rule.
>> What can I check ?
>> Any suggestions?
>>
>> Thanks in advance for your time.
>>
>>
>>
>>
>

Re: nifi 1.1.1 : Unable to empty queues

[Matt Gilman]

If your instance is clustered, you'll need to authorize the nodes with the
data policies as well. Any 'data-plane' endpoint (where data or meta-data
is returned or modified) will require the nodes in the request chain to
also be approved for access as the data will traverse some of them when the
request is replicated. 'Control-plane' endpoints (where data flow
configuration is retrieved/modified) are not subject to the additional
check.

FYI, there was a recent improvement committed the master branch that
provides some level of details regarding the insufficient privileges. This
should make it into the next release (1.2.0).

Matt

On Tue, Jan 10, 2017 at 9:36 AM, Alessio Palma <
alessio.pa...@docomodigital.com> wrote:

> Hello all,
> I'm checking nifi 1.1.1 before moving it to production and something weird
> is happening when I try to empty a queue.
> System replies telling I'm unable to perform the desired action due to
> insufficient permissions. My user has any available rule.
> What can I check ?
> Any suggestions?
>
> Thanks in advance for your time.
>
>
>
>