Re: How can NIFI dynamically update SAS Tokens for Azure Blob Storage processors.

[Kevin Doran]

I agree, the Controller Service needs to be enhanced to support rotating
SAS tokens. I know of a few users that have run into this exact problem.

Technically, it should possible to write a custom implementation of
the ADLSCredentialsService API that did this, but it would be ideal if NiFi
provided some capability out of the box.

On Mar 13, 2023 at 21:28:33, Adam Taft  wrote:

> Hi Mike,
>
> You're right, I don't think you're going to get anywhere with the
> flowfile attribute providing the value, as like you said,
> ListAzureBlobStorage is a source processor and doesn't accept any input
> flowfiles.
>
> So currently, you're really looking at two choices, which you've already
> identified:
> 1.  Manually rotate the SAS token through the user interface.
> 2.  Make an HTTP call to the NiFI API causing an update to the controller
> service SAS token property.
>
> Now, #2 is not quite as bad as you think it is. Luckily, NiFi will deal
> very nicely with the temporary restart of the backing controller service.
> NiFi handles shutting down all the processors that are connected to the
> controller service and then bringing them back up (from multiple API
> calls). It will queue up the flowfiles for the short time this operation
> occurs (but yes, it does need to stop those processors before any changes
> can be made). In theory, you shouldn't have any corruption issues going
> this path. The problem side of it will be in writing your script to connect
> with the API and issuing the request to make this change. A bit painful,
> but possible. Your script would need to locate or know how to find the
> correct controller service (typically via its uuid).
>
> All this really speaks to a feature request, to be honest. It would be
> ideal that the controller service can reconfigure itself by grabbing the
> new SAS token for your storage account from Azure. Presumably the best
> practice would be to pick up the new SAS token from Key Vault (since Key
> Vault can be configured to own SAS Token generation for the storage
> account)? I guess Azure AD / OAuth is also a possible (alternative)
> solution?
>
> Anyway, short term, consider the API approach. Long term, a contribution
> to the project along these lines would probably be gratefully accepted. At
> minimum, a Jira ticket to suggest the new feature would be a good starting
> place.
>
> /Adam
>
>
>
> On Wed, Mar 8, 2023 at 9:53 AM Mike Rutlin  wrote:
>
>>
>> I have a general question on how to dynamically update and sync SAS
>> tokens being utilized by the AzureBlogStorage processors (
>> List,Fetch,Delete, etc) .
>>
>> From our NIFI we are accessing a storage container in a different cloud
>> and thus cannot use AZURE MI as standard for storage containers in the same
>> cloud. Thus we need to use the SAS Tokens attribute / parameter supported
>> by the controller service or by the processors.
>>
>> We pull the SAS token on a periodic basis, before the old SAS token
>> expires.
>>
>> In manual testing of this we set a sensitive Parameter to the SAS token
>> and set up the Controller service SAS Token attribute / parameter to use
>> the Parameter we created.  Everything worked as it should.
>>
>> The problem/issue occurs when the SAS token expires and we pull the new
>> one. How do I modify the Parameter ? Using the NIFI API seems messy and
>> potential to corrupt the flow configuration. along with the fact that it
>> stops and starts service / processors with the Parameter we are updating.
>>
>> I would like to use a flowfile attribute ( which the documentation
>> suggests is possible ). However the ListAzureBlobStorage processor is a
>> start flow processor ( no inputs ). Thus I can't run a script to get the
>> new key and set an attribute that could then be used by the SAS Token
>> attribute / parameter in the processor.
>>
>> Any ideas on how I can do this? It seems this would be a common practice
>> to roll keys and then inject the new key into the data flow for either AWS
>> or Azure.
>>
>> If the ListAzureBlobStoreage had an input I coud update with new keys OR
>> if the list processor had a failure output I could update the key upon
>> expiration.
>>
>> Thanks for any and all guidance.
>>
>> Mike R
>>
>>

Re: Problems using registry configured with git

[Kevin Doran]

 Hi David,

Are you using nested versioned process groups, i.e., does the PG you
version control in Step 3 contain a nested PG that is independently
versioned?

If you configure the GitFlowPersisence provider for a brand new / empty
registry, it should pull the metadata for buckets from git on first start.

Hope this helps,
Kevin

On Feb 14, 2023 at 10:55:04, David Early via users 
wrote:

>
> Hi all,
>
> I accidentally submitted this with no subject a while back, so this is my
> redo.
>
> We are setting up a Nifi Registry (1.16.3 currently) that is tied to git.
>
> I get MOST of it working, but run into an issue when I actually try and
> use a registry entry.
>
> Steps:
>
> 1. Set up registry
> 2. Configure git
> 3. Start version control on a process group
> 4. Check git (it is entered and committed, as expected)
> 5. Try and drop the registry item onto the canvas and I get this:
>
> [image: image.png]
>
> In the registry:
> [image: image.png]
>
> When I had this configured to use the file based storage, it worked fine
> and I could install things from the registry.
>
> In addition, there is this log message:
>
> 2023-02-09 21:56:05,784 WARN [NiFi Registry Web Server-16]
> o.a.n.registry.service.RegistryService The specified bucket id
> [8ef77681-b71d-4687-b627-29e0738d979b] does not exist.
>
> The problem is that is an OLD bucket I reset the registry completely
> and even searched the flow.xml for the 2 systems trying to use the registry
> and I am STILL getting this error.  I even cleaned out the git repo.  I am
> using the same repo, but deleted everything and commited the delete before
> starting the registry up.
>
> I am a bit lostany thoughts/ideas?
>
> David
>
>

Re: Error reading remote resource when building NIFI from source

[Kevin Doran]

 Do you have any global maven settings that could be affecting this build,
i.e., a non-empty ~/.m2/settings.xml file?

On Feb 12, 2023 at 03:48:02, oyeyemi rafiu owolabi via users <
users@nifi.apache.org> wrote:

> Thank you for the reply Aldrin Piri.
> I have updated my maven as required.
> And i am getting another concerning the below screenshot.
>
>
>
>
>
>
> On Saturday, February 11, 2023 at 04:18:53 PM GMT+1, Aldrin Piri <
> ald...@apache.org> wrote:
>
>
> Your Maven version is below the required 3.8.6 and the recommended 3.9.0.
> Could you upgrade and give it another attempt?
>
> If that does not solve the problem, please provide more of the build log
> for review.
>
> On Sat, Feb 11, 2023 at 05:14 oyeyemi rafiu owolabi via users <
> users@nifi.apache.org> wrote:
>
> Hello,
>  Please, can someone help me with this error i encountered when building
> Nifi from source.
>
> Platform: WSL2 Ubuntu 22.04
> Java Version: Java version: 17.0.5
> Maven version: Apache Maven 3.8.5
>
> These are the procedure i followed to build Nifi.
> 1 git clone GitHub - apache/nifi: Apache NiFi
> 
>
> GitHub - apache/nifi: Apache NiFi
>
> Apache NiFi. Contribute to apache/nifi development by creating an account
> on GitHub.
> 
>
> 2 cd nifi
>
> 3 mvn -T 2.0C clean install -DskipTests
>
> The error i encountered:
> Thank you.
>
>

RE: [EXTERNAL] Re: Disabling flows - nifi registry

[Kevin Doran]

 No, in 1.20.0, component enabled/disabled state is still treated as a
change that affects versioning.

Do let us know if you continue to see performance issues for your flow on
recent versions of NiFi with all processors enabled. That is something that
should be addressed.

Regards,
Kevin

On Feb 9, 2023 at 21:38:46, "Chirthani, Deepak Reddy" <
c-deepakreddy.chirth...@charter.com> wrote:

> Kevin,
>
>
>
> Out of Curiosity, has this been taken care in Nifi 1.20.0 which was
> released today?
>
>
>
> Thanks
>
> Deepak
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> *From:* Chirthani, Deepak Reddy
> *Sent:* Tuesday, December 13, 2022 1:53 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry
>
>
>
> The version of Nifi where I had UI performance issues was 1.11.4
> The current version of our nifi instance is 1.15.3
>
>
> One other disadvantage of having Process Groups with Stopped components is
> that when a dataflow manager or a developer inadvertently right click and
> select “start” on the root canvas page, that will start all the components
> in the child process groups.
>
> Thanks
>
> Deepak
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> *From:* Kevin Doran 
> *Sent:* Tuesday, December 13, 2022 10:26 AM
> *To:* users@nifi.apache.org
> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Thanks for sharing Deepak. That's interesting. I think we may look into a
> way of disabling components that doesn't impact version control, or else
> fix performance issues such that disabling is not necessary.
>
>
>
> Just out of curiosity, what version of NiFi are you on? The reason I ask
> is that I know there have been some UI performance improvements made in
> recent versions of NiFi, so I'm wondering if some of this might already
> have been addressed...
>
>
>
> Thanks,
>
> Kevin
>
>
>
> On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" <
> c-deepakreddy.chirth...@charter.com> wrote:
>
> Hi Kevin,
>
> Thank you for your quick response. The only reason why I prefer processors
> to be disabled is to improve the NiFi UI performance by simply disabling
> stopped processors which I am not using.
>
>
> https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550
>
> I had issues with Nifi UI performance in the past where the refresh wheel
> keeps on spinning like for 10 seconds and disabling stopped processors
> really helped me.
>
> Thanks
> Deepak
>
>
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> *From:* Kevin Doran 
> *Sent:* Friday, December 9, 2022 9:33 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXTERNAL] Re: Disabling flows - nifi registry
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Hi Deepak,
>
>
>
> So far, we have been honoring the following policy for what constitutes a
> change in version control:
>
>
>
>1. stopped/started does not count as a "local change"
>2. enabled/disabled does count as a change, and that state is captured
>in the flow snapshot json version saved to registry.
>
>
>
> One reason for this is that some users want to setup CI/CD to deploy from
> Registry and automatically start a flow in the target NiFi. If there are
> components they don't want to start, the disabled state gives them a way to
> capture that configuration in the flow.
>
>
>
> Is there a reason you prefer disable over stop in your lower environment?
> I leave flows stopped in dev environments all the time and have never run
> into an issue, but of course, everyone's workflow and use case is slightly
> different, so I'm interested in hearing your perspective on this to see if
> we need to consider something more flexible.
>
>
>
> Cheers,
>
> Kevin
>
>
>
> On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" <
> c-deepakreddy.chirth...@charter.com> wrote:
>
> Hey 

Re: How to auto-create users based on OIDC group membership?

[Kevin Doran]

 Hi Dave,

I understand the type of auto-user registration for authenticated users
that you're describing, which a lot of OIDC-based web apps support.

Unfortunately, NiFi cannot support that at this time. It would be a cool
feature. It's not impossible that NiFi could support it one day. We would
have to enhance the NiFi Identity Provider, User Group Provider, and
Authorizer implementations for OIDC to work together to support this.

Currently, the closest you can get in NiFi is to use something like the
LdapUserGroupProvider to point at the same user directory as Keycloak, then
the users will be automatically synced from the same directory, including
their group membership for setting nifi-specific access at the group level.
But the OIDC identity would just be used for authentication, not group
mapping or access policy loading (that would still come from the NiFi User
Group Provider and Access Policy Provider).

Hope this helps.
Kevin

On Jan 12, 2023 at 19:04:49, David Dean via users 
wrote:

> Hi -
>
> Is it possible for NiFi to automatically grant user access to NiFi based
> on an OIDC authenticated users group membership matching a group in NiFi?
>
> I'm using the latest 1.19.1 with OIDC enabled and integrated with Keycloak.
>
> In Keycloak I have created a test user and assigned them to group "Test
> Group".
>
> In NiFi I have created a group called "Test Group" and granted it some
> policies.
>
> I have enabled the "nifi.security.user.oidc.claim.groups" config option
> to obtain the OIDC groups from Keycloak.
>
> If I pre-create a user account in NiFi and add them to "Test Group" then
> they can successfully login via OIDC and get the required policies.
>
> But what I want is to not have to pre-create the users.
>
> Instead I would like NiFi to evaluate an authenticated users OIDC group
> membership, and if a group name in OIDC matches one in NiFi then it should
> allow them access to NiFi using the policies assigned to the matching group
> in NiFi.
>
> Is this possible?
>
> Appreciate your help!
>
> Dave
>
>
>
>

Re: [EXTERNAL] Re: Disabling flows - nifi registry

[Kevin Doran]

 Yeah I agree that disabling/enabling a processor should count as a change
for the purpose of version control / nifi registry for the reasons Bryan
and Stephen mentioned.

I could see us introducing something like "pause tracking changes" which
would change the icon for disabled PGs to something that just indicates it
is disconnected from registry. And it would have to be "unpaused" prior to
committing a new version. Would that be useful in this case or in general?
Personally I don't mind having "local/dirty" state for a PG in a dev
environment as I can always check local changes to see what is included,
and IIRC, enabling a PG will remove the local changes icon.

On Dec 15, 2022 at 12:58:19, Bryan Bende  wrote:

> Yea, the other side of this is when someone has disabled a portion of
> their flow in a lower environment and committed to registry, and then
> imports to their higher environment and expects that part to remain
> disabled. Otherwise that change has to be redone in higher environment.
>
> On Thu, Dec 15, 2022 at 12:55 PM stephen.hindmarch.bt.com via users <
> users@nifi.apache.org> wrote:
>
>> Personally I am against it. I feel that it is more important to protect
>> production than development, and the operators running production will have
>> less time and possibly less skill to look in depth at what should and
>> should not be running, than the developers who are running development.
>>
>>
>>
>> My use case is that I want my operators to be able to press the run
>> button without worrying about which processors or ports might need to be
>> stopped. They should be able to observe the top level groups and see there
>> are no stopped processors.
>>
>>
>>
>> As a developer, if I think a processor should not be run in production,
>> then I will disable it before I commit it to the registry. These will
>> include test injectors and fix/feedback loops that would be used often in
>> development, but may also have a role in production during troubleshooting.
>> The trigger to version control when the enable/disable state of a processor
>> changes is an extra mechanism to warn if a processor has been used in
>> production and has been accidentally left enabled. I prefer to review all
>> changes in production before committing so that I am not accidentally
>> committing untested “workarounds” that got me out of trouble this morning
>> but would not like to persist in the code base.
>>
>>
>>
>> *Steve Hindmarch*
>>
>>
>>
>> *From:* Kevin Doran 
>> *Sent:* 15 December 2022 17:00
>> *To:* users@nifi.apache.org
>> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry
>>
>>
>>
>> Thanks, Deepak. I'll have to think about this one. I'm not sure if anyone
>> else has any thoughts on this workflow / use case?
>>
>>
>>
>> On Dec 13, 2022 at 14:52:31, "Chirthani, Deepak Reddy" <
>> c-deepakreddy.chirth...@charter.com> wrote:
>>
>> The version of Nifi where I had UI performance issues was 1.11.4
>> The current version of our nifi instance is 1.15.3
>>
>>
>> One other disadvantage of having Process Groups with Stopped components
>> is that when a dataflow manager or a developer inadvertently right click
>> and select “start” on the root canvas page, that will start all the
>> components in the child process groups.
>>
>> Thanks
>>
>> Deepak
>>
>> *[image: image005]*
>>
>> *Deepak Reddy* | Data Engineer
>> ​IT Centers of Excellence
>> 13736 Riverport Dr., Maryland Heights, MO 63043
>>
>>
>>
>> *From:* Kevin Doran 
>> *Sent:* Tuesday, December 13, 2022 10:26 AM
>> *To:* users@nifi.apache.org
>> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry
>>
>>
>>
>> *CAUTION:* The e-mail below is from an external source. Please exercise
>> caution before opening attachments, clicking links, or following guidance.
>>
>> Thanks for sharing Deepak. That's interesting. I think we may look into a
>> way of disabling components that doesn't impact version control, or else
>> fix performance issues such that disabling is not necessary.
>>
>>
>>
>> Just out of curiosity, what version of NiFi are you on? The reason I ask
>> is that I know there have been some UI performance improvements made in
>> recent versions of NiFi, so I'm wondering if some of this might already
>> have been addressed...
>>
>>
>>
>> Thanks,
>>
>> Kevin
>>
>>
>>
>> O

RE: [EXTERNAL] Re: Disabling flows - nifi registry

[Kevin Doran]

 Thanks, Deepak. I'll have to think about this one. I'm not sure if anyone
else has any thoughts on this workflow / use case?

On Dec 13, 2022 at 14:52:31, "Chirthani, Deepak Reddy" <
c-deepakreddy.chirth...@charter.com> wrote:

> The version of Nifi where I had UI performance issues was 1.11.4
> The current version of our nifi instance is 1.15.3
>
>
> One other disadvantage of having Process Groups with Stopped components is
> that when a dataflow manager or a developer inadvertently right click and
> select “start” on the root canvas page, that will start all the components
> in the child process groups.
>
> Thanks
>
> Deepak
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> *From:* Kevin Doran 
> *Sent:* Tuesday, December 13, 2022 10:26 AM
> *To:* users@nifi.apache.org
> *Subject:* RE: [EXTERNAL] Re: Disabling flows - nifi registry
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Thanks for sharing Deepak. That's interesting. I think we may look into a
> way of disabling components that doesn't impact version control, or else
> fix performance issues such that disabling is not necessary.
>
>
>
> Just out of curiosity, what version of NiFi are you on? The reason I ask
> is that I know there have been some UI performance improvements made in
> recent versions of NiFi, so I'm wondering if some of this might already
> have been addressed...
>
>
>
> Thanks,
>
> Kevin
>
>
>
> On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" <
> c-deepakreddy.chirth...@charter.com> wrote:
>
> Hi Kevin,
>
> Thank you for your quick response. The only reason why I prefer processors
> to be disabled is to improve the NiFi UI performance by simply disabling
> stopped processors which I am not using.
>
>
> https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550
>
> I had issues with Nifi UI performance in the past where the refresh wheel
> keeps on spinning like for 10 seconds and disabling stopped processors
> really helped me.
>
> Thanks
> Deepak
>
>
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> *From:* Kevin Doran 
> *Sent:* Friday, December 9, 2022 9:33 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXTERNAL] Re: Disabling flows - nifi registry
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Hi Deepak,
>
>
>
> So far, we have been honoring the following policy for what constitutes a
> change in version control:
>
>
>
>1. stopped/started does not count as a "local change"
>2. enabled/disabled does count as a change, and that state is captured
>in the flow snapshot json version saved to registry.
>
>
>
> One reason for this is that some users want to setup CI/CD to deploy from
> Registry and automatically start a flow in the target NiFi. If there are
> components they don't want to start, the disabled state gives them a way to
> capture that configuration in the flow.
>
>
>
> Is there a reason you prefer disable over stop in your lower environment?
> I leave flows stopped in dev environments all the time and have never run
> into an issue, but of course, everyone's workflow and use case is slightly
> different, so I'm interested in hearing your perspective on this to see if
> we need to consider something more flexible.
>
>
>
> Cheers,
>
> Kevin
>
>
>
> On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" <
> c-deepakreddy.chirth...@charter.com> wrote:
>
> Hey guys,
>
> So, once I fully develop and parameterize my nifi dataflow, let’s say in
> dev environment, I enable the version control, import the flow in higher
> environment and turn on the dataflow. In most of the cases both the flows
> in lower and higher environments will be running. Let says dev nifi
> connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub.
> However, in some cases, we do want to stop and disable the flow in lower
> env. When I do that the registry is identifying that local changes are made
> to the flow which is nothing but all the components are disabled. I don’t
> want to keep the processors in stopped state on the canvas(regi

RE: [EXTERNAL] Re: Disabling flows - nifi registry

[Kevin Doran]

Thanks for sharing Deepak. That's interesting. I think we may look into a
way of disabling components that doesn't impact version control, or else
fix performance issues such that disabling is not necessary.

Just out of curiosity, what version of NiFi are you on? The reason I ask is
that I know there have been some UI performance improvements made in recent
versions of NiFi, so I'm wondering if some of this might already have been
addressed...

Thanks,
Kevin

On Dec 13, 2022 at 09:52:47, "Chirthani, Deepak Reddy" <
c-deepakreddy.chirth...@charter.com> wrote:

> Hi Kevin,
>
> Thank you for your quick response. The only reason why I prefer processors
> to be disabled is to improve the NiFi UI performance by simply disabling
> stopped processors which I am not using.
>
>
> https://community.cloudera.com/t5/Support-Questions/Nifi-UI-Working-very-Slow-How-to-increase-performance-of-a/td-p/214550
>
> I had issues with Nifi UI performance in the past where the refresh wheel
> keeps on spinning like for 10 seconds and disabling stopped processors
> really helped me.
>
> Thanks
> Deepak
>
>
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> *From:* Kevin Doran 
> *Sent:* Friday, December 9, 2022 9:33 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXTERNAL] Re: Disabling flows - nifi registry
>
>
>
> *CAUTION:* The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
> Hi Deepak,
>
>
>
> So far, we have been honoring the following policy for what constitutes a
> change in version control:
>
>
>
>- stopped/started does not count as a "local change"
>- enabled/disabled does count as a change, and that state is captured
>in the flow snapshot json version saved to registry.
>
>
>
> One reason for this is that some users want to setup CI/CD to deploy from
> Registry and automatically start a flow in the target NiFi. If there are
> components they don't want to start, the disabled state gives them a way to
> capture that configuration in the flow.
>
>
>
> Is there a reason you prefer disable over stop in your lower environment?
> I leave flows stopped in dev environments all the time and have never run
> into an issue, but of course, everyone's workflow and use case is slightly
> different, so I'm interested in hearing your perspective on this to see if
> we need to consider something more flexible.
>
>
>
> Cheers,
>
> Kevin
>
>
>
> On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" <
> c-deepakreddy.chirth...@charter.com> wrote:
>
> Hey guys,
>
> So, once I fully develop and parameterize my nifi dataflow, let’s say in
> dev environment, I enable the version control, import the flow in higher
> environment and turn on the dataflow. In most of the cases both the flows
> in lower and higher environments will be running. Let says dev nifi
> connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub.
> However, in some cases, we do want to stop and disable the flow in lower
> env. When I do that the registry is identifying that local changes are made
> to the flow which is nothing but all the components are disabled. I don’t
> want to keep the processors in stopped state on the canvas(registry do not
> identify for stopping) but want to disable them. Any workaround for
> registry not to identify local changes when flow is disabled?
>
>
>
> Thanks
>
> Deepak
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>

Re: Error on nifi start

[Kevin Doran]

 Yep, was just about to say the same. NiFi 1.14.0 does not run on Java 17,
but recent versions of NiFi do. So either move to a NiFi >= 1.16.3 which
does support on Java 17, or else downgrade Java to 8 or 11.

On Dec 13, 2022 at 10:47:00, Joe Witt  wrote:

> We support 8, 11, and 17 now.  As far as NiFi 1.14 which James mentioned
> he was using we didn't support it then.  Support started in NiFi 1.16.
>
> Thanks
>
> On Tue, Dec 13, 2022 at 8:43 AM James Srinivasan <
> james.sriniva...@gmail.com> wrote:
>
>> I think nifi currently supports java 8 or 11, not 17:
>>
>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_requirements
>>
>> On Tue, 13 Dec 2022, 12:45 James McMahon,  wrote:
>>
>>> I am using an Ansible role from Ansible GALAXY that has been tested and
>>> validated up through Apache NiFi v1.14.0. I download and install 1.14.0.bin
>>> from the Apache NiFi archives fir this reason.
>>>
>>> I am using ansible to install on and AWS EC2 instance. My java version
>>> on this instance is:
>>>
>>> openjdk 17.0.5 2022-10-18 LTS
>>>
>>> OpenJDK Runtime Environment Corretto-17.0.5.8.1
>>>
>>> The install goes well. But when nifi attempts to start, it fails with
>>> the following error message. Is this error indicating a compatibility issue
>>> with the java installation on AWS? How should I proceed to get nifi to
>>> start?
>>>
>>> 2022-12-13 02:50:39,316 ERROR [main] org.apache.nifi.NiFi Failure to
>>> launch NiFi due to org.xerial.snappy.SnappyError:
>>> [FAILED_TO_LOAD_NATIVE_LIBRARY] Unable to make p
>>>
>>> rotected final java.lang.Class
>>> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
>>> throws java.lang.ClassFormatError acce
>>>
>>> ssible: module java.base does not "opens java.lang" to unnamed module
>>> @31b289da
>>>
>>> org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] Unable to
>>> make protected final java.lang.Class
>>> java.lang.ClassLoader.defineClass(java.lang.String,byte[]
>>>
>>> ,int,int,java.security.ProtectionDomain) throws
>>> java.lang.ClassFormatError accessible: module java.base does not "opens
>>> java.lang" to unnamed module @31b289da
>>>
>>> at
>>> org.xerial.snappy.SnappyLoader.injectSnappyNativeLoader(SnappyLoader.java:297)
>>>
>>> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:227)
>>>
>>> at org.xerial.snappy.Snappy.(Snappy.java:48)
>>>
>>> at
>>> org.apache.nifi.processors.hive.PutHiveStreaming.(PutHiveStreaming.java:158)
>>>
>>> at java.base/java.lang.Class.forName0(Native Method)
>>>
>>> at java.base/java.lang.Class.forName(Class.java:467)
>>>
>>> at
>>> org.apache.nifi.nar.StandardExtensionDiscoveringManager.getClass(StandardExtensionDiscoveringManager.java:328)
>>>
>>> at
>>> org.apache.nifi.documentation.DocGenerator.documentConfigurableComponent(DocGenerator.java:100)
>>>
>>> at
>>> org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:65)
>>>
>>> at
>>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1126)
>>>
>>> at org.apache.nifi.NiFi.(NiFi.java:159)
>>>
>>> at org.apache.nifi.NiFi.(NiFi.java:71)
>>>
>>> at org.apache.nifi.NiFi.main(NiFi.java:303)
>>>
>>>
>>>
>>>
>>>
>>>
>>>

Re: Disabling flows - nifi registry

[Kevin Doran]

 Hi Deepak,

So far, we have been honoring the following policy for what constitutes a
change in version control:


   - stopped/started does not count as a "local change"
   - enabled/disabled does count as a change, and that state is captured in
   the flow snapshot json version saved to registry.


One reason for this is that some users want to setup CI/CD to deploy from
Registry and automatically start a flow in the target NiFi. If there are
components they don't want to start, the disabled state gives them a way to
capture that configuration in the flow.

Is there a reason you prefer disable over stop in your lower environment? I
leave flows stopped in dev environments all the time and have never run
into an issue, but of course, everyone's workflow and use case is slightly
different, so I'm interested in hearing your perspective on this to see if
we need to consider something more flexible.

Cheers,
Kevin

On Dec 9, 2022 at 10:26:27, "Chirthani, Deepak Reddy" <
c-deepakreddy.chirth...@charter.com> wrote:

> Hey guys,
>
> So, once I fully develop and parameterize my nifi dataflow, let’s say in
> dev environment, I enable the version control, import the flow in higher
> environment and turn on the dataflow. In most of the cases both the flows
> in lower and higher environments will be running. Let says dev nifi
> connects to dev gcp pubsub and prod nifi connects to prod gcp pubsub.
> However, in some cases, we do want to stop and disable the flow in lower
> env. When I do that the registry is identifying that local changes are made
> to the flow which is nothing but all the components are disabled. I don’t
> want to keep the processors in stopped state on the canvas(registry do not
> identify for stopping) but want to disable them. Any workaround for
> registry not to identify local changes when flow is disabled?
>
>
>
> Thanks
>
> Deepak
>
>
>
> *[image: image005]*
>
> *Deepak Reddy* | Data Engineer
> ​IT Centers of Excellence
> 13736 Riverport Dr., Maryland Heights, MO 63043
>
>
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>

Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows

[Kevin Doran]

 I believe as soon as this is merged Joe Witt is planning to prepare the
1.19.1 RC candidate and open the vote. We should have a release by next
week unless something comes up


On Dec 1, 2022 at 07:44:29, josef.zahn...@swisscom.com wrote:

> Hi Bence
>
>
>
> Ok got it, thanks a lot, So we will try to sync as less as possible. Let’s
> hope that we will see a NiFi 1.19.1 soon…
>
>
>
> Cheers Josef
>
>
>
> *From: *Simon Bence 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Thursday, 1 December 2022 at 13:42
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi Registry Bug which brakes the flow sync with NiFi
> 1.18.0 (and same version of the registry) on nested flows
>
>
>
> Hi Josef,
>
>
>
> I would like to inform you that there is an adjustment for the ticket I
> mentioned in my previous mail, which is needed for the proper behaviour of
> the function in every case. The change is in PR phase but I hope it will be
> merged soon: https://github.com/apache/nifi/pull/6741 I would not
> recommend to go forward without this additional change!
>
>
>
> Regards,
>
> Bence
>
>
>
> On 2022. Nov 29., at 16:21, Simon Bence  wrote:
>
>
>
> Hi Josef,
>
>
>
> Your welcome! In your case it looks like the information is lost during
> transformations of the internal representation.
>
>
>
> For technical details, here you can take a look on the code changes:
> https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2
>
>
>
> Regards,
>
> Bence
>
>
>
> On 2022. Nov 29., at 16:08,  <
> josef.zahn...@swisscom.com> wrote:
>
>
>
> Hi Bence, Joe
>
>
>
> Thank you guys for your fast response.
>
>
>
> @Bence you are right, the whole issue seems to be related to nested flows
> which we try to sync. We are relying heavily on this as we are doing the
> integration on one NiFi system and then sync it to production via the NiFi
> Registry. Our parent flow contains multiple nested flows.
>
>
>
> Ok, as NIFI-10874 will be included in the next release we will skip the
> 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than
> a few days until we see the next release... Do you have an idea why the
> flow with the nested flow could lose the storageLocation? Because, we just
> change a small thing in one of the nested flows and suddenly one of the
> other nested flows have been losing it’s storageLocation in the parent flow
> and from this point the whole flow was broken. Now we are scared to change
> anything as it could brake anytime again.
>
>
>
> Cheers Josef
>
>
>
>
>
>
>
> *From: *Simon Bence 
> *Reply to: *"users@nifi.apache.org" 
> *Date: *Tuesday, 29 November 2022 at 13:31
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: NiFi Registry Bug which brakes the flow sync with NiFi
> 1.18.0 (and same version of the registry) on nested flows
>
>
>
> Hi Josef,
>
>
>
> Thanks for your patience!
>
>
>
> I took a deeper look on what you were writing. In general, this is a sign
> for the case where the registry client cannot find the nested flow. As far
> as I understood in your case this happens when you are having a sync. Based
> on this you may be hitting
> https://issues.apache.org/jira/browse/NIFI-10874 which is not part of
> NiFi 1.19.0. The next release should contain the fix that addresses your
> issue.
>
>
>
> Regards,
>
> Bence
>
>
>
>
> On 2022. Nov 28., at 17:48, Simon Bence  wrote:
>
> Hi Josef,
>
> Thank you for raising the attention to this.
>
> The storageLocation is a new concept to generalise the information used to
> locate nested flows in a versioned flow. In general it can be there and
> does not cause issues, as of now NiFi falls back the previous way to locate
> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to
> double check if it relates to this situation. Please give me some time, I
> will find you back.
>
> Regards,
> Bence
>
>
>
> On 2022. Nov 28., at 17:17, Joe Witt  wrote:
>
> Josef
>
> Sorry for the challenges you've hit there.  I do think in 1.18 we had a
> bug/regression as we refactored our registry client.  That issue should be
> resolved in 1.19 which just went live thanks to
> https://issues.apache.org/jira/browse/NIFI-10787.  However, I am not
> positive if this will solve the scenario you've hit now but please if able
> try it out.
>
> Thanks
>
> On Mon, Nov 28, 2022 at 8:50 AM  wrote:
> Hi guys
>
>
>
> We had the following bug twice already and it broke the whole sync with
> the NiFi registry for the Flow/PG. First time was directly after we have
> upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we
> thought it could be because of the upgrade, however it occurred again after
> a few NiFi Registry commits on NiFi 1.18.0… The error was the following
> when we tried to change the flow version or when we tried to start from
> scratch with that version from the NiFi Registry, so the version was broken
> in the NiFi Registry:
>
>
>
> 
>
>
>
>
>
> We investigated the last NiFi Registry Commit and we saw in our GIT repo
> (we sync the “flow_s

Re: Trouble configuring logging

[Kevin Doran]

 Thanks Dylan - Thanks for your interest in contributing to a solution for
the issue! Let's collaborate on the Jira.

Cheers,
Kevin

On Oct 3, 2022 at 12:59:02, Dylan Klomparens 
wrote:

> Pardon me, I forgot to include the link to the JIRA ticket:
> https://issues.apache.org/jira/browse/NIFI-10579
> --
> *From:* Dylan Klomparens 
> *Sent:* Monday, October 3, 2022 12:16 PM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Trouble configuring logging
>
> Everyone, thank you for the feedback on this topic.
>
> I have created a JIRA Improvement ticket about this topic. I proposed that
> there be a way to disable redirection of standard out and standard error to
> logback.
>
> If there is a positive response from the community and Apache foundation
> members, I will implement this improvement - following the contribution
> guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide>
> - and submit a pull request.
>
>
> --
> *From:* Chris Sampson 
> *Sent:* Thursday, September 29, 2022 6:52 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Trouble configuring logging
>
> We had similar logging configuration issues when using the apache/nifi
> image in a Kubernetes environment, so had to adjust how we log things via
> the logback.xml to make formats a little more consistent, remove some logs
> we didn't (generally) find so useful and hide some things in log files
> within the container because it would cause policy/GDPR issues if it was
> output to StdOut where it could potentially be exposed to other tenants
> within our multi-tenant cloud.
>
> A cutdown/sanitised version of this logback.xml file has been uploaded as
> a gist [1] in case it helps anyone in future - pay attention to the
> comments within the XML if you want to use the file as it's not complete
> (I've removed some duplicated appender definitions where the only
> difference was the appender name and output log file, for example).
>
> We also customised the image with a different startup script that is
> effectively the same as the apache/nifi start.sh but doesn't include the
> "tail" of the "nifi-app.log" file (as we no longer have that file in our
> config and the logback.xml sends everything to StdOut anyway).
>
> In terms of a future approach, I wonder whether taking a similar approach
> to some other applications, like Elasticsearch, would be sensible where
> they include different default configurations in the various distribution
> mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
> work to make more things configurable via environment variables would also
> be worth considering if someone has time because Spring Boot/Logback can
> definitely handle that, allowing users to then configure their deployment
> with little more than environment variables, etc.
>
>
> [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
> On Thu, 29 Sept 2022 at 01:21, Kevin Doran  wrote:
>
> Yeah, if it helps at all, this is how we work around this in the Apache
> Dockerfile (I knew realize why this method was used rather than a standard
> console appender in logback):
>
> # Continuously provide logs so that 'docker logs' can produce them
> "${NIFI_HOME}/bin/nifi.sh" run &
> nifi_pid="$!"
> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>
>
> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>
>
> Something to look into. As more NiFi deployments become containerized,
> this may be an area for improvement over time.
>
> Thanks,
> Kevin
>
> On Sep 28, 2022 at 19:40:35, Dylan Klomparens 
> wrote:
>
> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to

Re: Trouble configuring logging

[Kevin Doran]

Thanks Chris! This looks very helpful.

In terms of a future approach, I wonder whether taking a similar approach
> to some other applications, like Elasticsearch, would be sensible where
> they include different default configurations in the various distribution
> mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
> work to make more things configurable via environment variables would also
> be worth considering if someone has time because Spring Boot/Logback can
> definitely handle that, allowing users to then configure their deployment
> with little more than environment variables, etc.
>

Yeah I love that idea. I linked this email thread on our Docker
Improvements page [1], where we have a placeholder for logging
improvements. I actually may have time to make some contributions to the
Docker Hub image in coming months 🤞

[1]
https://cwiki.apache.org/confluence/display/NIFI/NiFi+Docker+Container+Improvements


Cheers,
Kevin

On Sep 29, 2022 at 06:52:57, Chris Sampson 
wrote:

> We had similar logging configuration issues when using the apache/nifi
> image in a Kubernetes environment, so had to adjust how we log things via
> the logback.xml to make formats a little more consistent, remove some logs
> we didn't (generally) find so useful and hide some things in log files
> within the container because it would cause policy/GDPR issues if it was
> output to StdOut where it could potentially be exposed to other tenants
> within our multi-tenant cloud.
>
> A cutdown/sanitised version of this logback.xml file has been uploaded as
> a gist [1] in case it helps anyone in future - pay attention to the
> comments within the XML if you want to use the file as it's not complete
> (I've removed some duplicated appender definitions where the only
> difference was the appender name and output log file, for example).
>
> We also customised the image with a different startup script that is
> effectively the same as the apache/nifi start.sh but doesn't include the
> "tail" of the "nifi-app.log" file (as we no longer have that file in our
> config and the logback.xml sends everything to StdOut anyway).
>
>
>
>
> [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
> On Thu, 29 Sept 2022 at 01:21, Kevin Doran  wrote:
>
>> Yeah, if it helps at all, this is how we work around this in the Apache
>> Dockerfile (I knew realize why this method was used rather than a standard
>> console appender in logback):
>>
>> # Continuously provide logs so that 'docker logs' can produce them
>> "${NIFI_HOME}/bin/nifi.sh" run &
>> nifi_pid="$!"
>> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>>
>>
>> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>>
>>
>> Something to look into. As more NiFi deployments become containerized,
>> this may be an area for improvement over time.
>>
>> Thanks,
>> Kevin
>>
>> On Sep 28, 2022 at 19:40:35, Dylan Klomparens <
>> dklompar...@foodallergy.org> wrote:
>>
>>> Mark and Kevin, thank you for your insightful comments. That information
>>> allowed me to piece together the puzzle. Indeed, anything that is sent to
>>> standard out subsequently *causes* the log message to come from the the
>>> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
>>> allowed me to find the exact line of code that does this
>>> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
>>> .
>>>
>>> From my perspective this is an unfortunate design choice for NiFi
>>> because it does not allow standard out to be redirected easily. (For
>>> example, flowing onward to Docker, and forwarded to AWS CloudWatch using
>>> Docker's built-in log driver, in my case). I suppose I'll have to find an
>>> alternative logback appender class to output the logs to, then find a way
>>> to forward them on from there.
>>>
>>> Thanks again for the additional information that put the picture into
>>> view.
>>> --
>>> *From:* Mark Payne 
>>> *Sent:* Wednesday, September 28, 2022 10:14 AM
>>> *To:* users 
>>> *Subject:* Re: Trouble configuring logging
>>>
>>>
>>> [EXTERNAL]
>>> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
>>> start

Re: Trouble configuring logging

[Kevin Doran]

 Yeah, if it helps at all, this is how we work around this in the Apache
Dockerfile (I knew realize why this method was used rather than a standard
console appender in logback):

# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &

https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129


Something to look into. As more NiFi deployments become containerized, this
may be an area for improvement over time.

Thanks,
Kevin

On Sep 28, 2022 at 19:40:35, Dylan Klomparens 
wrote:

> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
> to forward them on from there.
>
> Thanks again for the additional information that put the picture into view.
> --
> *From:* Mark Payne 
> *Sent:* Wednesday, September 28, 2022 10:14 AM
> *To:* users 
> *Subject:* Re: Trouble configuring logging
>
>
> [EXTERNAL]
> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
> start) it doesn’t directly start the NiFi process.
> Instead, it starts a different processor, which is called RunNiFi. This
> RunNiFi process is responsible for doing a lot of things, including
> monitoring the nifi process and if it dies restarting it.
> Anything written to NiFi’s Standard Out goes to this processor, which then
> logs it.
> So you’d probably need to update the logging for the appender writing to
> the bootstrap file:
>  class="ch.qos.logback.core.rolling.RollingFileAppender”>
>
> And redirect that to standard out
>
> Thanks
> -Mark
>
>
> On Sep 28, 2022, at 9:48 AM, Kevin Doran  wrote:
>
> Dylan - I looked into this and am yet unable to offer an explaination.
> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
> some light, or else I will keep digging when I have a block of time. To
> help in my understanding: Which Docker image are you using? Is it the
> apace/nifi image or a custom one, and if custom, can you share the
> Dockerfile?
>
> Thanks,
> Kevin
>
> On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
> wrote:
>
> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
> 
> 
> 
>
> 
> true
> 
>
> 
> 
> %date %level [%thread] %logger{40} %msg%n
> 
> 
>
> 
> 
>  level="INFO"/>
>  level="INFO"/>
>  name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
> 
> 
>  level="ERROR" />
>  level="ERROR" />
> 
> 
>  level="ERROR" />
>  level="ERROR" />
>
> 
>
>  level="OFF" />
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>  level="ERROR"/>
>
> 
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
>
> 
> 
> 
>
> 
>
>
>

Re: Trouble configuring logging

[Kevin Doran]

 Dylan - I looked into this and am yet unable to offer an explaination.
Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
some light, or else I will keep digging when I have a block of time. To
help in my understanding: Which Docker image are you using? Is it the
apace/nifi image or a custom one, and if custom, can you share the
Dockerfile?

Thanks,
Kevin

On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
wrote:

> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
> 
> 
> 
>
> 
> true
> 
>
> 
> 
> %date %level [%thread] %logger{40} %msg%n
> 
> 
>
> 
> 
>  level="INFO"/>
>  level="INFO"/>
>  name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
> 
> 
>  level="ERROR" />
>  level="ERROR" />
> 
> 
>  level="ERROR" />
>  level="ERROR" />
>
> 
>
>  level="OFF" />
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>  level="ERROR"/>
>
> 
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
>
> 
> 
> 
>
> 
>

Re: ConsumeIMAP Invalid credentials error and Gmail disabling less secured apps access

[Kevin Doran]

 Hi Valentina,

I've not tried this myself, so I cannot be sure it is a solution here, but
have you tried generating a Google App Password [1] for the Gmail account
to use in NiFi in place of the user password?

I'm not sure if it is a drop-in replacement or not, but worth trying to see
if it works with NiFi IMAP/SMTP processors as-is. If not, we should
probably capture this as an Apache NiFi Jira if it has not been already, as
this is essentially a feature request to add that capability as an
option/alternative form of authentication for IMAP/SMTP processors.

[1] https://support.google.com/accounts/answer/185833?hl=en

Hope this helps,
Kevin

On Jun 13, 2022 at 05:02:19, Valentina Ivanova 
wrote:

> Hello!
>
> I am using ConsumeIMAP processor to access a gmail mailbox. In order to
> authenticate with user and password I had to turn on the 'Allow less secure
> apps access' setting in Gmail.
> However, since May 30th, this setting is no longer available which leads
> to Invalid credentials error in ConsumeIMAP.
>
> What would be a solution to this issue? How to retrieve emails from a
> Gmail mailbox from Nifi?
>
> Many thanks & have a great week ahead,
> Valentina
>
>

Re: R: How to see the request header created by InvokeHTTP?

[Kevin Doran]

 Glad to hear you were able to solve your issue, Luca!

Cheers,
Kevin

On Mar 23, 2022 at 09:01:35, Luca Giovannini 
wrote:

>
>
> Thank you so much, Kevin!!
>
>
>
> With that I could solve it in a minute!
>
> The issue was that InvokeHTTP was sending a “user-agent” header with an
> empty value and that caused the endpoint to issue and error message.
>
> By setting the “useragent” parameter of InvokeHTTP to a random non-null
> value (i.e. “none”), the endpoint responded instead with the expected
> answer.
>
>
>
> Thanks again,
>
>
>
> Luca
>
>
>
>
>
> *Luca Giovannini*
> Information Systems Analyst
> *Dedagroup Public Services*
>
> www.linkedin.com/in/lucagio/
>
> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
> Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del
> Lavoro 67
>
> www.dedagroup.it/public-services
>
>
>
> <http://www.dedagroup.it/home>
>
>
>
> *Da:* Kevin Doran 
> *Inviato:* mercoledì 2 marzo 2022 16:39
> *A:* users@nifi.apache.org
> *Oggetto:* Re: How to see the request header created by InvokeHTTP?
>
>
>
> **ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta
> attenzione ad eventuali link o allegati al suo interno.
>
>
>
> Hi Luca,
>
>
>
> Based on looking at the code here [1], I believe you need to enable
> debug-level logging for this logger:
>
>
>
> org.apache.nifi.processors.standard.InvokeHTTP
>
>
>
> Instructions for changing log level for individual component loggers can
> be found here [2]. (Ignore the part about attaching a remote debugger if
> you are just interested in the logging part.)
>
> Basically, you need to add this line to your NiFi conf/logback.xml file:
>
>
>
>  level="DEBUG"/>
>
>
> [1]
> https://github.com/apache/nifi/blob/4cc20e6c0687277394f021ca1f3ececd430c5a0e/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L1277
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fblob%2F4cc20e6c0687277394f021ca1f3ececd430c5a0e%2Fnifi-nar-bundles%2Fnifi-standard-bundle%2Fnifi-standard-processors%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fnifi%2Fprocessors%2Fstandard%2FInvokeHTTP.java%23L1277&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hWA12fczMaiCmi7xXiYQI1oyYzrz2SCcX8Rv%2Bj7iQbw%3D&reserved=0>
>
>
> [2]
> https://community.cloudera.com/t5/Community-Articles/NiFi-Debugging-Tutorial/ta-p/246082
> <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.cloudera.com%2Ft5%2FCommunity-Articles%2FNiFi-Debugging-Tutorial%2Fta-p%2F246082&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mOI4b0IRYfbrfuYVtbMVQNkl%2BgVhOvodiie5zNL7mss%3D&reserved=0>
>
>
>
> Hope this helps!
>
> Kevin
>
> On Mar 2, 2022 at 04:26:10, Luca Giovannini 
> wrote:
>
>
>
> Dear All,
>
>
>
> I am having troubles replicating with InvokeHTTP (1.12.1) a call to an
> HTTP endpoint that I am successfully able to make with Postman.
>
> I use InvokeHTTP with its default values and I just add two attributes for
> ‘Authorization’ and ‘Host’.
>
>
>
> I would like to see the actual request header created by InvokeHTTP, so
> that I can compare it with the one created by Postman and try to have some
> insights on my problem from the comparison…
>
> So, how can I access the request header created by InvokeHTTP?
>
> I had a look in the nifi-app log but no detail on the call was provided.
>
>
>
> Thank you for your help!
>
>
>
> Luca
>
>
>
>
>
> *Luca Giovannini*
> Information Systems Analyst
> *Dedagroup Public Services*
>
> www.linkedin.com/in/lucagio/
> <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flucagio%2F&data=04%7C01%7CLuca.Giovannini%40dedagroup.it%7Ce1a122e0f71b45a1c49908d9fc62c06d%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637818323354462607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=OlfQuGLwbYUy97U6%2F8%2BPdbmiq1tIKQwbNmgg4DjWnz8%3D&reserved=0>
>
> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
> Dedagroup Public Services Srl – Se

Re: How to see the request header created by InvokeHTTP?

[Kevin Doran]

 Hi Luca,

Based on looking at the code here [1], I believe you need to enable
debug-level logging for this logger:

org.apache.nifi.processors.standard.InvokeHTTP

Instructions for changing log level for individual component loggers can be
found here [2]. (Ignore the part about attaching a remote debugger if you
are just interested in the logging part.)
Basically, you need to add this line to your NiFi conf/logback.xml file:



[1]
https://github.com/apache/nifi/blob/4cc20e6c0687277394f021ca1f3ececd430c5a0e/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L1277

[2]
https://community.cloudera.com/t5/Community-Articles/NiFi-Debugging-Tutorial/ta-p/246082


Hope this helps!
Kevin

On Mar 2, 2022 at 04:26:10, Luca Giovannini 
wrote:

>
>
> Dear All,
>
>
>
> I am having troubles replicating with InvokeHTTP (1.12.1) a call to an
> HTTP endpoint that I am successfully able to make with Postman.
>
> I use InvokeHTTP with its default values and I just add two attributes for
> ‘Authorization’ and ‘Host’.
>
>
>
> I would like to see the actual request header created by InvokeHTTP, so
> that I can compare it with the one created by Postman and try to have some
> insights on my problem from the comparison…
>
> So, how can I access the request header created by InvokeHTTP?
>
> I had a look in the nifi-app log but no detail on the call was provided.
>
>
>
> Thank you for your help!
>
>
>
> Luca
>
>
>
>
>
> *Luca Giovannini*
> Information Systems Analyst
> *Dedagroup Public Services*
>
> www.linkedin.com/in/lucagio/
>
> T +39.051.278.928 | M +39.347.799.3183 | VoIP 951.128
> Dedagroup Public Services Srl – Sede di Casalecchio di Reno, Via del
> Lavoro 67
>
> www.dedagroup.it/public-services
>
>
>
> 
>
>
> Le informazioni contenute in questo messaggio di posta elettronica sono
> riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o
> forma. Qualora Lei non fosse la persona destinataria del presente
> messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone
> gentilmente comunicazione al mittente.
>
> The information included in this e-mail and any attachments are
> confidential and may also be privileged. If you are not the correct
> recipient, you are kindly requested to notify the sender immediately, to
> cancel it and not to disclose the contents to any other person.
>

Re: Running unsecured Nifi in Docker

[Kevin Doran]

I’ve tried the docker compose yaml config you provided as well as the one I
sent you, and both are working for me with the latest nifi image. Is there
any other relevant part of your config that could be causing this (e.g.,
changes since your initial email?) Have you started over from a clean state
by running `docker compose down`?

On Feb 16, 2022 at 09:45:27, Jean-Sebastien Vachon 
wrote:

> Here you go... thanks for the fast response.
> I've looked at the start script to see what is being done and set the
> different environment variables to go through the proper sections in the
> file.
>
>
> ...
> /extensions/nifi-server-nar-1.15.3.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-ui-1.15.3.war}
>
> stack-nifi2-1| 2022-02-16 14:23:46,854 INFO [main]
> o.e.j.a.AnnotationConfiguration Scanning elapsed time=158ms
> stack-nifi2-1| 2022-02-16 14:23:46,917 INFO [main]
> o.e.j.s.handler.ContextHandler._nifi_api No Spring
> WebApplicationInitializer types detected on classpath
> stack-nifi2-1| 2022-02-16 14:23:46,995 INFO [main]
> o.e.j.s.handler.ContextHandler._nifi_api Initializing Spring root
> WebApplicationContext
> stack-nifi2-1| 2022-02-16 14:23:51,275 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 198 properties from
> /opt/nifi/nifi-current/./conf/nifi.properties
> stack-nifi2-1| 2022-02-16 14:23:55,907 INFO [main]
> o.a.n.r.v.FileBasedVariableRegistry Loaded 93 properties from system
> properties and environment variables
> stack-nifi2-1| 2022-02-16 14:23:55,908 INFO [main]
> o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 93 properties.
> Including precedence overrides effective accessible registry key size is 93
> stack-nifi2-1| 2022-02-16 14:23:56,178 WARN [main]
> o.a.nifi.security.util.SslContextFactory Some keystore properties are
> populated (, , , null) but not valid
> stack-nifi2-1| 2022-02-16 14:23:56,179 ERROR [main]
> o.apache.nifi.controller.FlowController Unable to start the flow controller
> because the TLS configuration was invalid: The keystore properties are not
> valid
> stack-nifi2-1| 2022-02-16 14:23:56,657 ERROR [main]
> o.s.web.context.ContextLoader Context initialization failed
> stack-nifi2-1|
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression
> parsing failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.security.configuration.JwtAuthenticationSecurityConfiguration':
> Unsatisfied dependency expressed through constructor parameter 3; nested
> exception is org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowController': FactoryBean threw exception on
> object creation; nested exception is java.lang.IllegalStateException: Flow
> controller TLS configuration is invalid
>
>
>
>
> *Jean-Sébastien Vachon *
> Co-Founder & Architect
>
>
> *Brizo Data, Inc. www.brizodata.com
> <https://outlook.office365.com/mail/options/mail/messageContent/www.brizodata.com>
> *
> --
> *From:* Kevin Doran 
> *Sent:* Wednesday, February 16, 2022 9:39 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Running unsecured Nifi in Docker
>
> There have been some changes recently, and NiFi is now secure by default
> with a self-signed cert I believe. It could be that NIFI_WEB_HTTP_PORT
> conflicts with the expected NIFI_WEB_HTTPS_PORT.
>
> Try this:
>
>   nifi:
> image: apache/nifi:latest
> ports:
>   - "8443:8443" # UI
>   - "1" # Site-to-Site Input Port
> environment:
>   SINGLE_USER_CREDENTIALS_USERNAME: admin
>   SINGLE_USER_CREDENTIALS_PASSWORD: some_password
>   NIFI_SENSITIVE_PROPS_KEY: some_other_password
>
> If that does not work, can you please share the exact startup error?
>
> On Feb 16, 2022 at 09:28:55, Jean-Sebastien Vachon 
> wrote:
>
> Hi all,
>
> I'm trying to start a simple unsecured Nifi instance in a c

Re: Running unsecured Nifi in Docker

[Kevin Doran]

 There have been some changes recently, and NiFi is now secure by default
with a self-signed cert I believe. It could be that NIFI_WEB_HTTP_PORT
conflicts with the expected NIFI_WEB_HTTPS_PORT.

Try this:

  nifi:
image: apache/nifi:latest
ports:
  - "8443:8443" # UI
  - "1" # Site-to-Site Input Port
environment:
  SINGLE_USER_CREDENTIALS_USERNAME: admin
  SINGLE_USER_CREDENTIALS_PASSWORD: some_password
  NIFI_SENSITIVE_PROPS_KEY: some_other_password

If that does not work, can you please share the exact startup error?

On Feb 16, 2022 at 09:28:55, Jean-Sebastien Vachon 
wrote:

> Hi all,
>
> I'm trying to start a simple unsecured Nifi instance in a container as
> part of a larger docker compose stack and I'm stuck with an error regarding
> the TLS configuration and/or keystore properties. Here is the relevant part
> of my docker-compose file... what am I missing? Please make me feel stupid
> 😉
>
>  nifi:
> image: apache/nifi:latest
> # command:
> ports:
> - "8080:8080"
> - "1:1"
>
> restart: always
> command:
> /bin/bash
> environment:
> NIFI_REMOTE_INPUT_HOST: 0.0.0.0
> NIFI_WEB_HTTP_HOST: 0.0.0.0
> SINGLE_USER_CREDENTIALS_USERNAME: admin
> SINGLE_USER_CREDENTIALS_PASSWORD: some_password
> NIFI_WEB_HTTP_PORT: 8080
> AUTH: none
>
>
>
>
>
> *Jean-Sébastien Vachon *
> Co-Founder & Architect
>
>
> *Brizo Data, Inc. www.brizodata.com
> 
> *
>

Re: Penalty feature of Processor (Disable)

[Kevin Doran]

Seems like a nice improvement. I would lean towards introducing the 
RetryableLookupFailureException. At the same time, I think we should add catch 
blocks for more specific subclasses of SQLException, such as 
SQLTransientException, which would also trigger a 
RetryableLookupFailureException instead of a LookupFailureException.

> On Oct 27, 2021, at 08:55, Bryan Bende  wrote:
> 
> I'd consider changing DatabaseRecordLookupService.. the way it is
> currently implemented there is no way to tell the difference between a
> retryable exception like IOException vs a non-retryable exception like
> SQLException because the columns are wrong and will never work. We
> could introduce a new RetryableLookupFailureException and have both
> services catch IOException and throw the retryable exception, or both
> services can let IOException be thrown and let the callers decide what
> to do.
> 
> On Wed, Oct 27, 2021 at 5:07 AM Edward Armes  wrote:
>> 
>> Hi Bilal,
>> 
>> Thanks for confirming, it looks like my hunch was correct, and there is a 
>> discrepancy in the lookup service code itself.
>> 
>> In DatabaseRecordLookupService we catch an IOException and return a lookup 
>> failure like is done for an SQLExeception. This isn't done in the 
>> SimpleDatabaseLookupService. I think in this case its worth adding the 
>> IOException catch to SimpleDatabaseLookupService to bringing it in line with 
>> DatabaseRecordLookupService.
>> 
>> Edward
>> 
>> On Wed, 27 Oct 2021, 09:11 Bilal Bektas,  wrote:
>>> 
>>> Hi Edward,
>>> 
>>> 
>>> 
>>> Thank you for helping.
>>> 
>>> 
>>> 
>>> You can find the information which you want:
>>> 
>>> 
>>> 
>>> * LookupAttribute processor uses SimpleDatabaseLookupService, bundle of 
>>> which is “org.apache.nifi - nifi-lookup-services-nar”
>>> 
>>> * SimpleDatabaseLookupService uses DBCPConnectionPool, bundle of which is 
>>> “org.apache.nifi - nifi-dbcp-service-nar”. There is no custom build service 
>>> or processor on NiFi. All are the default bundle.
>>> 
>>> 
>>> * Teradata JDBC version: 16.20.00.13
>>> 
>>> * Oracle JDBC version: 12.2.0.1.0
>>> 
>>> 
>>> 
>>> 
>>> 
>>> In addition, I have done similar test on LookupAttribute (Oracle) 
>>> processor. The same situation happened; flow files were penalized and the 
>>> queue on upstream connection of LookupAttribute (Oracle) increased..
>>> 
>>> 
>>> 
>>> Thank you in advance,
>>> 
>>> 
>>> 
>>> --Bilal
>>> 
>>> 
>>> 
>>> 
>>> 
>>> From: Edward Armes 
>>> Sent: 26 Ekim 2021 Salı 23:40
>>> To: users@nifi.apache.org
>>> Subject: Re: Penalty feature of Processor (Disable)
>>> 
>>> 
>>> 
>>> Hi Bilal,
>>> 
>>> 
>>> 
>>> Can you just confirm that your connection to Teradata is done using the 
>>> DatabaseRecordLookupService and is using one of the 2 DBCPConnectionPools 
>>> which is using the Teradata JDBC driver or are you using custom built 
>>> service?
>>> 
>>> 
>>> 
>>> The reason for asking is that I want to do a quick check to make sure by 
>>> were not masking an issue in one of the underlying services that hasn't 
>>> been caught correctly for some reason
>>> 
>>> 
>>> 
>>> Edward
>>> 
>>> 
>>> 
>>> On Tue, 26 Oct 2021, 10:29 Bilal Bektas,  wrote:
>>> 
>>> Hi Community,
>>> 
>>> 
>>> 
>>> Thank you for detailed solutions and analysis.
>>> 
>>> 
>>> 
>>> @Dev Team, do you think to add a new feature (Rollback On Failure) for 
>>> LookupAttribute processor like PutHiveQL processor?
>>> 
>>> 
>>> 
>>> Thank you for helping,
>>> 
>>> 
>>> 
>>> --Bilal
>>> 
>>> 
>>> 
>>> 
>>> 
>>> From: Edward Armes 
>>> Sent: 26 Ekim 2021 Salı 01:32
>>> To: users@nifi.apache.org
>>> Subject: Re: Penalty feature of Processor (Disable)
>>> 
>>> 
>>> 
>>> Having a quick look at the lookupAttribute code it looks like it takes a 
>>> Optional<> from the call to the configured service. So I wonder if its 
>>> worth adding the logic to service instead so that on erroring it can either 
>>> return a missing value or throw an exception that would trigger the 
>>> roleback. That would achieve the same goal without affecting other users of 
>>> the LookupAttribute processor, where such logic isn't needed or wanted 
>>> (e.g. SimpleLookupService).
>>> 
>>> 
>>> 
>>> Edward
>>> 
>>> 
>>> 
>>> On Mon, 25 Oct 2021, 21:54 Matt Burgess,  wrote:
>>> 
>>> The approach in #1 is already present in a few Put processors like
>>> PutHive3QL, the property is named "Rollback on Failure" and takes a
>>> boolean value. The docs explain that if set to false, the flowfile is
>>> routed to failure, and if true will throw an exception and rollback
>>> the session. Check RollbackOnFailure.java for more details.
>>> 
>>> Regards,
>>> Matt
>>> 
>>> On Mon, Oct 25, 2021 at 4:46 PM Bryan Bende  wrote:
 
 The try/catch for IOException in LookupAttribute is after already
 calling session.get(), so it is separate from loading a flow file.
 
 The SimpleDatabaseLookupService catches SQLException and throws
 LookupFailureException which is the indicator to route to failure, 

Re: Help on setting up authentication between NiFi and NiFi Registry

[Kevin Doran]

Hi Tony,

The typical approach I have seen is to allow NiFi to authenticate to NiFi 
Registry using a client certificate. This can be achieved alongside LDAP 
authentication by configuring  NiFi Registry using a 
ConfiguableCompositeUserGroupProvider, with a FileUserGroupProvider to manage 
certificate identities for NiFi and and an LDAPUserGroupProvider for users.

Does that make sense? For the NiFi side of things, here is a helpful 
walkthrough from Drew: https://youtu.be/qD03ao3R-a4

Hope this helps with the general approach. If there are questions on the 
details, let me know!

Kevin


From: Sim, Yoosuk 
Sent: Monday, March 22, 2021 4:09:52 PM
To: users@nifi.apache.org 
Subject: Help on setting up authentication between NiFi and NiFi Registry


Hello everyone,



I am trying to setup NiFi and NiFi Registry to talk to each other. They are 
both using LDAP to authenticate and authorize the users and they seem to talk 
to each other over SSL. However, I can’t seem to figure out how to let 
NiFi-Registry to authenticate NiFi.

After some debugging, the following are the information I get from NiFi 
Registry when NiFi tries to do version control to it.

2021-03-22 19:58:24,665 DEBUG [NiFi Registry Web Server-19] 
o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using 
X509IdentityProvider

2021-03-22 19:58:24,665 DEBUG [NiFi Registry Web Server-19] 
o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request.

2021-03-22 19:58:24,677 DEBUG [NiFi Registry Web Server-19] 
o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using 
JwtIdentityProvider

2021-03-22 19:58:24,692 DEBUG [NiFi Registry Web Server-19] 
o.a.n.r.s.a.BearerAuthIdentityProvider HTTP Bearer Auth credentials not 
present. Not attempting to extract credentials for authentication.

2021-03-22 19:58:24,692 DEBUG [NiFi Registry Web Server-19] 
o.a.n.r.w.s.a.AnonymousIdentityFilter Populated SecurityContextHolder with 
anonymous token: 'anonymous'



I understand that I need to make an account in NiFi-Registry for NiFi, but how 
would I go about doing that when I am relying on LDAP instead of credential for 
authentication? Or is it even possible?



Cheers,



Tony Sim

Re: Run Nifi in IntelliJ to debug?

[Kevin Doran]

Hi, Darren -

I am not aware of a method of running a full NiFi instance directly in 
IntelliJ, but I use the method Matt mentioned: attaching IntelliJ as a remote 
debugger to a running NiFi instance.

I put together a guide a few years back for folks unfamiliar with this process 
or new to IntelliJ:
https://community.cloudera.com/t5/Community-Articles/NiFi-Debugging-Tutorial/ta-p/246082
 


The screenshots are for an older IntelliJ, but the same basic steps should 
still work. 

Hope this helps!
Kevin

> On Oct 26, 2020, at 12:12, Darren Govoni  wrote:
> 
> Thanks Matt. I think if i can attach remotely and step through the code that 
> will satisfy my needs. Let me give it a try.
> 
> I also found how to run mvnDebug and attach to that from intellij. Just need 
> to find a maven goal that runs nifi but i havent seen one yet.
> 
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for Android 
> From: Matt Burgess 
> Sent: Monday, October 26, 2020 12:05:03 PM
> To: users@nifi.apache.org 
> Subject: Re: Run Nifi in IntelliJ to debug?
>  
> Sorry I misread the part where you wanted to run NiFi inside IntelliJ,
> I was talking about running it externally (from the command-line,
> e.g.) and connecting the IntelliJ debugger. I haven't run NiFi itself
> using IntelliJ, maybe someone else can chime in for that.
> 
> On Mon, Oct 26, 2020 at 12:03 PM Matt Burgess  wrote:
> >
> > Yes, that's a pretty common operation amongst NiFi developers. In
> > conf/bootstrap.conf there's a section called Enable Remote Debugging
> > and a commented-out line something like:
> >
> > java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
> >
> > You can remove the comment from that line and set things like the
> > address to the desired port, whether to suspend the JVM until a
> > debugger connects, etc. Then in IntelliJ you can create a new
> > configuration of type Remote, point it at the port you set in the
> > above line, and connect the debugger. It will then stop at breakpoints
> > and you can do all the debugging stuff like add Watches, execute
> > expressions (to change values at runtime), etc.
> >
> > Regards,
> > Matt
> >
> > On Mon, Oct 26, 2020 at 11:52 AM Darren Govoni  wrote:
> > >
> > > Hi
> > >Is it possible to run Nifi from inside IntelliJ with debugging such 
> > > that I can hit the app from my browser and trigger breakpoints?
> > >
> > > If anyone has done this can you please share any info?
> > >
> > > Thanks in advance!
> > > Darren
> > >
> > > Sent from my Verizon, Samsung Galaxy smartphone
> > > Get Outlook for Android

Re: Securing NiFI behind a proxy (NGINX).

[Kevin Doran]

Hi,

Your understanding is correct: In order to get the connections you want, NGINX 
will have to be recognized by NiFi as an authorized proxy. The client 
certificate DN will be used for each request, provided NGINX terminates that 
TLS connection from the client and passes the DN of the certificate in the 
X-ProxiedEntitiesChain header to NiFi.

There are a few examples here:
https://github.com/ijokarumawak/nifi-reverseproxy/tree/master/nginx 


Here is an example of configuring NGINX to pass the client Cert DN to NiFi:
https://github.com/ijokarumawak/nifi-reverseproxy/blob/master/nginx/standalone-secure-http/nginx.conf
 


The FQDN of NGINX should match the external hostname of the machine (i.e., what 
the client uses to send requests).

Hope this helps,
Kevin



> On Sep 16, 2020, at 02:04, scotty  wrote:
> 
> Hi Vijay,
> 
> After realizing that the reverse proxy was the problem, I've got NiFi,
> standalone, secured with certificates by removing the reverse proxy out of
> the mix.
> 
> Is there some example, somewhere, of using a NGINX reverse proxy so that I
> can have the following setup?
> 
> client > https > NGINX > https > NiFi
> 
> My understanding is that NGINX needs a client certificate and that the FQDN
> of that certificate needs to be setup to proxy user requests in the NiFi UI.
> I've done both of these things as well as setup the nifi.web.proxy.host and
> nifi.web.proxy.context.path in the nifi.properties file.
> 
> Is there a specific FQDN that NGINX is supposed to have?
> 
> Thanks.
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: In memoriam of Jeff Storck

[Kevin Doran]

Jeff, you were a fantastic collaborator and friend. You will be dearly missed. 
Thank you for all your contributions, and for all you’ve  shown and taught me 
over the years. You’ve left behind a great legacy that will continue to have a 
positive impact on the world for years to come, not just your work but your way 
of working with others, and for that we are all grateful. RIP.

> On Jun 15, 2020, at 3:30 PM, Pierre Villard  
> wrote:
> 
> I can't say how much we will miss you Jeff. You were a great guy, always nice 
> and helpful with everyone. You always went the extra mile to make things 
> easier and more robust.
> 
> RIP Jeff
> 
> Le lun. 15 juin 2020 à 21:13, Jeremy Dyer  > a écrit :
> This is shocking and heartbreaking news. Jeff was a great guy and will be 
> deeply missed. 
> 
> The last time I saw Jeff in person was with Aldrin. We were eating at Bonchon 
> chicken and he was mocking me for how little spice I could handle XD. I could 
> always count on him for a good Dumb and Dumber reference and laugh. We also 
> shared a common hatred for conference food.
> 
> RIP Jeff
> 
> On Mon, Jun 15, 2020 at 2:33 PM Joe Witt  > wrote:
> You will be greatly missed.  Your impact to this community has been 
> tremendous.  The items Andy summarizes were huge efforts that you drove over 
> periods of many many months if not a year or more and they make NiFi so much 
> more accessible than before.
> 
> RIP Jeff.
> 
> 
> 
> On Mon, Jun 15, 2020 at 11:24 AM Andy LoPresto  > wrote:
> It is with a heavy heart that I write to the NiFi community today. Jeff 
> Storck, a PMC member, committer, and genuine and helpful presence in the 
> community, has passed away. 
> 
> I was lucky enough to know Jeff personally for many years, and his absence is 
> a huge loss to all of us who did. Jeff was incredibly intelligent, but also 
> kind and willing to share his experience with everyone. Whether playing 
> volleyball (I am nowhere near as good but he humored me), discussing the best 
> ramen and sushi spots, or evaluating a new exercise regime, Jeff brought 
> passion to everything. A number of us are sharing stories of our favorite 
> times with Jeff, and I am touched by how many people have a memory of Jeff 
> reaching out and patiently helping them when they were new or struggling with 
> a task. 
> 
> While other colleagues would happily transition to any topic _but_ work when 
> we went to a nearby brewery at the end of a long day, Jeff would sit down 
> next to me and say with a smile, "Ok Andy, work's done, now we can _really_ 
> talk about Groovy unit testing." He never shied away from expressing his 
> perspective and stood on conviction, but he was also open and genuinely 
> wanted to hear other views to expand his mind. 
> 
> If you come across a Spock test in the NiFi codebase, that was most likely 
> Jeff's work. He was intimately involved in much of the most challenging code 
> - especially Kerberos integration, making the difficult but critical 
> processes easier for our users. Anyone running NiFi on Java 11 should thank 
> Jeff, as that was a labor of love, pushing against the headwinds of so many 
> compatibility issues and language changes. The ease with which NiFi runs on 
> multiple versions and platforms belies the immense amount of effort and 
> dedication that he put into making this happen. 
> 
> There are so many aspects to Jeff that a note like this could never capture, 
> but one that stands above the rest to me is Jeff's passion for learning and 
> growth. He devoted himself to doing the best he could and constantly 
> improving that. That is a noble philosophy that I know I will remember and 
> admire moving forward. I’ve already started learning Kotlin because of Jeff’s 
> enthusiasm and encouragement. 
> 
> Jeff’s family has created a GoFundMe page [1] and there they describe their 
> intent to celebrate his life. I think that message is very positive and 
> uplifting. To anyone wondering how they can honor Jeff's legacy, I suggest 
> offering a helping hand to someone who needs it. Something as simple as 
> responding to an extra "newbie" mailing list question at the end of a long 
> day, or taking on a challenging task because your neighbor has their plate 
> full. That's how Jeff lived, and he made the world a better place. 
> 
> 
> 
> Andy
> 
> [1] https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 

Re: Multiple access login NiFi

[Kevin Doran]

Hi Daniel,

This is unexpected, and I’ve never heard of someone experiencing the behavior 
you describe until now.

NiFi does allow multiple, simultaneous connections from different users without 
any special configuration to enable that.

Can you share some details about your NiFi version and security configuration 
(nifi.properties, login-identity-providers.xml, authorizers.xml), as well as 
steps to reproduce?

Thanks,
Kevin

> On Apr 21, 2020, at 06:03, Daniel Arnedo  wrote:
> 
> Hi,
>  
> I just configured NiFi with LDAP. I have all the roles/groups and users with 
> its memberships correctly only from LDAP but I have a problem. If there is 
> some user logged and another tries to log in too, the first user 
> automatically logs out.
> How could I configure NiFi to allow multiple connections from different users 
> at the same time?
>  
> Thanks,
> Daniel

Re: Suggestions for Flow Development Lifestyle

[Kevin Doran]

I’ve always thought along the lines Otto suggests, that eventually, given some 
way of formatting the diff, there would also be some visual tool in the 
ecosystem that would help visualize that diff and could be used specifically in 
the context of reviewing/merging changes.

Lots of good discussion on this thread, thanks all!

Kevin

> On Feb 25, 2020, at 15:44, Otto Fowler  wrote:
> 
> -or- when diffing, some other representation could be presented other than 
> straight xml.
> 
> The diff ( if run / visualized in nifi ) needn’t be byte for byte, it can be 
> logical can’t it? 
> 
> On February 25, 2020 at 14:58:24, Eric Secules (esecu...@gmail.com 
> ) wrote:
> 
>> Hi Ken,
>>  
>> I feel like there are also issues in trying to build a pull-request-like 
>> workflow, in that most NiFi design is done visually on the canvas but there 
>> isn't a good way, that I know of, to represent differences between versions 
>> in a visual way. Even if you enabled pull request-like functionality, I 
>> don't know that I would expect users to approve changes by examining diffs 
>> in the XML files.
>> 
>> I agree, doing PR can be improved by starting up a nifi instance and 
>> registry on top of the feature branch, so you can view the changes in a safe 
>> environment before approving. But this doesn't solve the issue of merging 
>> diverged versions. For that we'd need a visual representation of a diff. 
>> Maybe entering a 3 way diff mode with side-by-side canvases and no active 
>> processors where the differences are highlighted. Doing merge resolution at 
>> the XML level is a non-starter for me. 
>> 
>> I've tried out restarting the registry which works for receiving new 
>> changes. I have also had some trouble with loosing previous version history 
>> when restarting a registry, but the main issue would still be merging 
>> diverged branches. Maybe it would help to have some checkout mechanism and 
>> return to the days of centralized version control if merging is too 
>> difficult to implement.
>> 
>> -Eric
>> 
>> On Tue, Feb 25, 2020 at 9:36 AM Ken Swanson > > wrote:
>> I have a lot of interest in this too, as my team has run into these issues 
>> as well.
>> 
>> I feel like there are also issues in trying to build a pull-request-like 
>> workflow, in that most NiFi design is done visually on the canvas but there 
>> isn't a good way, that I know of, to represent differences between versions 
>> in a visual way. Even if you enabled pull request-like functionality, I 
>> don't know that I would expect users to approve changes by examining diffs 
>> in the XML files.
>> 
>> However, I did have one thing to add that I hope might help:
>> On Tue, Feb 25, 2020 at 12:32 AM Eric Secules > > wrote:
>> I've also tried backing up my local registry to a separate branch in git and 
>> manually merging it with the branch that central-reg backs up to, but these 
>> git branches are glorified backups and the registry doesn't seem to be built 
>> to pull updates from them. On top of that doing a code review on the 
>> generated JSON describing a process group is difficult and I ran into 
>> several merge conflicts testing out a very simple merge where the target 
>> branch diverged slightly from the feature branch.
>> 
>> From what I've seen, the registry will use any updates that are in the repo 
>> when it starts up. So you should be able to do something like this:
>> Make the registry pull (or clone) from the backup as part of its startup 
>> process
>> Restart the registry when a merge to master is made. AFAIK this should not 
>> affect any NiFi instances that are connected to the registry; they should 
>> continue to work once the registry comes back up
>> It's a bit clunky but I think it works.
>> 
>> -Ken

Re: OIDC Secured NiFi with Secure NiFi Registry (certs?)

[Kevin Doran]

Yep exactly as you said it. Let me know if you run into trouble.



From: Ryan H 
Sent: Thursday, October 24, 2019 8:56 AM
To: users@nifi.apache.org
Subject: Re: OIDC Secured NiFi with Secure NiFi Registry (certs?)

Pierre/Kevin,

Thanks for the additional info on this. Yes, this makes sense to me. I wasn't 
sure if what I was wanting to do worked or was supported at this time, but now 
I see how it will. To summarize, I just need to spin up the registry with an 
initial admin user which will have a cert created to access the registry UI. 
From the UI, the initial admin can add in users for the NiFi Nodes and any 
users that should have access to Registry (with whatever bucket permissions 
desired). When connecting NiFi to Registry, the nodes will identify themselves 
via their node certs which will succeed as long as corresponding node users 
have been created on Registry. When users place something under version 
control, they will only be able to access Buckets that they have been granted 
permissions for via their corresponding/matching user identities as created on 
Registry via the initial admin user. I hope I summarized this correctly.

As always, thanks for the quick responses and help.


Cheers,

Ryan H

On Thu, Oct 24, 2019 at 8:33 AM Kevin Doran 
mailto:kdo...@apache.org>> wrote:
This is a very good question, and Pierre gives a good summary of how
to go about solving for it.

Essentially, you need to configure NiFi Registry for how to know about
the users and groups that will be passed to it. That is the
authorizers.xml file Pierre mentioned. There are two options for a
UserGroupProvider: File based and LDAP based. If your NiFi OIDC
provider is backed by an LDAP directory you can hook up to directly,
that would be an option, even if you are not using LDAP for
authentication in Registry. If that's not the case, then configuring
the FileUserGroupProvider and an initial admin (for example, a client
cert authenticated admin), will let you manually define users through
the Registry UI that match the identities of the OIDC users that will
be passed by NiFi.

Best,
Kevin

On Thu, Oct 24, 2019 at 5:54 AM Pierre Villard
mailto:pierre.villard...@gmail.com>> wrote:
>
> Hi Ryan,
>
> NiFi nodes will use their own certificates as identities to authenticate 
> against the NiFi Registry and the NiFi nodes will then proxy the users 
> connected to the NiFi instances for the interactions with the registry. You 
> have to configure the NiFi node identities as well as where to get the 
> users/groups informations using the authorizers.xml file [1]. Once the 
> users/groups are known in the NiFi Registry you can define the authorizations 
> as you described for the users and groups and it will reflected for the 
> users/groups when they connect to NiFi. If, however, you want to allow users 
> to authenticate on the NiFi Registry UI (to create buckets for instance), 
> then you'd have to also configure the authentication parts on the Registry 
> [2] (note that OIDC is not supported yet [3]).
>
> Hope this helps a bit.
>
> [1] 
> https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#authorizers-setup
> [2] 
> https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#user_authentication
> [3] https://issues.apache.org/jira/browse/NIFIREG-313
>
> Le jeu. 24 oct. 2019 à 03:54, Ryan H 
> mailto:ryan.howell.developm...@gmail.com>> 
> a écrit :
>>
>> Hi All,
>>
>> We currently have a multi-node NiFi cluster (1.8.0) that is secured using 
>> the OIDC provider for authentication. We are setting up a secure NiFi 
>> Registry (0.5.0) which our secure NiFi cluster will connect to.
>>
>> What is the recommended way to connect the OIDC secured NiFi instance to the 
>> secure NiFi Registry (only option looks to be using certs since we are not 
>> using LDAP or Kerb)? I am assuming the only way is to do a cert import to 
>> NiFi which will then open up all buckets to the entire cluster (based on the 
>> permissions of the user tied to the certificate).
>>
>> We are operating in a multi-tenant environment and would like to achieve 
>> bucket level permissions for the various users of the system. Accessing the 
>> UI of the NiFi Registry instance isn't super important, except for maybe a 
>> couple users for which generating a couple certs isn't a big deal. However, 
>> allowing users to only access certain buckets may be important.
>>
>> For now just being able to get this hooked up is ideal. Thoughts?
>>
>>
>> Thanks in Advance,
>>
>> Ryan H.
>>

Re: OIDC Secured NiFi with Secure NiFi Registry (certs?)

[Kevin Doran]

This is a very good question, and Pierre gives a good summary of how
to go about solving for it.

Essentially, you need to configure NiFi Registry for how to know about
the users and groups that will be passed to it. That is the
authorizers.xml file Pierre mentioned. There are two options for a
UserGroupProvider: File based and LDAP based. If your NiFi OIDC
provider is backed by an LDAP directory you can hook up to directly,
that would be an option, even if you are not using LDAP for
authentication in Registry. If that's not the case, then configuring
the FileUserGroupProvider and an initial admin (for example, a client
cert authenticated admin), will let you manually define users through
the Registry UI that match the identities of the OIDC users that will
be passed by NiFi.

Best,
Kevin

On Thu, Oct 24, 2019 at 5:54 AM Pierre Villard
 wrote:
>
> Hi Ryan,
>
> NiFi nodes will use their own certificates as identities to authenticate 
> against the NiFi Registry and the NiFi nodes will then proxy the users 
> connected to the NiFi instances for the interactions with the registry. You 
> have to configure the NiFi node identities as well as where to get the 
> users/groups informations using the authorizers.xml file [1]. Once the 
> users/groups are known in the NiFi Registry you can define the authorizations 
> as you described for the users and groups and it will reflected for the 
> users/groups when they connect to NiFi. If, however, you want to allow users 
> to authenticate on the NiFi Registry UI (to create buckets for instance), 
> then you'd have to also configure the authentication parts on the Registry 
> [2] (note that OIDC is not supported yet [3]).
>
> Hope this helps a bit.
>
> [1] 
> https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#authorizers-setup
> [2] 
> https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#user_authentication
> [3] https://issues.apache.org/jira/browse/NIFIREG-313
>
> Le jeu. 24 oct. 2019 à 03:54, Ryan H  a 
> écrit :
>>
>> Hi All,
>>
>> We currently have a multi-node NiFi cluster (1.8.0) that is secured using 
>> the OIDC provider for authentication. We are setting up a secure NiFi 
>> Registry (0.5.0) which our secure NiFi cluster will connect to.
>>
>> What is the recommended way to connect the OIDC secured NiFi instance to the 
>> secure NiFi Registry (only option looks to be using certs since we are not 
>> using LDAP or Kerb)? I am assuming the only way is to do a cert import to 
>> NiFi which will then open up all buckets to the entire cluster (based on the 
>> permissions of the user tied to the certificate).
>>
>> We are operating in a multi-tenant environment and would like to achieve 
>> bucket level permissions for the various users of the system. Accessing the 
>> UI of the NiFi Registry instance isn't super important, except for maybe a 
>> couple users for which generating a couple certs isn't a big deal. However, 
>> allowing users to only access certain buckets may be important.
>>
>> For now just being able to get this hooked up is ideal. Thoughts?
>>
>>
>> Thanks in Advance,
>>
>> Ryan H.
>>

Re: Problem with Context Path Whitelisting

[Kevin Doran]

Swarup,

First, thanks for the great email. Nice job troubleshooting this and
sharing your findings with the community.

I'm more familiar with how these types of things get configured on
NiFi Registry than NiFi, so I'm not as much help as others. But I did
take a look and one thing I noticed was a difference between the
startup config and the per-request config.

On Startup, the whitelisted context paths are coming from the
ServletContext FilterConfig [1].

During request handling, the whitelisted context paths are coming from
the ApplicationContext, directly from NiFi Properties [2]

[1] 
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/filter/SanitizeContextPathFilter.java#L41
[2] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ApplicationResource.java#L165

Ultimately, my assumption is that both of these property values
*should* be backed by the same nifi.properties file. But it appears
something is happening in your case/environment/situation that is
causing the ServletContext and ApplicationContext to get
configured/initialized differently. This could be something specific
to your environment or it could be uncovering an edge-case bug in
NiFi.

I think others on this mailing list who are more familiar with how the
ServletContext gets setup in NiFi might be able to help further on
this and determine if there is a solution/workaround or bug that needs
patching.

Thanks,
Kevin

On Fri, Oct 11, 2019 at 4:55 AM Swarup Karavadi  wrote:
>
> Greetings,
>
> I have deployed a single node unsecured NiFi cluster (I say cluster because 
> nifi.cluster.is.node is set to "true") as a stateful set on Kubernetes (AWS 
> EKS to be specific). The NiFi cluster sits behind an Nginx ingress. I have 
> configured the Nginx ingress to forward the appropriate headers to NiFi (when 
> deployed behind a reverse proxy) as described in the documentation.
>
> The path on the Nginx ingress which proxies traffic to the NiFi UI is 
> "/pie/ip". This same path has been whitelisted by setting the 
> "nifi.web.proxy.context.path" property to "/pie/ip". The way I am expecting 
> this setup to work is that when users navigate to http://foo.com/pie/ip in 
> the browser, they are shown a simple HTML page with redirect info and then 
> automatically redirected to http://foo.com/pie/ip/nifi where they can view 
> the NiFi canvas. Instead, the users are being redirected to 
> http://foo.com/nifi which results in a 404 response because there is no 
> '/nifi' path that has been configured on the Nginx ingress.
>
> I set the NiFi and Jetty Server log levels to DEBUG to understand what was 
> happening under the hood and this is what I got -
>
> On Startup (when the SanitizeContextPathFilter is initialized) -
> 2019-10-11 06:07:26,206 DEBUG [main] o.a.n.w.filter.SanitizeContextPathFilter 
> SanitizeContextPathFilter received provided whitelisted context paths from 
> NiFi properties: /pie/ip
>
> On Request (when the actual request is made) -
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils Context path:
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils On the request, the following context paths 
> were parsed from headers:
>  X-ProxyContextPath: /pie/ip
> X-Forwarded-Context: null
> X-Forwarded-Prefix: null
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils Determined context path: /pie/ip
> 2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils The provided context path [/pie/ip] was not 
> whitelisted []
> 2019-10-11 06:45:45,556 ERROR [NiFi Web Server-23] 
> org.apache.nifi.web.util.WebUtils Error determining context path on JSP page: 
> The provided context path [/pie/ip] was not whitelisted []
> 2019-10-11 06:45:45,556 DEBUG [NiFi Web Server-23] 
> o.a.n.w.filter.SanitizeContextPathFilter SanitizeContextPathFilter set 
> contextPath:
>
> You will notice from the above log entries that the path '/pie/ip' was 
> successfully whitelisted. Yet, when handling the request, the whitelisted 
> context paths array is empty and this causes the wrong redirect to happen on 
> the browser - and I can't figure out why this is happening or how I can fix 
> it. Has anyone come across this kind of problem before? Any help on this is 
> much appreciated.
>
> Cheers,
> Swarup.

Re: In nifi-registry, why can't I edit other users privileges

[Kevin Doran]

Ouch. That is understandably frustrating, and can be improved. I’ll look into 
replacing that with a case-insensitive match, as well as logging warnings for 
unrecognized properties.

Sorry for the difficultly you experienced in getting this working. Thanks for 
sharing the resolution. Let me know if you have any other questions.



From: Nicolas Delsaux 
Sent: Thursday, September 5, 2019 2:47 AM
To: users@nifi.apache.org
Subject: Re: In nifi-registry, why can't I edit other users privileges

Well, in fact, I had a number of issues with configuration files.

So I took the time to verify all those files, and I took the time to
understand Nifi registry UI for permissions (which is as user-friendly
as nifi one). And I finally understood what problem I had.

In fact, the worst part came when I tried to understand why my nifi
runner couldn't connect to nifi registry.

Which was simply due to the fact that, on nifi registry side, in
authorizers.xml, I used a property called "Nifi identify 1", whereas I
should have used "NiFi Identity 1". Can you spot the difference ?

For me, it took one phase of reading authorization code, then running
the regexp for that property in an online editor.

TO my mind, this would deserve a bug, because really, using property
names this way is really too much error-prone.

I would at least add code to detect nearby texts (through Levenshtein
distance, as an example) and show a BIG warning to explain the user what
is wrong.

But I'm only a user ;-) (a little grumpy, this morning, indeed)

Le 04/09/2019 à 18:59, Kevin Doran a écrit :
> Hi Nicolas,
>
> Is it possible you changed the initial admin identity at some point?
> If so, you will need to delete authorizations.xml and restart NiFi
> Registry to allow it to be recreated with the new initial admin.
>
> Also, nifi registry never allows modifying the permissions for the
> current user. you would have to login as another admin to change your
> permissions.
>
> Hope this helps,
> Kevin
>
> On Mon, Sep 2, 2019 at 8:56 AM Nicolas Delsaux  wrote:
>> Hi all
>>
>> I'm still trying to connect nifi to registry with both of them using
>> authentication.
>>
>> So far, i've understood that, like in Nifi, I have to set
>> identity-providers.xml and authorizers.xml to have connection to ldap
>> configured.
>>
>> And I can connect to the registry using my ldap, so it works (to a
>> certain extend).
>>
>> *However*, it seems like my user is not really an admin, as I can't
>> manage other users.
>>
>> To say things more clearly, nifi-registry UI allow me to view my user
>> privileges, but I can't edit my permissions, and I can edit none of the
>> other users permissions. I can no more add/remove users.
>>
>> Which is weird, cosnidering I'm the initial admin of nifi-registry.
>>
>> Is there smothing I forgot ?
>>
>>
>> Here is my authorizers.xml for nifi-registry
>>
>> 
>>  
>> file-user-group-provider
>> org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
>> ./conf/users.xml
>> cn=nifi-runner.mycompany.com, ou=0008 43120727, ou=ssl infra server,
>> o=mycompany, c=fr
>> 
>> 
>> ldap-user-group-provider
>> org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider
>> LDAPS
>>
>> uid=dont-ask-me,ou=applicationAccounts,o=mycompany.com
>> YOU_KIDDIN___DO_YOU
>>
>> 
>> 
>> 
>> /opt/certs/cacerts.jks
>> pfeblelep
>> JKS
>> 
>> TLSv1
>> 
>>
>> FOLLOW
>> 10 secs
>> 10 secs
>>
>> > name="Url">ldaps://ldapserver.my.company.com:636
>> 
>> 30 mins
>>
>> 
>> 
>> OBJECT
>> 
>> 
>> 
>> 
>>
>> cn=NIFI-ADMIN,ou=DATAou=applicationRole,ou=role,ou=OU,o=mycompany.com
>> groupofuniquenames
>> SUBTREE
>> 
>> cn
>> uniqueMember
>> 
>> 
>> 
>> composite-user-group-provider
>> org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider
>> ldap-user-group-provider
>> file-user-group-provider
>> 
>> 
>> file-access-policy-provider
>> org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
>> composite-user-group-provider
>> ./conf/authorizations.xml
>> uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com
>> cn=nifi-psh.adeo.com, ou=0002
>> 421206079, ou=ssl infra server, o=adeo services, c=fr
>> 
>> 
>> managed-authorizer
>> org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer
>> file-access-policy-provider
>> 
>> 
>>
>> Thanks for your help
>>

Re: In nifi-registry, why can't I edit other users privileges

[Kevin Doran]

Hi Nicolas,

Is it possible you changed the initial admin identity at some point?
If so, you will need to delete authorizations.xml and restart NiFi
Registry to allow it to be recreated with the new initial admin.

Also, nifi registry never allows modifying the permissions for the
current user. you would have to login as another admin to change your
permissions.

Hope this helps,
Kevin

On Mon, Sep 2, 2019 at 8:56 AM Nicolas Delsaux  wrote:
>
> Hi all
>
> I'm still trying to connect nifi to registry with both of them using
> authentication.
>
> So far, i've understood that, like in Nifi, I have to set
> identity-providers.xml and authorizers.xml to have connection to ldap
> configured.
>
> And I can connect to the registry using my ldap, so it works (to a
> certain extend).
>
> *However*, it seems like my user is not really an admin, as I can't
> manage other users.
>
> To say things more clearly, nifi-registry UI allow me to view my user
> privileges, but I can't edit my permissions, and I can edit none of the
> other users permissions. I can no more add/remove users.
>
> Which is weird, cosnidering I'm the initial admin of nifi-registry.
>
> Is there smothing I forgot ?
>
>
> Here is my authorizers.xml for nifi-registry
>
> 
> 
> file-user-group-provider
> org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider
>  ./conf/users.xml
>  cn=nifi-runner.mycompany.com, ou=0008 43120727, ou=ssl infra server,
> o=mycompany, c=fr
>  
>  
> ldap-user-group-provider
> org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider
>  LDAPS
>
>  uid=dont-ask-me,ou=applicationAccounts,o=mycompany.com
>  YOU_KIDDIN___DO_YOU
>
>  
>  
>  
>  /opt/certs/cacerts.jks
>  pfeblelep
>  JKS
>  
>  TLSv1
>  
>
>  FOLLOW
>  10 secs
>  10 secs
>
>   name="Url">ldaps://ldapserver.my.company.com:636
>  
>  30 mins
>
>  
>  
>  OBJECT
>  
>  
>  
>  
>
>  cn=NIFI-ADMIN,ou=DATAou=applicationRole,ou=role,ou=OU,o=mycompany.com
>  groupofuniquenames
>  SUBTREE
>  
>  cn
>  uniqueMember
>  
>  
>  
> composite-user-group-provider
> org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider
>  ldap-user-group-provider
>  file-user-group-provider
>  
>  
> file-access-policy-provider
> org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider
>  composite-user-group-provider
>  ./conf/authorizations.xml
>  uid=20008203,ou=people,ou=go-lm,o=corp.leroymerlin.com
>  cn=nifi-psh.adeo.com, ou=0002
> 421206079, ou=ssl infra server, o=adeo services, c=fr
>  
>  
>  managed-authorizer
> org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer
>  file-access-policy-provider
>  
> 
>
> Thanks for your help
>

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

[Kevin Doran]

My apologies, the link I sent in my last message [1] may not be
accessible unless you are signed into Jira. If you cannot view that
link, see [2].
[2] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12345213

On Tue, Apr 2, 2019 at 1:18 PM Kevin Doran  wrote:
>
> For what it's worth, v1.9.2 is going to include a number of bug fixes
> in addition to NIFI-6150. While the other bugs are not nearly as
> critical (depending on your usage of NiFi), all the more the reason to
> upgrade to 1.9.2 rather than patching a 1.9.1 install. For a full list
> of what's been included in the RC for 1.9.2 so far, see [1]:
>
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12345213
>
> Cheers,
> Kevin
>
> On Tue, Apr 2, 2019 at 1:04 PM William Gosse
>  wrote:
> >
> > "Although I'm not sure why that would be preferred over getting the 
> > officially released fix in 1.9.2" - I don't prefer applying a patch but I 
> > do have some systems already setup with 1.9.1. I'll probably bite the 
> > bullet and moved to 1.9.2 as soon as its available.  We are in the process 
> > of containerizing nifi with docker so hopefully  that will make moving to a 
> > newer release not a big deal.
> >
> > -Original Message-
> > From: Bryan Bende 
> > Sent: Tuesday, April 2, 2019 12:25 PM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi 1.9.1 release contains a bug causing content repos to 
> > fill...
> >
> > [CAUTION: This email originated from outside of Kodak Alaris. Do not click 
> > links or open attachments unless you recognize the sender and know the 
> > content is safe.] 
> >
> > The associated JIRA is NIFI-6150 [1] and it leads to the content repo not 
> > properly cleaning up and eventually filling up disks.
> >
> > There will be a 1.9.2 release to address this. Of course you are free to 
> > take the patch for the fix [2], apply it to the 1.9.1 release tag [3], and 
> > build your own fix, although I'm not sure why that would be preferred over 
> > getting the officially released fix in 1.9.2.
> >
> > [1] 
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__issues.apache.org_jira_browse_NIFI-2D6150%26d%3DDwIFaQ%26c%3DskUQKLuTFULvQUMqV0uyBaBvPxsA7Wk8MKjvMTLjY3w%26r%3DWm89MSTpkAcFFP1M0uvGefI8slyO_VIaIza2IJwAlsU%26m%3DIcEOLvLqMRwVGHZrj8MeEuGvHHj6WxiOyETFetIDgS8%26s%3D26O4gEjALxwRdlVmLjatxiCTiApEi5UN76no-9D--yQ%26e&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866&sdata=emHvuWSerL0yVKRSam5metw3KBFYCphFpI%2BlHEpfmOc%3D&reserved=0=
> > [2] 
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__patch-2Ddiff.githubusercontent.com_raw_apache_nifi_pull_3391.patch%26d%3DDwIFaQ%26c%3DskUQKLuTFULvQUMqV0uyBaBvPxsA7Wk8MKjvMTLjY3w%26r%3DWm89MSTpkAcFFP1M0uvGefI8slyO_VIaIza2IJwAlsU%26m%3DIcEOLvLqMRwVGHZrj8MeEuGvHHj6WxiOyETFetIDgS8%26s%3D2_4CR7XulBNmfR2MlPlDcdTgKD6NIiP6S_u8YG17qO0%26e&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866&sdata=dFHMeTTGT%2FwzbfoPBizqRUJ1dMXhoGS2Q1d%2F3v%2FOBx8%3D&reserved=0=
> > [3] 
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__github.com_apache_nifi_tree_rel_nifi-2D1.9.1%26d%3DDwIFaQ%26c%3DskUQKLuTFULvQUMqV0uyBaBvPxsA7Wk8MKjvMTLjY3w%26r%3DWm89MSTpkAcFFP1M0uvGefI8slyO_VIaIza2IJwAlsU%26m%3DIcEOLvLqMRwVGHZrj8MeEuGvHHj6WxiOyETFetIDgS8%26s%3DVP_sGtxIOZIh_pm1wipeONZ6l1i3G5epzYUT_J_K2dk%26e&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866&sdata=9B2wS%2FXeHsS9U02A72CKE4MWcr9mQXXY2LIGnFD8ME0%3D&reserved=0=
> >
> > On Tue, Apr 2, 2019 at 12:17 PM William Gosse  
> > wrote:
> > >
> > > Also can someone explain more what this issue is and how serious it is.
> > >
> > > Can a patch be made available to fix this issue for  existing 1.9.1 
> > > installations?
> > >
> > >
> > >
> > > From: Joe Witt 
> > > Sent: Saturday, March 30, 2019 12:32 PM
> > > To: users@nifi.apache.org
> > > Subject: Re: NiFi 1.9.1 release contains a bug causing content repos to 
> > > fill...
> > >
> > >
> > >
> > > [CAUTION: This email originated from outside of Kodak Alaris. Do not
&

Re: NiFi 1.9.1 release contains a bug causing content repos to fill...

[Kevin Doran]

For what it's worth, v1.9.2 is going to include a number of bug fixes
in addition to NIFI-6150. While the other bugs are not nearly as
critical (depending on your usage of NiFi), all the more the reason to
upgrade to 1.9.2 rather than patching a 1.9.1 install. For a full list
of what's been included in the RC for 1.9.2 so far, see [1]:

[1] https://issues.apache.org/jira/projects/NIFI/versions/12345213

Cheers,
Kevin

On Tue, Apr 2, 2019 at 1:04 PM William Gosse
 wrote:
>
> "Although I'm not sure why that would be preferred over getting the 
> officially released fix in 1.9.2" - I don't prefer applying a patch but I do 
> have some systems already setup with 1.9.1. I'll probably bite the bullet and 
> moved to 1.9.2 as soon as its available.  We are in the process of 
> containerizing nifi with docker so hopefully  that will make moving to a 
> newer release not a big deal.
>
> -Original Message-
> From: Bryan Bende 
> Sent: Tuesday, April 2, 2019 12:25 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi 1.9.1 release contains a bug causing content repos to 
> fill...
>
> [CAUTION: This email originated from outside of Kodak Alaris. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.] 
>
> The associated JIRA is NIFI-6150 [1] and it leads to the content repo not 
> properly cleaning up and eventually filling up disks.
>
> There will be a 1.9.2 release to address this. Of course you are free to take 
> the patch for the fix [2], apply it to the 1.9.1 release tag [3], and build 
> your own fix, although I'm not sure why that would be preferred over getting 
> the officially released fix in 1.9.2.
>
> [1] 
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__issues.apache.org_jira_browse_NIFI-2D6150%26d%3DDwIFaQ%26c%3DskUQKLuTFULvQUMqV0uyBaBvPxsA7Wk8MKjvMTLjY3w%26r%3DWm89MSTpkAcFFP1M0uvGefI8slyO_VIaIza2IJwAlsU%26m%3DIcEOLvLqMRwVGHZrj8MeEuGvHHj6WxiOyETFetIDgS8%26s%3D26O4gEjALxwRdlVmLjatxiCTiApEi5UN76no-9D--yQ%26e&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866&sdata=emHvuWSerL0yVKRSam5metw3KBFYCphFpI%2BlHEpfmOc%3D&reserved=0=
> [2] 
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__patch-2Ddiff.githubusercontent.com_raw_apache_nifi_pull_3391.patch%26d%3DDwIFaQ%26c%3DskUQKLuTFULvQUMqV0uyBaBvPxsA7Wk8MKjvMTLjY3w%26r%3DWm89MSTpkAcFFP1M0uvGefI8slyO_VIaIza2IJwAlsU%26m%3DIcEOLvLqMRwVGHZrj8MeEuGvHHj6WxiOyETFetIDgS8%26s%3D2_4CR7XulBNmfR2MlPlDcdTgKD6NIiP6S_u8YG17qO0%26e&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866&sdata=dFHMeTTGT%2FwzbfoPBizqRUJ1dMXhoGS2Q1d%2F3v%2FOBx8%3D&reserved=0=
> [3] 
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__github.com_apache_nifi_tree_rel_nifi-2D1.9.1%26d%3DDwIFaQ%26c%3DskUQKLuTFULvQUMqV0uyBaBvPxsA7Wk8MKjvMTLjY3w%26r%3DWm89MSTpkAcFFP1M0uvGefI8slyO_VIaIza2IJwAlsU%26m%3DIcEOLvLqMRwVGHZrj8MeEuGvHHj6WxiOyETFetIDgS8%26s%3DVP_sGtxIOZIh_pm1wipeONZ6l1i3G5epzYUT_J_K2dk%26e&data=02%7C01%7Cwilliam.gosse%40aifoundry.com%7C65f8067fa3cf41f6eb6e08d6b78b241d%7Cd29b7a9b6edb472099a83c5c6c3eeeb0%7C0%7C0%7C636898205580491866&sdata=9B2wS%2FXeHsS9U02A72CKE4MWcr9mQXXY2LIGnFD8ME0%3D&reserved=0=
>
> On Tue, Apr 2, 2019 at 12:17 PM William Gosse  
> wrote:
> >
> > Also can someone explain more what this issue is and how serious it is.
> >
> > Can a patch be made available to fix this issue for  existing 1.9.1 
> > installations?
> >
> >
> >
> > From: Joe Witt 
> > Sent: Saturday, March 30, 2019 12:32 PM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi 1.9.1 release contains a bug causing content repos to 
> > fill...
> >
> >
> >
> > [CAUTION: This email originated from outside of Kodak Alaris. Do not
> > click links or open attachments unless you recognize the sender and
> > know the content is safe.]
> >
> > 
> >
> > mathieu
> >
> >
> >
> > just a matter of finding the chunk of time to knock it out.  perhaps this 
> > evening.
> >
> >
> >
> > thanks
> >
> > On Sat, Mar 30, 2019, 11:27 AM Louis-Ferdinand, Mathieu 
> >  wrote:
> >
> > Hi Joe,
> >
> >
> >
> > Do you have an idea of the timeline for 1.9.2 RC?
> >
> >
> >
> > Cheers,
> >
> > Mathieu.
> >
> >
> >
> > From: Joe Witt [mailto:joe.w...@gmail.com]
> > Sent: Wednesday, March 27, 2019 11:00 AM
> > To: users@nifi.apache.org
> > Subject: NiFi 1.9.1 release contains a bug causing content repos to fill...
> >
> >
> >
> > All,
> >
> >
> >
> > We will work to promptly produce an Apache NiFi 1.9.2 release which will 
> > correct a regression introduced in the 1.9.1 release.  The issue, NIFI-6150 
> > has been identified and fixed on master a

Re: NiFi registry - encrypt providers.xml?

[Kevin Doran]

Terrific, glad it is working for you!



From: David Gallagher 
Sent: Thursday, March 28, 2019 5:26 PM
To: users@nifi.apache.org
Subject: RE: NiFi registry - encrypt providers.xml?

Working and awesome, as usual with NiFi! Thanks!

-Original Message-
From: Kevin Doran 
Sent: Thursday, March 28, 2019 4:00 PM
To: users@nifi.apache.org
Subject: Re: NiFi registry - encrypt providers.xml?

Yeah sorry, I should have specified that your ssh key needs to be password-less 
for this to work (your suspicion was correct!). So we recommend doing that and 
securing the key using OS filesystem permissions (ie, only the run as user 
should have read access to that ssk key file)

On Thu, Mar 28, 2019 at 3:54 PM David Gallagher  
wrote:
>
> Thanks for the reply! I've set up ssh access to the repository, and
> I've confirmed that it works by pushing commits on the terminal.
> However, nifi-registry isn't performing the push automatically; the
> log file indicates
>
> 2019-03-28 15:48:19,029 ERROR [GitFlowMetaData Push thread]
> o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin
> due to org.eclipse.jgit.api.errors.TransportException:
> ssh://git@.git: USERAUTH fail
>
> As per the instructions, I created a file at ${USER_HOME}/.ssh/config with a 
> Host entry and pointer to the key file, where ${USER} is the user running the 
> nifi registry service. Is there some additional setting that I need to make 
> in providers.xml? One thing, there's a passphrase associated with my key, 
> could that be causing the issue?
>
> Thanks,
>
> Dave
>
>
>
> -Original Message-
> From: Kevin Doran 
> Sent: Thursday, March 28, 2019 12:20 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi registry - encrypt providers.xml?
>
> This is not possible at this time. We recommend configuring your git repo so 
> that the remote and push using ssh key pairs of the NiFi Registry run-as user 
> and protecting the private key on disk. in other words, if on the linux host 
> as the user that nifi registry runs as, you should be able to push using 
> ssh+git instead of username / password. then NiFi Registry will be able to do 
> the same.
>
> On Thu, Mar 28, 2019 at 12:09 PM David Gallagher 
>  wrote:
> >
> > Hi – does anyone know if it is possible to somehow encrypt the 
> > providers.xml file? I’m using the Git persistence provider and want to 
> > store the remote user and password there, but I don’t want to leave that 
> > value in there unencrypted if I don’t have to. I downloaded the 
> > nifi-toolkit, but I don’t see it as an option.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Dave

Re: NiFi registry - encrypt providers.xml?

[Kevin Doran]

Yeah sorry, I should have specified that your ssh key needs to be
password-less for this to work (your suspicion was correct!). So we
recommend doing that and securing the key using OS filesystem
permissions (ie, only the run as user should have read access to that
ssk key file)

On Thu, Mar 28, 2019 at 3:54 PM David Gallagher
 wrote:
>
> Thanks for the reply! I've set up ssh access to the repository, and I've 
> confirmed that it works by pushing commits on the terminal. However, 
> nifi-registry isn't performing the push automatically; the log file indicates
>
> 2019-03-28 15:48:19,029 ERROR [GitFlowMetaData Push thread] 
> o.a.n.r.p.flow.git.GitFlowMetaData Failed to push commits to origin due to 
> org.eclipse.jgit.api.errors.TransportException: ssh://git@.git: 
> USERAUTH fail
>
> As per the instructions, I created a file at ${USER_HOME}/.ssh/config with a 
> Host entry and pointer to the key file, where ${USER} is the user running the 
> nifi registry service. Is there some additional setting that I need to make 
> in providers.xml? One thing, there's a passphrase associated with my key, 
> could that be causing the issue?
>
> Thanks,
>
> Dave
>
>
>
> -Original Message-
> From: Kevin Doran 
> Sent: Thursday, March 28, 2019 12:20 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi registry - encrypt providers.xml?
>
> This is not possible at this time. We recommend configuring your git repo so 
> that the remote and push using ssh key pairs of the NiFi Registry run-as user 
> and protecting the private key on disk. in other words, if on the linux host 
> as the user that nifi registry runs as, you should be able to push using 
> ssh+git instead of username / password. then NiFi Registry will be able to do 
> the same.
>
> On Thu, Mar 28, 2019 at 12:09 PM David Gallagher 
>  wrote:
> >
> > Hi – does anyone know if it is possible to somehow encrypt the 
> > providers.xml file? I’m using the Git persistence provider and want to 
> > store the remote user and password there, but I don’t want to leave that 
> > value in there unencrypted if I don’t have to. I downloaded the 
> > nifi-toolkit, but I don’t see it as an option.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Dave

Re: NiFi registry - encrypt providers.xml?

[Kevin Doran]

This is not possible at this time. We recommend configuring your git
repo so that the remote and push using ssh key pairs of the NiFi
Registry run-as user and protecting the private key on disk. in other
words, if on the linux host as the user that nifi registry runs as,
you should be able to push using ssh+git instead of username /
password. then NiFi Registry will be able to do the same.

On Thu, Mar 28, 2019 at 12:09 PM David Gallagher
 wrote:
>
> Hi – does anyone know if it is possible to somehow encrypt the providers.xml 
> file? I’m using the Git persistence provider and want to store the remote 
> user and password there, but I don’t want to leave that value in there 
> unencrypted if I don’t have to. I downloaded the nifi-toolkit, but I don’t 
> see it as an option.
>
>
>
> Thanks,
>
>
>
> Dave

Re: LDAP User Authentication Issue - Unable to validate supplied credentials

[Kevin Doran]

Hello,

The IdentityAccessException indicates that the NiFi server did successfully
receive and recognize LDAP username and password credentials in the
request, but was unable to communicate with the LDAP server in order to
authenticate them. The nested exception "socket closed" does not give a lot
of information. It could be that the protocol/host/port set in the "Url"
property are incorrect, or that NiFi could not authenticate to the LDAP
server as the LDAP manager/admin service account.

NiFi LDAP integration supports several types of authentication to the LDAP
server:

   - ANONYMOUS - if the LDAP server allows access to any client without
   credentials (probably not the case)
   - SIMPLE - Manager DN and Manager Password are used to authentication
   - LDAPS / START_TLS - Both use a private key to authenticate by doing a
   ssl/tls handshake that authenticates both the client and server to each
   other - similar to the way HTTPS with TLS Mutual Auth works. I believe
   LDAPS is an older protocol (though still used by some LDAP servers) and
   START_TLS is its replacement.

You must find out from the LDAP server admin which one is used, and
configure this in NiFi using the "Authentication Strategy" property. If
using ANONYMOUS, no additional properties are required. If using SIMPLE,
you must set the Manager DN and Manager Password. If using LDAPS or
START_TLS, you must set the TLS - * properties to specify the keystore and
truststore details so that the client can successfully perform the key
exchange handshake with the server. These keys or stores are usually
provided to you but the LDAP server admin/operator.

Note the LDAP Url must start with the correct protocol (ldap:// or
ldaps://) depending on what the LDAP server supports.

I hope this helps! Please reach back out if you are still stuck or need
help after getting past this issue. Once login authentication is working,
you typically would want to configure your authorizers.xml file with
similar LDAP settings so that you can sync users and groups in order to
define access policies for authenticated users. (In other words,
login-identity-providers.xml configures LDAP authentication, and
authorizers.xml configures authorization).

Regards,
Kevin

On Mon, Mar 25, 2019 at 9:46 AM  wrote:

> Hi,
>
>
>
>
>
> I would like to state that I am not NiFi-savvy or LDAP-savvy by any means.
> I am having issues setting up my local NiFi instance with an LDAP provided
> to me (I can ensure you that there are no issues with the LDAP as I use it
> for other services). The outline of my issue is stated below:
>
> *Use Case*:
>
> Force users to login before accessing the site (if they have permissions).
> Users should be authenticated through the LDAP that the NiFi is setup to
> point to.
>
>
>
> *Setup*:
>
> 1.  I’ve setup a local instance (localhost:8080/nifi) of NiFi 1.9.0
> and have no issues accessing the GUI. (I am running on windows)
>
>
>
> 2.  I followed an article (
> https://community.hortonworks.com/articles/58233/using-the-tls-toolkit-to-simplify-security.html)
> to get HTTPS setup on my local instance and the next necessary steps (setup
> Initial Admin) to have a working NiFi instance with an admin user. It
> worked and I had no issues using the instance with my Admin user.
>
>
>
> 3.  I followed Peirre Villard’s article (
> https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/) to
> get NiFi setup with a *local LDAP server*, which worked fine.
>
>
>
> 4.  I tried getting NiFi setup with an LDAP I’ve been provided (the
> ldap address and specific search base filters) using an *Anonymous 
> *authentication
> strategy (I’ve attempted with SIMPLE and ran into other issues, but cannot
> confirm that the user I used had User Authentication permissions), but
> receive an error when trying to login into a user.
>
>
>
> *NOTE: *When I am using *Anonymous* authentication strategy I have no
> Manager DN set (could this be the issue?)
>
>
> *Error*:
> Caused by:
> org.apache.nifi.authentication.exception.IdentityAccessException: Unable to
> validate the supplied credentials. Please contact the system administrator.
>
>at
> org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:340)
>
>at
> org.apache.nifi.web.security.spring.LoginIdentityProviderFactoryBean$1.authenticate(LoginIdentityProviderFactoryBean.java:315)
>
>at
> org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:728)
>
>... 83 common frames omitted
>
> Caused by:
> org.springframework.security.authentication.InternalAuthenticationServiceException:
> (*ADDRESS*:*PORT)*; socket closed; nested exception is
> javax.naming.ServiceUnavailableException: (*ADDRESS*:*PORT)*; socket
> closed
>
>at
> org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:206)
>
>at
> org.springframework

Re: Empty "nifi users" page.

[Kevin Doran]

How many users and groups do you expect to sync? Are you able to test
your search base and filter are correct using a command line tool such
as ldapsearch? I'm not sure if a search filter is required (although
it would probably fail with an error if it was), but maybe try setting
a "match all" filter such as (uid=*).

Hope this helps,
Kevin

On Fri, Mar 22, 2019 at 11:08 AM DEHAY Aurelien
 wrote:
>
> Hello.
>
>
>
> My “nifi users” page is empty, with only the spinner spinning at the bottom 
> of the page.
>
>
>
> I’ve tried a lot of conf, and for now everything work fine, except that. 
> There is nothing in the log, even when I tried to put “debug” for 
> authorization parts in logback.
>
>
>
> I had the same behavior with 1.7, 1.8, 1.9 and 1.9.1.
>
>
>
> My configuration authorizers.xml and the part of user mapping in the 
> nifi.properties is 
> https://gist.github.com/zorel/f76c3924f2b266e8a878397271a54f24
>
>
>
> I’ve read Pierre Villard blog post, nifi doc and a lot of post in hortonworks 
> site, but I’ve must I missed something. Any pointer is appreciated.
>
>
>
> Thanks.
>
>
> This electronic transmission (and any attachments thereto) is intended solely 
> for the use of the addressee(s). It may contain confidential or legally 
> privileged information. If you are not the intended recipient of this 
> message, you must delete it immediately and notify the sender. Any 
> unauthorized use or disclosure of this message is strictly prohibited.  
> Faurecia does not guarantee the integrity of this transmission and shall 
> therefore never be liable if the message is altered or falsified nor for any 
> virus, interception or damage to your system.

Re: Problems with NiFi Registry Conflicts after Processor Upgrades

[Kevin Doran]

Chad,

I wanted to echo Bryan and say thanks for sharing the details of an
upgrade process that works. Until we have improved NiFi to handle the
upgrades regardless of order of steps, this is a helpful reference for
the community for a sequence that can work.

Cheers,
Kevin

On Tue, Mar 19, 2019 at 11:40 AM Bryan Bende  wrote:
>
> Chad,
>
> Thanks for reporting your experience and glad to hear that there is a
> good process to follow.
>
> We do want to make this situation better and there is a JIRA to track
> the issue [1].
>
> Thanks,
>
> Bryan
>
> [1] https://issues.apache.org/jira/browse/NIFI-6028
>
>
>
> On Tue, Mar 19, 2019 at 11:23 AM Chad Woodhead  wrote:
> >
> > Hey guys,
> >
> > I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was 
> > important to me. I've done some testing and wanted to share my results.
> >
> > Test environment setup:
> > 2 NiFi Clusters
> > 1 NiFi Registry
> >
> > Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
> > Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
> >
> > Working upgrade method:
> > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
> > 2. All Dev NiFi versioned flows showed local changes due to new properties 
> > in processors
> > 3. Commit all Dev NiFi changes for all versioned process groups
> > 4. In NiFi Registry, view and see all committed changes from Dev NiFi
> > 5. On Cert NiFi, all versioned PGs show newer version available.
> > 6. On Cert NiFi, change all versioned process groups to use new version. 
> > PGs now show "Flow version is current" (Note: the processors DO NOT become 
> > invalid and they do not show the new properties yet)
> > 7. Upgrade Cert NiFi from 1.8.0 to 1.9.1
> > 8. On Cert NiFi, all versioned process groups still show they are on the 
> > latest version and the processors DO show the new properties now
> >
> > Not working upgrade method:
> > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
> > 2. All Dev NiFi versioned flows showed local changes due to new properties 
> > in processors
> > 3. Commit all Dev NiFi changes to all versioned process groups
> > 4. In NiFi Registry, view and see all committed changes from Dev NiFi
> > 5. Upgrade Cert NiFi from 1.8.0 to 1.9.1
> > 6. On Cert NiFi, versioned process groups show "Local changes have been 
> > made and a newer version of this flow is available". When clicking on PG 
> > and selecting "Version", only options are "Show local changes", "Revert 
> > local changes", and "Stop version control".
> > "Show local changes" allows you to see what has changed (all new properties 
> > in processors).
> > "Revert local changes" does nothing as these changes are required since 
> > they are new properties from the upgrade.
> > 7. My only 2 options at this point are
> > -Stop version control of the PG and start it back up, causing me to lose 
> > all history
> > -Delete the PG on Cert and then re-import from NiFi Registry. This option 
> > really isn't an option when in Prod as I don't want to stop/delete 
> > production flows that need to keep ingesting data.
> >
> > So as shown above, when upgrading NiFi with versioned flows in NiFi 
> > Registry, the steps are very important and as long pull in the latest 
> > commit to Cert before upgrading Cert, your process groups will work as 
> > expected.
> >
> > Thanks,
> > Chad
> >
> >>
> >>
> >>
> >> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)"  wrote:
> >>
> >> *External Message* - Use caution before opening links or attachments
> >>
> >> Bryan,
> >>
> >> I agree, that is probably a solution. Unfortunately, there is no mass 
> >> upgrade option, so we'd have to manually touch ever versioned process 
> >> group (or scripted it).
> >>
> >> Thanks,
> >>   Peter
> >>
> >> -Original Message-
> >> From: Bryan Bende [mailto:bbe...@gmail.com]
> >> Sent: Thursday, November 29, 2018 1:29 PM
> >> To: users@nifi.apache.org
> >> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after 
> >> Processor Upgrades
> >>
> >> Peter,
> >>
> >> I feel like this came up before, and unfortunately I'm not sure there 
> >> is currently a solution.
> >>
> >> I think ultimately there needs to be some kind of force upgrade so you 
> >> can ignore the local changes and take whatever is available.
> >>
> >> The only thing I can think of, but haven't tried, is if you had 
> >> upgraded the PG in the second instance before upgrading NiFi itself, it 
> >> would bring in the new properties that are not valid in that version and 
> >> the processor would show as invalid, then upgrade NiFi and it would be 
> >> valid again.
> >>
> >> -Bryan
> >> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) 
> >>  wrote:
> >> >
> >> > Ran into a NiFi Registry issue while upgrading our instances to NiFi 
> >> 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so 
> >> after upgrading our, our versioned processor groups show as having local 
> >> changes, which 

Re: How to integrate Secured Ragistry with Secured Nifi

[Kevin Doran]

Hi Tom,

If you ping nifi-registry from nifi using "localhost", then localhost is the 
the hostname you should use when adding the Registry client to NiFi.

Let me know if this does not work for you.

Kevin


On February 19, 2019 at 03:33:51, Tomislav Novosel (to.novo...@gmail.com) wrote:
> Hi again,
> 
> @Kevin, I chacked all you mentioned, containers are on the same docker
> network and I can ping from the Nifi container host where is Nifi registry,
> although they are on the
> same server, it is then localhost I believe.
> When adding nifi registry client I'm using hostname from the
> nifi-registry.properties whisch is the same as hostname where the registry
> and nifi instance are (because I added flag --hostname
> while running containers and hostname is the same as the server hostname
> where the containers are running).
> 
> I also checked yours and Mike's docker-compose files and everything is
> preety much the same regarding, hosts and ports.
> 
> BR,
> Tom
> 
> On Tue, 19 Feb 2019 at 06:51, Tomislav Novosel wrote:
> 
> > Hi Mike, Kevin
> >
> > Thank you for your answers, I appreciate it.
> >
> > @Mike, why are you setting WEB_HTTP_HOST and WEB_HTTP_PORT when you are
> > using secured nifi? Shouldnt that be empty and only HTTPS host and port
> > used?
> >
> > BR,
> > Tom
> >
> > On Mon, 18 Feb 2019, 23:56 Mike Thomsen > >
> >> Tom,
> >>
> >> > Note: both Registry and Nifi are in Docker containers on the same node.
> >> I tried with IP address, but nothing.
> >>
> >> Each docker container has its own IP address. You need to link the two
> >> containers. I always use Docker Compose, so I can't help you on how to set
> >> it up manually. That said, I did a sample last year connecting a few
> >> different NiFi nodes and a registry w/ SSL here:
> >>
> >>
> >> https://github.com/MikeThomsen/nifi-docker-compose/blob/master/docker-compose-registry.yml
> >>  
> >>
> >> I can't remember if I kept the LDAP docker container referenced in it,
> >> but you should be able to look at it and figure out how to link everything
> >> up from that with Docker Compose.
> >>
> >> Mike
> >>
> >> On Mon, Feb 18, 2019 at 12:00 PM Kevin Doran wrote:
> >>
> >>>
> >>> Hi Tom,
> >>>
> >>> Given that you are getting a Connection refused exception and not an
> >>> HTTP 401 or 403, I suspect that the problem is networking related and not
> >>> authentication/authorization.
> >>>
> >>> Are the two docker containers on the same docker network? Can you
> >>> resolve/ping the Registry container from the NiFi container, and when you
> >>> create the Registry client in NiFi, are you using the hostname that the
> >>> NiFi server/container would use to address Registry (ie, not the host a
> >>> REgistry UI use might use if you are using port mapping to the docker
> >>> container with the host).
> >>>
> >>> Here is an example repo in which I have an example of connecting NiFi
> >>> and Registry and docker conatiners using docker-compose:
> >>>
> >>> https://github.com/kevdoran/flowops
> >>>
> >>> Hope this helps,
> >>> Kevin
> >>>
> >>>
> >>> On February 18, 2019 at 10:08:54, Tomislav Novosel (to.novo...@gmail.com)
> >>> wrote:
> >>> > Hi all,
> >>> >
> >>> > I generated standalone certificate with nifi-toolkit for my two Nifi
> >>> > instances and for Nifi registry instance. All are on the same domain
> >>> so I
> >>> > used one certificate and its credentials for properties file (trustore
> >>> > path, keystore path, keystore passw, trustore passw).
> >>> >
> >>> > Auth is configured through domain LDAP server and everything works.
> >>> >
> >>> > On both Nifi node and Registry node I configured authorizers.xml file
> >>> on
> >>> > property "Node Identity 1" with value from keystore.jks on "Owner"
> >>> > attribute.
> >>> >
> >>> > Owner: <>
> >>> > In Nifi registry I added that as user and gave rights to read and
> >>> modify
> >>> > buckets.
> >>> >
> >>> > When I add Registry Client on Nifi node and Hit Start version control
> >>> on
> >>> > process group I got error:
> >>> >
> >>> > Error
> >>> >
> >>> > Unable to obtain listing of buckets: java.net.ConnectException:
> >>> Connection
> >>> > refused (Connection refused)
> >>> >
> >>> >
> >>> > I missed something in configuration, please help me.
> >>> >
> >>> > Note: both Registry and Nifi are in Docker containers on the same
> >>> node. I
> >>> > tried with IP address, but nothing.
> >>> >
> >>> >
> >>> > Thank you,
> >>> >
> >>> > Tom
> >>> >
> >>>
> >>>
> 

Re: How to integrate Secured Ragistry with Secured Nifi

[Kevin Doran]

Hi Tom,

Given that you are getting a Connection refused exception and not an HTTP 401 
or 403, I suspect that the problem is networking related and not 
authentication/authorization.

Are the two docker containers on the same docker network? Can you resolve/ping 
the Registry container from the NiFi container, and when you create the 
Registry client in NiFi, are you using the hostname that the NiFi 
server/container would use to address Registry (ie, not the host a REgistry UI 
use might use if you are using port mapping to the docker container with the 
host).

Here is an example repo in which I have an example of connecting NiFi and 
Registry and docker conatiners using docker-compose:

https://github.com/kevdoran/flowops

Hope this helps,
Kevin


On February 18, 2019 at 10:08:54, Tomislav Novosel (to.novo...@gmail.com) wrote:
> Hi all,
> 
> I generated standalone certificate with nifi-toolkit for my two Nifi
> instances and for Nifi registry instance. All are on the same domain so I
> used one certificate and its credentials for properties file (trustore
> path, keystore path, keystore passw, trustore passw).
> 
> Auth is configured through domain LDAP server and everything works.
> 
> On both Nifi node and Registry node I configured authorizers.xml file on
> property "Node Identity 1" with value from keystore.jks on "Owner"
> attribute.
> 
> Owner: <> 
> In Nifi registry I added that as user and gave rights to read and modify
> buckets.
> 
> When I add Registry Client on Nifi node and Hit Start version control on
> process group I got error:
> 
> Error
> 
> Unable to obtain listing of buckets: java.net.ConnectException: Connection
> refused (Connection refused)
> 
> 
> I missed something in configuration, please help me.
> 
> Note: both Registry and Nifi are in Docker containers on the same node. I
> tried with IP address, but nothing.
> 
> 
> Thank you,
> 
> Tom
> 

Re: Automate NiFi Ranger Policies

[Kevin Doran]

That's a good point, Chad.

I don't know if there might be an easy way to get that info.

The only thing I can think of is that when running with Apache Ranger,
you have the option
to enable Audit Log in which NiFi reports to Ranger any time an access
policy is checked for authorization.

I'm not sure if it would have the level of detail you need, but
assuming it does, those audit logs should be accessible via the Ranger
REST API, so when creating the policies for the Application PG, you
could also check for a WRITE to the parent Ingest Channel PG, and try
to correlate it to the creation of the Application PG based on a close
timestamp? It seems a bit fiddly and hard to code up, but it could
work...

Others might chime in if they know of an easier way to get the
creator/owner of a PG or component in general. We've talk a bit in the
past about a new flow storage model that would store individual
changes in sequence (to enable undo/redo on the canvas UI), and in
that model we would have the author of each change. So this use case
could be an additional motivation to work towards such a model. In the
meantime, Ranger audit logs would be what I would look at.

Regards,
Kevin


On February 15, 2019 at 14:35:01, Chad Woodhead (chadwoodh...@gmail.com) wrote:
> Kevin,
>
> Thanks for the high level thought process. Seems like a feasible solution.
> Do you know if I would be able to get the user who created the "Application
> PG" to add them to the Ranger policy so they don't lose access to their own
> application? Does NiFi keep that information?
>
> Thanks,
> Chad
>
> On Fri, Feb 15, 2019 at 2:02 PM Kevin Doran wrote:
>
> > Hi Chad,
> >
> > I've never done this, but if I were to go about it I would create a
> > script / cron job to poll the NiFi REST API [1] periodically, and upon
> > detection of a new "Application PG", create the corresponding policies
> > in Ranger via its REST API [2].
> >
> > You'll have to create service accounts in both NiFI and Ranger for
> > this script to run as and authenticate to both REST APIs, so you'll
> > need a secure server to run it from that (1) has access to both NiFi
> > and Ranger services and (2) has a way of restricting access to that
> > management server, or at least the service account credentials that
> > are stored on the server. And of course I would take care when
> > automating access policy creation in any service!
> >
> > Tools like NiPyAPI [3][4] can help with scripting access to the NiFi
> > REST API to poll for process groups. I'm not sure if a similar tool
> > exists on the Ranger side, although they do have a published Swagger
> > spec of their REST API [5][6], so generating something similar to
> > NiPyApi using swagger-codegen [7] might be possible. Then again you
> > only need to authenticate and access a few endpoints, so any scripting
> > language with a decent HTTP client library should be sufficient for
> > this type of thing.
> >
> > [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> > [2] https://ranger.apache.org/apidocs/index.html
> > [3] https://pypi.org/project/nipyapi/
> > [4] https://github.com/Chaffelson/nipyapi
> > [5] https://ranger.apache.org/apidocs/ui/swagger.json
> > [6] https://ranger.apache.org/apidocs/ui/index.html
> > [7] https://swagger.io/tools/swagger-codegen/
> >
> > Hope this helps,
> > Kevin
> >
> > On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodh...@gmail.com)
> > wrote:
> > > We use Ranger with NiFi for security and we are looking to automate the
> > > creation of our Ranger policies.
> > >
> > > The way we organize our flows is like this:
> > > NiFi Root Canvas > Ingest Channel PG > Application PG
> > >
> > > We create 3 Ranger Policies per Application PG:
> > > -/process-groups/
> > > -/data/process-groups/
> > > -/provenance-data/process-groups/
> > >
> > > Admins create the Ingest Channel PGs and developers create the
> > Application
> > > PGs. We were thinking of automating any time a new Application PG is
> > > created inside any of the Ingest Channel PGs, create the 3 corresponding
> > > Ranger policies. Was curious to see if anyone else has implemented
> > anything
> > > like this and if so, any tips/suggestions of how to do it?
> > >
> > > Thanks,
> > > Chad
> > >
> >
>

Re: Automate NiFi Ranger Policies

[Kevin Doran]

Hi Chad,

I've never done this, but if I were to go about it I would create a
script / cron job to poll the NiFi REST API [1] periodically, and upon
detection of a new "Application PG", create the corresponding policies
in Ranger via its REST API [2].

You'll have to create service accounts in both NiFI and Ranger for
this script to run as and authenticate to both REST APIs, so you'll
need a secure server to run it from that (1) has access to both NiFi
and Ranger services and (2) has a way of restricting access to that
management server, or at least the service account credentials that
are stored on the server. And of course I would take care when
automating access policy creation in any service!

Tools like NiPyAPI [3][4] can help with scripting access to the NiFi
REST API to poll for process groups. I'm not sure if a similar tool
exists on the Ranger side, although they do have a published Swagger
spec of their REST API [5][6], so generating something similar to
NiPyApi using swagger-codegen [7] might be possible. Then again you
only need to authenticate and access a few endpoints, so any scripting
language with a decent HTTP client library should be sufficient for
this type of thing.

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
[2] https://ranger.apache.org/apidocs/index.html
[3] https://pypi.org/project/nipyapi/
[4] https://github.com/Chaffelson/nipyapi
[5] https://ranger.apache.org/apidocs/ui/swagger.json
[6] https://ranger.apache.org/apidocs/ui/index.html
[7] https://swagger.io/tools/swagger-codegen/

Hope this helps,
Kevin

On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodh...@gmail.com) wrote:
> We use Ranger with NiFi for security and we are looking to automate the
> creation of our Ranger policies.
>
> The way we organize our flows is like this:
> NiFi Root Canvas > Ingest Channel PG > Application PG
>
> We create 3 Ranger Policies per Application PG:
> -/process-groups/
> -/data/process-groups/
> -/provenance-data/process-groups/
>
> Admins create the Ingest Channel PGs and developers create the Application
> PGs. We were thinking of automating any time a new Application PG is
> created inside any of the Ingest Channel PGs, create the 3 corresponding
> Ranger policies. Was curious to see if anyone else has implemented anything
> like this and if so, any tips/suggestions of how to do it?
>
> Thanks,
> Chad
>

Re: Nifi registry Kerberos Auth with Docker

[Kevin Doran]

Glad it is working for you! I replied to the points you raised inline below.


On February 14, 2019 at 00:52:06, Tomislav Novosel (to.novo...@gmail.com) wrote:
> Hi Kevin,
>  
> Thank you for your suggestions. I succeeded to get everything working now.
> As you described, now is everything exectly like that in files you
> mentioned.
>  
> One strange thing. At first stratup of container, I can login into UI
> without problems, but I cannot add new users and policies. After I
> refreshed UI in browser, I was able to do that. So just after refreshing. ??
> 

I haven't seen this, though it is possible your browser was caching assets / 
data that was changed when you refreshed? Still that is odd. If you experience 
this as an ongoing issue plese let us know so we can dig into it.

> And also, Im not able to modify my initial admin and user privileges, I
> mean for myself, but for new added user I can.

This is by design. NiFI Registry does not let the current user modify their own 
privileges or delete their account, to prevent the initial admin from removing 
permissions and locking themselves out. As a workaround, you can create another 
admin account and use that to modify (or delete) the initial admin account that 
was created on first load of the server.

>  
> I read on some forums that it can be slow snyc beetwen Nifi and AD. Im on
> my company's domain and there are couple of hundreds users.
> 

A couple hundred users should sync just fine. I have seen the slow sync 
scenario that some users have experienced and reported, but they had thousands 
of users and groups in their enterprise directories.

A few tips to avoid this issue and tune your LDAP sync (should you give it a 
try):

- You do not need to sync your entire AD. Set the User Search Base/Filter and 
Group Search Base/Filter properties to be as restrictive as possible to avoid 
syncing unecessary users.
- Tune the Page Size and Sync Interval settings for the LdapUserGroupProvider. 
The page size will affect the number and duration of search queries to your AD 
server needed to pull all users and groups -- ie, a larger page size will use 
fewer retrievals with more results in each page, a smaller page size will have 
more retrievals, but each one will be for a smaller set of results. The Sync 
Interval is more important, you can increase this to run the sync less 
frequently if you are worried about background processing or load on your AD 
server.


Cheers,
Kevin

> BR,
> Tom
>  
> On Wed, 13 Feb 2019, 15:29 Kevin Doran >  
> > Hi Tom,
> >
> > How are you configuring the various config files? Through the docker
> > container's environment variables, or through modifying those files
> > directly? If modifying those files, are you injecting them through a volume
> > or something like that? Trying to determine if there is something else at
> > play here overwritting your settings on startup...
> >
> > It sounds like you are able to configure authentication/login
> > successfully, and are just running into a snag on the authorization /
> > initial admin side of things.
> >
> > Try this:
> >
> > 1. In authorizers.xml, set the "Initial User Identity 1" and "Initial
> > Admin Identity" properties to exactly match the user identity recognized by
> > NiFi (the one you see in the upper-right corner of the UI after logging
> > in). Make sure whitespace and capitalization all agree.
> >
> > 2. Delete users.xml and authorizations.xml files and restart NiFI Registry.
> >
> > If all goes successfully, your users.xml file should be regenerated to
> > hold a user with an identity matching "Initial User Identity 1", and
> > authorizations.xml should be regenerated to hold the policies for the
> > "Initial Admin Identity".
> >
> > If you get that working, you can improve things a bit by configuring the
> > LdapUserGroupProvider to sync users and groups from LDAP, letting you set
> > policies in the UI without having to manually create users that match the
> > LDAP directory users.
> >
> > Hope this helps,
> > Kevin
> >
> >
> > On February 13, 2019 at 03:56:52, Tomislav Novosel (to.novo...@gmail.com)
> > wrote:
> > > Also, FYI.
> > >
> > > If I set for INITIAL_ADMIN_IDENTITY my user's full DN,
> > cn=...,ou=...,dc=...
> > > I can also login into UI, but there is no properties button upper right
> > in
> > > the UI.
> > >
> > > [image: 1.PNG]
> > >
> > > If I set only USERNEMA to be u21g46, I can see properties button, but I
> > > can't add new users.
> > >
> > > BR,
> > > Tom
> > >

Re: Nifi registry Kerberos Auth with Docker

[Kevin Doran]

y Web Server-14]
> > o.a.n.r.w.m.UnauthorizedExceptionMapper
> > org.apache.nifi.registry.web.exception.UnauthorizedException: The supplied 
> > client credentials are not valid.. Returning Unauthorized response.
> > >> >
> > >> > Not sure what is going on here.
> > >> >
> > >> > Regards,
> > >> > Tom
> > >> >
> > >> >
> > >> > On Fri, 8 Feb 2019 at 11:36, Daniel Chaffelson 
> > wrote:
> > >> >>
> > >> >> Hi Tomislav,
> > >> >> I created that build a long time ago before the official apache one
> > was up, and it is out of date sorry.
> > >> >> Can I suggest you switch to the official apache image that Kevin
> > mentioned and try again? It is an up to date version and recommended by the
> > community.
> > >> >>
> > >> >> On Thu, Feb 7, 2019 at 5:54 PM Tomislav Novosel <
> > to.novo...@gmail.com> wrote:
> > >> >>>
> > >> >>> Hi Kevin,
> > >> >>>
> > >> >>> I'm using image from Docker hub on this link:
> > >> >>> https://hub.docker.com/r/chaffelson/nifi-registry
> > >> >>>
> > >> >>> I think I know where is the problem. The problem is in config file
> > where
> > >> >>> http host and http port property remains even if I manually set
> > https host and htpps port.
> > >> >>> I deleted http host and http port to be empty, but when I started
> > container again, those values are again there.
> > >> >>>
> > >> >>> I don't know what the author of image wanted to say with this:
> > >> >>>
> > >> >>> The Docker image can be built using the following command:
> > >> >>>
> > >> >>> .
> > ~/Projects/nifi-dev/nifi-registry/nifi-registry-docker/dockerhub/DockerBuild.sh
> >  
> > >> >>>
> > >> >>> What does this commend mean?
> > >> >>>
> > >> >>> And this:
> > >> >>>
> > >> >>> Note: The default version of NiFi-Registry specified by the
> > Dockerfile is typically that of one that is unreleased if working from
> > source. To build an image for a prior released version, one can override
> > the NIFI_REGISTRY_VERSIONbuild-arg with the following command:
> > >> >>>
> > >> >>> docker build --build-arg=NIFI_REGISRTY_VERSION={Desired
> > NiFi-Registry Version} -t apache/nifi-registry:latest .
> > >> >>>
> > >> >>> For this command above you need to have Dockerfile. I tried with
> > Dockerfile from docker hub, but there are errors in execution on this line:
> > >> >>>
> > >> >>> ADD sh/ ${NIFI_REGISTRY_BASE_DIR}/scripts/
> > >> >>>
> > >> >>> On the other hand, If I manage to get the image with first
> > command, I will get Nifi registry version 0.1.0 which I don't want.
> > >> >>>
> > >> >>> I'm little bit confused here, sorry for longer mail.
> > >> >>>
> > >> >>> Thanks.
> > >> >>>
> > >> >>> Regards,
> > >> >>> Tom
> > >> >>>
> > >> >>> On Thu, 7 Feb 2019 at 17:38, Kevin Doran wrote:
> > >> >>>>
> > >> >>>> Hi Tom,
> > >> >>>>
> > >> >>>> Are you using the apache/nifi-registry image or a custom image for
> > this?
> > >> >>>>
> > >> >>>> Have you configured TLS?
> > >> >>>> Can you share your complete conf dir (removing sensitive values
> > such as password or domains)?
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> Kevin
> > >> >>>>
> > >> >>>>
> > >> >>>> On February 7, 2019 at 05:57:37, Tomislav Novosel (
> > to.novo...@gmail.com) wrote:
> > >> >>>> > Hi all,
> > >> >>>> >
> > >> >>>> > I'm trying to configure Nifi registry authentication with
> > Kerberos while
> > >> >>>> > Nifi registry runs
> > >> >>>> > inside Docker container.
> > >> >>>> >
> > >> >>>> > I configured all security properties in
> > nifi-registry.properties, login
> > >> >>>> > identity provider and
> > >> >>>> > authorizers.xml. Everything the same as for Nifi registry
> > running as a
> > >> >>>> > service without Docker container.
> > >> >>>> >
> > >> >>>> > When I open UI in browser and type in login data, login dose not
> > pass.
> > >> >>>> >
> > >> >>>> > In /logs/nifi-registry-app.log I see error:
> > >> >>>> >
> > >> >>>> > An Authentication object was not found in the SecurityContext
> > Returning
> > >> >>>> > 401 response
> > >> >>>> > java.lang.IllegalStateException: Access tokens are only issued
> > over HTTPS
> > >> >>>> >
> > >> >>>> > nifi.registry.web.https.host property is default because of
> > Docker:
> > >> >>>> > ae24ea32faef
> > >> >>>> > nifi.registry.web.https.port=18080
> > >> >>>> >
> > >> >>>> > How can I resolve this?
> > >> >>>> > Thanks.
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > BR,
> > >> >>>> > Tom
> > >> >>>> >
> > >> >>>>
> >
> 

Re: Nifi registry Kerberos Auth with Docker

[Kevin Doran]

Hi Tom,

Are you using the apache/nifi-registry image or a custom image for this?

Have you configured TLS?
Can you share your complete conf dir (removing sensitive values such as 
password or domains)?

Thanks,
Kevin


On February 7, 2019 at 05:57:37, Tomislav Novosel (to.novo...@gmail.com) wrote:
> Hi all,
> 
> I'm trying to configure Nifi registry authentication with Kerberos while
> Nifi registry runs
> inside Docker container.
> 
> I configured all security properties in nifi-registry.properties, login
> identity provider and
> authorizers.xml. Everything the same as for Nifi registry running as a
> service without Docker container.
> 
> When I open UI in browser and type in login data, login dose not pass.
> 
> In /logs/nifi-registry-app.log I see error:
> 
> An Authentication object was not found in the SecurityContext Returning
> 401 response
> java.lang.IllegalStateException: Access tokens are only issued over HTTPS
> 
> nifi.registry.web.https.host property is default because of Docker:
> ae24ea32faef
> nifi.registry.web.https.port=18080
> 
> How can I resolve this?
> Thanks.
> 
> 
> BR,
> Tom
> 

RE: Java exception when attempting to update version from registry

[Kevin Doran]

Russ,

Actually, I think that would be fine. Mark Payne can chime in and correct me if 
I'm wrong, but as far as I know, when changing a flow to a new version, not 
every series of changes is applied, but rather, the diff between the current 
version and the new version is computed and applied. 

So in this case, if you rename the port that is causing the 
IllegalStateException to give it a unique name, and then change from the 
previous version to your newest version (skipoping the offending change that 
triggers the bug case), it _should_ avoid the issue.

Hope this helps get you past the current blocker!

Kevin


On January 10, 2019 at 16:50:41, Russ Weiher (russ.wei...@progenity.com) wrote:
> Kevin -
>  
> Glad to hear there is a root cause for this. Am I correct in assuming that I 
> won't be able  
> to just rename the offending input port to get this current change pushed 
> out? Basically  
> the "bad thing" is already embedded in the change history of the flow?
>  
> Russ Weiher
> BI CI Solutions Architect
> Progenity, Inc.
>  
>  
> -Original Message-
> From: Kevin Doran [mailto:kdo...@apache.org]
> Sent: Thursday, January 10, 2019 4:41 PM
> To: users@nifi.apache.org; Russ Weiher  
> Subject: RE: Java exception when attempting to update version from registry
>  
> Russ,
>  
> The diff was perfect. It showed the problem was two input ports with the same 
> name but different  
> IDs, which I was able to reproduce. Your stack trace and flow diff helped us 
> uncover a bug  
> in our update flow logic -- thanks!
>  
> I verified it still affects the latest version of NiFi and I opened a Jira 
> issue [1] that  
> captures this. Feel free to sign up to ASF Jira to "watch" that issue if you 
> want to see when  
> this gets patched. In the meantime, you can avoid this by avoiding 
> deleting/adding input  
> ports with the same name (during a single version change) within a versioned 
> process  
> group. It's a pretty rare case that should not come up often, so I don't 
> think this should  
> impact your workflow for moving flows from dev to higher envs in general. If 
> you do run  
> into other failures in your workflow, please let us know.
>  
> [1] 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-5950&data=01%7C01%7CRuss.Weiher%40progenity.com%7Cb1c40b5ddbbb45dd4ec208d677445b2a%7C0666349c4979449297792a34e68103b3%7C0&sdata=kN4WhTPPOGlO%2FlogPv5BB7716DnzEPyrfkMjSeedQ2E%3D&reserved=0
>   
>  
> Thanks!
> Kevin
>  
> On January 10, 2019 at 15:45:43, Russ Weiher (russ.wei...@progenity.com) 
> wrote:
> > Kevin -
> >
> > Hopefully the diff will provide some insight. As you have suggested, I
> > don't think I could actually provide the whole flow outside of an 
> > established customer  
> support situation.
> >
> > Here is the diff:
> >
> > {
> > "bucketId": "251301de-a141-46f5-b4ac-106e751f0dc9",
> > "componentDifferenceGroups": [
> > {
> > "componentId": "99d38ec9-4c72-3395-9f14-10d6d115f7a8",
> > "componentName": "",
> > "componentType": "Connection",
> > "differences": [
> > {
> > "changeDescription": "Connection was removed",
> > "differenceType": "COMPONENT_REMOVED",
> > "differenceTypeDescription": "Component Removed",
> > "valueA": "99d38ec9-4c72-3395-9f14-10d6d115f7a8"
> > }
> > ],
> > "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
> > },
> > {
> > "componentId": "4f281a67-5d32-38b1-bb49-071a3005f559",
> > "componentName": "",
> > "componentType": "Connection",
> > "differences": [
> > {
> > "changeDescription": "Connection was added",
> > "differenceType": "COMPONENT_ADDED",
> > "differenceTypeDescription": "Component Added",
> > "valueB": "4f281a67-5d32-38b1-bb49-071a3005f559"
> > }
> > ],
> > "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
> > },
> > {
> > "componentId": "00088125-f087-3f98-a58a-f4e102f2bc0e",
> > "componentName": "",
> > "componentType": "Connection",
> > "differences": [
> > {
> > "changeDescription": "Connection was added",
> > "diff

RE: Java exception when attempting to update version from registry

[Kevin Doran]

ENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "20855d76-0fe1-3f2a-9936-ba580d3e2493"
> }
> ],
> "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
> },
> {
> "componentId": "250abf10-86b6-3d8c-a02a-5474c3bb7e98",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "250abf10-86b6-3d8c-a02a-5474c3bb7e98"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "641896df-1af4-351f-bcd6-009d52961599",
> "componentName": "FlowfileInput",
> "componentType": "Input Port",
> "differences": [
> {
> "changeDescription": "Input Port was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "641896df-1af4-351f-bcd6-009d52961599"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "4dd008c1-7ce0-3cce-9622-4edd668172f5",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "4dd008c1-7ce0-3cce-9622-4edd668172f5"
> }
> ],
> "processGroupId": "4ec2bdf2-11dc-3135-8218-e7714e998826"
> },
> {
> "componentId": "93af327e-d129-3bf5-b6cf-c461ae8affa7",
> "componentName": "ReplaceText - OrderChoiceReportFileInfo",
> "componentType": "Processor",
> "differences": [
> {
> "changeDescription": "Processor was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "93af327e-d129-3bf5-b6cf-c461ae8affa7"
> }
> ],
> "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
> },
> {
> "componentId": "cc69de3d-623b-3e43-9e9f-6d2b47ed7bdd",
> "componentName": "ExtractText",
> "componentType": "Processor",
> "differences": [
> {
> "changeDescription": "Position was changed",
> "differenceType": "POSITION_CHANGED",
> "differenceTypeDescription": "Position Changed",
> "valueA": "[x=771.01306938, y=956.859275745]",
> "valueB": "[x=771.01306938, y=955.541856044762]"
> }
> ],
> "processGroupId": "1785e36e-6b5d-3451-b3bc-465ab0f21e81"
> },
> {
> "componentId": "819bb5e2-d98e-3608-8d70-526409c72790",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "819bb5e2-d98e-3608-8d70-526409c72790"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "e7fac33a-b85e-3ec1-ac8a-a2b9d313ab9c",
> "componentName": "Copia data by date",
> "componentType": "Process Group",
> "differences": [
> {
> "changeDescription": "Process Group was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "e7fac33a-b85e-3ec1-ac8a-a2b9d313ab9c"
> }
> ],
> "processGroupId": "4ec2bdf2-11dc-3135-8218-e7714e998826"
> },
> {
> "componentId": "48a09a92-3a68-3df6-a4f5-1127f00818e9",
> "componentName": "ExtractText",
> "componentType": "Processor",
> "differences": [
> {
> "changeDescription&quo

RE: Java exception when attempting to update version from registry

[Kevin Doran]

FiServiceFacadeLock.java:66)
> at sun.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
> at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
> at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$f3a45649.updateProcessGroupContents()
> at 
> org.apache.nifi.web.api.VersionsResource.updateFlowVersion(VersionsResource.java:1473)
> at 
> org.apache.nifi.web.api.VersionsResource.lambda$null$19(VersionsResource.java:1139)
> at 
> org.apache.nifi.web.api.concurrent.AsyncRequestManager$2.run(AsyncRequestManager.java:109)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Russ Weiher
> BI CI Solutions Architect
> Progenity, Inc.
>
> -Original Message-
> From: Kevin Doran [mailto:kdoran.apa...@gmail.com]
> Sent: Thursday, January 10, 2019 11:16 AM
> To: users@nifi.apache.org; Russ Weiher
> Subject: Re: Java exception when attempting to update version from registry
>
> Sorry, forgot to add -- can you include the full stack trace for the 
> IllegalStateException
> as well? It should be in the nifi-app.log file.
>
> Thanks,
> Kevin
>
>
> On January 10, 2019 at 11:13:33, Kevin Doran (kdo...@apache.org) wrote:
> > Hi Russ,
> >
> > You mentioned you're using NiFi 1.6. What version of NiFi Registry are
> > you running? Is the error message consistent?
> >
> > What is the sequence of steps you are using to move the flow snapshot
> > from DEV to QA? Is it all through the NiFi UI (i.e., both environments
> > are pointed to the same NiFi Registry instance, and you save a version
> > from DEV and then change version in QA), or is there more to it than that?
> >
> > I find it interesting that change version is failing but delete and import 
> > is working...
> > That seems to indicate that the flow snapshot is being saved to
> > registry correctly and completly, but there is some error in the logic
> > of change version that is applying the changes in the wrong order.
> >
> > I know this may not be super helpful, but you may be running into a
> > bug in the Change Version logic that has alreay been fixed in the
> > latest version of NiFi. (To others on the mailing list, does that ring
> > any bells?) If you're not able to test this on the NiFi 1.8 and Registry 
> > 0.3, can you provide
> the steps to reproduce it so that I can try?
> >
> > Thanks,
> > Kevin
> >
> >
> > On January 10, 2019 at 10:57:19, Russ Weiher (russ.wei...@progenity.com) 
> > wrote:
> > > Hello -
> > >
> > >
> > >
> > > I'm trying to move changes from my DEV instance to the QA instance
> > > (via the= Registry), but it's giving me the following error:
> > >
> > >
> > >
> > > "Failed to update flow to new version due to
> > > java.lang.IllegalStateException: The input port name or identifier is not 
> > > available
> to be added."
> > >
> > >
> > >
> > > I had a similar issue in the past where it wouldn't update to a new
> > > version= , but due to
> > deadlines
> > > I ended up just deleting the QA version and re-impor= ting the flow from 
> > > the registry.
> > > However, that approach means that dozens o= f variables have to be
> > > double checked and/or updated for the new environmen= t (QA and then 
> > > again for PROD
> which gives the same error).
> > >
> > 

Re: Java exception when attempting to update version from registry

[Kevin Doran]

Sorry, forgot to add -- can you include the full stack trace for the
IllegalStateException as well? It should be in the nifi-app.log file.

Thanks,
Kevin


On January 10, 2019 at 11:13:33, Kevin Doran (kdo...@apache.org) wrote:
> Hi Russ,
>
> You mentioned you're using NiFi 1.6. What version of NiFi Registry are you 
> running? Is
> the error message consistent?
>
> What is the sequence of steps you are using to move the flow snapshot from 
> DEV to QA? Is it
> all through the NiFi UI (i.e., both environments are pointed to the same NiFi 
> Registry
> instance, and you save a version from DEV and then change version in QA), or 
> is there more
> to it than that?
>
> I find it interesting that change version is failing but delete and import is 
> working...
> That seems to indicate that the flow snapshot is being saved to registry 
> correctly and
> completly, but there is some error in the logic of change version that is 
> applying the
> changes in the wrong order.
>
> I know this may not be super helpful, but you may be running into a bug in 
> the Change Version
> logic that has alreay been fixed in the latest version of NiFi. (To others on 
> the mailing
> list, does that ring any bells?) If you're not able to test this on the NiFi 
> 1.8 and Registry
> 0.3, can you provide the steps to reproduce it so that I can try?
>
> Thanks,
> Kevin
>
>
> On January 10, 2019 at 10:57:19, Russ Weiher (russ.wei...@progenity.com) 
> wrote:
> > Hello -
> >
> >
> >
> > I'm trying to move changes from my DEV instance to the QA instance (via 
> > the= Registry),
> > but it's giving me the following error:
> >
> >
> >
> > "Failed to update flow to new version due to 
> > java.lang.IllegalStateException: The
> > input port name or identifier is not available to be added."
> >
> >
> >
> > I had a similar issue in the past where it wouldn't update to a new 
> > version= , but due to
> deadlines
> > I ended up just deleting the QA version and re-impor= ting the flow from 
> > the registry.
> > However, that approach means that dozens o= f variables have to be double 
> > checked and/or
> > updated for the new environmen= t (QA and then again for PROD which gives 
> > the same error).
> >
> >
> >
> > If this is going to keep happening, I'd rather try and get to the bottom 
> > of= it, but haven't
> > been able to find other examples of people having issues w= hen updating 
> > flow versions
> > using the registry.
> >
> >
> >
> > Currently running Nifi version 1.6.
> >
> >
> > Russ Weiher
> > BI CI Solutions Architect
> > Progenity, Inc.
> >
> > 5230 S. State Road
> > Ann Arbor, MI 48108
> > www.progenity.com
> >
> > (734) 794-9488 Office
> >
> > IMPORTANT: This email (and any attachments) is intended for the use of only 
> > the person
> > or entity to which it is addressed, and may contain information that is 
> > privileged and
> > confidential. You, the recipient, are obligated to maintain it in a safe, 
> > secure and
> > confidential manner. Unauthorized redisclosure or failure to maintain 
> > confidentiality
> > may subject you to federal and state penalties. If you are not the intended 
> > recipient,
> > please immediately notify us by return email, and delete this message from 
> > your computer.
> >
>

Re: Java exception when attempting to update version from registry

[Kevin Doran]

Hi Russ,

You mentioned you're using NiFi 1.6. What version of NiFi Registry are you 
running? Is the error message consistent?

What is the sequence of steps you are using to move the flow snapshot from DEV 
to QA? Is it all through the NiFi UI (i.e., both environments are pointed to 
the same NiFi Registry instance, and you save a version from DEV and then 
change version in QA), or is there more to it than that?

I find it interesting that change version is failing but delete and import is 
working... That seems to indicate that the flow snapshot is being saved to 
registry correctly and completly, but there is some error in the logic of 
change version that is applying the changes in the wrong order.

I know this may not be super helpful, but you may be running into a bug in the 
Change Version logic that has alreay been fixed in the latest version of NiFi. 
(To others on the mailing list, does that ring any bells?) If you're not able 
to test this on the NiFi 1.8 and Registry 0.3, can you provide the steps to 
reproduce it so that I can try?

Thanks,
Kevin


On January 10, 2019 at 10:57:19, Russ Weiher (russ.wei...@progenity.com) wrote:
> Hello -
>  
>  
>  
> I'm trying to move changes from my DEV instance to the QA instance (via the= 
> Registry),  
> but it's giving me the following error:
>  
>  
>  
> "Failed to update flow to new version due to java.lang.IllegalStateException: 
> The  
> input port name or identifier is not available to be added."
>  
>  
>  
> I had a similar issue in the past where it wouldn't update to a new version= 
> , but due to deadlines  
> I ended up just deleting the QA version and re-impor= ting the flow from the 
> registry.  
> However, that approach means that dozens o= f variables have to be double 
> checked and/or  
> updated for the new environmen= t (QA and then again for PROD which gives the 
> same error).  
>  
>  
>  
> If this is going to keep happening, I'd rather try and get to the bottom of= 
> it, but haven't  
> been able to find other examples of people having issues w= hen updating flow 
> versions  
> using the registry.
>  
>  
>  
> Currently running Nifi version 1.6.
>  
>  
> Russ Weiher
> BI CI Solutions Architect
> Progenity, Inc.
>  
> 5230 S. State Road
> Ann Arbor, MI 48108
> www.progenity.com
>  
> (734) 794-9488 Office
>  
> IMPORTANT: This email (and any attachments) is intended for the use of only 
> the person  
> or entity to which it is addressed, and may contain information that is 
> privileged and  
> confidential. You, the recipient, are obligated to maintain it in a safe, 
> secure and  
> confidential manner. Unauthorized redisclosure or failure to maintain 
> confidentiality  
> may subject you to federal and state penalties. If you are not the intended 
> recipient,  
> please immediately notify us by return email, and delete this message from 
> your computer.  
>  

Re: Truststore/Trusted hostname

[Kevin Doran]

Hi Walter,

I could be mistaken, but my interpretation of the Trusted Hostname 
configuration option is that it is designed to work with/in-addition-to the 
truststore, not instead of a truststore as an alternative trust mechanism. 

Basically, I think it is to be used in situations when the default hostname 
verifier (i.e., the remote hostname must match the hostname/SANs of the 
certificate) prevents the connection. IF you have a reason the hostname does 
not match the cert (for example, a dev/test environment) you could whitelist an 
alternative hostname while still making aan HTTPS connection.

Note that when using this option there are man-in-the-middle attack 
implications you should consider.

Hope this helps!

Cheers,
Kevin


On January 9, 2019 at 03:28:45, Vos, Walter (walter@ns.nl) wrote:
> Hi,
>  
> I'm trying to use the invokeHttp processor to POST to an https site through a 
> proxy. The  
> proxy is http. Through some googling I found references that Java is rather 
> finicky with  
> SSL connections and wants the target server certificate in its truststore, 
> but InvokeHttp  
> also offers the trusted hostname parameter.
>  
> Because I don't have CLI access to the server that NiFi runs on, that seemed 
> like the way  
> to get what I want and I added the hostname to the Trusted Hostname. The 
> domain is in a form  
> of subsub.sub.domain.tld and I've tried it just like, as well as 
> *.sub.domain.tld and  
> *.domain.tld and domain.tld, but I keep getting this Java exception:
>  
> sun.security.validator.ValidatorException: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException:  
> unable to find valid certification path to requested target
>  
> Am I doing something wrong? Is truststore really the only way to go? We're 
> working with  
> HDF 3.1.0 / NiFi 1.5.0.*
>  
> Cheers, Walter
>  
> 
>  
> Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor 
> (gebruik door)  
> de geadresseerde. De e-mail kan persoonlijke of vertrouwelijke informatie 
> bevatten.  
> Openbaarmaking, vermenigvuldiging, verspreiding en/of verstrekking van (de 
> inhoud  
> van) deze e-mail (en eventuele bijlagen) aan derden is uitdrukkelijk niet 
> toegestaan.  
> Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk verzocht 
> degene die  
> de e-mail verzond hiervan direct op de hoogte te brengen en de e-mail (en 
> eventuele bijlagen)  
> te vernietigen.
>  
> Informatie vennootschap
>  

Re: Expression language - Convert ISO 8601 to unixtimestamp

[Kevin Doran]

Hi Faisal,

Out of curiosity, does the upstream system(s) guarantee that these timestamps 
will be unique, even at the microsecond level? (I’ve seen some systems that 
write a “microsecond precision” which is really just a millisecond value + a 
sub-millisecond value that increments every time it is queried within a certain 
time window and thus guaranteeing uniqueness within a set of values taken from 
the same clock).

Disclaimer on all of the below: I’m not an HBase expert, so I’m just applying 
my knowledge of using other databases to try to help you troubleshoot this 
situation.

Do you have control over the HBase Schema, and if so, is there another 
field/value you could use for transaction id other than timestamp?  According 
to the little I have read, one should avoid using timestamps for row keys 
anyway [1]. That aside, it seems like you have a uniqueness challenge 
truncating your microseconds, so you’ll have to consider one of the following:


  1.  Change the HBase schema to use some other key. If you can,  just derive a 
primary key to use as a transaction id for the Hbase record. You could just use 
the string representation of the microsecond timestamp (assuming you have 
guarantees that is going to be unique in your input data) or run it through 
some hash algorithm and store the digest, which might also improve the HBase 
region distribution when bulk writing. You can still generate a millisecond 
timestamp to write to Hbase, but you can drop the unique requirement for the 
data as you will have some other key. If the upstream system is not 
guaranteeing that the timestamps will be unique, it sounds to me like you 
should probably not be relying on the timestamp as a transaction id at all.
  2.  If you don’t control the target Hbase system and you’re stuck using 
millisecond timestamps as transactions ids, then you could consider modifying 
the data you are inserting so that the timestamps are always unique. This 
may/may not be an option depending on how many transactions per second you are 
dealing with in the input data. If the rate of transactions is low (e.g., much 
less than 1000 per second), then if you get a millisecond collision when 
truncating microseconds you can deal with that by modifying the duplicate 
timestamp (i.e., the second one to get truncated to the same value) to the 
closest unused millisecond value. This adds overhead (you have to keep track of 
milliseconds used within a time window / batch, or have a failure path for data 
flow that tries to insert into Hbase until successful, modifyingthe key each 
time), and again, doesn’t work if you have more than 1000 transactions per 
second as then you will inevitably start skewing the timestamps by quite a bit.

If it were me, I would try hard to use something other than timestamp as your 
row key / transaction id to avoid this altogether… even if the timestamps is 
part of whatever you end up using as that id.

Hope this helps and good luck,
Kevin

[1] http://hbase.apache.org/book.html#rowkey.design

From: Faisal Durrani 
Reply-To: "users@nifi.apache.org" 
Date: Monday, December 3, 2018 at 22:03
To: "users@nifi.apache.org" 
Subject: Re: Expression language - Convert ISO 8601 to unixtimestamp

Hi Kevin,

Thank you for your mail and suggestions. I was also able to achieve the same by 
using the below expression

${current_ts:substring(0,19):toDate("-MM-dd'T'HH:mm:ss","UTC"):toNumber():plus(${current_ts:substring(20,23)})}

So I am taking out the millisecond part of the microsecond time and adding into 
the unix time generated from -MM-dd'T'HH:mm:ss. While this worked for the 
one of the case i was working with, sadly it failed for another one. As it 
turns out I need to have a microsecond precision so e.g. I have one transaction 
that is coming at current_ts: 2018-11-11T00:17:27.937000 and then I have 
another one coming at current_ts: 2018-11-11T00:17:27.937001 . The Unix 
timestamp converted from the above expression is in millisecond (13 digit epoch 
number) so as i understand it cannot be more precise then this. If it take out 
the microsecond precision the target system which is Hbase in my case considers 
it as the same transaction and over writes the record received at the earlier 
time. Hbase timestamp too can only accept unix time in millisecond (13 digit 
epoch number) precision. So i am lost at this point how to order these 
transactions :(. Please do me know if you have any suggestion on this.

Regards,
Faisal

On Tue, Dec 4, 2018 at 1:49 AM Kevin Doran 
mailto:kdoran.apa...@gmail.com>> wrote:
Hi Faisal,

It appears whatever is writing these date strings that you have for inputs is 
writing microseconds, not milliseconds,

So when you are using `.SS`, that is, in this case `937000` microseconds 
being interpreted by the expression language java parser as 937000 milliseconds 
(15 minutes 37 seconds), hence the “slightly off” factor

Re: Expression language - Convert ISO 8601 to unixtimestamp

[Kevin Doran]

Hi Faisal,

It appears whatever is writing these date strings that you have for inputs is 
writing microseconds, not milliseconds,

So when you are using `.SS`, that is, in this case `937000` microseconds 
being interpreted by the expression language java parser as 937000 milliseconds 
(15 minutes 37 seconds), hence the “slightly off” factor you are experiencing 
for the minutes/seconds of the result.

The difference in hour/day can be explained by timezone. Your input timestring 
must be localtime, but you are treating it as UTC. Specify the correct timezone 
when calling toDate() and that will shift the hours.

So, my suggestion is to run these date strings through and expression that 
first “converts” microseconds to milliseconds via truncation (using substring 
to drop off the trailing “000” is equivalent to dividing by 1000 and dropping 
the remainder/fraction), and then using your expression with the timezone 
specified. Given that your example was off by 9 hours (in addition to the ~15 
minutes), I used UTC+9 time zone and got this expression, which worked for me:

${current_ts:substring(0,${current_ts:length():minus(3)}):toDate("-MM-dd'T'HH:mm:ss.SS",
 "AWDT"):toNumber()}

Lastly, depending on how you are using the result of toNumber(), keep in mind 
that some systems expect seconds since epoch (not milliseconds, which 
toNumber() outputs) for a Unix timestamp.

Cheers,
Kevin

From: Faisal Durrani 
Reply-To: "users@nifi.apache.org" 
Date: Sunday, December 2, 2018 at 21:59
To: "users@nifi.apache.org" 
Subject: Expression language - Convert ISO 8601 to unixtimestamp

Hi Guys,

I am having trouble converting the ISO 8601 to a unixtimestamp . Here is what i 
have tried

current_ts: 2018-11-11T00:17:27.937000

Using updateAttribute, I have configured the below property

${current_ts:toDate("-MM-dd'T'HH:mm:ss.SS"):toNumber()}
This gives the Output value : 1541863984000
if converted back from epochcoverter, its giving

Assuming that this timestamp is in milliseconds:

GMT: Saturday, 10 November 2018 15:33:04

Your time zone: Sunday, 11 November 2018 00:33:04 
GMT+09:00

Relative: 22 days ago



I have also tried using ${current_ts:toDate("-MM-dd'T'HH:mm:ss.SS", 
"UTC"):toNumber()} but the result is the same and its completely wrong. Kindly 
help finding out correct expression.

Regards,
Faisal

Re: Multiple NiFi clusters with 1 NiFi Rigistry

[Kevin Doran]

Good suggestion, Koji.

On 11/20/18, 20:38, "Koji Kawamura"  wrote:

I agree with Bryan and Kevin. This is a good feature request.
Filed a JIRA for further discussion.
https://issues.apache.org/jira/browse/NIFIREG-212

In the meantime, as a work-around I'd deploy a reverse proxy (Nginx)
in front of NiFi Registry to only pass mutation requests
(POST/PUT/DELETE) from dev NiFi hosts.
https://serverfault.com/questions/152745/nginx-proxy-by-request-method

Thanks,
Koji
On Tue, Nov 20, 2018 at 11:33 PM Kevin Doran  
wrote:
>
> I think Bryan’s correct that this makes a good feature request for 
Registry.
>
>
>
> One idea is if you are able to set separate policies for production NiFi 
and non-production NiFi, then you could limit the user policies to read only 
for the NiFi canvas / process group and only allow a service account or admin 
have write access to that NiFi.
>
>
>
> NiFi registry would still technically be writable by those users, but it 
would prevent them from doing something like making a change in the production 
NiFi and then saving that version to Registry. It would be a safe guard to make 
sure changes get introduced to the Registry copy from non-production NiFis.
>
>
>
> Regards,
>
> Kevin
>
>
>
> From: Bryan Bende 
> Reply-To: "users@nifi.apache.org" 
> Date: Tuesday, November 20, 2018 at 08:28
> To: "users@nifi.apache.org" 
> Subject: Re: Multiple NiFi clusters with 1 NiFi Rigistry
>
>
>
> I think we would need to build some type of feature into registry to 
truly support this. Possibly a more specific policy for proxies so that we 
could say Dev NiFi can proxy read and write requests, and prod NiFi can only 
proxy read requests. Currently it would only really work if you had separate 
Kerberos domains, or weren’t using ranger.
>
>
>
> On Tue, Nov 20, 2018 at 6:25 AM Woodhead, Chad  
wrote:
>
> Hi Koji,
>
> Unfortunately all of my NiFi clusters use the same Kerberos domain, which 
is making this harder.
>
> Using NiFi identity mappings to map the same Kerberos principal to 
environment aware ones seems like a good idea, but I’m thinking there will then 
be a disconnect for Ranger (used for NiFi authorization) and NiFi Registry. I 
say this because we use ldap user/group sync for Ranger and ldap sync for NiFi 
Registry using ldap-user-group-provider.
>
> So in Ranger and NiFi Registry, the users are just listed as ‘bob’ and 
not ‘bob@dev’. That means I would manually have to add users to Ranger and NiFi 
Registry to add the ‘@dev’ part right? Or is there a way to customize that too?
>
> Hope I’m not overcomplicating this!
>
> Thanks,
> Chad
>
> On 11/19/18, 8:26 PM, "Koji Kawamura"  wrote:
>
> *External Message* - Use caution before opening links or attachments
>
> Hi Chad,
>
> NiFi Registry uses NiFi user's identity to authorize request.
> Registry also checks NiFi instance's identity to authorize proxying
> user requests, but this can only authorize proxy capability. In order
> to control access such as bucket read/write, Registry uses NiFi user's
> identity.
>
> I assume the kerberos identities are different among those
> environments (Dev, Cert and Prod). And Registry will have different
> user identities for those.
> E.g. for user Bob, Registry would have bob@dev, bob@cert and bob@prod
> Then you can define dev, cert and prod groups at Registry to give
> certain access to buckets.
> E.g. give read/write access to dev group, and give only read access to
> cert and prod groups for a bucket.
>
> In case your NiFi clusters use the same Kerberos domain, you can use
> NiFi identity mapping to map the same Kerberos principal to
> environment aware ones, so that the above authorization can be
> configured at NiFi Registry.
> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_html_administration-2Dguide.html-23identity-2Dmapping-2Dproperties&d=DwIFaQ&c=gJN2jf8AyP5Q6Np0yWY19w&r=MJ04HXP0mOz9-J4odYRNRx3ln4A_OnHTjJvmsZOEG64&m=lFWN8OtWhTL6eW3O-K1-lwIDlC0ZViDDrxxFSys2-Lw&s=_4v5XehCxEQZKSAr800935KuCtiECO-BQGkPknz5Gg4&e=
>
> Thanks,
> Koji
>
>
> On Tue, Nov 20, 2018 at 1:38 AM Woodhead, Chad 
 wrote:
> >
> > I am standing up 3 new HDF 3.2 clusters

Re: Multiple NiFi clusters with 1 NiFi Rigistry

[Kevin Doran]

I think Bryan’s correct that this makes a good feature request for Registry.

One idea is if you are able to set separate policies for production NiFi and 
non-production NiFi, then you could limit the user policies to read only for 
the NiFi canvas / process group and only allow a service account or admin have 
write access to that NiFi.

NiFi registry would still technically be writable by those users, but it would 
prevent them from doing something like making a change in the production NiFi 
and then saving that version to Registry. It would be a safe guard to make sure 
changes get introduced to the Registry copy from non-production NiFis.

Regards,
Kevin

From: Bryan Bende 
Reply-To: "users@nifi.apache.org" 
Date: Tuesday, November 20, 2018 at 08:28
To: "users@nifi.apache.org" 
Subject: Re: Multiple NiFi clusters with 1 NiFi Rigistry

I think we would need to build some type of feature into registry to truly 
support this. Possibly a more specific policy for proxies so that we could say 
Dev NiFi can proxy read and write requests, and prod NiFi can only proxy read 
requests. Currently it would only really work if you had separate Kerberos 
domains, or weren’t using ranger.

On Tue, Nov 20, 2018 at 6:25 AM Woodhead, Chad 
mailto:chad.woodh...@ncr.com>> wrote:
Hi Koji,

Unfortunately all of my NiFi clusters use the same Kerberos domain, which is 
making this harder.

Using NiFi identity mappings to map the same Kerberos principal to environment 
aware ones seems like a good idea, but I’m thinking there will then be a 
disconnect for Ranger (used for NiFi authorization) and NiFi Registry. I say 
this because we use ldap user/group sync for Ranger and ldap sync for NiFi 
Registry using ldap-user-group-provider.

So in Ranger and NiFi Registry, the users are just listed as ‘bob’ and not 
‘bob@dev’. That means I would manually have to add users to Ranger and NiFi 
Registry to add the ‘@dev’ part right? Or is there a way to customize that too?

Hope I’m not overcomplicating this!

Thanks,
Chad

On 11/19/18, 8:26 PM, "Koji Kawamura" 
mailto:ijokaruma...@gmail.com>> wrote:

*External Message* - Use caution before opening links or attachments

Hi Chad,

NiFi Registry uses NiFi user's identity to authorize request.
Registry also checks NiFi instance's identity to authorize proxying
user requests, but this can only authorize proxy capability. In order
to control access such as bucket read/write, Registry uses NiFi user's
identity.

I assume the kerberos identities are different among those
environments (Dev, Cert and Prod). And Registry will have different
user identities for those.
E.g. for user Bob, Registry would have bob@dev, bob@cert and bob@prod
Then you can define dev, cert and prod groups at Registry to give
certain access to buckets.
E.g. give read/write access to dev group, and give only read access to
cert and prod groups for a bucket.

In case your NiFi clusters use the same Kerberos domain, you can use
NiFi identity mapping to map the same Kerberos principal to
environment aware ones, so that the above authorization can be
configured at NiFi Registry.

https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_html_administration-2Dguide.html-23identity-2Dmapping-2Dproperties&d=DwIFaQ&c=gJN2jf8AyP5Q6Np0yWY19w&r=MJ04HXP0mOz9-J4odYRNRx3ln4A_OnHTjJvmsZOEG64&m=lFWN8OtWhTL6eW3O-K1-lwIDlC0ZViDDrxxFSys2-Lw&s=_4v5XehCxEQZKSAr800935KuCtiECO-BQGkPknz5Gg4&e=

Thanks,
Koji


On Tue, Nov 20, 2018 at 1:38 AM Woodhead, Chad 
mailto:chad.woodh...@ncr.com>> wrote:
>
> I am standing up 3 new HDF 3.2 clusters (Dev, Cert, and Prod) and we will 
be focusing on NiFi (1.7.0) + NiFi Registry (0.2.0). We are using git as our 
FlowPersistenceProvider. My plan is to use 1 NiFi Registry (the Prod NiFi 
registry) for all 3 clusters, rather than having 3 NiFi Registries and trying 
to keep the DB’s in sync between the 3 NiFi Registry instances.
>
>
>
> Is there a way to implement some type of authorization so that users can 
only PUSH/PULL changes from Dev NiFi to Prod NiFi Registry, and only PULL from 
Cert and Prod NiFi from Prod NiFi Registry?
>
>
>
> NiFi and NiFi Registry both use the ‘kerberos-identity-provider’ for 
authentication, and the Prod NiFi Registry authenticates with git via a ssh 
access key.
>
>
>
> Thanks,
>
> Chad

--
Sent from Gmail Mobile

Re: variable url for remote process group

[Kevin Doran]

Hi,

Can you put the remote process group outside what is versioned to NiFi 
Registry? For example, if the remote process group is a sink, put everything 
upstream of it in a process group and version that as the thing that moves 
between environments, leaving the RPG, and the connection from the versioned PG 
to the RPG, as static in each environment. (Likewise, if the RPG is the source, 
put everything downstream of it in a process group that is versioned.)

Hope this helps,
Kevin

On 10/5/18, 20:25, "evanthx"  wrote:

i saw this and thought I'd reply as I have a similar issue.

We have a staging and production cluster. I have a remote process group in
them, and have our flow backed up in the registry as that seems to be the
approved method for moving flows from staging to production.

Since the remote process group can't be a variable, however, that means that
we can't alter that variable between production and staging? So our
production cluster keeps trying to send data to our staging cluster. We can
fix it, but then the Registry is out of sync.

We don't want to change it dynamically, for clarity - we just need a way to
version our flow using the registry. If you have any advice I would
appreciate it!



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Nifi Registry + Git: target registry ignores git flows

[Kevin Doran]

ty.util.thread.strategy.ExecuteProduceConsume.produceCo
> nsume(ExecuteProduceConsume.java:201)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(Execu
> teProduceConsume.java:133)
> [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool
> .java:672) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.
> java:590) [jetty-util-9.4.3.v20170317.jar:9.4.3.v20170317]
>
> at java.lang.Thread.run(Unknown Source) [na:1.8.0_181]
>
>
>
>
>
> If I restart the registry service, I’m getting this error in the log:
>
> 2018-09-12 18:34:12,592 INFO [main]
> o.a.n.r.provider.StandardProviderFactory
> Instantiated FlowPersistenceProvider with class name
> org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
>
> 2018-09-12 18:34:13,206 WARN [main] org.eclipse.jgit.util.FS_Win32
> null
>
> 2018-09-12 18:34:13,563 WARN [main] o.a.n.r.p.flow.git.GitFlowMetaData
> Git object id for Flow 929cd7fd-a095-44fc-9eb6-e09335e29c2d version 1
> with path CleverReports\Logons_to_CleverReports.snapshot in bucket
> CleverReports:b8561db0-2138-4054-ace7-ae5c6e8856d5 was not found.
> Ignoring this entry.
>
> 2018-09-12 18:34:13,563 INFO [main] o.a.n.r.p.flow.git.GitFlowMetaData
> Configured GitFlowPersistenceProvider with Flow Storage Directory
> D:\nifi-registry-0.2.0\.\cleverreportsnifi
>
>
>
> This is a new system, so all the ids should match. Any idea what’s going on?
>
>
>
> Thanks,
>
>
>
> Dave
>
>
>
>
>
>
>
>
>
> From: Bryan Bende 
> Sent: Tuesday, September 11, 2018 6:04 PM
>
>
> To: users@nifi.apache.org
> Subject: Re: Nifi Registry + Git: target registry ignores git flows
>
>
>
> Assuming you have a dev DB & git repo and then a prod DB & git repo,
> each DB goes with the corresponding repo and has ids that are unique
> to the environment, so you can't take one of the DBs and point at the
> other’s git repo, that is why you need the export/import process with the CLI.
>
>
>
> The reason everything looked ok on the registry side of things is
> because it doesn’t need any of the flow content at that point, so it
> was just showing what was in the DB, but then when you said to import
> the flow into NiFi it then used the info from the DB to locate the
> flow in the git repo, and it didn’t exist because the bucket id and flow id 
> are not the same.
>
>
>
> On Sep 11, 2018, at 4:55 PM, David Gallagher
> 
> wrote:
>
>
>
> Ok, Last question on this one (promise!). I’ve set up the dev / git
> repository and it’s working great. I can export flows from dev using
> the cli and import them into Prod. One thing though, following up on
> an earlier suggestion from Mike below – I tried stopping the dev
> registry, copying its database, and using it to overwrite the prod
> registry’s database. It seemed to work fine – I started up the prod
> registry, and the buckets and flows were all present. However, when I
> tried to import a flow from Prod into NiFi, I got an error message (in a 
> popup):
>
>
>
> The Flow Registry with ID ca62a0a5-0165-1000-256b-521c462b06f8 reports
> that no Flow exists with Bucket d4438d9e-4aef-43f3-866b-55c4a2089e19,
> Flow b24d2adf-5d2f-4cfd-a160-88a4de99a6d0, Version 1
>
>
>
> I tried using the cli tool, list buckets and flows works as expected
> but I can’t use the export function. Do I have to do anything special
> to back up the metadata database correctly?
>
>
>
> Thanks,
>
>
>
> Dave
>
>
>
> From: Kevin Doran 
> Sent: Tuesday, September 11, 2018 10:25 AM
> To: users@nifi.apache.org
> Subject: Re: Nifi Registry + Git: target registry ignores git flows
>
>
>
> Happy to help anytime. Glad it’s working for you!
>
>
>
> Kevin
>
>
>
> From: David Gallagher 
> Reply-To: "users@nifi.apache.org" 
> Date: Tuesday, September 11, 2018 at 10:18
> To: "users@nifi.apache.org" 
> Subject: RE: Nifi Registry + Git: target registry ignores git flows
>
>
>
> OK, I think I’ve got it now – I didn’t realize that you had to create
> a bucket and empty flow in the ‘prd’ repository before you could
> import a version into that flow. For some reason I thought ‘Import
> Flow’ would create the bucket and flow if it didn’t already exist. Thanks for 
> the help!
>
>
>
> Dave
>
>
>
> From: Kevin Doran 
> Sent: Monday, September 10, 2018 10:18 AM
> To: users@nifi.apache.org
> Subject: Re: Nifi R

Re: Nifi Registry + Git: target registry ignores git flows

[Kevin Doran]

Happy to help anytime. Glad it’s working for you!

Kevin

From: David Gallagher 
Reply-To: "users@nifi.apache.org" 
Date: Tuesday, September 11, 2018 at 10:18
To: "users@nifi.apache.org" 
Subject: RE: Nifi Registry + Git: target registry ignores git flows

OK, I think I’ve got it now – I didn’t realize that you had to create a bucket 
and empty flow in the ‘prd’ repository before you could import a version into 
that flow. For some reason I thought ‘Import Flow’ would create the bucket and 
flow if it didn’t already exist. Thanks for the help!

Dave

From: Kevin Doran 
Sent: Monday, September 10, 2018 10:18 AM
To: users@nifi.apache.org
Subject: Re: Nifi Registry + Git: target registry ignores git flows

Hi Dave,

Either persistence provider should work fine for your use case.  The error 
message indicates that flow ID does not exist in that bucket. You can browse 
the available buckets and flows in the NiFi Registry UI (in the default view), 
where clicking on a flow will display details including the flow id and bucket 
id.

You can also get this information from the CLI using the 'nifi-registy 
list-buckets' and 'nifi-registy list-flows' commands.

If you are using the correct ids and still getting this error, it means 
something is corrupted creating a mismatch between the flow persistence 
provider and the metadata persistence provider. I’ve never seen that happen 
before. My guess would be it could be caused due to a configuration error, a 
change from one provider to another without foiloowing the right steps, or 
manually modifying the persistence files outside of the NiFi Registry API/UI

I hope this helps you get on the right track. Let me know if you need further 
assistance.

Cheers,
Kevin

From: David Gallagher 
mailto:dgallag...@cleverdevices.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Date: Sunday, September 9, 2018 at 11:06
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: RE: Nifi Registry + Git: target registry ignores git flows

Thanks for the assistance! I tried the CLI and it worked up until I tried to 
export a flow from ‘Dev’, got this message:

2018-09-09 10:52:45,911 ERROR [NiFi Registry Web Server-19] 
o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: 
org.apache.nifi.registry.flow.FlowPersistenceException: Flow ID 
0df59975-2683-4e84-91f5-f616bbed207a version 1 was not found in bucket 
CleverGit:34658e23-c138-4085-9280-6359f322672d.. Returning Internal Server 
Error response.
org.apache.nifi.registry.flow.FlowPersistenceException: Flow ID 
0df59975-2683-4e84-91f5-f616bbed207a version 1 was not found in bucket 
CleverGit:34658e23-c138-4085-9280-6359f322672d.

Am I going about this the wrong way? Should I just be using the default flow 
persistence provider?

Thanks,

Dave

From: Kevin Doran mailto:kdoran.apa...@gmail.com>>
Sent: Saturday, September 8, 2018 1:54 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: Nifi Registry + Git: target registry ignores git flows

Mike is correct – it is likely due to mismatch of metadata (bucket ids and flow 
ids) in the two environments. The recommended way to move flow snapshots across 
air-gapped environments at this time is and export/import with the NiFi CLI 
[1], a tool included the NiFi Toolkit, which you can download here [2]. The 
import step will create the necessary metadata for you.

Let me know if you have any questions!

[1] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli
[2] https://nifi.apache.org/download.html

Kevin


From: Mike Thomsen mailto:mikerthom...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Date: Saturday, September 8, 2018 at 13:17
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: Re: Nifi Registry + Git: target registry ignores git flows

It probably has to do with the database the registry used not being migrated to 
your new environment. Try setting up a snapshot of that too.
On Sat, Sep 8, 2018 at 1:06 PM David Gallagher 
mailto:dgallag...@cleverdevices.com>> wrote:
Hi – I have a dev registry server (1.7.1) set up with git, and am trying to use 
git bundle to move the buckets and flows to a test registry instance (I’m using 
git bundle because there’s going to be an air gap in production). Everything 
looks fine, but when I start the production registry it ignores the buckets 
from the clone. I’ve got this in the error message:

2018-09-08 11:57:39,084 INFO [main] o.a.n.r.provider.StandardProviderFactory 
Instantiated FlowPersistenceProvider with class name 
org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
2018-09-08 11:57:40,077 WARN [main] o.a.n.r.p.flow.

Re: Nifi Registry + Git: target registry ignores git flows

[Kevin Doran]

Hi Dave,

Either persistence provider should work fine for your use case.  The error 
message indicates that flow ID does not exist in that bucket. You can browse 
the available buckets and flows in the NiFi Registry UI (in the default view), 
where clicking on a flow will display details including the flow id and bucket 
id.

You can also get this information from the CLI using the 'nifi-registy 
list-buckets' and 'nifi-registy list-flows' commands.

If you are using the correct ids and still getting this error, it means 
something is corrupted creating a mismatch between the flow persistence 
provider and the metadata persistence provider. I’ve never seen that happen 
before. My guess would be it could be caused due to a configuration error, a 
change from one provider to another without foiloowing the right steps, or 
manually modifying the persistence files outside of the NiFi Registry API/UI

I hope this helps you get on the right track. Let me know if you need further 
assistance.

Cheers,
Kevin

From: David Gallagher 
Reply-To: "users@nifi.apache.org" 
Date: Sunday, September 9, 2018 at 11:06
To: "users@nifi.apache.org" 
Subject: RE: Nifi Registry + Git: target registry ignores git flows

Thanks for the assistance! I tried the CLI and it worked up until I tried to 
export a flow from ‘Dev’, got this message:

2018-09-09 10:52:45,911 ERROR [NiFi Registry Web Server-19] 
o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: 
org.apache.nifi.registry.flow.FlowPersistenceException: Flow ID 
0df59975-2683-4e84-91f5-f616bbed207a version 1 was not found in bucket 
CleverGit:34658e23-c138-4085-9280-6359f322672d.. Returning Internal Server 
Error response.
org.apache.nifi.registry.flow.FlowPersistenceException: Flow ID 
0df59975-2683-4e84-91f5-f616bbed207a version 1 was not found in bucket 
CleverGit:34658e23-c138-4085-9280-6359f322672d.

Am I going about this the wrong way? Should I just be using the default flow 
persistence provider?

Thanks,

Dave

From: Kevin Doran 
Sent: Saturday, September 8, 2018 1:54 PM
To: users@nifi.apache.org
Subject: Re: Nifi Registry + Git: target registry ignores git flows

Mike is correct – it is likely due to mismatch of metadata (bucket ids and flow 
ids) in the two environments. The recommended way to move flow snapshots across 
air-gapped environments at this time is and export/import with the NiFi CLI 
[1], a tool included the NiFi Toolkit, which you can download here [2]. The 
import step will create the necessary metadata for you.

Let me know if you have any questions!

[1] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli
[2] https://nifi.apache.org/download.html

Kevin


From: Mike Thomsen mailto:mikerthom...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Date: Saturday, September 8, 2018 at 13:17
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" 
mailto:users@nifi.apache.org>>
Subject: Re: Nifi Registry + Git: target registry ignores git flows

It probably has to do with the database the registry used not being migrated to 
your new environment. Try setting up a snapshot of that too.
On Sat, Sep 8, 2018 at 1:06 PM David Gallagher 
mailto:dgallag...@cleverdevices.com>> wrote:
Hi – I have a dev registry server (1.7.1) set up with git, and am trying to use 
git bundle to move the buckets and flows to a test registry instance (I’m using 
git bundle because there’s going to be an air gap in production). Everything 
looks fine, but when I start the production registry it ignores the buckets 
from the clone. I’ve got this in the error message:

2018-09-08 11:57:39,084 INFO [main] o.a.n.r.provider.StandardProviderFactory 
Instantiated FlowPersistenceProvider with class name 
org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
2018-09-08 11:57:40,077 WARN [main] o.a.n.r.p.flow.git.GitFlowMetaData Git 
object id for Flow 0df59975-2683-4e84-91f5-f616bbed207a version 1 with path 
CleverGit\LogonTransactions_to_CleverReports.snapshot in bucket 
CleverGit:34658e23-c138-4085-9280-6359f322672d was not found. Ignoring this 
entry.

I can see that the structure is present and (seemingly) correct on the ‘Prod’ 
registry. Anything obvious that I’m missing, or things I can try out?

Thanks,

Dave

Re: Nifi Registry + Git: target registry ignores git flows

[Kevin Doran]

Mike is correct – it is likely due to mismatch of metadata (bucket ids and flow 
ids) in the two environments. The recommended way to move flow snapshots across 
air-gapped environments at this time is and export/import with the NiFi CLI 
[1], a tool included the NiFi Toolkit, which you can download here [2]. The 
import step will create the necessary metadata for you.

Let me know if you have any questions!

[1] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli
[2] https://nifi.apache.org/download.html

Kevin


From: Mike Thomsen 
Reply-To: "users@nifi.apache.org" 
Date: Saturday, September 8, 2018 at 13:17
To: "users@nifi.apache.org" 
Subject: Re: Nifi Registry + Git: target registry ignores git flows

It probably has to do with the database the registry used not being migrated to 
your new environment. Try setting up a snapshot of that too.
On Sat, Sep 8, 2018 at 1:06 PM David Gallagher 
mailto:dgallag...@cleverdevices.com>> wrote:
Hi – I have a dev registry server (1.7.1) set up with git, and am trying to use 
git bundle to move the buckets and flows to a test registry instance (I’m using 
git bundle because there’s going to be an air gap in production). Everything 
looks fine, but when I start the production registry it ignores the buckets 
from the clone. I’ve got this in the error message:

2018-09-08 11:57:39,084 INFO [main] o.a.n.r.provider.StandardProviderFactory 
Instantiated FlowPersistenceProvider with class name 
org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
2018-09-08 11:57:40,077 WARN [main] o.a.n.r.p.flow.git.GitFlowMetaData Git 
object id for Flow 0df59975-2683-4e84-91f5-f616bbed207a version 1 with path 
CleverGit\LogonTransactions_to_CleverReports.snapshot in bucket 
CleverGit:34658e23-c138-4085-9280-6359f322672d was not found. Ignoring this 
entry.

I can see that the structure is present and (seemingly) correct on the ‘Prod’ 
registry. Anything obvious that I’m missing, or things I can try out?

Thanks,

Dave

Re: Adding processor property broke ability to commit changes to the registry

[Kevin Doran]

Hi Mike,

The menu items in the Version submenu are enabled/disabled based on the state 
of the versioned PG, which is an enum [1].

The menu options you are seeing, review local changes or revert, usually 
correspond to the “LOCALLY_MODIFIED_AND_STALE” state, which indicates that your 
NiFi instance is not on the latest version of the flow and also has local 
uncommitted changes. Before digging further into this issue, can you verify 
which version is saved to NiFi Registry and which version you are on in NiFi? 
Can you verify the state NiFi thinks the versioned PG is in?

If you are not on the latest version, then this is actually correct behavior, 
the only option currently is to revert, upgrade to latest, and then introduce 
the changes again. For this, you may have to install the old version of the 
custom processor (you can have both installed simultaneously) in order to 
revert.

If you are on the latest version that has been saved to NiFi Registry, then 
this sounds to me like a bug in determining the versioned PG state / flow diff 
and we’ll need to keep digging into the root cause.

[1] 
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/registry/flow/VersionedFlowState.java

From: Mike Thomsen 
Reply-To: "users@nifi.apache.org" 
Date: Friday, August 24, 2018 at 10:55
To: "users@nifi.apache.org" 
Subject: Re: Adding processor property broke ability to commit changes to the 
registry

Kevin,

1. We added a new property definition in the Java code. Yes, the version number 
changes from 1.5.X to 1.6.X (these don't track NiFi releases FWIW).

2. We have ~ 3 layers of process groups. Only one of the inner PGs is 
versioned. The rest are all versioned at the parent level. When we added this 
new property, it was detected as a "local change" in one of those process 
groups that was not versioned on its own. The parent level one where the 
versioning is done only gives us two choices: review local changes or revert. 
Revert obviously doesn't work because it's a code change. The commit option is 
missing. It's like the registry doesn't know what to do since we added that new 
property.

So yeah, it very much knows something changed, but it just won't let us commit. 
It doesn't even show that option.

Thanks,

Mike

On Fri, Aug 24, 2018 at 10:00 AM Kevin Doran 
mailto:kdo...@apache.org>> wrote:
Hi Mike,

I’ve never seen this before. I have a few questions for you to help me 
understand what could be going on:

1. When you say the custom processor got a new property, do you mean that the 
custom processor was modified to include a new property definition, and was 
recompiled/bundled and installed to overwrite the existing nar? If so, when the 
custom processor was updated, did the version number change?

2. Can you clarify what you mean by “It detects the new property at that level, 
but won't let us ‘commit local changes.’ ” – where has it detected local 
changes? In the parent PG? Does it give you the option to “show local changes” 
and if so what does it list? Does it give you the option to commit local 
changes, but that operation fails, or does it not give you the option at all 
(ie, it thinks it is in sync)

If it’s not giving you the option to commit local changes because it can’t 
detect the change, one possible workaround off the top of my head would be to 
change something else (even just something like the position of the processor). 
Another possible workaround would be to version the processor so the old and 
new can be installed simultaneously and then change versions of the processor. 
If that wasn’t done it might be confusing the flow diff logic. Just a thought, 
happy to help dig into this with you more so we can file the appropriate JIRA 
to get it fixed so it doesn’t require special handling.

Thanks,
Kevin

On Fri, Aug 24, 2018 at 9:23 AM Mike Thomsen 
mailto:mikerthom...@gmail.com>> wrote:
We have a custom processor that got a new property. It's part of an embedded PG 
and the parent PG is version controlled. It detects the new property at that 
level, but won't let us "commit local changes." So we're stuck unable to 
commit. Has anyone seen this before or have any ideas on what is happening/work 
around?

If not, I'll see if I can cleanly replicate and post a JIRA.

Thanks,

Mike

Re: Adding processor property broke ability to commit changes to the registry

[Kevin Doran]

Hi Mike,



I’ve never seen this before. I have a few questions for you to help me
understand what could be going on:



1. When you say the custom processor got a new property, do you mean that
the custom processor was modified to include a new property definition, and
was recompiled/bundled and installed to overwrite the existing nar? If so,
when the custom processor was updated, did the version number change?



2. Can you clarify what you mean by “It detects the new property at that
level, but won't let us ‘commit local changes.’ ” – where has it detected
local changes? In the parent PG? Does it give you the option to “show local
changes” and if so what does it list? Does it give you the option to commit
local changes, but that operation fails, or does it not give you the option
at all (ie, it thinks it is in sync)



If it’s not giving you the option to commit local changes because it can’t
detect the change, one possible workaround off the top of my head would be
to change something else (even just something like the position of the
processor). Another possible workaround would be to version the processor
so the old and new can be installed simultaneously and then change versions
of the processor. If that wasn’t done it might be confusing the flow diff
logic. Just a thought, happy to help dig into this with you more so we can
file the appropriate JIRA to get it fixed so it doesn’t require special
handling.



Thanks,
Kevin

On Fri, Aug 24, 2018 at 9:23 AM Mike Thomsen  wrote:

> We have a custom processor that got a new property. It's part of an
> embedded PG and the parent PG is version controlled. It detects the new
> property at that level, but won't let us "commit local changes." So we're
> stuck unable to commit. Has anyone seen this before or have any ideas on
> what is happening/work around?
>
> If not, I'll see if I can cleanly replicate and post a JIRA.
>
> Thanks,
>
> Mike
>

Re: Re:

[Kevin Doran]

>
> *Explaining to your end users that you should skip the first Certificate
> Prompt but accept the second but only when you haven't logged in the
> current session is really painful*


Wow, that sounds terrible. Confusing, accident prone, and frustrating to
correct mistakes (at least in my experience, forcing a browser to forget
client certificate preferences is difficult).

Thanks for sharing those details about your deployment scenario. This can
definitely be improved and I have some ideas for how to do it. I've cloned
the issue to NiFi to make sure we are tracking it for both projects [1][2]

[1] https://issues.apache.org/jira/browse/NIFIREG-189
[2] https://issues.apache.org/jira/browse/NIFI-5504

On Thu, Aug 9, 2018 at 11:54 AM, Shawn Weeks 
wrote:

> The project I'm on is running into this issue as well and it gets
> particularly painful when all of your server's are signed by the same root
> ca that signs your smart card logins and your using something like KnoxSSO.
> Explaining to your end users that you should skip the first Certificate
> Prompt but accept the second but only when you haven't logged in the
> current session is really painful and shows major shortcoming between the
> back end authentication between servers and front end ui authentication.
>
>
> We can't even considering putting it behind our identify reverse proxies
> because we can't turn off two way ssl.
>
>
> Thanks
>
> Shawnk
> --
> *From:* Kevin Doran 
> *Sent:* Thursday, August 9, 2018 10:47:56 AM
> *To:* users@nifi.apache.org
> *Subject:* Re:
>
> sorry forgot the link. here it is:
>
> [1] https://issues.apache.org/jira/projects/NIFIREG/issues/NIFIREG-189
>
> On Thu, Aug 9, 2018 at 11:47 AM, Kevin Doran  wrote:
>
> Hi Curtis,
>
> This has come up a few times. Unfortunately I don’t think there is
> currently an easy way to disable X509-based identity extraction in NiFi
> today. There is an open JIRA for the same issue in NiFi Registry [1]. NiFi
> Registry follows the same AuthN/AuthZ design (and a fair amount of code) as
> NiFi, so this ticket should apply to NiFi as well.
>
> Perhaps you could share more about your needs and use case on that ticket
> so that when it gets implemented we could take that scenario with reverse
> proxies and OIDC into account?
>
> Thanks,
> Kevin
>
> On Mon, Aug 6, 2018 at 10:23 AM, Curtis Ruck 
> wrote:
>
> I'm trying to setup OIDC authentication, but with Nifi service existing
> behind a reverse proxy, and for our other apps we use SSL Client
> Authentication between reverse proxy and application, Nifi is picking up
> the Reverse Proxy's SSL Certificate and falling into X509 Authentication
> instead of OIDC. Any idea how I can disable X509 authentication in Nifi?
>
> Connecting directly to nifi, it triggers the proper OIDC redirects.
>
> --
> Curtis Ruck
>
>
>
>

Re:

[Kevin Doran]

sorry forgot the link. here it is:

[1] https://issues.apache.org/jira/projects/NIFIREG/issues/NIFIREG-189

On Thu, Aug 9, 2018 at 11:47 AM, Kevin Doran  wrote:

> Hi Curtis,
>
> This has come up a few times. Unfortunately I don’t think there is
> currently an easy way to disable X509-based identity extraction in NiFi
> today. There is an open JIRA for the same issue in NiFi Registry [1]. NiFi
> Registry follows the same AuthN/AuthZ design (and a fair amount of code) as
> NiFi, so this ticket should apply to NiFi as well.
>
> Perhaps you could share more about your needs and use case on that ticket
> so that when it gets implemented we could take that scenario with reverse
> proxies and OIDC into account?
>
> Thanks,
> Kevin
>
> On Mon, Aug 6, 2018 at 10:23 AM, Curtis Ruck 
> wrote:
>
>> I'm trying to setup OIDC authentication, but with Nifi service existing
>> behind a reverse proxy, and for our other apps we use SSL Client
>> Authentication between reverse proxy and application, Nifi is picking up
>> the Reverse Proxy's SSL Certificate and falling into X509 Authentication
>> instead of OIDC. Any idea how I can disable X509 authentication in Nifi?
>>
>> Connecting directly to nifi, it triggers the proper OIDC redirects.
>>
>> --
>> Curtis Ruck
>>
>
>

Re:

[Kevin Doran]

Hi Curtis,

This has come up a few times. Unfortunately I don’t think there is
currently an easy way to disable X509-based identity extraction in NiFi
today. There is an open JIRA for the same issue in NiFi Registry [1]. NiFi
Registry follows the same AuthN/AuthZ design (and a fair amount of code) as
NiFi, so this ticket should apply to NiFi as well.

Perhaps you could share more about your needs and use case on that ticket
so that when it gets implemented we could take that scenario with reverse
proxies and OIDC into account?

Thanks,
Kevin

On Mon, Aug 6, 2018 at 10:23 AM, Curtis Ruck  wrote:

> I'm trying to setup OIDC authentication, but with Nifi service existing
> behind a reverse proxy, and for our other apps we use SSL Client
> Authentication between reverse proxy and application, Nifi is picking up
> the Reverse Proxy's SSL Certificate and falling into X509 Authentication
> instead of OIDC. Any idea how I can disable X509 authentication in Nifi?
>
> Connecting directly to nifi, it triggers the proper OIDC redirects.
>
> --
> Curtis Ruck
>

Re: Pushing to bitbucket from nifi registry

[Kevin Doran]

Hi,



Glad to hear you are finding the NiFi Registry features useful. Regarding
your question, the “Remote Access User” and “Remote Access Password”
properties are only used when the remote URL is an HTTPS url. When it is an
SSH url, it is expected that password-less SSH has been configured on your
NiFi Registry instance, and the user/password properties are ignored.



If it is possible, I would suggest you try one of the following:

   1. Setup password-less SSH (e.g., if you can ensure no one else has
   access to the private ssh key through some other mechanism, such as file
   permissions)
   2. If your remote bitbucket accepts username/password authentication
   over https, change your remote to be https based rather than ssh based.



I hope this helps! If you have any other trouble/questions let me know.



Regards,

Kevin


On Sun, Jul 29, 2018 at 11:18 AM, Krish  wrote:

> Hello,
>
> I've just upgraded from NiFi 1.2.0 to 1.7.1 and been playing around with
> the NiFi registry features which are great!
>
> I've been trying to get it to push automatically to my bitbucket
> repository but I keep getting the following error: "Failed to push commits
> to origin due to org.eclipse.jgit.api.errors.TransfportException
> ssh://git@bitbucketserver:1234/nifi-registry.git Auth Fail"
>
> In my configuration I have the folllowing setttings in my providers.xml:
> 
> org.apache.nifi.registry.provider.flow.git.
> GitFlowPersistenceProvider
> .
> */storage/nifi_registry_repo*
> *origin*
> *ssh_key_user*
> *ssh key password*
> 
> 
>
> The nifi_registry_repo is a git repo which I have cloned down from my
> bitbucket server.
> I've setup a SSH and loaded into the bitbucket repo, I can successfully
> push and pull from the command line on the nifi registry server using the
> ssh key.
>
> When I change a flow in the NiFi registry UI it commits successfully to
> the repo but fails to push the changes up to the bitbucket server.
>
> Has anyone else done this successfully? Or is there mistake I've made in
> my configs?
>
> Thanks
>
> K
>
>
>

Re: nifi-registry.sh doesn't find user

[Kevin Doran]

Never seen that before - definitely odd. What platform are you running on?



Thanks,
Kevin

On Fri, Jul 20, 2018 at 1:58 PM, Geoff Craig  wrote:

> Hello,
>
>
>
> I’m trying to run nifi-registry (0.2.0) using run.as in bootstrap.conf
> and the nifi-registry.sh run command is throwing this error:
>
>
>
> does not exist. Exiting.
>
>
>
> The only reference I see to this is here:
>
>
>
> if ! id -u "${run_as_user}" >/dev/null 2>&1; then
>
> echo "The specified run.as user ${run_as_user} does not
> exist. Exiting."
>
> exit 1
>
> fi
>
>
>
> I have ran id -u with the run.as username and it comes back with the user
> id.  I put in some echo commands and the run_as_user is reporting
> appropriately.  Any suggestions?
>
>
>
> Thank you,
>
>
>
>
>

Re: NiFi Registry with nested PGs

[Kevin Doran]

Hi Mike,

Yes, this is expected behavior.

Let's say I have a PG A that has a nested versioned PG B, both are at
version 1. Because PG B is versioned, the full definition of PG A does not
extend down into PG B, it stops at a reference to "PG B:v1". Because PG B
is versioned independently, a reference to that versioned PG id at version
number fully defines PG A. Let's say I change the definition of PG B. PG B
will show up on my canvas as "version 1 with local changes", but until I
commit those changes, it is still at version 1 (modified), so PG A is still
defined both locally and in NiFi Registry as "containing PG B:v1".
Therefore, PG A, at this point in time, does not have any local changes
(because compared to Registry, it has the same definition), which is why it
shows up as such.

Once I commit PG B's local changes, PG B on my canvas is now at version 2,
and my local PG A's definition has changed to include a reference to PG
B:v2. This differs from what is in Registry for PG A:v1, so now PG A shows
local changes. If I revert local changes, I will go back to PG A:v1
referencing PG B:v1. If I commit local changes, PG A is now also at version
2, which references PG B:v2.

Fair to say this can be surprising/confusing behavior at first, even though
technically correct. This is why the top level version indicators on the
global status bar are useful to see if there are any changes, nested or
otherwise, in the overall flow.

Best,
Kevin


On Wed, Jul 18, 2018 at 9:24 PM, Mike Thomsen 
wrote:

> We have 0.2 hooked up to a NiFi instance that has nested PGs. All PGs are
> versioned. When one of the inner ones has local changes, the out of sync
> icon doesn't appear on the parent PGs. Is that expected behavior? No one
> really minds it, but I didn't have an answer as to whether we stumbled onto
> a bug or it's expected behavior.
>
> Thanks,
>
> Mike
>

Re: Merging output of multiple processors

[Kevin Doran]

Hi Jean-Sébastien,

 

Sorry you’re running into trouble. NiFi can have a bit of a learning curve at 
first, but once you are comfortable with the components it comes with and how 
to use them effectively, it gets much faster to accomplish tasks such as your 
example.

 

In general, don’t worry too much about reducing the amount of data going 
around. Unless you modify the flow file contents, all content data is passed by 
reference in NiFi, meaning there is only one physical copy of it (stored in the 
flow file content repository) in NiFi’s storage at any time. In most cases, 
this is very efficient and you do not need to optimize the data for NiFi, it 
will try to do the intelligent thing for you. If you start to experience system 
resource exhaustion (e.g., run out of storage in content repository), or if 
copying data becomes a performance bottleneck in your flow, then take the time 
to optimize that aspect or tune the system to meet the demands of your data 
flow.

 

Keeping that in mind, here are a few pointers to help you get started:

 
Rather than split parts of your JSON object apart into separate flow files, 
keep the entire object together. From there, take advantage of processors that 
are designed to interpret (and manipulate) the flow file contents as JSON. Look 
at the JSON processors (you can filter processors by name, e.g., “Json” when 
adding one to the canvas), such as JoltTransformJSON, EvaluateJsonPath, 
FlattenJson, SplitJson, JsonPathReader.
Processing multiple JSON records at a time can often be more efficient than a 
single JSON object per flow file. If you have or can construct flow files in 
this manner (e.g., a single flow file contains an array of JSON elements), then 
from that point on you can use the record-oriented processors. Record 
processors (again, you can find them by searching/filtering for “Record” when 
adding processors or controller services) require defining a schema to apply to 
your data, but once that is done, you can read/write/modify various record 
formats, including Json. Here is an example of doing JSON record enrichment in 
this style [1].
Lastly, in general, don’t worry too much about reducing the amount of data 
going around. Unless you modify the flow file contents, all content data is 
passed by reference in NiFi, meaning there is only one physical copy of it 
(stored in the flow file content repository) in NiFi’s storage at any time. In 
most cases, this is very efficient and you do not need to optimize the data for 
NiFi, it will try to do the intelligent thing for you. If you start to 
experience system resource exhaustion (e.g., run out of storage in content 
repository), or if copying data becomes a performance bottleneck in your flow, 
then take the time to optimize that aspect or tune the system to meet the 
demands of your data flow.
 

[1] 
https://community.hortonworks.com/articles/138632/data-flow-enrichment-with-nifi-lookuprecord-proces.html
 

 

Hope this helps!

Kevin

 

From: Jean-Sebastien Vachon 
Reply-To: 
Date: Thursday, July 12, 2018 at 12:12
To: "users@nifi.apache.org" 
Subject: Merging output of multiple processors

 

Hi,

 

I am pretty new to Nifi and I’m struggling on something that (in my mind) 
should be very easy to do 😉

My flow consists of a Json file being processed by different processors to 
extract different information and enrich the data. Each processors have been 
implemented as ExecuteStreamCommand and will output the information extracted 
in a JSON like element. As an example, one of the module determines the 
language of one of the field in the original JSON and will output something 
like: 

 

{ “language” : “en” }

 

Every module is extracting a different piece of information and my goal was to 
reduce the amount of data going around.

 

What would be the best way of merging the responses on all modules into my JSON 
when everything has been processed? The resulting JSON will then continue in 
the flow for further processing.

 

I tried using the MergeContent module but the output format can not be in JSON 
so I’m a bit stuck. Right now, the merge strategy is set to “Bin-Packing 
algorithm” with the Correlation attribute set to ${filename}. The min and max 
entries are set to the expected number of elements to merge (6 in this case).

 

I tried the “Defragment” strategy as well but the system was complaining about 
missing fragment.index attribute (which I tried to provide through an 
UpdateAttribute processor but that does not seem to work either

 

Thanks

--

Jean-Sébastien Vachon



vacho...@gmail.com

jsvac...@brizodata.com

www.brizodata.com

 

Re: NiFi ExecuteScript vs multiple processors vs custom processor

[Kevin Doran]

There's a lot of great discussion on this thread.

I’ll add that if you intend to use NiFi Registry with NiFi (which has lot of 
advantages, some of which have already been discussed), you’ll want to consider 
what is going to work best with NiFi Registry and your flow 
deployment/promotion strategy.

Here are some considerations as of today (NiFi 1.7.0 and NiFi Registry 0.2.0):

•   Using chains of built-in processors inside a process group (PG) will 
work out of the box with any NiFi and NiFi Registry instance, so things become 
very portable. Flow composition via nested/reusable process groups – that is, 
being able to build a PG out of a few processors, save it to NiFi Registry as a 
reusable component, and then importing it into other flows or multiple places 
in one flow -- is really powerful capability. Boris, to your point, nesting 
versioned process groups is an available feature (and has been since NiFi 
Registry 0.1 actually).

•   If you use ISP or ES, changes to the Script Body processor property 
will be saved/read with your flow definition in NiFi Registry, but if you are 
invoking an external script using the Script File property, only the filename 
will be saved to NiFi Registry, so changes to the script file contents need to 
be versioned and synced outside of NiFi Registry.

•   Likewise, if you use custom processors, those need to be 
versioned/deployed/installed on each NiFi separately from your flow definition. 
This is a limitation of NiFi Registry today as it only handles flow 
definitions, although this will probably not be a limitation in the future once 
the Extension Registry capabilities [1] that have been discussed in various 
forums [2] are implemented.

•   Also, not directly related to your question but may be helpful to folks 
reading this thread – if you haven’t looked at record-oriented processors 
[3][4] , they may solve this problem for you. In places that previously 
required string together long chains of processors or using a custom/ES/ISP 
processor, you may now be able to do the equivalent logic with just one or two 
record processors (and very efficiently in terms of performance too!)

At the end of the day, all these considerations can be overcome, so if you have 
a particular problem to solve my recommendation is always to use what is the 
best fit in terms of simplicity/understandability/maintainability and 
performance. In situations where those factors are more-or-less equal, these 
factors of overall ecosystem start to come into play as well. 

I like Mike’s suggestion of sticking to built-in processors when possible, and 
when not possible using ISP/ES to prototype custom logic that you ultimately 
migrate into custom processor. Once NiFi Registry gets Extension capabilities, 
that workflow should be even better! 

[1] 
https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
[2] 
http://apache-nifi.1125220.n5.nabble.com/DISCUSS-Apache-NiFi-distribution-has-grown-too-large-td20861.html
 
[3] https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi 
[4] 
https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries
 

Hope this helps ,
Kevin

On 7/10/18, 11:37, "Bryan Bende"  wrote:

You can do nested versioning since the beginning in 0.1.0.

A common scenario might be to have several teams build different
versioned flows, and then someone who is in charge of deploying them
will create another version PG that combines the nested versioned
process groups of each of these teams.

The outer versioned flow in registry does not fully contain the
others, just pointers to the actual versioned flows, which technically
could come from another registry if desired.

On Tue, Jul 10, 2018 at 11:22 AM, Boris Tyukin  
wrote:
> thanks Bryan. I saw your blog post on that. I think with registry 0.1 it 
was
> not possible to version nested PGs within parent PGs so I could not have
> "templatized" PG which has it is own version and use that PG with other
> versioned PGs. Has it changed with registry 0.2 now out?
>
> On Tue, Jul 10, 2018 at 11:08 AM Bryan Bende  wrote:
>>
>> Boris,
>>
>> Regarding templates being limited... templates were really made as a
>> way to share example flows, or help with debugging if you need to send
>> someone your flow. Unfortunately they turned into a deployment
>> mechanism since there wasn't a better solution at the time.
>>
>> Using NiFi Registry should now be the preferred solution, and you can
>> sync changes in-place. You can have many versioned process groups tied
>> to the same versioned flow in a registry and update them all.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Tue, Jul 10, 2018 at 10:50 AM, Boris Tyukin 
>> wrote:
>> > I like Ed's recommendations and doing s

Re: Store external data into NiFi Registry

[Kevin Doran]

[answered on SO as well]

Kumar,

As of today, it is not currently possible to store data/objects in NiFi 
Registry other than a NiFi Flow and its configuration (component properties, 
default variable values, controller services, etc).

There have been discussions about extending NiFi Registry's storage 
capabilities to include other items. Often discussed is NiFi extensions, such 
as NAR bundles which are the archive format for components such as custom 
processors. This would allow custom components to be versioned in the same 
place as a flow and downloaded at runtime based on a flow definition rather 
than pre-installed on a NiFi/MiNiFi instances.

Today though, only Flows are supported. Other data or components has to be 
stored/versioned somewhere else.

If you have data you want to associate with a specific flow version snapshot, 
here is a suggestion: You could store that data externally in another service 
and use the flow version snapshot comment field to store a URI/link to where 
the associated data resides. If you use a machine parsable format such as JSON 
in the snapshot comment to store this URI metadata, an automated process could 
retrieve this data from an external system by reading this field when doing an 
operation involving a specific flow snapshot version.

Cheers,
Kevin



From: kumar r 
Sent: Tuesday, July 3, 2018 7:31:11 AM
To: users@nifi.apache.org
Subject: Store external data into NiFi Registry


Is that possible to store external data (not NiFi flow) into NiFi Registry 
using REST API?

https://nifi.apache.org/docs/nifi-registry-docs/index.html

As i know, NiFi Registry designed for versioning NiFi flow. But i want to know 
whether it is capable of storing other data into NiFi registry and retrieve it 
based on versions.

https://stackoverflow.com/questions/51151363/store-other-data-into-nifi-registry

Re: Checking out specific branch of nifi

[Kevin Doran]

Hi Vitaly,

All releases are tagged as “rel/”, so you can see all tags by running 
“git tag” and then use “git checkout ” to checkout the version you 
want.

Cheers,
Kevin

From: Vitaly Krivoy 
Reply-To: 
Date: Thursday, June 28, 2018 at 11:54
To: "users@nifi.apache.org" 
Subject: Checking out specific branch of nifi

 

If I do “git checkout master”, that would check out the next version of nifi 
which is 1.7 

How do I check out current version 1.6? I don’t see it when I list all 
branches. Thanks.

Vitaly Krivoy

 

 

STATEMENT OF CONFIDENTIALITY The information contained in this email message 
and any attachments may be confidential and legally privileged and is intended 
for the use of the addressee(s) only. If you are not an intended recipient, 
please: (1) notify me immediately by replying to this message; (2) do not use, 
disseminate, distribute or reproduce any part of the message or any attachment; 
and (3) destroy all copies of this message and any attachments.

Re: HandleHttpRequest and Allowed Paths

[Kevin Doran]

Hi Boris,

 

Are those urls correct? It looks like the path and port are out of place, e.g.:

 

localhost:8011/admin

localhost:8011/info 

 

-Kevin

 

From: Boris Tyukin 
Reply-To: 
Date: Thursday, June 14, 2018 at 16:26
To: 
Subject: HandleHttpRequest and Allowed Paths

 

Hi,

 

I am following the example here with a simple web service

https://pierrevillard.com/2016/04/10/url-shortener-service-with-apache-nifi/

 

and want to make different endpoints like 

localhost/admin:8011

localhost/info:8011 

 

etc.

 

Looks like I need to set "Allowed Paths" property on HandleHttpRequest 
processor but cannot make it work.

 

it works fine if I call just localhost:8011 



I run NiFi on VM..



Thanks!

Re: Fun with DistributeLoad

[Kevin Doran]

Thanks for the additional details. It sounds like you have already explored 
alternatives quite a bit and have found the best path. :) Looks like Mark has 
some good advice for making this flow manageable, so if this is working for 
you, I’d take his suggestions where it makes sense and run with it for a while 
until you come up against some reason to change it.  Good luck!

 

From: Martijn Dekkers 
Reply-To: 
Date: Wednesday, June 13, 2018 at 10:37
To: 
Subject: Re: Fun with DistributeLoad

 

Hi Kevin!

 

Thanks for your reply.

 

Can you share more about the details of what your DistributeLoad process group 
is doing and how the 24 endpoints of the particular S3-compatible storage 
service work? Are they fixed or could they change? Just hoping to understand 
what are the constraints you have to work within.

 

The endpoints won't change, this is an on-prem system. The whole setup is 
really just a poor-mans' load-balancer :) we use the DistributeLoad processor 
as a next-available distribution mechanism, and all PutS3Object processors 
point to a different Override URL.

 

 

I’m assuming that each of the PutS3Object in your flow is configured with a 
different Endpoint Override URL property. As that property supports values that 
use NiFi’s Expression Language, one approach you could take would be to use a 
single PutS3Object processes group that gets the remote endpoint from the 
incoming flowfile attribute using an expression. You would have to set that 
property on each flow file upstream in your flow, maybe in place of what your 
DistributeLoad PG is doing now in order to round-robin through available 
endpoints as flowfiles pass through, or something like that.

 

We did consider something like this idea, using a dynamic value for the URL, 
but we are seeing a very nice throughput increase with this setup. Each 
flowfile is a 16MB image, so we can upload 24 images in parallel. We are 
considering dealing with unavailable endpoints through the API, which would be 
more graceful, but this would be at a later stage. 

 

  

Another approach would be to introduce a service discovery mechanism. This is 
an alternative to the server-side load balancing approach that you mentioned 
you wanted to avoid, except that instead of a proxy distributing the load to a 
backend that is hidden from the client, the client uses the service discovery 
API to discover an endpoint to use from a pool of available endpoints (service 
discovery solutions usually have some way of doing simple distribution such as 
round-robin). This would be especially useful if the set of 24 endpoints can 
change (or one could become unavailable and you want to dynamically take it out 
of rotation). The service discovery directory would keep track of the 
available, healthy endpoints, and in your flow you leverage it so that the 
endpoint used by PutS3Object is rotated (this could be done explicitly in your 
flow by querying a service discovery directory, or transparently if you use a 
mechanism such as DNS discovery that rotates the results of a hostname lookup). 
The failure path of your flow could even reach back to the service discovery 
API to mark an endpoint as unhealthily if it sees a particular type of or 
number of failures, automatically removing it from the pool.

 

We use DCOS/Mesosphere in-house so we have a nice service discovery environment 
already available. The issue is that we have a lot of limitations on the 
machines where we will be deploying this. They run Windows, and not much good 
quality loadbalancing software exists for that. We experimented with HAProxy on 
Windows, but determined it to be too fragile. 

 

Lastly, I wouldn’t entirely rule out server-side load balancing via a proxy. 
True, the proxy could become a bottleneck or point of failure, but there are a 
lot of great load balancers available today that attempt to solve this class of 
problem in a reliable, scalable manner.

 

Not so concerned about SPOF - everything is HA, but the throughput bottleneck 
is a real concern here. We will have many systems generating this data, and 
previous work with centralizing this particular data flow has proven 
problematic for this reason. 

 

These are just some ideas. The right solution depends on a number of factors 
and constraints particular to your flow and environment. Maybe what I’ve 
outlined here will give you an idea for what that might be in your case.

 

Thank you so much, it is really appreciated!

 

Martijn

Re: Fun with DistributeLoad

[Kevin Doran]

Hi Martijn,

 

Can you share more about the details of what your DistributeLoad process group 
is doing and how the 24 endpoints of the particular S3-compatible storage 
service work? Are they fixed or could they change? Just hoping to understand 
what are the constraints you have to work within.

 

I’m assuming that each of the PutS3Object in your flow is configured with a 
different Endpoint Override URL property. As that property supports values that 
use NiFi’s Expression Language, one approach you could take would be to use a 
single PutS3Object processes group that gets the remote endpoint from the 
incoming flowfile attribute using an expression. You would have to set that 
property on each flow file upstream in your flow, maybe in place of what your 
DistributeLoad PG is doing now in order to round-robin through available 
endpoints as flowfiles pass through, or something like that.

 

Another approach would be to introduce a service discovery mechanism. This is 
an alternative to the server-side load balancing approach that you mentioned 
you wanted to avoid, except that instead of a proxy distributing the load to a 
backend that is hidden from the client, the client uses the service discovery 
API to discover an endpoint to use from a pool of available endpoints (service 
discovery solutions usually have some way of doing simple distribution such as 
round-robin). This would be especially useful if the set of 24 endpoints can 
change (or one could become unavailable and you want to dynamically take it out 
of rotation). The service discovery directory would keep track of the 
available, healthy endpoints, and in your flow you leverage it so that the 
endpoint used by PutS3Object is rotated (this could be done explicitly in your 
flow by querying a service discovery directory, or transparently if you use a 
mechanism such as DNS discovery that rotates the results of a hostname lookup). 
The failure path of your flow could even reach back to the service discovery 
API to mark an endpoint as unhealthily if it sees a particular type of or 
number of failures, automatically removing it from the pool.

 

Lastly, I wouldn’t entirely rule out server-side load balancing via a proxy. 
True, the proxy could become a bottleneck or point of failure, but there are a 
lot of great load balancers available today that attempt to solve this class of 
problem in a reliable, scalable manner.

 

These are just some ideas. The right solution depends on a number of factors 
and constraints particular to your flow and environment. Maybe what I’ve 
outlined here will give you an idea for what that might be in your case.

 

Hope this helps!

 

Kevin

 

From: Martijn Dekkers 
Reply-To: 
Date: Wednesday, June 13, 2018 at 02:49
To: 
Subject: Re: Fun with DistributeLoad

 

see https://imgur.com/c4MDP7o

 

The success routes are not yet in place - each PutS3Object needs to be routed 
to a success handling set of processors.

 

Thanks

 

Martijn

 

On 13 June 2018 at 08:42, Sivaprasanna  wrote:

Is it possible to share screenshots of the flow which feels cluttered? I have a 
hard time picturing how the PutS3Objects are routed to failures and successes. 
A picture would certainly help.

 

Thanks.

 

On Wed, Jun 13, 2018 at 12:07 PM, Martijn Dekkers  
wrote:

Thanks, I already use process groups specifically for the PutS3Object 
processors. However, with 24 of those, all needing a failure and success 
connection, this screen is very cluttered. 

 

Thanks

Martijn

 

On 13 June 2018 at 08:30, Sivaprasanna  wrote:

Martijn,

 

One clean up approach that comes immediately to my mind is to use 'Process 
Groups'. Using which, you can group processors together that perform a related 
sequence of actions. You can think of them as 'functions' or 'methods' in 
programming terms. And since you mentioned that you are using 24 PutS3Object 
processors, you can even club them together under a processor group and name it 
something like 'Write to S3' and have it abstracted from the developer's 
immediate view. so it would like DistributeLoad -> Write to S3 [Process Group] 
-> Failure handling flow. If the failure handling involves a sequence of steps, 
you can again put them together under a processor group and name it "Failure 
Handler".

 

Thanks,

Sivaprasanna

 

 

On Wed, Jun 13, 2018 at 11:41 AM, Martijn Dekkers  
wrote:

All,

 

I have a more general question. We will be uploading files to an S3 compatible 
storage system. In our case, this system presents 24 endpoints to upload to. 
Given the volume of data we are sending to this device, we want to avoid using 
a loadbalancer like HAProxy for some use-cases, to avoid bottlenecks. The 
use-case is to enable direct, parallel uploads against all S3 endpoints. We 
will have roughly 100 systems generating data, each running this Flow. 

 

We did consider running a local loadbalancer on each of the systems, but this 
is unfortunately not possible due to other constraints

Re: LDAP/AD User groups

[Kevin Doran]

Cool, glad it’s working for you! Thanks for sharing your final config, I’m sure 
that might help future users who are searching for AD integration help.

 

Regards,

Kevin

 

From: Martijn Dekkers 
Reply-To: 
Date: Wednesday, June 6, 2018 at 00:38
To: 
Subject: Re: LDAP/AD User groups

 

Bingo! Thank you so much Kevin,

 

In the end not using a Group Search Filter is what made this work for AD.

 

Here is the relevant snippet:

 

OU=groups,OU=Dev,DC=xxx,DC=,DC=net

group

ONE_LEVEL



cn

member



 

Thanks again!

 

Martijn

 

On 6 June 2018 at 05:58, Kevin Doran  wrote:

Whoops, in my example, disregard the class for the ldap-user-group-provider 
(org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider). I took 
that snippet from one of my NiFi Registry configuration files, which is the 
same as NiFi except for the class name ☺. Just mentioning that so it doesn’t 
confuse anyone or you don’t copy/paste from my example.

 

From: Kevin Doran 
Reply-To: 
Date: Tuesday, June 5, 2018 at 23:52
To: 
Subject: Re: LDAP/AD User groups

 

Hi Martijn,

 

You’re really close to having this configured correctly. Basically, you just 
need to set a few more Group properties in your ldap-user-group-provider config.

 

The way user and group loading works in NiFI:

 
The users are synced from LDAP using the “User Search *” (and related) 
properties 
The groups are synced from LDAP using the “Group Search * (and related) 
properties
The user <-> group relationships are created using the “User Group Name 
Attribute” or “Group Member Attribute” properties
 

The full description of these properties is available in the NiFi Admin Guide 
[1].

 

It looks like you’ve fully specified the User related properties and it sound 
like that is working the way you expect.

 

Next you’ll want to specify the Group properties that NiFi needs to sync the 
group records (search base, search filter, etc.) This will be very similar to 
your user search, just change the base and filter. This will create the groups 
in NiFi, but not the information about which users are in each group.

 

For that, you’ll need to specify the User Group Name or Group Member Attribute 
properties. Which ones to set and how to configure them depends on the 
structure of your LDAP/AD. From the NiFi Admin Guide documentation [1]:

 

User Group Name Attribute - Attribute to use to define group membership (i.e. 
memberof). Optional. If not set group membership will not be calculated through 
the users. Will rely on group membership being defined through 'Group Member 
Attribute' if set. The value of this property is the name of the attribute in 
the user ldap entry that associates them with a group. The value of that user 
attribute could be a dn or group name for instance. What value is expected is 
configured in the 'User Group Name Attribute - Referenced Group Attribute'.

 

Group Member Attribute - Attribute to use to define group membership (i.e. 
member). Optional. If not set group membership will not be calculated through 
the groups. Will rely on group membership being defined through 'User Group 
Name Attribute' if set. The value of this property is the name of the attribute 
in the group ldap entry that associates them with a user. The value of that 
group attribute could be a dn or memberUid for instance. What value is expected 
is configured in the 'Group Member Attribute - Referenced User Attribute'. 
(i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1)

 

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup
 

 

---

 

Here is just one example of how this might work.

 

For this directory structure (in LDIF format):


# define dc=example,dc=com, etc ...

 

dn: uid=user1,dc=example,dc=com
sn: User1
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
uid: user1
cn: User 1

 

dn: uid=user2,dc=example,dc=com
sn: User2
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
uid: user1
cn: User 2

dn: ou=group1,dc=example,dc=com
uniqueMember: uid=user1,dc=example,dc=com
uniqueMember: uid=user2,dc=example,dc=com
ou: group1
cn: Group 1
objectClass: groupOfUniqueNames
objectClass: top
 
The ldap-user-group-provider in authorizers.xml would look like this:

 

ldap-user-group-provider

org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider
 
dc=example,dc=com
person
ONE_LEVEL
(uid=*)
uid


dc=example,dc=com
groupOfUniqueNames
ONE_LEVEL
(ou=*)
ou
uniqueMember

 

 

This is just an example of course. Using this properties depends on your 
directory. There will be some unused properties (this is because the 
configuration has to be flexible enough to support almost any directory 
structure), but make sure something is 

Re: LDAP/AD User groups

[Kevin Doran]

Whoops, in my example, disregard the class for the ldap-user-group-provider 
(org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider). I took 
that snippet from one of my NiFi Registry configuration files, which is the 
same as NiFi except for the class name ☺. Just mentioning that so it doesn’t 
confuse anyone or you don’t copy/paste from my example.

 

From: Kevin Doran 
Reply-To: 
Date: Tuesday, June 5, 2018 at 23:52
To: 
Subject: Re: LDAP/AD User groups

 

Hi Martijn,

 

You’re really close to having this configured correctly. Basically, you just 
need to set a few more Group properties in your ldap-user-group-provider config.

 

The way user and group loading works in NiFI:

 
The users are synced from LDAP using the “User Search *” (and related) 
properties 
The groups are synced from LDAP using the “Group Search * (and related) 
properties
The user <-> group relationships are created using the “User Group Name 
Attribute” or “Group Member Attribute” properties
 

The full description of these properties is available in the NiFi Admin Guide 
[1].

 

It looks like you’ve fully specified the User related properties and it sound 
like that is working the way you expect.

 

Next you’ll want to specify the Group properties that NiFi needs to sync the 
group records (search base, search filter, etc.) This will be very similar to 
your user search, just change the base and filter. This will create the groups 
in NiFi, but not the information about which users are in each group.

 

For that, you’ll need to specify the User Group Name or Group Member Attribute 
properties. Which ones to set and how to configure them depends on the 
structure of your LDAP/AD. From the NiFi Admin Guide documentation [1]:

 

User Group Name Attribute - Attribute to use to define group membership (i.e. 
memberof). Optional. If not set group membership will not be calculated through 
the users. Will rely on group membership being defined through 'Group Member 
Attribute' if set. The value of this property is the name of the attribute in 
the user ldap entry that associates them with a group. The value of that user 
attribute could be a dn or group name for instance. What value is expected is 
configured in the 'User Group Name Attribute - Referenced Group Attribute'.

 

Group Member Attribute - Attribute to use to define group membership (i.e. 
member). Optional. If not set group membership will not be calculated through 
the groups. Will rely on group membership being defined through 'User Group 
Name Attribute' if set. The value of this property is the name of the attribute 
in the group ldap entry that associates them with a user. The value of that 
group attribute could be a dn or memberUid for instance. What value is expected 
is configured in the 'Group Member Attribute - Referenced User Attribute'. 
(i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1)

 

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup
 

 

---

 

Here is just one example of how this might work.

 

For this directory structure (in LDIF format):


# define dc=example,dc=com, etc ...

 

dn: uid=user1,dc=example,dc=com
sn: User1
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
uid: user1
cn: User 1

 

dn: uid=user2,dc=example,dc=com
sn: User2
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
uid: user1
cn: User 2

dn: ou=group1,dc=example,dc=com
uniqueMember: uid=user1,dc=example,dc=com
uniqueMember: uid=user2,dc=example,dc=com
ou: group1
cn: Group 1
objectClass: groupOfUniqueNames
objectClass: top
 
The ldap-user-group-provider in authorizers.xml would look like this:

 

ldap-user-group-provider

org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider
 
dc=example,dc=com
person
ONE_LEVEL
(uid=*)
uid


dc=example,dc=com
groupOfUniqueNames
ONE_LEVEL
(ou=*)
ou
uniqueMember

 

 

This is just an example of course. Using this properties depends on your 
directory. There will be some unused properties (this is because the 
configuration has to be flexible enough to support almost any directory 
structure), but make sure something is defining group membership.

 

Hope this helps solve your issue!

 

Regards,
Kevin

 

From: Martijn Dekkers 
Reply-To: 
Date: Tuesday, June 5, 2018 at 22:01
To: 
Subject: LDAP/AD User groups

 

Hello all,

 

I have set up a secure NiFi that works using an AD LDAP server. I can log in 
with different users, and authenticate, assign policies - all great. 

 

The one thing I am not managing to get to work at all is groups and group 
membership. Users that are part of the NiFiUsers group show up, and no others - 
as I want it to be. Unfortunately, the groups defined in AD (in the correct OU) 
just don't show, and in the userlist 

Re: LDAP/AD User groups

[Kevin Doran]

Hi Martijn,

 

You’re really close to having this configured correctly. Basically, you just 
need to set a few more Group properties in your ldap-user-group-provider config.

 

The way user and group loading works in NiFI:

 
The users are synced from LDAP using the “User Search *” (and related) 
properties 
The groups are synced from LDAP using the “Group Search * (and related) 
properties
The user <-> group relationships are created using the “User Group Name 
Attribute” or “Group Member Attribute” properties
 

The full description of these properties is available in the NiFi Admin Guide 
[1].

 

It looks like you’ve fully specified the User related properties and it sound 
like that is working the way you expect.

 

Next you’ll want to specify the Group properties that NiFi needs to sync the 
group records (search base, search filter, etc.) This will be very similar to 
your user search, just change the base and filter. This will create the groups 
in NiFi, but not the information about which users are in each group.

 

For that, you’ll need to specify the User Group Name or Group Member Attribute 
properties. Which ones to set and how to configure them depends on the 
structure of your LDAP/AD. From the NiFi Admin Guide documentation [1]:

 

User Group Name Attribute - Attribute to use to define group membership (i.e. 
memberof). Optional. If not set group membership will not be calculated through 
the users. Will rely on group membership being defined through 'Group Member 
Attribute' if set. The value of this property is the name of the attribute in 
the user ldap entry that associates them with a group. The value of that user 
attribute could be a dn or group name for instance. What value is expected is 
configured in the 'User Group Name Attribute - Referenced Group Attribute'.

 

Group Member Attribute - Attribute to use to define group membership (i.e. 
member). Optional. If not set group membership will not be calculated through 
the groups. Will rely on group membership being defined through 'User Group 
Name Attribute' if set. The value of this property is the name of the attribute 
in the group ldap entry that associates them with a user. The value of that 
group attribute could be a dn or memberUid for instance. What value is expected 
is configured in the 'Group Member Attribute - Referenced User Attribute'. 
(i.e. member: cn=User 1,ou=users,o=nifi vs. memberUid: user1)

 

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup
 

 

---

 

Here is just one example of how this might work.

 

For this directory structure (in LDIF format):


# define dc=example,dc=com, etc ...

 

dn: uid=user1,dc=example,dc=com
sn: User1
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
uid: user1
cn: User 1

 

dn: uid=user2,dc=example,dc=com
sn: User2
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
uid: user1
cn: User 2

dn: ou=group1,dc=example,dc=com
uniqueMember: uid=user1,dc=example,dc=com
uniqueMember: uid=user2,dc=example,dc=com
ou: group1
cn: Group 1
objectClass: groupOfUniqueNames
objectClass: top
 
The ldap-user-group-provider in authorizers.xml would look like this:

 

    ldap-user-group-provider
    
org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider
     
dc=example,dc=com
    person
    ONE_LEVEL
    (uid=*)
    uid
    

    dc=example,dc=com
    groupOfUniqueNames
    ONE_LEVEL
    (ou=*)
    ou
    uniqueMember

 

 

This is just an example of course. Using this properties depends on your 
directory. There will be some unused properties (this is because the 
configuration has to be flexible enough to support almost any directory 
structure), but make sure something is defining group membership.

 

Hope this helps solve your issue!

 

Regards,
Kevin

 

From: Martijn Dekkers 
Reply-To: 
Date: Tuesday, June 5, 2018 at 22:01
To: 
Subject: LDAP/AD User groups

 

Hello all,

 

I have set up a secure NiFi that works using an AD LDAP server. I can log in 
with different users, and authenticate, assign policies - all great. 

 

The one thing I am not managing to get to work at all is groups and group 
membership. Users that are part of the NiFiUsers group show up, and no others - 
as I want it to be. Unfortunately, the groups defined in AD (in the correct OU) 
just don't show, and in the userlist in NiFi "Member of:" is empty for the 
listed users.

 

Any suggestions appreciated. I followed Pierre Villard's excellent guide.

 

Many thanks. 

 

Martijn

 

My authorizers.xml:

 



 





file-user-group-provider

org.apache.nifi.authorization.FileUserGroupProvider

./conf/users.xml







 



ldap-user-group-provider

org.apache.nifi.ldap.tenants.LdapUserGroupProvider

START_TLS

CN=LDAP 
MGR,OU=people,OU=Dev,DC=xxx,DC=,DC=net

   

Re: LDAP issues

[Kevin Doran]

Hi Scott,

I have not run into or heard of this issue. NiFi should run fine on Ubuntu, 
including the LDAP login identity provider. I would guess is an 
environmental/networking or configuration issue. Can you provide some more 
details, such as: 

- the full stack trace for the IllegalArgumentException you shared in your 
original email
- are there any other warnings or errors in your nifi-app.log on startup?
- did Ansible report any issues / warnings during deployment?
- any more details about what may be different/changed from the working 
deployment to the new deployment (for example, the LDAP server the same? If so, 
is it visible from the new instance - it could be an environment issue. If it 
is a different LDAP server, it could be an issue on the LDAP server side.)

Thanks,
Kevin

On 5/21/18, 10:50, "Scott Howell"  wrote:

I am having some issues with LDAP when I switched from running Nifi on 
Amazon Linux 2 to running on Ubuntu. I don’t think there is any issue with the 
actual OS but seeing some weird issues when spinning up servers with Ubuntu vs 
Amazon Linux 2. 

o.a.n.w.a.c.IllegalArgumentExceptionMapper 
java.lang.IllegalArgumentException: The supplied username and password are not 
valid.. Returning Bad Request response.

That is the above error I am getting on the server when I try to do my 
initial login. I am using Ansible to configure the servers so there should be 
no difference in the configuration of the servers between my Amazon Linux 
server and my Ubuntu server. 

Was wondering if anyone has ran into this issue.

Thanks,

Scott

Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control

[Kevin Doran]

Thanks a lot, Bryan. PR merged.

 

From: Ryan H 
Reply-To: 
Date: Monday, May 7, 2018 at 15:21
To: 
Subject: Re: Error Reference When Creating Template From PG Under NiFi Registry 
Version Control

 

Awesome, thanks for sending out. Great turn around!

-Ryan

 

On Mon, May 7, 2018 at 2:58 PM, Bryan Bende  wrote:

FYI, created a JIRA and PR here:

https://issues.apache.org/jira/browse/NIFI-5163


On Mon, May 7, 2018 at 10:03 AM, Bryan Bende  wrote:
> The 1.6 toolkit should work fine with 1.5.0, even the latest CLI from
> master should work with 1.5 or 1.6.
>
> Even though I do agree that we should be moving away from using
> templates for deployment, I do think we should be clearing the version
> control DTO when creating a template, it was not left in there
> intentionally.
>
> On Mon, May 7, 2018 at 9:04 AM, Kevin Doran  wrote:
>>> It seems that the version control information (registry references)
>>> shouldn't be included when creating templates or there should be an option
>>> to include/exclude when creating the template. I would guess that there are
>>> a lot of use cases where a template is being created to be used in
>>> environments other than the one which it was created in.
>>
>>
>>
>> Ryan, I agree with you on this point. New features (NiFi Registry
>> integration) should not break old features (NiFi flow templates), unless the
>> community explicitly deprecates those features, in which case that should be
>> documented, and, ideally, migration tools and guides should be provided.
>> This case you’ve encountered is definitely one where we could improve the
>> documentation, and possibly even NiFi’s handling of the Registry metadata
>> when exporting or importing templates if it is not a huge effort.
>>
>>
>>
>> Thanks,
>>
>> Kevin
>>
>>
>>
>> From: Ryan H 
>> Reply-To: 
>> Date: Monday, May 7, 2018 at 08:30
>> To: 
>> Subject: Re: Error Reference When Creating Template From PG Under NiFi
>> Registry Version Control
>>
>>
>>
>> Hi Kevin,
>>
>> Thanks for the multiple options as it relates to FDLC; good articles. The
>> workaround for our current error was to manually search/destroy all
>> references to the registry in the created template xml file
>> ( tag in the template). Then when the template is
>> imported into a the new environment, the issue was resolved as all of the
>> bad references were not ported into the new environment.
>>
>> It seems that the version control information (registry references)
>> shouldn't be included when creating templates or there should be an option
>> to include/exclude when creating the template. I would guess that there are
>> a lot of use cases where a template is being created to be used in
>> environments other than the one which it was created in. While it may not
>> fully align with best practices/suggested methods for FDLC, this does cause
>> issues if done this way. It may just be the case that this is documented as
>> being and issue and the recommendation is to follow one of the suggested
>> practices to avoid issues such as this one.
>>
>> Cheers,
>>
>> Ryan
>>
>>
>>
>> On Mon, May 7, 2018 at 7:10 AM, Kevin Doran  wrote:
>>
>> Hi Ryan,
>>
>>
>>
>> I’ve never tried creating a template from a process group that was saved to
>> a NiFi Registry, so I haven’t run into this exact error. However, there are
>> users that cannot connect multiple environments (e.g., dev and production)
>> to the same NiFi Registry and therefore have the need you’ve described of
>> having to move flow definitions without a shared NiFi Registry instance.
>>
>>
>>
>> From what you’ve described and the error you’ve encountered, I’ve gathered
>> that the approach you are using to migrate flows across environments is:
>>
>>
>>
>> 1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s
>> NiFi Registry as needed
>>
>> 2. [Env A] Export to template
>>
>> 3. [Env B] Import template into NiFI
>>
>> 4. [Env B] Save to environment’s NiFi Registry
>>
>>
>>
>> Is that more or less correct? If so, instead of this workflow, would it be
>> possible to take the following path to migrate the flow version, which I
>> have seen others use:
>>
>>
>>
>> [Env A] NiFi  >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env B]
>> NiFi
>>
>>
>>
>> In this FDLC workflow, templates are never used, because in Env B the
&

Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control

[Kevin Doran]

> It seems that the version control information (registry references) shouldn't 
> be included when creating templates or there should be an option to 
> include/exclude when creating the template. I would guess that there are a 
> lot of use cases where a template is being created to be used in environments 
> other than the one which it was created in.

 

Ryan, I agree with you on this point. New features (NiFi Registry integration) 
should not break old features (NiFi flow templates), unless the community 
explicitly deprecates those features, in which case that should be documented, 
and, ideally, migration tools and guides should be provided. This case you’ve 
encountered is definitely one where we could improve the documentation, and 
possibly even NiFi’s handling of the Registry metadata when exporting or 
importing templates if it is not a huge effort.

 

Thanks,

Kevin

 

From: Ryan H 
Reply-To: 
Date: Monday, May 7, 2018 at 08:30
To: 
Subject: Re: Error Reference When Creating Template From PG Under NiFi Registry 
Version Control

 

Hi Kevin,

Thanks for the multiple options as it relates to FDLC; good articles. The 
workaround for our current error was to manually search/destroy all references 
to the registry in the created template xml file ( 
tag in the template). Then when the template is imported into a the new 
environment, the issue was resolved as all of the bad references were not 
ported into the new environment. 

It seems that the version control information (registry references) shouldn't 
be included when creating templates or there should be an option to 
include/exclude when creating the template. I would guess that there are a lot 
of use cases where a template is being created to be used in environments other 
than the one which it was created in. While it may not fully align with best 
practices/suggested methods for FDLC, this does cause issues if done this way. 
It may just be the case that this is documented as being and issue and the 
recommendation is to follow one of the suggested practices to avoid issues such 
as this one. 

Cheers,

Ryan

 

On Mon, May 7, 2018 at 7:10 AM, Kevin Doran  wrote:

Hi Ryan,

 

I’ve never tried creating a template from a process group that was saved to a 
NiFi Registry, so I haven’t run into this exact error. However, there are users 
that cannot connect multiple environments (e.g., dev and production) to the 
same NiFi Registry and therefore have the need you’ve described of having to 
move flow definitions without a shared NiFi Registry instance.

 

>From what you’ve described and the error you’ve encountered, I’ve gathered 
>that the approach you are using to migrate flows across environments is:

 

1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s NiFi 
Registry as needed

2. [Env A] Export to template

3. [Env B] Import template into NiFI

4. [Env B] Save to environment’s NiFi Registry

 

Is that more or less correct? If so, instead of this workflow, would it be 
possible to take the following path to migrate the flow version, which I have 
seen others use:

 

[Env A] NiFi  >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env B] NiFi

 

In this FDLC workflow, templates are never used, because in Env B the Registry 
is populated first and then imported into NiFi using the import from Registry 
feature.

 

If this sounds like it could work for your use case, and you would be able to 
avoid templates entirely (i.e., there is no other reason you need them other 
than migrating flows across environments), then there are lots of great write 
ups by others on how to achieve this workflow. The trick is in the Registry A 
to Registry B step, and for that there are a few approaches and tools that 
others have documented. 

 

A great write up about this “one Registry per environment” FDLC approach is 
included in a blog post by Pierre Villard, “Automate workflow deployment in 
Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can use the 
NiFi CLI [1], included in the NiFi Toolkit as a separate executable since NiFi 
1.6.0 [2] as a tool to migrate flow version snapshots from one NiFi Registry to 
another. Specifically, the CLI tools “registry export-flow-version” and 
“registry import-flow-version” commands.

 

An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client SDK 
that makes the REST API for NiFi and NiFi Registry easier to work with from 
Python automation scripts. There’s a good write up on using this to interact 
with flows in NiFi Registry instances from Timothy Spann [5].

 

I hope this helps you workaround the issue you are running into. It certainly 
does not solve the specific error you have documented, but perhaps it is a 
viable alternative that avoids that exception.

 

Best,
Kevin

 

[1] 
https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/

[2

Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control

[Kevin Doran]

Hi Ryan,

 

I’ve never tried creating a template from a process group that was saved to a 
NiFi Registry, so I haven’t run into this exact error. However, there are users 
that cannot connect multiple environments (e.g., dev and production) to the 
same NiFi Registry and therefore have the need you’ve described of having to 
move flow definitions without a shared NiFi Registry instance.

 

>From what you’ve described and the error you’ve encountered, I’ve gathered 
>that the approach you are using to migrate flows across environments is:

 

1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s NiFi 
Registry as needed

2. [Env A] Export to template

3. [Env B] Import template into NiFI

4. [Env B] Save to environment’s NiFi Registry

 

Is that more or less correct? If so, instead of this workflow, would it be 
possible to take the following path to migrate the flow version, which I have 
seen others use:

 

[Env A] NiFi  >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env B] NiFi

 

In this FDLC workflow, templates are never used, because in Env B the Registry 
is populated first and then imported into NiFi using the import from Registry 
feature.

 

If this sounds like it could work for your use case, and you would be able to 
avoid templates entirely (i.e., there is no other reason you need them other 
than migrating flows across environments), then there are lots of great write 
ups by others on how to achieve this workflow. The trick is in the Registry A 
to Registry B step, and for that there are a few approaches and tools that 
others have documented. 

 

A great write up about this “one Registry per environment” FDLC approach is 
included in a blog post by Pierre Villard, “Automate workflow deployment in 
Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can use the 
NiFi CLI [1], included in the NiFi Toolkit as a separate executable since NiFi 
1.6.0 [2] as a tool to migrate flow version snapshots from one NiFi Registry to 
another. Specifically, the CLI tools “registry export-flow-version” and 
“registry import-flow-version” commands.

 

An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client SDK 
that makes the REST API for NiFi and NiFi Registry easier to work with from 
Python automation scripts. There’s a good write up on using this to interact 
with flows in NiFi Registry instances from Timothy Spann [5].

 

I hope this helps you workaround the issue you are running into. It certainly 
does not solve the specific error you have documented, but perhaps it is a 
viable alternative that avoids that exception.

 

Best,
Kevin

 

[1] 
https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/

[2] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli

[3] https://nifi.apache.org/download.html 

[4] https://github.com/Chaffelson/nipyapi 

[5] 
https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au.html
 

 

 

From: Ryan H 
Reply-To: 
Date: Sunday, May 6, 2018 at 21:05
To: 
Subject: Error Reference When Creating Template From PG Under NiFi Registry 
Version Control

 

Hi All,

We have found what seems to be an issue when creating a template of a Process 
Group that is under version control with NiFi Registry. What seems to happen is 
that there is a hard reference to the Registry Instance from the Env that the 
template was created from within the template xml file. When importing the 
template into a new Env the reference to the registry goes with the template 
and looks to stay with the new process group, is written to the flow.xml.gz, 
and causes an issue when clustered (the nodes end up not being able to sync 
after a while because of the error and goes into a disconnected state).

I'll say upfront that this somewhat defeats the purpose of having the registry 
and I'm guessing that there will be suggestions that we should either go the 
template FLDC route or go the registry FLDC route. For this purpose we cannot 
share the registry instance between the two environments and may be the case 
that we should not use the registry at this time. 

Should this be an error? Is a better way other than going in a 
searching/destroying all the old registry references out of the template and 
flow xml files? Should we manually change the registry instances in each of the 
environments to be the same guid id? 

 

As always any help or direction is greatly appreciated.

Cheers,

Ryan Howell


template.xml snippet (created template to download, which will then be uploaded 
into new env) (seems that the error is automatically created knowing that it is 
going to be a problem importing):
...


1f26b7fd-8987-491d-a2ac-4f87292ab29b
Dev-Test

c8e54d77-ce35-4be7-aff6-339a96b58e7d
MySuperCoolFlow

Re: Nifi Registry LDAP

[Kevin Doran]

That’s odd, but…. Glad it’s working for you? ☺

 

Thanks for letting me know. I’ll leave this for now, let me know if you run 
into further issues and I can look into them if the need arises again.

 

Cheers,
Kevin

 

From: Scott Howell 
Reply-To: 
Date: Tuesday, April 10, 2018 at 16:22
To: 
Subject: Re: Nifi Registry LDAP

 

So interesting thing just happened. I added my TLS parts of the 
identity-provider.xml and I restarted the server and everything is working fine.

 

I don’t want you digging into it too much but it is a strange issue I was 
receiving. 



On Apr 10, 2018, at 3:21 PM, Kevin Doran  wrote:

 

Thanks; that certainly narrows it down. It could be that you’ve uncovered a bug 
with the LdapIdentityProvider when using START_TLS. I’ll try to recreate it and 
dig into it on my end. Thanks for sharing all this info.

 

Kevin

 

From: Scott Howell 
Reply-To: 
Date: Tuesday, April 10, 2018 at 16:05
To: 
Subject: Re: Nifi Registry LDAP

 

I was able to remove the TLS information in the identity-provider.xml and was 
able to use my remote LDAP to login. So I think I am narrowing down the issue.

 

 




On Apr 10, 2018, at 2:57 PM, Kevin Doran  wrote:

 

Thanks Scott,

 

I don’t see anything wrong with your configuration. I created a free jumpcloud 
account, so I’ll try to recreate this issue and get back to you if I have any 
other insights.

 

Kevin

 

From: Scott Howell 
Reply-To: 
Date: Tuesday, April 10, 2018 at 15:54
To: 
Subject: Re: Nifi Registry LDAP

 

I was able to switch back to my local LDAP server and was able to login 
successfully. The provider I am using in identity-providers.xml is as follows:

 



ldap-identity-provider


org.apache.nifi.registry.security.ldap.LdapIdentityProvider

SIMPLE

 

cn=Manager,dc={redacted},dc=com

{redacted}

 

 

FOLLOW

10 secs

10 secs

 

ldap://{redacted}:389

ou=users,dc={redacted},dc=com

uid={0}

 

USE_DN

12 hours



 

This is a super strange issue as to why nifi works with the remote LDAP and 
nifi-registry does not. 

 



On Apr 10, 2018, at 2:18 PM, Scott Howell  wrote:

 

Thanks Kevin for sending that back,

 

This is what I see when looking at the Headers on the login. 



 

The version of Nifi-Registry I am running is 0.1.0. What confuses me is that 
this was working with my local LDAP fine. It just stopped working when I 
switched to setting up the identity-provider.xml with the same credentials as 
my nifi-cluster. 

 

 





On Apr 10, 2018, at 2:10 PM, Kevin Doran  wrote:

 

If everything is configured correctly, this error usually indicates that the 
server did not locate your login credentials when processing the login request. 
That usually means it will not even attempt to authenticate the credentials, so 
I'm not sure it is an LDAP configuration error.

 

If you want to check this manually using developer tools in a browser (e.g., 
Chrome or Firefox) you can look at the HTTP traffic to see if credentials are 
being passed to the server. NiFi Registry uses the HTTP Basic Auth protocol to 
login (credentials are encoded in the Authorization header and passed to the 
server from the login page to generate a temporary authentication token). 

 

So after clicking "Login", you should look for an HTTP POST to 
/nifi-registry-api/access/token/login, which should have an 
"Authorization" header with the value "Basic {encoded-username-and-password}"

 

If the credentials are there, it is likely something is misconfigured on the 
server side with the identity provider so that login credentials are not even 
being looked for. If the credentials are not there... well I've never seen 
that. I would probably as if your NiFi Registry Server running behind a load 
balancer or proxy that could be interfering with HTTP headers?

 

What version of NiFi Registry are you using? 0.1.0 or a version built from 
source?

 

Hope this helps,

Kevin

 

 

On 4/10/18, 14:59, "Scott Howell"  wrote:

 

Yes I did, I had Nifi-registry working with a local instances of LDAP 
running. It’s now not cooperating since I moved to using Jumpcloud. 

    

> On Apr 10, 2018, at 1:56 PM, Kevin Doran  wrote:

> 

> Hi Scott,

> 

> Did you configure nifi-registry.properties with:

> 

> nifi.registry.security.identity.provider=ldap-identity-provider

> 

> On 4/10/18, 14:53, "Scott Howell"  wrote:

> 

>Thanks for the all the help yesterday standing up LDAP for NIFI. I was 
able to troubleshoot and fix the issues myself. I am running into a unique 
issue with my Nifi-Registry when I try to login with my LDAP credentials like I 
do for the nifi cluster I get in my logs with this:

> 

>2018-04-10 18:43:15,303 INFO [NiFi Registry Web Server-18] 
o.a.n.r.w.s.NiFiRegistr

Re: Nifi Registry LDAP

[Kevin Doran]

Thanks; that certainly narrows it down. It could be that you’ve uncovered a bug 
with the LdapIdentityProvider when using START_TLS. I’ll try to recreate it and 
dig into it on my end. Thanks for sharing all this info.

 

Kevin

 

From: Scott Howell 
Reply-To: 
Date: Tuesday, April 10, 2018 at 16:05
To: 
Subject: Re: Nifi Registry LDAP

 

I was able to remove the TLS information in the identity-provider.xml and was 
able to use my remote LDAP to login. So I think I am narrowing down the issue.

 

 



On Apr 10, 2018, at 2:57 PM, Kevin Doran  wrote:

 

Thanks Scott,

 

I don’t see anything wrong with your configuration. I created a free jumpcloud 
account, so I’ll try to recreate this issue and get back to you if I have any 
other insights.

 

Kevin

 

From: Scott Howell 
Reply-To: 
Date: Tuesday, April 10, 2018 at 15:54
To: 
Subject: Re: Nifi Registry LDAP

 

I was able to switch back to my local LDAP server and was able to login 
successfully. The provider I am using in identity-providers.xml is as follows:

 



ldap-identity-provider


org.apache.nifi.registry.security.ldap.LdapIdentityProvider

SIMPLE

 

cn=Manager,dc={redacted},dc=com

{redacted}

 

 

FOLLOW

10 secs

10 secs

 

ldap://{redacted}:389

ou=users,dc={redacted},dc=com

uid={0}

 

USE_DN

12 hours



 

This is a super strange issue as to why nifi works with the remote LDAP and 
nifi-registry does not. 

 


On Apr 10, 2018, at 2:18 PM, Scott Howell  wrote:

 

Thanks Kevin for sending that back,

 

This is what I see when looking at the Headers on the login. 



 

The version of Nifi-Registry I am running is 0.1.0. What confuses me is that 
this was working with my local LDAP fine. It just stopped working when I 
switched to setting up the identity-provider.xml with the same credentials as 
my nifi-cluster. 

 

 




On Apr 10, 2018, at 2:10 PM, Kevin Doran  wrote:

 

If everything is configured correctly, this error usually indicates that the 
server did not locate your login credentials when processing the login request. 
That usually means it will not even attempt to authenticate the credentials, so 
I'm not sure it is an LDAP configuration error.

 

If you want to check this manually using developer tools in a browser (e.g., 
Chrome or Firefox) you can look at the HTTP traffic to see if credentials are 
being passed to the server. NiFi Registry uses the HTTP Basic Auth protocol to 
login (credentials are encoded in the Authorization header and passed to the 
server from the login page to generate a temporary authentication token). 

 

So after clicking "Login", you should look for an HTTP POST to 
/nifi-registry-api/access/token/login, which should have an 
"Authorization" header with the value "Basic {encoded-username-and-password}"

 

If the credentials are there, it is likely something is misconfigured on the 
server side with the identity provider so that login credentials are not even 
being looked for. If the credentials are not there... well I've never seen 
that. I would probably as if your NiFi Registry Server running behind a load 
balancer or proxy that could be interfering with HTTP headers?

 

What version of NiFi Registry are you using? 0.1.0 or a version built from 
source?

 

Hope this helps,

Kevin

 

 

On 4/10/18, 14:59, "Scott Howell"  wrote:

 

Yes I did, I had Nifi-registry working with a local instances of LDAP 
running. It’s now not cooperating since I moved to using Jumpcloud. 

    

> On Apr 10, 2018, at 1:56 PM, Kevin Doran  wrote:

> 

> Hi Scott,

> 

> Did you configure nifi-registry.properties with:

> 

> nifi.registry.security.identity.provider=ldap-identity-provider

> 

> On 4/10/18, 14:53, "Scott Howell"  wrote:

> 

>Thanks for the all the help yesterday standing up LDAP for NIFI. I was 
able to troubleshoot and fix the issues myself. I am running into a unique 
issue with my Nifi-Registry when I try to login with my LDAP credentials like I 
do for the nifi cluster I get in my logs with this:

> 

>2018-04-10 18:43:15,303 INFO [NiFi Registry Web Server-18] 
o.a.n.r.w.s.NiFiRegistrySecurityConfig AuthenticationEntryPoint invoked as no 
user identity credentials were found in the request.

> 

>My identity-providers.xml is this:

>

>  

>  ldap-identity-provider  


 org.apache.nifi.registry.security.ldap.LdapIdentityProvider 

>  START_TLS

&g

Re: Nifi Registry LDAP

[Kevin Doran]

Thanks Scott,

 

I don’t see anything wrong with your configuration. I created a free jumpcloud 
account, so I’ll try to recreate this issue and get back to you if I have any 
other insights.

 

Kevin

 

From: Scott Howell 
Reply-To: 
Date: Tuesday, April 10, 2018 at 15:54
To: 
Subject: Re: Nifi Registry LDAP

 

I was able to switch back to my local LDAP server and was able to login 
successfully. The provider I am using in identity-providers.xml is as follows:

 



ldap-identity-provider


org.apache.nifi.registry.security.ldap.LdapIdentityProvider

SIMPLE

 

cn=Manager,dc={redacted},dc=com

{redacted}

 

 

FOLLOW

10 secs

10 secs

 

ldap://{redacted}:389

ou=users,dc={redacted},dc=com

uid={0}

 

USE_DN

12 hours



 

This is a super strange issue as to why nifi works with the remote LDAP and 
nifi-registry does not. 

 

On Apr 10, 2018, at 2:18 PM, Scott Howell  wrote:

 

Thanks Kevin for sending that back,

 

This is what I see when looking at the Headers on the login. 



 

The version of Nifi-Registry I am running is 0.1.0. What confuses me is that 
this was working with my local LDAP fine. It just stopped working when I 
switched to setting up the identity-provider.xml with the same credentials as 
my nifi-cluster. 

 

 



On Apr 10, 2018, at 2:10 PM, Kevin Doran  wrote:

 

If everything is configured correctly, this error usually indicates that the 
server did not locate your login credentials when processing the login request. 
That usually means it will not even attempt to authenticate the credentials, so 
I'm not sure it is an LDAP configuration error.

 

If you want to check this manually using developer tools in a browser (e.g., 
Chrome or Firefox) you can look at the HTTP traffic to see if credentials are 
being passed to the server. NiFi Registry uses the HTTP Basic Auth protocol to 
login (credentials are encoded in the Authorization header and passed to the 
server from the login page to generate a temporary authentication token). 

 

So after clicking "Login", you should look for an HTTP POST to 
/nifi-registry-api/access/token/login, which should have an 
"Authorization" header with the value "Basic {encoded-username-and-password}"

 

If the credentials are there, it is likely something is misconfigured on the 
server side with the identity provider so that login credentials are not even 
being looked for. If the credentials are not there... well I've never seen 
that. I would probably as if your NiFi Registry Server running behind a load 
balancer or proxy that could be interfering with HTTP headers?

 

What version of NiFi Registry are you using? 0.1.0 or a version built from 
source?

 

Hope this helps,

Kevin

 

 

On 4/10/18, 14:59, "Scott Howell"  wrote:

 

Yes I did, I had Nifi-registry working with a local instances of LDAP 
running. It’s now not cooperating since I moved to using Jumpcloud. 

    

> On Apr 10, 2018, at 1:56 PM, Kevin Doran  wrote:

> 

> Hi Scott,

> 

> Did you configure nifi-registry.properties with:

> 

> nifi.registry.security.identity.provider=ldap-identity-provider

> 

> On 4/10/18, 14:53, "Scott Howell"  wrote:

> 

>Thanks for the all the help yesterday standing up LDAP for NIFI. I was 
able to troubleshoot and fix the issues myself. I am running into a unique 
issue with my Nifi-Registry when I try to login with my LDAP credentials like I 
do for the nifi cluster I get in my logs with this:

> 

>2018-04-10 18:43:15,303 INFO [NiFi Registry Web Server-18] 
o.a.n.r.w.s.NiFiRegistrySecurityConfig AuthenticationEntryPoint invoked as no 
user identity credentials were found in the request.

> 

>My identity-providers.xml is this:

>

>  

>  ldap-identity-provider  


 org.apache.nifi.registry.security.ldap.LdapIdentityProvider 

>  START_TLS

>  uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com

>  {redacted} 

>   

>  

>  /opt/certs/jumpcloud.jks 

>  {redacted} 

> JKS 

>   

>  TLSv1.2

>  

>  FOLLOW 

>  10 
secs 

>  10 secs 

>   

Re: Nifi Registry LDAP

[Kevin Doran]

If everything is configured correctly, this error usually indicates that the 
server did not locate your login credentials when processing the login request. 
That usually means it will not even attempt to authenticate the credentials, so 
I'm not sure it is an LDAP configuration error.

 

If you want to check this manually using developer tools in a browser (e.g., 
Chrome or Firefox) you can look at the HTTP traffic to see if credentials are 
being passed to the server. NiFi Registry uses the HTTP Basic Auth protocol to 
login (credentials are encoded in the Authorization header and passed to the 
server from the login page to generate a temporary authentication token). 

 

So after clicking "Login", you should look for an HTTP POST to 
/nifi-registry-api/access/token/login, which should have an 
"Authorization" header with the value "Basic {encoded-username-and-password}"

 

If the credentials are there, it is likely something is misconfigured on the 
server side with the identity provider so that login credentials are not even 
being looked for. If the credentials are not there... well I've never seen 
that. I would probably as if your NiFi Registry Server running behind a load 
balancer or proxy that could be interfering with HTTP headers?

 

What version of NiFi Registry are you using? 0.1.0 or a version built from 
source?

 

Hope this helps,

Kevin

 

 

On 4/10/18, 14:59, "Scott Howell"  wrote:

 

    Yes I did, I had Nifi-registry working with a local instances of LDAP 
running. It’s now not cooperating since I moved to using Jumpcloud. 

    

> On Apr 10, 2018, at 1:56 PM, Kevin Doran  wrote:

    > 

> Hi Scott,

    > 

> Did you configure nifi-registry.properties with:

    > 

> nifi.registry.security.identity.provider=ldap-identity-provider

    > 

> On 4/10/18, 14:53, "Scott Howell"  wrote:

    > 

>    Thanks for the all the help yesterday standing up LDAP for NIFI. I was 
able to troubleshoot and fix the issues myself. I am running into a unique 
issue with my Nifi-Registry when I try to login with my LDAP credentials like I 
do for the nifi cluster I get in my logs with this:

    > 

>    2018-04-10 18:43:15,303 INFO [NiFi Registry Web Server-18] 
o.a.n.r.w.s.NiFiRegistrySecurityConfig AuthenticationEntryPoint invoked as no 
user identity credentials were found in the request.

    > 

>    My identity-providers.xml is this:

    >    

    >      

>      ldap-identity-provider  

        
 org.apache.nifi.registry.security.ldap.LdapIdentityProvider 

>      START_TLS

    >      uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com

    >      {redacted} 

>       

>      

    >      /opt/certs/jumpcloud.jks 

>      {redacted}     

>     JKS 

>       

>      TLSv1.2

    >      

    >      FOLLOW 

>      10 
secs 

>      10 secs 

>      ldap://ldap.jumpcloud.com:389 

>      ou=Users,o={redacted},dc=jumpcloud,dc=com 

>      uid={0} 

>      USE_USERNAME 

>      12 
hours 

>      

    >    

    > 

>    For the most part I grabbed most of this from my Nifi node 
login-identity-providers.xml but I seem to have something messed up.

    > 

> 





 

Re: Nifi Registry LDAP

[Kevin Doran]

Hi Scott,

Did you configure nifi-registry.properties with:

nifi.registry.security.identity.provider=ldap-identity-provider

On 4/10/18, 14:53, "Scott Howell"  wrote:

Thanks for the all the help yesterday standing up LDAP for NIFI. I was able 
to troubleshoot and fix the issues myself. I am running into a unique issue 
with my Nifi-Registry when I try to login with my LDAP credentials like I do 
for the nifi cluster I get in my logs with this:

2018-04-10 18:43:15,303 INFO [NiFi Registry Web Server-18] 
o.a.n.r.w.s.NiFiRegistrySecurityConfig AuthenticationEntryPoint invoked as no 
user identity credentials were found in the request.

My identity-providers.xml is this:

 
 ldap-identity-provider
 
org.apache.nifi.registry.security.ldap.LdapIdentityProvider 
 START_TLS
 uid=nifi,ou=Users,o={redacted},dc=jumpcloud,dc=com
 {redacted} 
  
 
 /opt/certs/jumpcloud.jks 
 {redacted}
JKS 
  
 TLSv1.2
 
 FOLLOW 
 10 secs 
 10 secs 
 ldap://ldap.jumpcloud.com:389 
 ou=Users,o={redacted},dc=jumpcloud,dc=com 
 uid={0} 
 USE_USERNAME 
 12 hours 
 


For the most part I grabbed most of this from my Nifi node 
login-identity-providers.xml but I seem to have something messed up.

Re: Integrating nifi with cloud based LDAP JumpCloud

[Kevin Doran]

Scott,

I've never implemented NiFi with JumpCloud, but speculating as to what could be 
the cause of your error, it could be the User Search Base/Filter configuration 
values. Can you share the contents of your login-identity-providers.xml 
(removing any sensitive values such as ldap credentials)?

Thanks,
Kevin

On 4/9/18, 14:53, "Scott Howell"  wrote:

I was wondering if there was anyone on the user group that had successfully 
integrated their NIFI authentication to work with Jumpcloud LDAP. I have 
followed the steps Jumpcloud provides with adding the correct credentials to 
the the NIFI login-identity-providers.xml but I am getting an error of “Unable 
to validate the supplied credentials. Please contact the system administrator.” 
In the UI in my nifi-user.log I am seeing [LDAP: error code 32 - No Such 
Object] when its trying to look up the LDAP user.

Scott

Re: Ranger-plugin authorises "anonymous" for /flow

[Kevin Doran]

Good catch, Bryan. It does sound like that could explain what is going on in 
these cases.

Regarding  this:

>I think one way to support this might be to bring back the "Allow
>Anonymous" flag in the Ranger Authorizer config.
>
>This way if this value is set to false, then before doing anything
>else the authorizer would check if the incoming identity is
>"anonymous" and reject the request before even checking the actual
>policies from Ranger.

Would this be a good setting to have in general, perhaps for managed authorizer 
or in nifi.properties, so that it could be toggled for any configured 
authorizer?

On 4/9/18, 11:32, "Bryan Bende"  wrote:

Actually, thinking about this more, I think the issue is that you
currently can't use Ranger's {USER} concept because NiFi always has an
"anonymous" user, and there is no way in Ranger to exclude "anonymous"
from {USER}.

What is happening is something like this...

- You enter the NiFi URL in your browser which loads some of the front-end 
code
- The front-end code then attempts to access a server side resource
that requires authorization
- This request is happening as the anonymous user, and if it fails
with an unauthorized response, then the front-end knows to present the
login page
- In this case, the request is succeeding because of the policy you
have with {USER}

I think one way to support this might be to bring back the "Allow
Anonymous" flag in the Ranger Authorizer config.

This way if this value is set to false, then before doing anything
else the authorizer would check if the incoming identity is
"anonymous" and reject the request before even checking the actual
policies from Ranger.

On Mon, Apr 9, 2018 at 11:10 AM, Bryan Bende  wrote:
> Ok, so you end up in the NiFi UI with the identity in the top right
> saying "Anonymous user" ? (as show in the screenshots of that hwx
> forum)
>
> I think we need to dig into what is happening during authentication...
>
> Can you do a fresh login to NiFi using your LDAP credentials, and then
> capture the content from nifi-user.log that occurred during this
> login?
>
>
> On Mon, Apr 9, 2018 at 10:37 AM, Meixner, Johannes
>  wrote:
>>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>
>> Good question. NiFi uses its LdapProvider (with Authentication Strategy =
>> SIMPLE, Identity Strategy = USE_USERNAME) to identify and the
>> RangerNifiAuthorizer to authorise users.
>>
>> I'm not quite sure yet what authenticates the anonymous user, but I've 
found
>> similar reports in a Hortonworks thread [1] without obvious solution.
>>
>> [1]
>> 
https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html
>>
>> On Mon, Apr 9, 2018 at 4:09 PM, Bryan Bende  wrote:
>>>
>>> Ah thanks for the info, didn't know that.
>>>
>>> So you have a policy in Ranger where the resource is "/flow", the
>>> action is READ, and the users/groups is "{USER}", and then you are
>>> saying an "anonymous" user can retrieve the flow?
>>>
>>> I'm assuming that since your Ranger is backed by an LDAP, that you
>>> also have configured NiFi's LDAP Login Identity Provider?
>>>
>>> If so, then how are you authenticating to NiFi as an "anonymous" user?
>>>
>>> In a secure NiFi you should always have to authenticate as some
>>> identity, the anonymous user is only used in an unsecured NiFi when
>>> there is not authentication/authorization taking place.
>>>
>>>
>>> On Mon, Apr 9, 2018 at 9:51 AM, Meixner, Johannes
>>>  wrote:
>>> > Hi Bryan,
>>> >
>>> > It's a placeholder in Ranger for usernames, see
>>> >
>>> > 
https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable
>>> >
>>> > which is used by certain matcher classes.
>>> >
>>> > On Mon, Apr 9, 2018 at 3:28 PM, Bryan Bende  wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> I don't see any issue with the code you linked to. It's saying "if 
the
>>> >> ranger policies say the operation is allowed, then return approved".
>>> >>
>>> >> Is '{USER}' a special syntax in Ranger? or are you using that as a
>>> >> placeholder in email so you don't have to provide the real user
>>> >> identity?
>>> >>
>>> >> I haven't seen that syntax before so just trying to understand what
>>> >> {USER} and {OWNER} mean here.
>>> >>
>>> >> -Bryan
>>> >>
>>> >>
>>> >> On Mon, Apr 9, 2018 at 3:55 AM, Meixner, Johannes
>>> >>  wrote:
>>> >> > I'm trying to harden my NiFi instance's authorizations and auditing
>>> >> > using
>>> >> > Ranger (which is backed by an LDAP instance).
>>> >> >
>>> >> > In Ranger I have defined

Re: AWS CloudWatch

[Kevin Doran]

Hi Laurens,

I've never done this but here are some ideas you could experiment with.

Assuming the logs are coming from something like an application running on an 
EC2 instance, there are a number of ways you could probably expose them to NiFi 
without going through CloudWatch logs. There are a number of articles and blog 
posts [1] that describe how to do this. For instance, if your app logging 
framework supports an appender that can go direct to NiFi, or logging locally 
and running a local MiNiFi agent running a simple flow that tails a log file 
and sends the contents to NiFi using the site to site protocol. This would have 
the advantage of attaching provenance metadata to your logs right at the 
source, in case that is valuable for your use case.

I'm assuming you want to use CloudWatch for some other reason/integration as 
part of your overall architecture, or that the source is not something you 
could run a MiNiFi agent on (i.e., another AWS service). There is a backlogged 
NiFi JIRA for reading from a Kinesis Stream [2], but in absence of that feature 
being implemented, you would have to have something between a Kinesis 
Stream/Firehose carrying the log data and NiFi.  Some ideas include:

- Kinesis > S3 > NiFi (as you suggested) could work
- Kinesis > Redshift > NiFi using one of NiFi's SQL processors (e.g., 
QueryDatabaseTable) to get data out of Redshift. 
- Kinesis > AWS Lambda > NiFi using one of NiFi's listener processors 
(ListenUDP or HTTP) on the NiFi side to create an endpoint an AWS Lambda 
function could post to.
- Kinesis > [any streaming/messaging framework that NiFi can integrate with, 
such as Kafka or JMS] > NiFi

The last one is kind of a shot in the dark, but given the number of libraries 
and protocols out there that are interoperable, it might be possible to 
leverage one if you can find a good third-party tool for getting a data stream 
out of Kinesis into a system that natively interoperates with another messaging 
protocol.

I'd be interested to hear if you come up with an approach that works well for 
your use case. Hope this helps!

Kevin

[1] 
https://bryanbende.com/development/2015/05/17/collecting-logs-with-apache-nifi
[2] https://issues.apache.org/jira/browse/NIFI-2892

On 3/23/18, 14:15, "Laurens Vets"  wrote:

Hi list,

Has anyone tried to setup NiFi to get real-time CloudWatch logs somehow?

I can export CloudWatch logs to S3, but it might take up to 12 hours for 
them to become available. I suspect the only other option is to go 
through AWS Kinesis Firehose to stream to S3 and have NiFi pick up the 
logs there?

Any ideas/comments/suggestions are highly appreciated :)

Re: Unknown user w/ Docker image

[Kevin Doran]

It’s of course not a recommended best practice for real-word use, but for a 
demo, yes that should. Keep in mind that if you add those files, you cannot 
change the initial admin identity using the Docker environment variables (as 
the authorizers.xml initial admin only gets applied when 
users/authorizations.xml files are empty at startup), so if you need to 
generate a new cert for your initial admin or something, you’ll need to edit 
those files or regenerate them.

 

Glad it’s working for you!

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 10:24
To: 
Subject: Re: Unknown user w/ Docker image

 

Thanks. Now test can login. One last question... if I copy users.xml and 
authorizations.xml to the host file system and inject them into later runs of 
the Docker image through volume references, would that work for making a 
reproducible demo?

 

On Thu, Mar 22, 2018 at 10:08 AM, Kevin Doran  wrote:

Yeah, from looking at your Docker compose file, your LDAP search base/filter is 
configured as:

LDAP_USER_SEARCH_BASE='ou=people,dc=nifi,dc=com'
LDAP_USER_SEARCH_FILTER='uid={0}'

 

This means that NiFi is going to search the directory for any nodes that are 
children of 'ou=people,dc=nifi,dc=com', match any LDAP entry with an attribute 
uid={0}, and use the substring matched by {0} as the identity.

 

In this case that is “test”. So that is the user identity (“test”) you want to 
add to NiFi using the initial admin. Grant them access to the right resources 
(e.g., the UI), and then you should be able to login with test/password.

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 10:03


To: 
Subject: Re: Unknown user w/ Docker image

 

I added two entries:

uid=test

cn=test, ou=people, dc=nifi, dc=com

 

Tried logging in w/ test/password (what the LDIF uses)

 

Got: Unknown user with identity 'test'. Contact the system administrator.

 

Any ideas?

 

On Thu, Mar 22, 2018 at 9:34 AM, Kevin Doran  wrote:

Mike,

 

To my knowledge, the Docker image does not yet have support for adding the 
LdapUserGroupProvider to authorizers.xml. It only adds the LdapProvider to 
login-identity-provider.xml. This means you should be able to 
login/authenticate as an LDAP user, but users and group will not sync in order 
to create authorization rules. You will have to manually add users (with 
identities that match how your login-identity-provider is configured) using the 
initial admin.

 

I’ve opened a JIRA to add LdapUserGroupProvider support to the NiFi Docker 
image [1].

 

Also, it looks like there is already a JIRA for the AUTH=ldap documentation 
issue [2].

 

Kevin

 

[1] https://issues.apache.org/jira/browse/NIFI-5002 

[2] https://issues.apache.org/jira/browse/NIFI-4934 

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 09:26


To: 
Subject: Re: Unknown user w/ Docker image

 

Thanks. I fixed that, but it's still not returning any users from the LDAP. 
It's weird because the LDAP docker image is set up using the same configuration 
from Pierre's blog posts that I've gotten to work outside of Docker. I'm also 
not seeing anything in the logs indicating that it's trying the LDAP query.

 

On Thu, Mar 22, 2018 at 8:30 AM, Kevin Doran  wrote:

Sorry, meant to include the link to start.sh, which is in our codebase [1].  
I’m only pointing it out b/c it looked like in your Docker compose file that 
you wanted this to be an LDAP demo.

 

[1] 
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L30

 

From: Kevin Doran 
Date: Thursday, March 22, 2018 at 08:27
To: 
Subject: Re: Unknown user w/ Docker image

 

Good eye, Pierre.

 

Mike, unrelated to the initial admin question, but anticipating something you 
might run int o after you get that part working. Change the "AUTH=tls" 
environment variable value to "AUTH=ldap". (I know the README file for the 
docker image uses ‘AUTH=tls’ in the documentation for LDAP setup; that is an 
error.­ I’ll open a PR to correct the documentation. To confirm how it works, 
look at the start.sh file)

 

Cheers,
Kevin

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 08:25
To: 
Subject: Re: Unknown user w/ Docker image

 

They were. I did a copy from the Docker Hub page and didn't think they'd harm 
anything in the YAML. Removing them got initialAdmin to work.

 

On Thu, Mar 22, 2018 at 8:20 AM, Pierre Villard  
wrote:

Hmmm no... the single quotes must be the issue here... I would expect
identity="CN=initialAdmin, OU=NIFI"

In your yaml file, I'd try to use double quotes around your property values.

 

2018-03-22 13:16 GMT+01:00 Mike Thomsen :

Yeah, that's the weird part. It looks valid to me:

 















 

 

On Thu, Mar 22, 2018 at 8:07 AM, Pierre Villard  
wrote:

Hey Mike,

Can you check the users.xml file created by NiF

Re: Unknown user w/ Docker image

[Kevin Doran]

Yeah, from looking at your Docker compose file, your LDAP search base/filter is 
configured as:


LDAP_USER_SEARCH_BASE='ou=people,dc=nifi,dc=com'
LDAP_USER_SEARCH_FILTER='uid={0}'

 

This means that NiFi is going to search the directory for any nodes that are 
children of 'ou=people,dc=nifi,dc=com', match any LDAP entry with an attribute 
uid={0}, and use the substring matched by {0} as the identity.

 

In this case that is “test”. So that is the user identity (“test”) you want to 
add to NiFi using the initial admin. Grant them access to the right resources 
(e.g., the UI), and then you should be able to login with test/password.

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 10:03
To: 
Subject: Re: Unknown user w/ Docker image

 

I added two entries:

uid=test

cn=test, ou=people, dc=nifi, dc=com

 

Tried logging in w/ test/password (what the LDIF uses)

 

Got: Unknown user with identity 'test'. Contact the system administrator.

 

Any ideas?

 

On Thu, Mar 22, 2018 at 9:34 AM, Kevin Doran  wrote:

Mike,

 

To my knowledge, the Docker image does not yet have support for adding the 
LdapUserGroupProvider to authorizers.xml. It only adds the LdapProvider to 
login-identity-provider.xml. This means you should be able to 
login/authenticate as an LDAP user, but users and group will not sync in order 
to create authorization rules. You will have to manually add users (with 
identities that match how your login-identity-provider is configured) using the 
initial admin.

 

I’ve opened a JIRA to add LdapUserGroupProvider support to the NiFi Docker 
image [1].

 

Also, it looks like there is already a JIRA for the AUTH=ldap documentation 
issue [2].

 

Kevin

 

[1] https://issues.apache.org/jira/browse/NIFI-5002 

[2] https://issues.apache.org/jira/browse/NIFI-4934 

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 09:26


To: 
Subject: Re: Unknown user w/ Docker image

 

Thanks. I fixed that, but it's still not returning any users from the LDAP. 
It's weird because the LDAP docker image is set up using the same configuration 
from Pierre's blog posts that I've gotten to work outside of Docker. I'm also 
not seeing anything in the logs indicating that it's trying the LDAP query.

 

On Thu, Mar 22, 2018 at 8:30 AM, Kevin Doran  wrote:

Sorry, meant to include the link to start.sh, which is in our codebase [1].  
I’m only pointing it out b/c it looked like in your Docker compose file that 
you wanted this to be an LDAP demo.

 

[1] 
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L30

 

From: Kevin Doran 
Date: Thursday, March 22, 2018 at 08:27
To: 
Subject: Re: Unknown user w/ Docker image

 

Good eye, Pierre.

 

Mike, unrelated to the initial admin question, but anticipating something you 
might run int o after you get that part working. Change the "AUTH=tls" 
environment variable value to "AUTH=ldap". (I know the README file for the 
docker image uses ‘AUTH=tls’ in the documentation for LDAP setup; that is an 
error.­ I’ll open a PR to correct the documentation. To confirm how it works, 
look at the start.sh file)

 

Cheers,
Kevin

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 08:25
To: 
Subject: Re: Unknown user w/ Docker image

 

They were. I did a copy from the Docker Hub page and didn't think they'd harm 
anything in the YAML. Removing them got initialAdmin to work.

 

On Thu, Mar 22, 2018 at 8:20 AM, Pierre Villard  
wrote:

Hmmm no... the single quotes must be the issue here... I would expect
identity="CN=initialAdmin, OU=NIFI"

In your yaml file, I'd try to use double quotes around your property values.

 

2018-03-22 13:16 GMT+01:00 Mike Thomsen :

Yeah, that's the weird part. It looks valid to me:

 















 

 

On Thu, Mar 22, 2018 at 8:07 AM, Pierre Villard  
wrote:

Hey Mike,

Can you check the users.xml file created by NiFi when it started for the first 
time?

 

2018-03-22 12:41 GMT+01:00 Mike Thomsen :

I'm trying to use the Docker image to set up a secure NiFi demo, and am running 
into this error:

 

Unknown user with identity 'CN=initialAdmin, OU=NIFI'. Contact the system 
administrator.

 

SSL works, I verified that the owner in the cert is "CN=initialAdmin, OU=NIFI"

 

I've attached the Docker Compose configuration that I'm using. Any input would 
be appreciated.

 

Thanks,

 

Mike

 

 

 

 

 

 

Re: Unknown user w/ Docker image

[Kevin Doran]

Mike,

 

To my knowledge, the Docker image does not yet have support for adding the 
LdapUserGroupProvider to authorizers.xml. It only adds the LdapProvider to 
login-identity-provider.xml. This means you should be able to 
login/authenticate as an LDAP user, but users and group will not sync in order 
to create authorization rules. You will have to manually add users (with 
identities that match how your login-identity-provider is configured) using the 
initial admin.

 

I’ve opened a JIRA to add LdapUserGroupProvider support to the NiFi Docker 
image [1].

 

Also, it looks like there is already a JIRA for the AUTH=ldap documentation 
issue [2].

 

Kevin

 

[1] https://issues.apache.org/jira/browse/NIFI-5002 

[2] https://issues.apache.org/jira/browse/NIFI-4934 

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 09:26
To: 
Subject: Re: Unknown user w/ Docker image

 

Thanks. I fixed that, but it's still not returning any users from the LDAP. 
It's weird because the LDAP docker image is set up using the same configuration 
from Pierre's blog posts that I've gotten to work outside of Docker. I'm also 
not seeing anything in the logs indicating that it's trying the LDAP query.

 

On Thu, Mar 22, 2018 at 8:30 AM, Kevin Doran  wrote:

Sorry, meant to include the link to start.sh, which is in our codebase [1].  
I’m only pointing it out b/c it looked like in your Docker compose file that 
you wanted this to be an LDAP demo.

 

[1] 
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L30

 

From: Kevin Doran 
Date: Thursday, March 22, 2018 at 08:27
To: 
Subject: Re: Unknown user w/ Docker image

 

Good eye, Pierre.

 

Mike, unrelated to the initial admin question, but anticipating something you 
might run int o after you get that part working. Change the "AUTH=tls" 
environment variable value to "AUTH=ldap". (I know the README file for the 
docker image uses ‘AUTH=tls’ in the documentation for LDAP setup; that is an 
error.­ I’ll open a PR to correct the documentation. To confirm how it works, 
look at the start.sh file)

 

Cheers,
Kevin

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 08:25
To: 
Subject: Re: Unknown user w/ Docker image

 

They were. I did a copy from the Docker Hub page and didn't think they'd harm 
anything in the YAML. Removing them got initialAdmin to work.

 

On Thu, Mar 22, 2018 at 8:20 AM, Pierre Villard  
wrote:

Hmmm no... the single quotes must be the issue here... I would expect
identity="CN=initialAdmin, OU=NIFI"

In your yaml file, I'd try to use double quotes around your property values.

 

2018-03-22 13:16 GMT+01:00 Mike Thomsen :

Yeah, that's the weird part. It looks valid to me:

 















 

 

On Thu, Mar 22, 2018 at 8:07 AM, Pierre Villard  
wrote:

Hey Mike,

Can you check the users.xml file created by NiFi when it started for the first 
time?

 

2018-03-22 12:41 GMT+01:00 Mike Thomsen :

I'm trying to use the Docker image to set up a secure NiFi demo, and am running 
into this error:

 

Unknown user with identity 'CN=initialAdmin, OU=NIFI'. Contact the system 
administrator.

 

SSL works, I verified that the owner in the cert is "CN=initialAdmin, OU=NIFI"

 

I've attached the Docker Compose configuration that I'm using. Any input would 
be appreciated.

 

Thanks,

 

Mike

 

 

 

 

 

Re: Unknown user w/ Docker image

[Kevin Doran]

Sorry, meant to include the link to start.sh, which is in our codebase [1].  
I’m only pointing it out b/c it looked like in your Docker compose file that 
you wanted this to be an LDAP demo.

 

[1] 
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh#L30

 

From: Kevin Doran 
Date: Thursday, March 22, 2018 at 08:27
To: 
Subject: Re: Unknown user w/ Docker image

 

Good eye, Pierre.

 

Mike, unrelated to the initial admin question, but anticipating something you 
might run int o after you get that part working. Change the "AUTH=tls" 
environment variable value to "AUTH=ldap". (I know the README file for the 
docker image uses ‘AUTH=tls’ in the documentation for LDAP setup; that is an 
error.­ I’ll open a PR to correct the documentation. To confirm how it works, 
look at the start.sh file)

 

Cheers,
Kevin

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 08:25
To: 
Subject: Re: Unknown user w/ Docker image

 

They were. I did a copy from the Docker Hub page and didn't think they'd harm 
anything in the YAML. Removing them got initialAdmin to work.

 

On Thu, Mar 22, 2018 at 8:20 AM, Pierre Villard  
wrote:

Hmmm no... the single quotes must be the issue here... I would expect
identity="CN=initialAdmin, OU=NIFI"

In your yaml file, I'd try to use double quotes around your property values.

 

2018-03-22 13:16 GMT+01:00 Mike Thomsen :

Yeah, that's the weird part. It looks valid to me:

 















 

 

On Thu, Mar 22, 2018 at 8:07 AM, Pierre Villard  
wrote:

Hey Mike,

Can you check the users.xml file created by NiFi when it started for the first 
time?

 

2018-03-22 12:41 GMT+01:00 Mike Thomsen :

I'm trying to use the Docker image to set up a secure NiFi demo, and am running 
into this error:

 

Unknown user with identity 'CN=initialAdmin, OU=NIFI'. Contact the system 
administrator.

 

SSL works, I verified that the owner in the cert is "CN=initialAdmin, OU=NIFI"

 

I've attached the Docker Compose configuration that I'm using. Any input would 
be appreciated.

 

Thanks,

 

Mike

 

 

 

 

Re: Unknown user w/ Docker image

[Kevin Doran]

Good eye, Pierre.

 

Mike, unrelated to the initial admin question, but anticipating something you 
might run int o after you get that part working. Change the "AUTH=tls" 
environment variable value to "AUTH=ldap". (I know the README file for the 
docker image uses ‘AUTH=tls’ in the documentation for LDAP setup; that is an 
error.­ I’ll open a PR to correct the documentation. To confirm how it works, 
look at the start.sh file)

 

Cheers,
Kevin

 

From: Mike Thomsen 
Reply-To: 
Date: Thursday, March 22, 2018 at 08:25
To: 
Subject: Re: Unknown user w/ Docker image

 

They were. I did a copy from the Docker Hub page and didn't think they'd harm 
anything in the YAML. Removing them got initialAdmin to work.

 

On Thu, Mar 22, 2018 at 8:20 AM, Pierre Villard  
wrote:

Hmmm no... the single quotes must be the issue here... I would expect
identity="CN=initialAdmin, OU=NIFI"

In your yaml file, I'd try to use double quotes around your property values.

 

2018-03-22 13:16 GMT+01:00 Mike Thomsen :

Yeah, that's the weird part. It looks valid to me:

 















 

 

On Thu, Mar 22, 2018 at 8:07 AM, Pierre Villard  
wrote:

Hey Mike,

Can you check the users.xml file created by NiFi when it started for the first 
time?

 

2018-03-22 12:41 GMT+01:00 Mike Thomsen :

I'm trying to use the Docker image to set up a secure NiFi demo, and am running 
into this error:

 

Unknown user with identity 'CN=initialAdmin, OU=NIFI'. Contact the system 
administrator.

 

SSL works, I verified that the owner in the cert is "CN=initialAdmin, OU=NIFI"

 

I've attached the Docker Compose configuration that I'm using. Any input would 
be appreciated.

 

Thanks,

 

Mike

 

 

 

 

Re: Issue with AWS ELB on secure nifi-registry

[Kevin Doran]

Ok, that use case should be fine.

If it were an authorization issue you would see something in the logs saying 
that an authorization attempt failed and the server is responding with a 403.  
Just to be sure, can you enable debug logging if you haven't already, i.e., in 
your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to 
debug:




If there is nothing being written to nifi-registry-app.log, it points towards a 
connection issue, so I would double check your host, port, and TLS settings. 
You'll have to get an HTTPS cert from a root CA or configure your ELB to trust 
your company's self-signed cert (again, not sure if/how to do this, but I 
assume there should be some way to configure it. It might require settings not 
exposed in the AWS web console.)

On 3/19/18, 10:51, "Scott Howell"  wrote:

Thanks Kevin,

I am just using the ELB to go from the public subnet to the private subnet. 
I will not have multiple instances running of registry. 

I will say on my authorizers.xml there is one difference between my nifi 
instance. On my nifi instance I am using file-provider for 
nifi.security.user.authorizer in my nifi.properties. I don’t think from reading 
the documents for nifi-registry that I can use that. If there is a way that 
might be my problem. I was running into some issues with my nifi instance when 
I was using managed-authorizers instead of file-provider. 



> On Mar 19, 2018, at 9:35 AM, Kevin Doran  wrote:
> 
> Hey Scott,
> 
> Assuming you are using two-way TLS with client certificates for 
authentication, I recommend configuring your ELB for TCP passthrough so that 
the TLS handshake is between the end-client and the NiFi Registry Server (in 
other words, no decryption/termination of the TLS connection happens in the 
ELB). If you are using some other form of authentication (e.g., LDAP), you will 
need to configure your ELB to trust the self-signed key NiFi Registry is using. 
I'm not sure how to do that as I've never run an ELB with that configuration 
before.
> 
> Also, just a note about using an ELB with NiFi Registry:
> 
> NiFi Registry is currently only supports single-instance use as persisted 
data and in-memory state is not synced between multiple instances. Are you 
hoping to use the ELB for actual load balancing, or is it just to take 
advantage of other ELB features, such as forwarding and security group rules? 
If the plan is to load balance multiple Registry instances, just be aware that 
you will probably run into some unexpected behavior. (As you mentioned using 
authorization, that is one case where I know the in-memory cache of the 
persisted data will not refresh across instances, so even if you were using 
some sort of shared network file system attached to multiple Registry 
instances, such as EFS, it would not work the way you hope.)
> 
> Hope this helps,
> Kevin
> 
> On 3/19/18, 10:20, "Scott Howell"  wrote:
> 
>Thanks for the quick response.
> 
>A couple of things I am seeing.
> 
>1. There is no error, I don’t see anything in the logs once the 
service comes up. This is because the health check is not even hitting the 
instance when secure. 
> 
>2. Nothing interesting in the nifi-registry-app.logs. That was my 
concern because on my nifi instance I can see the health check hitting the 
instance from the ELB. This does not happen on the nifi-registry instance.  I 
see the service startup and it tells me what domain and port I can access the 
UI but nothing else after that.
> 
>3. When I am on an instances in the same private subnet I am able to 
curl to the instance I get the TLS SSL which tells me the keystore is on the 
server. I am using a JKS keystore that is self-signed by the company I work for.
> 
>> On Mar 19, 2018, at 9:10 AM, Bryan Bende  wrote:
>> 
>> Hello,
>> 
>> What error are you getting when you cannot access the UI?
>> 
>> Is there anything interesting in nifi-registry-app.log regarding
>> authentication/authorization when this happens?
>> 
>> Can you access the UI securely without going through the ELB?
>> 
>> Thanks,
>> 
>> Bryan
>> 
>> 
>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell 
 wrote:
>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. 
Everything was working great and was able to access the UI anonymously. I set 
up the authorization just like on my nifi instances along with the authorizers 
and identity-provider. The service comes up without errors and everything looks 
good but the health check does not pass and I cannot access the UI to login. I 
was wondering if anyone else has ran into this issue using nifi-registry.
> 
> 
> 
> 

Re: Issue with AWS ELB on secure nifi-registry

[Kevin Doran]

Hey Scott,

Assuming you are using two-way TLS with client certificates for authentication, 
I recommend configuring your ELB for TCP passthrough so that the TLS handshake 
is between the end-client and the NiFi Registry Server (in other words, no 
decryption/termination of the TLS connection happens in the ELB). If you are 
using some other form of authentication (e.g., LDAP), you will need to 
configure your ELB to trust the self-signed key NiFi Registry is using. I'm not 
sure how to do that as I've never run an ELB with that configuration before.

Also, just a note about using an ELB with NiFi Registry:

NiFi Registry is currently only supports single-instance use as persisted data 
and in-memory state is not synced between multiple instances. Are you hoping to 
use the ELB for actual load balancing, or is it just to take advantage of other 
ELB features, such as forwarding and security group rules? If the plan is to 
load balance multiple Registry instances, just be aware that you will probably 
run into some unexpected behavior. (As you mentioned using authorization, that 
is one case where I know the in-memory cache of the persisted data will not 
refresh across instances, so even if you were using some sort of shared network 
file system attached to multiple Registry instances, such as EFS, it would not 
work the way you hope.)

Hope this helps,
Kevin

On 3/19/18, 10:20, "Scott Howell"  wrote:

Thanks for the quick response.

A couple of things I am seeing.

1. There is no error, I don’t see anything in the logs once the service 
comes up. This is because the health check is not even hitting the instance 
when secure. 

2. Nothing interesting in the nifi-registry-app.logs. That was my concern 
because on my nifi instance I can see the health check hitting the instance 
from the ELB. This does not happen on the nifi-registry instance.  I see the 
service startup and it tells me what domain and port I can access the UI but 
nothing else after that.

3. When I am on an instances in the same private subnet I am able to curl 
to the instance I get the TLS SSL which tells me the keystore is on the server. 
I am using a JKS keystore that is self-signed by the company I work for.

> On Mar 19, 2018, at 9:10 AM, Bryan Bende  wrote:
> 
> Hello,
> 
> What error are you getting when you cannot access the UI?
> 
> Is there anything interesting in nifi-registry-app.log regarding
> authentication/authorization when this happens?
> 
> Can you access the UI securely without going through the ELB?
> 
> Thanks,
> 
> Bryan
> 
> 
> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell  
wrote:
>> I was able to stand up nifi-registry behind an AWS ELB non-secure. 
Everything was working great and was able to access the UI anonymously. I set 
up the authorization just like on my nifi instances along with the authorizers 
and identity-provider. The service comes up without errors and everything looks 
good but the health check does not pass and I cannot access the UI to login. I 
was wondering if anyone else has ran into this issue using nifi-registry.

Re: Error when sending HTTPS request to Nifi using the Restful API

[Kevin Doran]

Any secured (HTTPS) NiFi instance requires client authentication for every REST 
API request, so the first thing you need to know is how your NiFi server is 
configured to authenticate clients. There are lots of options, including client 
certificates, login with LDAP username and password, or authenticate with 
Kerberos ticket, Knox identity, or OpenId connect. These options are explained 
in the NiFi Administration Guide [1]. One way to determine this is by looking 
at the files in the /conf directory (i.e., nifi.properties).

 

Once you know how the NiFi server authenticates client requests, you must 
configure the REST API client you are using to perform the same authentication 
steps. This varies based on authentication mechanism, but is basically either a 
client certificate provided during the TLS connection handshake or obtaining a 
JWT access token using one of the /access/* REST API endpoints that uses some 
form of client credential to authenticate and generate a signed JWT token that 
is then passed back to the server on later requests.

 

You may find it helpful to use your browser’s developer console while using the 
NiFi Web UI to look at the REST API network interactions between the UI and the 
server. This may show you how to obtain an authentication token (assuming that 
is how your NiFi instance is configured) and also how to perform operations 
such as start or stop a processor.

 

Lastly, while this can be done using CURL, you may find it helpful to use one 
of the community clients, such as NiPyApi [2], which abstracts some of the 
details of interacting with the NiFi REST API.

 

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
 

[2] https://github.com/Chaffelson/nipyapi 

 

From: Tian TD Deng 
Reply-To: 
Date: Friday, March 16, 2018 at 11:20
To: 
Cc: 
Subject: Re: Error when sending HTTPS request to Nifi using the Restful API

 

Hi Kevin,

 

Thanks a lot for your help.

I tried to modify the property " nifi.security.needClientAuth " from true to 
false, unfortunately, it didn't work, displayed the same error.

 

However, I find that the  "nifi.security.user.login.identity.provider"  is not 
set to any value in the nifi.properties.

 

So in this case, what should I do to send a HTTPS request to the Nifi Restful 
API to do something like start/stop the processor using curl?

 

Thanks & Best Regards
 

Joey(Tian) Deng

---
IT Specialist
IBM China Global Delivery Center
Tel: (+86) 27 59269644
E-mail: dengt...@cn.ibm.com

WhatsApp: +86 15671158671
---

 

 

- Original message -
From: Kevin Doran 
To: 
Cc:
Subject: Re: Error when sending HTTPS request to Nifi using the Restful API
Date: Fri, Mar 16, 2018 9:58 PM
 

When using access tokens, make sure your NiFi instance is configured to not 
require client certificates for the TLS certificates:

 

nifi.security.needClientAuth=false

 

This is because you only want the NiFi server (not the client) to establish its 
identity with a TLS certificate; the client will establish identity after the 
TLS connection is established by passing the username and password credentials 
that will be validated by the Login Identity Provider you configured for NiFi. 
For example:

 

nifi.security.user.login.identity.provider=ldap-provider

 

where “ldap-provider” is configured in login-identity-providers.xml

 

Hope this helps!

Kevin

 

From: Tian TD Deng 
Reply-To: 
Date: Friday, March 16, 2018 at 04:51
To: 
Subject: Error when sending HTTPS request to Nifi using the Restful API

 

Dear All,

 

I was trying to send HTTPS request to Nifi using the Restful API via curl, 
however, it didn't work as expected.

 

curl -k --tlsv1.2  https://localhost:8443/nifi-api/access/token --data 
'username=test&password=password'

 

 

Could you please help to give some advice about this?

Thanks a lot!

 

Thanks & Best Regards
 

Joey(Tian) Deng

---
IT Specialist
IBM China Global Delivery Center
Tel: (+86) 27 59269644
E-mail: dengt...@cn.ibm.com

WhatsApp: +86 15671158671
---