Re: SSL/LDAP Configuration

[Andy LoPresto]

Ok to diagnose, look at the users.xml to see if there is a user matching that 
DN, and if so, it should have a UUID. Then in the authorizations.xml there 
should be policies defined in a hierarchical manner associating those users 
with a right on a specific resource (component/processor). If so, you can 
copy/paste as many as you want to define them. 

Again, this is not the ideal situation; most of this should be possible through 
the UI but I’m not sitting there to diagnose the issue. 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 22, 2020, at 16:56, White, Daniel  wrote:
> 
> 
> Hi Andy,
>  
> I tried removing users.xml and authorizations.xml but I’m still getting the 
> same error.
>  
> Suspect it’s something to do with authorizers.xml, but I can’t see any issues 
> with it.
>  
> I see this in the nifi-user.log :
>  
> 
> Thanks
> Dan
>  
> From: Andy LoPresto  
> Sent: 23 August 2020 00:12
> To: users@nifi.apache.org
> Subject: Re: SSL/LDAP Configuration
>  
> CAUTION: This email originated from outside of the organisation. Do not click 
> links or open attachments unless you recognise the sender and know the 
> content is safe.
>  
> Daniel,
>  
> A couple options:
>  
> The “easy way” is to shut down NiFi, delete “users.xml” and 
> “authorizations.xml” in the “conf/“ directory, and then restart NiFi. 
> Whatever user was specified as the IAI should have enough permissions to get 
> started now. 
>  
> Once you can access the main canvas, you’ll want to go into the global 
> policies dialog (global menu top right > policies) and give yourself the 
> specific view & modify permissions on the root process group. I understand 
> this manual effort is less than ideal, but the stages in which things are 
> defined has mandated this for now. 
>  
> I think the User Guide does a good job of explaining the theory here as well 
> as specific component steps (but doesn’t go soup to nuts on the process), so 
> I’d recommend that as well as the “end” (the last 3-4 steps) of the 
> Walkthrough guide section on securing NiFi. 
>  
> I’m on my phone so I don’t have all my usual resources available, but 
> hopefully this guides you in the right direction. If not, please let me know 
> and tomorrow I can provide more specific instructions. 
>  
>  
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> On Aug 22, 2020, at 16:05, White, Daniel  wrote:
> 
> 
> Hi Andy,
>  
> I’ve now managed to login to Nifi using my AD account but am getting the 
> following error :
>  
> Insufficient Permissions – No applicable policies could be found.
>  
> 
>  
> Any pointers would be gratefully received.
>  
> Thanks
> Dan
>  
> From: Andy LoPresto  
> Sent: 03 August 2020 03:07
> To: users@nifi.apache.org
> Subject: Re: SSL/LDAP Configuration
>  
> CAUTION: This email originated from outside of the organisation. Do not click 
> links or open attachments unless you recognise the sender and know the 
> content is safe.
>  
> Also, your authorizers.xml is not correct — you haven’t configured (or even 
> uncommented) the LDAP user group provider, so the specified user group 
> provider is the file users.xml, and you haven’t configured any initial 
> admins, so no users will be allowed to log in. Did you follow the steps in 
> the NiFi Admin Guide [3][4] for configuring this? Authentication and 
> authorization are decoupled in NiFi, and while you can use LDAP for both, 
> you’ll have to configure it for each. 
>  
> Also, your login-identity-providers.xml uses START_TLS as the authentication 
> strategy but does not specify any properties for the keystore or truststore, 
> which will be required. 
>  
> [3] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider
> [4] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider
>  
>  
>  
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> 
> On Aug 2, 2020, at 7:02 PM, Andy LoPresto  wrote:
>  
> Hi Daniel,
>  
> Did you verify that the provided credentials are correct? There will be two 
> sets — the “manager” DN and password which are provided as configuration 
> values in the authorizers.xml file, and the individual user credentials 
> provided on each login attempt. The manager credentials allow NiFi to make an 
> authenticated request to the LDAP service, and the request itself contains 
> the user’s credentials. 
>  
> You can verify these credentials by using the ldapsearch [1][2] tool from one 
> of the machines where NiFi is installed. This allows you to verify TLS, 
> ports, network reachability, and the correctness of the credentials 
> themselves. 
>  
> Something like:
>  
> $ ldapsearc

RE: SSL/LDAP Configuration

[White, Daniel]

Hi Andy,

I tried removing users.xml and authorizations.xml but I’m still getting the 
same error.

Suspect it’s something to do with authorizers.xml, but I can’t see any issues 
with it.

I see this in the nifi-user.log :

[cid:image001.png@01D678E8.3E715990]
Thanks
Dan

From: Andy LoPresto 
Sent: 23 August 2020 00:12
To: users@nifi.apache.org
Subject: Re: SSL/LDAP Configuration

CAUTION: This email originated from outside of the organisation. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.

Daniel,

A couple options:

The “easy way” is to shut down NiFi, delete “users.xml” and 
“authorizations.xml” in the “conf/“ directory, and then restart NiFi. Whatever 
user was specified as the IAI should have enough permissions to get started now.

Once you can access the main canvas, you’ll want to go into the global policies 
dialog (global menu top right > policies) and give yourself the specific view & 
modify permissions on the root process group. I understand this manual effort 
is less than ideal, but the stages in which things are defined has mandated 
this for now.

I think the User Guide does a good job of explaining the theory here as well as 
specific component steps (but doesn’t go soup to nuts on the process), so I’d 
recommend that as well as the “end” (the last 3-4 steps) of the Walkthrough 
guide section on securing NiFi.

I’m on my phone so I don’t have all my usual resources available, but hopefully 
this guides you in the right direction. If not, please let me know and tomorrow 
I can provide more specific instructions.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 22, 2020, at 16:05, White, Daniel 
mailto:daniel.wh...@lgim.com>> wrote:

Hi Andy,

I’ve now managed to login to Nifi using my AD account but am getting the 
following error :

Insufficient Permissions – No applicable policies could be found.



Any pointers would be gratefully received.

Thanks
Dan

From: Andy LoPresto mailto:alopre...@apache.org>>
Sent: 03 August 2020 03:07
To: users@nifi.apache.org
Subject: Re: SSL/LDAP Configuration

CAUTION: This email originated from outside of the organisation. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.

Also, your authorizers.xml is not correct — you haven’t configured (or even 
uncommented) the LDAP user group provider, so the specified user group provider 
is the file users.xml, and you haven’t configured any initial admins, so no 
users will be allowed to log in. Did you follow the steps in the NiFi Admin 
Guide [3][4] for configuring this? Authentication and authorization are 
decoupled in NiFi, and while you can use LDAP for both, you’ll have to 
configure it for each.

Also, your login-identity-providers.xml uses START_TLS as the authentication 
strategy but does not specify any properties for the keystore or truststore, 
which will be required.

[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider
[4] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Aug 2, 2020, at 7:02 PM, Andy LoPresto 
mailto:alopre...@apache.org>> wrote:

Hi Daniel,

Did you verify that the provided credentials are correct? There will be two 
sets — the “manager” DN and password which are provided as configuration values 
in the authorizers.xml file, and the individual user credentials provided on 
each login attempt. The manager credentials allow NiFi to make an authenticated 
request to the LDAP service, and the request itself contains the user’s 
credentials.

You can verify these credentials by using the ldapsearch [1][2] tool from one 
of the machines where NiFi is installed. This allows you to verify TLS, ports, 
network reachability, and the correctness of the credentials themselves.

Something like:

$ ldapsearch -x -b “dc=,dc=com" -H ldap:// -D 
"cn=admin,dc=,dc=com" -W

That will conduct a general search using the account provided by -D, and prompt 
for the password with -W. You can also switch out the account in -D for the 
specific user you’re trying to log in as to verify those credentials.

[1] 
https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811
[2] https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Aug 2

Re: SSL/LDAP Configuration

[Andy LoPresto]

Daniel,

A couple options:

The “easy way” is to shut down NiFi, delete “users.xml” and 
“authorizations.xml” in the “conf/“ directory, and then restart NiFi. Whatever 
user was specified as the IAI should have enough permissions to get started 
now. 

Once you can access the main canvas, you’ll want to go into the global policies 
dialog (global menu top right > policies) and give yourself the specific view & 
modify permissions on the root process group. I understand this manual effort 
is less than ideal, but the stages in which things are defined has mandated 
this for now. 

I think the User Guide does a good job of explaining the theory here as well as 
specific component steps (but doesn’t go soup to nuts on the process), so I’d 
recommend that as well as the “end” (the last 3-4 steps) of the Walkthrough 
guide section on securing NiFi. 

I’m on my phone so I don’t have all my usual resources available, but hopefully 
this guides you in the right direction. If not, please let me know and tomorrow 
I can provide more specific instructions. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 22, 2020, at 16:05, White, Daniel  wrote:
> 
> 
> Hi Andy,
>  
> I’ve now managed to login to Nifi using my AD account but am getting the 
> following error :
>  
> Insufficient Permissions – No applicable policies could be found.
>  
> 
>  
> Any pointers would be gratefully received.
>  
> Thanks
> Dan
>  
> From: Andy LoPresto  
> Sent: 03 August 2020 03:07
> To: users@nifi.apache.org
> Subject: Re: SSL/LDAP Configuration
>  
> CAUTION: This email originated from outside of the organisation. Do not click 
> links or open attachments unless you recognise the sender and know the 
> content is safe.
>  
> Also, your authorizers.xml is not correct — you haven’t configured (or even 
> uncommented) the LDAP user group provider, so the specified user group 
> provider is the file users.xml, and you haven’t configured any initial 
> admins, so no users will be allowed to log in. Did you follow the steps in 
> the NiFi Admin Guide [3][4] for configuring this? Authentication and 
> authorization are decoupled in NiFi, and while you can use LDAP for both, 
> you’ll have to configure it for each. 
>  
> Also, your login-identity-providers.xml uses START_TLS as the authentication 
> strategy but does not specify any properties for the keystore or truststore, 
> which will be required. 
>  
> [3] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider
> [4] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider
>  
>  
>  
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> On Aug 2, 2020, at 7:02 PM, Andy LoPresto  wrote:
>  
> Hi Daniel,
>  
> Did you verify that the provided credentials are correct? There will be two 
> sets — the “manager” DN and password which are provided as configuration 
> values in the authorizers.xml file, and the individual user credentials 
> provided on each login attempt. The manager credentials allow NiFi to make an 
> authenticated request to the LDAP service, and the request itself contains 
> the user’s credentials. 
>  
> You can verify these credentials by using the ldapsearch [1][2] tool from one 
> of the machines where NiFi is installed. This allows you to verify TLS, 
> ports, network reachability, and the correctness of the credentials 
> themselves. 
>  
> Something like:
>  
> $ ldapsearch -x -b “dc=,dc=com" -H ldap:// -D 
> "cn=admin,dc=,dc=com" -W 
>  
> That will conduct a general search using the account provided by -D, and 
> prompt for the password with -W. You can also switch out the account in -D 
> for the specific user you’re trying to log in as to verify those credentials. 
>  
> [1] 
> https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811
> [2] https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/
>  
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
> 
> On Aug 2, 2020, at 1:11 PM, White, Daniel  wrote:
>  
> Confidential
>  
> Hi All,
>  
> Looking for some assistance with setting up SSL/LDAP to enable user admin 
> within Nifi.
>  
> I’ve setup and configured my non-prod environment but am having issue login 
> in :
>  
> Unable to validate the supplied credentials. Please contact the system 
> administrator
>  
> I’ve followed the config guide and am stuck as to what the issue could be.
>  
> The steps I followed :
>  
> Generate keys etc using tls-toolkit.sh
> Updated nifi.properties to set 
> nifi.security.user.login.identity.provider=ldap-provider
> Modified login-identity-providers.xml (copy attach

RE: SSL/LDAP Configuration

[White, Daniel]

Hi Andy,

I’ve now managed to login to Nifi using my AD account but am getting the 
following error :

Insufficient Permissions – No applicable policies could be found.

[cid:image001.png@01D678E1.0E81F160]

Any pointers would be gratefully received.

Thanks
Dan

From: Andy LoPresto 
Sent: 03 August 2020 03:07
To: users@nifi.apache.org
Subject: Re: SSL/LDAP Configuration

CAUTION: This email originated from outside of the organisation. Do not click 
links or open attachments unless you recognise the sender and know the content 
is safe.

Also, your authorizers.xml is not correct — you haven’t configured (or even 
uncommented) the LDAP user group provider, so the specified user group provider 
is the file users.xml, and you haven’t configured any initial admins, so no 
users will be allowed to log in. Did you follow the steps in the NiFi Admin 
Guide [3][4] for configuring this? Authentication and authorization are 
decoupled in NiFi, and while you can use LDAP for both, you’ll have to 
configure it for each.

Also, your login-identity-providers.xml uses START_TLS as the authentication 
strategy but does not specify any properties for the keystore or truststore, 
which will be required.

[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider
[4] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 2, 2020, at 7:02 PM, Andy LoPresto 
mailto:alopre...@apache.org>> wrote:

Hi Daniel,

Did you verify that the provided credentials are correct? There will be two 
sets — the “manager” DN and password which are provided as configuration values 
in the authorizers.xml file, and the individual user credentials provided on 
each login attempt. The manager credentials allow NiFi to make an authenticated 
request to the LDAP service, and the request itself contains the user’s 
credentials.

You can verify these credentials by using the ldapsearch [1][2] tool from one 
of the machines where NiFi is installed. This allows you to verify TLS, ports, 
network reachability, and the correctness of the credentials themselves.

Something like:

$ ldapsearch -x -b “dc=,dc=com" -H ldap:// -D 
"cn=admin,dc=,dc=com" -W

That will conduct a general search using the account provided by -D, and prompt 
for the password with -W. You can also switch out the account in -D for the 
specific user you’re trying to log in as to verify those credentials.

[1] 
https://forums.opensuse.org/showthread.php/401522-performing-ldapsearch-over-tls-ssl-against-active-directory#post1908811
[2] https://devconnected.com/how-to-search-ldap-using-ldapsearch-examples/

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 2, 2020, at 1:11 PM, White, Daniel 
mailto:daniel.wh...@lgim.com>> wrote:

Confidential

Hi All,

Looking for some assistance with setting up SSL/LDAP to enable user admin 
within Nifi.

I’ve setup and configured my non-prod environment but am having issue login in :

Unable to validate the supplied credentials. Please contact the system 
administrator

I’ve followed the config guide and am stuck as to what the issue could be.

The steps I followed :


  1.  Generate keys etc using tls-toolkit.sh
  2.  Updated nifi.properties to set 
nifi.security.user.login.identity.provider=ldap-provider
  3.  Modified login-identity-providers.xml (copy attached)
  4.  Modified authorizers.xml (copy attached)

Nifi starts and I can get to the login page, just unable to login (with error 
shown above).

Any help will be very grateful.

Thanks

Dan White
Lead Technical Architect
Legal & General Investment Management
One Coleman Street, London, EC2R 5AA
Tel: +44 203 124 4048
Mob: +44 7980 027 656
www.lgim.com

This e-mail (and any attachments) may contain privileged and/or confidential 
information. If you are not the intended recipient please do not disclose, 
copy, distribute, disseminate or take any action in reliance on it. If you have 
received this message in error please reply and tell us and then delete it. 
Should you wish to communicate with us by e-mail we cannot guarantee the 
security of any data outside our own computer systems.

Any information contained in this message may be subject to applicable terms 
and conditions and must not be construed as giving investment advice within or 
outside the United Kingdom or Republic of Ireland.

Telephone Conversations may be recorded for your protection and to ensure 
quality of service

Legal & General Investment Management Limited (no 2091894), LGIM Real Assets 
(Operator) Limited (no 05522016), LGIM (International) Limited (no 77

Re: Flow.xml.gz gets overwritten with Empty Flow.xml

[Andy LoPresto]

Hi John, 

I’m glad you found a documented workaround which solved your problem. 

While we try to build NiFi to operate at high scale, I don’t think it was ever 
anticipated that a flow definition would be 70 GB. Gzip compression can often 
achieve > 10:1 ratio for structured text data like XML, so that could actually 
be ~ 1 TB of uncompressed data. The Encrypt-Config Toolkit usually runs with 8 
GB of heap space, far short of what is necessary to hold the deserialized data 
in memory to perform the encryption/decryption process for a flow of that size. 

There is an open Jira [1] which captures some of these issues and will direct 
resolution & improvements, but I am curious to gather more information about 
your scenario to hopefully provide the best solution. Usually when we encounter 
flow definitions of more than 100 MB, there are a few causes:

* A high number of components on the canvas (10k+ processors)
* A high number of saved templates

Some NiFi instances are deployed for multitenant access and “legitimately” need 
hundreds or thousands of processors in many flows. However, we have often seen 
duplication of identical or near-identical flows due to “copy/paste” rather 
than parameterizing a single property value, which leads to significant 
opportunities for flow refactoring and deduplication. An analogy in writing 
code would be writing a new method to print “Hello Andy”, “Hello John”, “Hello 
Yolanda”, etc. rather than a single method which accepts a name and prints 
“Hello ”. Reducing these duplicate flows within the definition will 
reduce the size. 

Templates aren’t constantly visible and thus can grow over time, adding sizable 
impact to the flow definition even if no longer necessary. With the 
introduction of the NiFi Registry, templates have been deprecated, and cleaning 
up unused templates via the Global Menu > Templates option will drastically 
reduce the size of the flow definition. 

As NiFi has become more suited for elastic scaling and containerization, we’ve 
also seen better success with orchestrated containerized deployments on 
infrastructure abstractions like Kubernetes, which allows “one (or a logical 
group of) flow per cluster” and provides resource isolation & contention 
management, easier authorization management, independent scaling & monitoring, 
etc. If your flow definition is 70 GB, you may be very interested in pursuing 
this approach.  


[1] https://issues.apache.org/jira/browse/NIFI-6999 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 21, 2020, at 8:09 PM, jgunvaldson  wrote:
> 
> Just FYI, I found the answer
> 
> https://community.cloudera.com/t5/Support-Questions/NiFi-Toolkit-Can-t-start-NiFi-with-a-75MB-gzipped-flow-xml/td-p/288712
> 
> Exact same problem with proper solution
> 
> Fixed…
> 
> 
>> On Aug 21, 2020, at 3:48 PM, jgunvaldson  wrote:
>> 
>> We have a production instance of NIFI (1.9.0.3.4.1.1-4 built 05/01/2019 
>> 02:15:30 UTC Tagged nifi-1.9.0-RC2) with an unusual symptom.
>> 
>> We know, that on a new instance of NIFI, the canvas comes up empty (result 
>> of a new flow.xml). Developers then start building process
>> groups and more from this start.
>> 
>> What we are experiencing (never seen this before) - is that on restart (via 
>> startup log file entries verified) of NIFI using AMBARI (or other means), 
>> NIFI is overwriting flow.xml which is about 70 GB, with new empty flow.xml? 
>> Each time we copy 
>> back the original Flow.xml and restart (single node cluster) - NIFI 
>> overwrites with an empty Flow.xml
>> 
>> Possibly related to toolkit and encryption of flow.xml.gz
>> 
>> Obviously we are down until we can solve this
>> 
>> Any ideas?
>> 
>> John
>