Etienne, If you feel like there would be nice additions to have in the documentation, please feel free to share your suggestions to improve it (though a JIRA / pull request). We don't want NiFi users/administrators to have a painful process when configuring all of this.
Thanks, Pierre Le jeu. 26 nov. 2020 à 12:01, Etienne Jouvin <lapinoujou...@gmail.com> a écrit : > Hello all. > > And finally I got it. > When I am not able to find in the documentation, because I was not > correctly searching offcourse, I read the source code and do remote debug. > > And the conclusion that it is all about identity mapping. > > For reminder, the objective was to have a secured cluster and everything > managed in the LDAP. > I follow the walk throughs documentation. But in it, it is only matter of > nodes managed in userGroupIdentity, based on files. > > In my case, users and groups are managed in the LDAP, even nodes for > cluster. > > With remote debugging, I touched the class X509AuthenticationProvider and > specially the function authenticate. By the way, this is the only class > that fires an exception with message Untrusted proxy ... > In it, identities are extracted from the request and linked certificates. > Ids extracted were : > > - uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch , for the "real" user > - CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, > DC=amexio, DC=ch, for the "proxy" node > > > But in the LDAP, the node as the entry DN : cn=mig1.amexio.ch > ,ou=users,ou=nifi;dc=amexio,dc=ch > > So when the provider validate identities, the NiFi identity "CN= > mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio, > DC=ch" is not found. No user/group, and does not belongs to any group. Then > the validation on policy "proxy", with "write" access, failed and the > connection was rejected. > > What I found also in this function, there is a identity mapping, > function mapIdentity from class NiFiAuthenticationProvider. > This will use function mapIdentity in class IdentityMappingUtil. > With this, the identity is normalized and this is the trick to transform > the idnetity retrieved from the certificate, with space and upper case, to > an identity matching the one retrieved from LDAP. > > > That was the point. Si in the file nifi.properties, for all nodes, here is > modifications : > nifi.security.identity.mapping.pattern.ldapCertNode=^CN=(.*?), OU=(.*?), > OU=(.*?), DC=(.*?), DC=(.*?)$ > > nifi.security.identity.mapping.value.ldapCertNode=cn=$1,ou=$2,ou=$3,dc=$4,dc=$5 > nifi.security.identity.mapping.transform.ldapCertNode=NONE > > Like this, the identity retrieved from certificate, for the proxy node, is > CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio, > DC=ch > After identity mapping it is cn=mig1.amexio.ch <http://mig1.assura.ch/> > ,ou=users,ou=nifi,dc=amexio,dc=ch > And it is exactly matching identity retrieved from the LDAP for the node. > > Et voilà. > > Hard work (for me) and happy to find the way to manage this. > > Thanks all for your help. > > > Etienne > > > > Le mer. 25 nov. 2020 à 19:50, Etienne Jouvin <lapinoujou...@gmail.com> a > écrit : > >> That's what I suspect for now. >> >> But this is strange to have a transformation with space introduced. >> In the certificate (I use KeyStore Explorer to visualize the certificate) >> there is no space. >> >> But I will give a try to use the identity mapping from here >> # Identity Mapping Properties # >> # These properties allow normalizing user identities such that identities >> coming from different identity providers >> # (certificates, LDAP, Kerberos) can be treated the same internally in >> NiFi. The following example demonstrates normalizing >> # DNs from certificates and principals from Kerberos into a common >> identity string: >> # >> # nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), >> L=(.*?), ST=(.*?), C=(.*?)$ >> # nifi.security.identity.mapping.value.dn=$1@$2 >> # nifi.security.identity.mapping.transform.dn=NONE >> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$ >> # nifi.security.identity.mapping.value.kerb=$1@$2 >> # nifi.security.identity.mapping.transform.kerb=UPPER >> >> # Group Mapping Properties # >> # These properties allow normalizing group names coming from external >> sources like LDAP. The following example >> # lowercases any group name. >> # >> # nifi.security.group.mapping.pattern.anygroup=^(.*)$ >> # nifi.security.group.mapping.value.anygroup=$1 >> # nifi.security.group.mapping.transform.anygroup=LOWER >> >> for now, I do not know what to put ;) >> >> Etienne >> >> >> >> Le mer. 25 nov. 2020 à 19:45, Bryan Bende <bbe...@gmail.com> a écrit : >> >>> The values have to match exactly, it’s case and white space sensitive. >>> There are identity transforms in Nifi properties that possibly help. >>> >>> The value from where it says “Attempting request for...” is the value >>> nifi is getting from the certificate. You can also use key tool to list the >>> contents of the keystore jks and it will probably be the same as the log. >>> >>> On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <lapinoujou...@gmail.com> >>> wrote: >>> >>>> Hello all. >>>> >>>> Still no luck. >>>> I follow steps from here : >>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit >>>> >>>> Reminder, everything is currently in LDAP, I do not have any file >>>> user/group provider. >>>> When only one node, I can connect, this is correct. >>>> >>>> I can see the group having nodes "users" from the LDAP and the group >>>> has the proxy policy. >>>> >>>> But one thing strange, may be not the cause. >>>> In the LDAP, the DN is something like : >>>> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch >>>> >>>> And when I try to connect, I can see thoses errors : >>>> In nifi-user.log for the node where I connect : >>>> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter >>>> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch >>>> >>>> In nifi-user.log on the second node : >>>> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Attempting >>>> request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN= >>>> mig1.amexio.ch <http://mig1.assura.ch>, OU=users, OU=nifi, DC=amexio, >>>> DC=ch>) GET https://mig2.amexio.ch:9442/nifi-api/flow/current-user >>>> (source ip: 127.0.0.1) >>>> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting >>>> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users, >>>> OU=nifi, DC=amexio, DC=ch >>>> >>>> The doubt is on the message, why the node DN has spaces between each >>>> level ? >>>> That does not match the value in the certificate and not the DN in LDAP. >>>> >>>> Maybe this is not the good route to investigate. Still searching. I >>>> have to find a way to "debug" the NiFiAuthenticationFilter and check the >>>> certificate comparison. >>>> >>>> Etienne >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Le mer. 25 nov. 2020 à 14:18, David Handermann < >>>> exceptionfact...@gmail.com> a écrit : >>>> >>>>> I am not as familiar with the LDAP user group provider, but based on >>>>> the "Untrusted proxy" message you are seeing, it sounds like the nodes are >>>>> not being identified properly as members of the "nodes" group from LDAP. >>>>> Just for testing purposes, you could try specifying the node distinguished >>>>> names in the "Node Identity N" properties of the access policy provider, >>>>> using "Node Identity 1", "Node Identity 2" and "Node Identity 3" to >>>>> specify >>>>> each node DN. If that works, then it sounds like a configuration issue >>>>> with the Node Group, either on the LDAP server, or in the way NiFi is >>>>> attempting to query LDAP. >>>>> >>>>> Regards, >>>>> David Handermann >>>>> >>>>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin < >>>>> lapinoujou...@gmail.com> wrote: >>>>> >>>>>> Just for information, did not have time to test it from now. >>>>>> I was not able to get this Walk Throughs documentation. >>>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html >>>>>> >>>>>> Hope I will find the error I have about certificate (I have a little >>>>>> idea) >>>>>> >>>>>> Etienne >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <lapinoujou...@gmail.com> >>>>>> a écrit : >>>>>> >>>>>>> Hello. >>>>>>> >>>>>>> I made some progress yesterday. >>>>>>> I did setup in LDAP groups and person >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Groups : >>>>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for >>>>>>> administrators >>>>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors >>>>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all >>>>>>> "person" representing NiFi nodes. >>>>>>> >>>>>>> Users : >>>>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node, >>>>>>> replacing X by the index, and with object class person >>>>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" >>>>>>> user used to connect on the platform, with object class inetOrgperson >>>>>>> >>>>>>> In NiFi configuration. >>>>>>> I did activate a userGroupProvider linked to the LDAP >>>>>>> <userGroupProvider> >>>>>>> <identifier>amexio-ldap-user-group-provider</identifier> >>>>>>> >>>>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> >>>>>>> <property name="Authentication Strategy">SIMPLE</property> >>>>>>> >>>>>>> <property name="Manager DN">uid=admin,ou=system</property> >>>>>>> <property name="Manager Password">secret</property> >>>>>>> >>>>>>> <property name="TLS - Keystore"></property> >>>>>>> <property name="TLS - Keystore Password"></property> >>>>>>> <property name="TLS - Keystore Type"></property> >>>>>>> <property name="TLS - Truststore"></property> >>>>>>> <property name="TLS - Truststore Password"></property> >>>>>>> <property name="TLS - Truststore Type"></property> >>>>>>> <property name="TLS - Client Auth"></property> >>>>>>> <property name="TLS - Protocol"></property> >>>>>>> <property name="TLS - Shutdown Gracefully"></property> >>>>>>> >>>>>>> <property name="Referral Strategy">FOLLOW</property> >>>>>>> <property name="Connect Timeout">10 secs</property> >>>>>>> <property name="Read Timeout">10 secs</property> >>>>>>> >>>>>>> <property name="Url">ldap://localhost:10389</property> >>>>>>> <property name="Page Size">50</property> >>>>>>> <!-- <property name="Sync Interval">30 mins</property> --> >>>>>>> <property name="Sync Interval">30 seconds</property> >>>>>>> <property name="Group Membership - Enforce Case >>>>>>> Sensitivity">false</property> >>>>>>> >>>>>>> <property name="User Search >>>>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property> >>>>>>> <property name="User Object Class">person</property> >>>>>>> <property name="User Search Scope">ONE_LEVEL</property> >>>>>>> <property name="User Search Filter"></property> >>>>>>> <property name="User Identity Attribute"></property> >>>>>>> <property name="User Group Name Attribute"></property> >>>>>>> <property name="User Group Name Attribute - Referenced Group >>>>>>> Attribute"></property> >>>>>>> >>>>>>> <property name="Group Search >>>>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property> >>>>>>> <property name="Group Object Class">groupOfNames</property> >>>>>>> <property name="Group Search Scope">ONE_LEVEL</property> >>>>>>> <property name="Group Search Filter"></property> >>>>>>> <property name="Group Name Attribute">cn</property> >>>>>>> <property name="Group Member Attribute">member</property> >>>>>>> <property name="Group Member Attribute - Referenced User >>>>>>> Attribute"></property> >>>>>>> </userGroupProvider> >>>>>>> >>>>>>> Of course, register it inside the accessPolicyProvider >>>>>>> <accessPolicyProvider> >>>>>>> <identifier>file-access-policy-provider</identifier> >>>>>>> >>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> >>>>>>> <!-- <property name="User Group >>>>>>> Provider">file-user-group-provider</property> --> >>>>>>> <property name="User Group >>>>>>> Provider">amexio-ldap-user-group-provider</property> >>>>>>> <property name="Authorizations >>>>>>> File">./conf/authorizations.xml</property> >>>>>>> <!-- <property name="Initial Admin Identity"></property> --> >>>>>>> <property name="Initial Admin >>>>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property> >>>>>>> <property name="Legacy Authorized Users File"></property> >>>>>>> <property name="Node Identity 1"></property> >>>>>>> <property name="Node Group">nodes</property> >>>>>>> </accessPolicyProvider> >>>>>>> >>>>>>> I am able to connect with the initial administrator account, when >>>>>>> the first node is started. >>>>>>> And all nodes are synchronized in the NiFi instance. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> As soon as I start an additional node, I can not connect to the >>>>>>> first node >>>>>>> Erreur message >>>>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, >>>>>>> DC=ch >>>>>>> >>>>>>> But I can connect on the second node. >>>>>>> >>>>>>> >>>>>>> So all this is about the certificate I guess. >>>>>>> for reminder, I use nls-toolkit to generate certificate on all nodes >>>>>>> with something like : >>>>>>> tls-toolkit.bat standalone -f >>>>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o >>>>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch >>>>>>> --nifiDnPrefix cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch >>>>>>> >>>>>>> Proxy is untrusted, ok fine. So may be I should not use the >>>>>>> standalone function of toolkit, but using server and client. In that >>>>>>> case, >>>>>>> I have to stay alive the server from toolkit ? >>>>>>> Also, it seems I did not add certificate from node1 inside node2 >>>>>>> trutstore, and node2 certificate inside node1 truststore ? >>>>>>> But in this case, if I have to add a new node, let's say node4, I >>>>>>> would have to push the certificate from node4 inside all existing nodes >>>>>>> ? >>>>>>> >>>>>>> I continue to search, but any idea / input will be appreciated. >>>>>>> >>>>>>> Etienne >>>>>>> >>>>>>> >>>>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bbe...@gmail.com> a >>>>>>> écrit : >>>>>>> >>>>>>>> Yes it will be the DN of the server's certificate which comes from >>>>>>>> the keystore. >>>>>>>> >>>>>>>> NiFi will get an incoming request, see that there is an X509 cert, >>>>>>>> take the DN and go to the user group provider and ask for the user >>>>>>>> with this identity. >>>>>>>> >>>>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin < >>>>>>>> lapinoujou...@gmail.com> wrote: >>>>>>>> > >>>>>>>> > Hum OK, >>>>>>>> > >>>>>>>> > I will give it a try. >>>>>>>> > But one more thing... >>>>>>>> > >>>>>>>> > If I only set the group node; >>>>>>>> > How NiFi will connect the node with the nodeId in the LDAP ? >>>>>>>> > Where does it take the nodeid value ? >>>>>>>> > Is it the value we set in the keystore / truststore, by default >>>>>>>> cn=localhost, dc=NIFI (something like this) ? >>>>>>>> > >>>>>>>> > Etienne >>>>>>>> > >>>>>>>> > >>>>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bbe...@gmail.com> a >>>>>>>> écrit : >>>>>>>> >> >>>>>>>> >> I don't really know the LDAP specifics too well, so I'm not >>>>>>>> actually sure. >>>>>>>> >> >>>>>>>> >> You just need the nodes to come back from the LDAP >>>>>>>> UserGroupProvider >>>>>>>> >> as if they were regular users and members of some group "foo", >>>>>>>> which >>>>>>>> >> you then put "foo" into the "Node Group". >>>>>>>> >> >>>>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin < >>>>>>>> lapinoujou...@gmail.com> wrote: >>>>>>>> >> > >>>>>>>> >> > Thanks Bryan. >>>>>>>> >> > >>>>>>>> >> > With your answer.... I will go to the Node Group and assign >>>>>>>> node identities. >>>>>>>> >> > Better for deployment and setup on the fly, I guess. >>>>>>>> >> > >>>>>>>> >> > One more point, you said "creating ldap entries for your nodes >>>>>>>> and assigning them group membership in ldap". What type of objectClass >>>>>>>> would you assign to the node in LDAP ? >>>>>>>> >> > This is not inetOrgPerson. The node should not have password. >>>>>>>> >> > If I create groupOfMembers for each node, is it correct ? >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > Thanks >>>>>>>> >> > >>>>>>>> >> > Etienne >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bbe...@gmail.com> >>>>>>>> a écrit : >>>>>>>> >> >> >>>>>>>> >> >> Hello, >>>>>>>> >> >> >>>>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in >>>>>>>> that it >>>>>>>> >> >> sets up the policies for the initial nodes to have >>>>>>>> permissions to >>>>>>>> >> >> proxy. >>>>>>>> >> >> >>>>>>>> >> >> If you are creating ldap entries for your nodes and assigning >>>>>>>> them >>>>>>>> >> >> group membership in ldap, then yes you could put that group >>>>>>>> name as >>>>>>>> >> >> the "Node Group" and then you don't need to specify the "Node >>>>>>>> >> >> Identities". >>>>>>>> >> >> >>>>>>>> >> >> If you are creating the node users in NiFi's file-based user >>>>>>>> group >>>>>>>> >> >> provider then you need to use node identities, and when >>>>>>>> adding a new >>>>>>>> >> >> node to the cluster you'd have to add the user first through >>>>>>>> the >>>>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to >>>>>>>> the >>>>>>>> >> >> cluster. >>>>>>>> >> >> >>>>>>>> >> >> Thanks, >>>>>>>> >> >> >>>>>>>> >> >> Bryan >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin < >>>>>>>> lapinoujou...@gmail.com> wrote: >>>>>>>> >> >> > >>>>>>>> >> >> > Hello all. >>>>>>>> >> >> > >>>>>>>> >> >> > >>>>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP >>>>>>>> authentication. >>>>>>>> >> >> > For now the accessPolicyProvider is the default one with >>>>>>>> the configuration template : >>>>>>>> >> >> > <accessPolicyProvider> >>>>>>>> >> >> > <identifier>file-access-policy-provider</identifier> >>>>>>>> >> >> > >>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> >>>>>>>> >> >> > <property name="User Group >>>>>>>> Provider">file-user-group-provider</property> >>>>>>>> >> >> > <property name="Authorizations >>>>>>>> File">./conf/authorizations.xml</property> >>>>>>>> >> >> > <property name="Initial Admin Identity"></property> >>>>>>>> >> >> > <property name="Legacy Authorized Users >>>>>>>> File"></property> >>>>>>>> >> >> > <property name="Node Identity 1"></property> >>>>>>>> >> >> > <property name="Node Group"></property> >>>>>>>> >> >> > </accessPolicyProvider> >>>>>>>> >> >> > >>>>>>>> >> >> > But I do not really understand the purpose of the Node >>>>>>>> Identity X property. >>>>>>>> >> >> > If I well understood, all nodes should have the same >>>>>>>> configuration file, and I should register all nodes identity. >>>>>>>> >> >> > >>>>>>>> >> >> > But what about if I want to add a new node in the cluster >>>>>>>> on the fly ? >>>>>>>> >> >> > Should I register a new node identity, and then I should >>>>>>>> change all nodes configurations ? >>>>>>>> >> >> > The comment, in the configuration file, mentions the >>>>>>>> configuration Node Group, The name of a group containing NiFi cluster >>>>>>>> nodes. The typical use for this is when nodes are dynamically >>>>>>>> added/removed >>>>>>>> from the cluster. >>>>>>>> >> >> > Should I just put a Node group name and this will do the >>>>>>>> trick ? >>>>>>>> >> >> > >>>>>>>> >> >> > What should I put ? At the following link, >>>>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html, >>>>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com >>>>>>>> >> >> > In that case, what should be the obejct class for the node >>>>>>>> cn=nifi-1 in the LDAP ? >>>>>>>> >> >> > >>>>>>>> >> >> > Any documentation links will be appreciated. >>>>>>>> >> >> > >>>>>>>> >> >> > Regards. >>>>>>>> >> >> > >>>>>>>> >> >> > Etienne Jouvin >>>>>>>> >>>>>>> -- >>> Sent from Gmail Mobile >>> >>
