Re: SSLPeerUnverifiedException Hostname "xyz" not verified
Can you clarify that statement? If you have a completely empty authorizers.xml file, would the node join the cluster and inherit the clusters authorizers.xml? Also, what would happen if the whole cluster was restarted? Are changes persisted? And if they are, if a node is restarted at the same time a new node is added -- would that node not have an outdated authorizers.xml and get rejected from rejoining? Thanks again. On Thu, Jul 19, 2018 at 4:55 AM, Peter Wilcsinszky < peterwilcsins...@gmail.com> wrote: > On Thu, Jul 19, 2018 at 12:38 AM Andy LoPresto > wrote: > > > Hi Jon, > > > > There are automatable, scalable, and non-restart-required ways to > > horizontally scale a secure cluster without requiring wildcard > > certificates. I should collect the various instructions / notes together > > into an article and people can reference that, but until that time, the > > Admin Guide [1] and the conversation Prashanth and I had on the ticket > [2] > > are the best documentation for this. > > > > Basically: > > > > * run the TLS toolkit as a server and have each node connect to it to > > obtain a valid cert. All of these will be signed by the same CA and each > > node will be able to communicate with all others. Add a new user with the > > node CN and RW on /proxy resource via the UI/API of the existing nodes. > > > > > > You should not need to modify authorizers.xml directly. > > > > I would highlight that in that case you should use the original > authorizers.xml that do not contain any nodes, not even the initial admin > identity. This way new nodes can figure to accept authorizers information > from the cluster instead of having a conflict and failing to start. > (This failure scenario happened to me when tried to spin up new nodes with > the same authorizers.xml that I used for the inital cluster.) > > > > * same permissions steps but use the toolkit in standalone mode. In this > > case, you must run it from the same directory every time (so it uses the > > same CA key to sign). > > * same permissions steps but run toolkit in standalone on each node. In > > this case, you must import the generated CA into existing node > truststores > > (requires restart). > > > > [1] > > https://nifi.apache.org/docs/nifi-docs/html/administration- > guide.html#tls-generation-toolkit > > [2] https://issues.apache.org/jira/browse/NIFI-5370 > > > > > > > > > > Andy LoPresto > > alopre...@apache.org > > *alopresto.apa...@gmail.com * > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > On Jul 18, 2018, at 2:49 PM, Jon Logan wrote: > > > > I saw this in the release notes...specifically that wildcard certs are > not > > supported. How do most people handle this in practice? We can run a cert > > server or get them from other means (AWS cert manager, etc) but am not > sure > > how to overcome the authorizers.xml issue -- would we need to have a > > provisioning script register each new server certificate with NiFi before > > it can actually do anything useful? Will new servers then have issues > > joining because their authorizers will not match the rest of the cluster? > > > > On Thu, Jul 5, 2018 at 8:04 AM, Pierre Villard < > > pierre.villard...@gmail.com> > > wrote: > > > > Hi Josef, > > > > I don't have a solution for you but it seems it has already been reported > > and a JIRA has been opened: > > https://issues.apache.org/jira/browse/NIFI-5370 > > > > Andy might be able to give more insights about it. > > > > Pierre > > > > 2018-07-05 13:19 GMT+02:00 Josefz : > > > > Hi expert > > > > I've just done an upgrade from NiFi 1.5.0 to 1.7.0 in a SSL secured > > > > cluster > > > > with LDAP authentication. Now I'm not anymore able to login into the > > webgui. > > After I have entered the login/password I'm getting the following > > > > message: > > > > > > > > > > And nifi-app.log reports the following error messages: > > > > > > > > I'm having a wildcard SSL certificate and I'm using the same > > keystore/truststore combination for three usecases: > > - for cluster connectivity (in nifi.properties) > > - in "authorizer.xml" > > - in "login-identity-providers.xml". > > > > The keystore.jks (private/public) keypair has been signed by our internal > > root CA and the root CA cert has been imported into the truststore.jks. > > > > As > > > > the ldap login works with certificates I'm more or less sure that the > > > > certs > > > > in general are fine. Has anybody an idea if wildcard CN and SAN names > > should > > work in a cluster or where the problem could be? I've tried the same > > > > certs > > > > as well in standalone mode, no issue at all. > > > > The following parameters in nifi.properties are enabled: > > nifi.security.needClientAuth=true > > nifi.cluster.protocol.is.secure=true > > > > Thanks in advance > > > > > > > > > > -- > > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ > > > > > > > > >
Re: SSLPeerUnverifiedException Hostname "xyz" not verified
The authorizers.xml supports many different options, such user group providers for file-based, ldap, or composite, and policy provider for file-based and ranger. The concept of inheritance only really applies to the file-based cases because in the other scenarios like LDAP and Ranger, the users and groups are stored external to NiFi, so want to be clear that we are talking about file-based scenarios here. In the file-based case, if you have a new node that has no flow, no users/groups, and no policies, then it will inherit everything from the cluster. The common issue is if all the above is true, but you defined an initial admin or node identity in the authorizers.xml on the new node, then before joining the cluster it will create the users and policies locally, and now it no longer matches the cluster. So that is why Peter pointed out to make sure the those are not populated locally on the new node. On Fri, Jul 20, 2018 at 1:44 PM, Jon Logan wrote: > Can you clarify that statement? If you have a completely empty > authorizers.xml file, would the node join the cluster and inherit the > clusters authorizers.xml? Also, what would happen if the whole cluster was > restarted? Are changes persisted? And if they are, if a node is restarted > at the same time a new node is added -- would that node not have an > outdated authorizers.xml and get rejected from rejoining? > > > Thanks again. > > On Thu, Jul 19, 2018 at 4:55 AM, Peter Wilcsinszky < > peterwilcsins...@gmail.com> wrote: > >> On Thu, Jul 19, 2018 at 12:38 AM Andy LoPresto >> wrote: >> >> > Hi Jon, >> > >> > There are automatable, scalable, and non-restart-required ways to >> > horizontally scale a secure cluster without requiring wildcard >> > certificates. I should collect the various instructions / notes together >> > into an article and people can reference that, but until that time, the >> > Admin Guide [1] and the conversation Prashanth and I had on the ticket >> [2] >> > are the best documentation for this. >> > >> > Basically: >> > >> > * run the TLS toolkit as a server and have each node connect to it to >> > obtain a valid cert. All of these will be signed by the same CA and each >> > node will be able to communicate with all others. Add a new user with the >> > node CN and RW on /proxy resource via the UI/API of the existing nodes. >> > >> >> >> > You should not need to modify authorizers.xml directly. >> > >> >> I would highlight that in that case you should use the original >> authorizers.xml that do not contain any nodes, not even the initial admin >> identity. This way new nodes can figure to accept authorizers information >> from the cluster instead of having a conflict and failing to start. >> (This failure scenario happened to me when tried to spin up new nodes with >> the same authorizers.xml that I used for the inital cluster.) >> >> >> > * same permissions steps but use the toolkit in standalone mode. In this >> > case, you must run it from the same directory every time (so it uses the >> > same CA key to sign). >> > * same permissions steps but run toolkit in standalone on each node. In >> > this case, you must import the generated CA into existing node >> truststores >> > (requires restart). >> > >> > [1] >> > https://nifi.apache.org/docs/nifi-docs/html/administration- >> guide.html#tls-generation-toolkit >> > [2] https://issues.apache.org/jira/browse/NIFI-5370 >> > >> > >> > >> > >> > Andy LoPresto >> > alopre...@apache.org >> > *alopresto.apa...@gmail.com * >> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> > >> > On Jul 18, 2018, at 2:49 PM, Jon Logan wrote: >> > >> > I saw this in the release notes...specifically that wildcard certs are >> not >> > supported. How do most people handle this in practice? We can run a cert >> > server or get them from other means (AWS cert manager, etc) but am not >> sure >> > how to overcome the authorizers.xml issue -- would we need to have a >> > provisioning script register each new server certificate with NiFi before >> > it can actually do anything useful? Will new servers then have issues >> > joining because their authorizers will not match the rest of the cluster? >> > >> > On Thu, Jul 5, 2018 at 8:04 AM, Pierre Villard < >> > pierre.villard...@gmail.com> >> > wrote: >> > >> > Hi Josef, >> > >> > I don't have a solution for you but it seems it has already been reported >> > and a JIRA has been opened: >> > https://issues.apache.org/jira/browse/NIFI-5370 >> > >> > Andy might be able to give more insights about it. >> > >> > Pierre >> > >> > 2018-07-05 13:19 GMT+02:00 Josefz : >> > >> > Hi expert >> > >> > I've just done an upgrade from NiFi 1.5.0 to 1.7.0 in a SSL secured >> > >> > cluster >> > >> > with LDAP authentication. Now I'm not anymore able to login into the >> > webgui. >> > After I have entered the login/password I'm getting the following >> > >> > message: >> > >> > >> > >> > >> > And nifi-app.log reports the following error messages: >> > >> >
Re: SSLPeerUnverifiedException Hostname "xyz" not verified
@Andy LoPresto I fully understand what you wrote regarding certs in the admin guide, however as you already mentioned, in my point of view this certificate stuff is really a pain. We have lost multiple days to get it running together with LDAP, just because of the complexity of the whole configuration. And after the upgrade to 1.7.0 we had again issues because of certs and the bug... Let me explain why we use wildcard certs. We have to use our company CA and we have to manually insert the CSR on a website (with some additional parameters) to get the certificate signed. If we have to do that for 20 nodes or even more, this would be a huge work. Additionally our network where the NiFi Nodes are, is a subnet secured by a firewall, so it's not possible to connect from outside through the cluster port. If an attacker is inside the subnet and is able to create a NiFi Node who can join the cluster (with the certificate and the password for the keystore), then we would anyway have bigger problems. But yes of course, wildcard certs are less secure. *Two questions for you:* 1. We used the wildcard certs already in NiFi 1.5.0 in our lab, however we would like to go live with 1.7.1 now. If we haven't seen any issues on NiFi 1.5.0 with the wildcard certs, how likely would it be that we see some issues on 1.7.1? 2. Somewhere I've read that in an optimal world (eg. with the NiFi TLS Certkit) we should have a Cert with a unique DN and as well use the same DN for the SAN per node. Would it be ok to have the following: 3-Node Cluster Environment: nifi-node-1, nifi-node-2, nifi-node-3 One Keystore Certificate for all NiFi nodes with the following attributes: -> DN "CN=NiFi Apache"; -> SAN = nifi-node-1, nifi-node-2, nifi-node-3 Background is the following, we are planning a loadbalancer in front of NiFi Webgui and I don't see any solution to get the whole thing work without the procedure above. Today we use wildcard, with that we are good to go. But as you already mentioned multiple times that wildcards are not supported we are looking for some alternatives. Thanks in advance Josef -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Listen* on Virtual Interfaces
While attempting to bind ListenTCP to a certain interface, it was discovered that it does not support subinterfaces (virtual interfaces) while validating the contents of the "Local Network Interface" property. I traced it back to the ListenerProperties class [1], which uses " NetworkInterface.getNetworkInterfaces()" to pull a list of interfaces. This method does not include subinterfaces, so further iteration using the "getSubInterfaces()" method [2] on each interface would be required to fully populate the list. However, IP addresses are not found on the subinterfaces when using "getInetAddresses()" on each interface. Instead, they can be found in the enumeration returned by the parent interface. Further (needlessly complex) logic would be required to determine which IP address is being requested when specifying a virtual interface. Additionally, in a scenario where someone has assigned a virtual interface on a host and entered the parent interface into the "Local Network Interface" property for ListenTCP, the code will find multiple IP addresses for that parent interface... but only select the first one in the list. [3] I'm uncertain as to how the IP addresses are ordered in the returned enumeration, but I suspect that unpredictable outcomes would result based upon that ordering. Since this logic resides in the abstract class, multiple Listen processors are affected. I have opened NIFI-5445 [4] for this issue and submitted PR #2909 on GitHub [5] which proposes a fix for this issue. Changing the property name to "Local IP Address" will allow a simpler designation of intended operation without complicating code. The IP address entry is validated against known IP addresses on the system, as it was with the network interfaces. I tested this change, and it binds as intended to IP addresses located on both virtual and physical interfaces. I'm not much of a developer, but I thought this was the best way to fix the problem. However, I'm posting here for input on any other ways this could be solved without changed property names... which I understand is not trivial. Also, please let me know if anything is wrong with the PR as this is my first contribution. Thank you. -Nick [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/listen/ListenerProperties.java#L41 [2] https://docs.oracle.com/javase/8/docs/api/java/net/NetworkInterface.html#getSubInterfaces-- [3] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventProcessor.java#L189 [4] https://issues.apache.org/jira/browse/NIFI-5445 [5] https://github.com/apache/nifi/pull/2909
Re: SSLPeerUnverifiedException Hostname "xyz" not verified
The other issue we have is that we would now have to run two additional separate services in any deployment we would run -- namely, a certificate server and LDAP. We try to reduce complexity of deployments, but it's stuff like this that quickly becomes a maintenance burden. On Fri, Jul 20, 2018 at 3:51 PM, Josefz wrote: > @Andy LoPresto > > I fully understand what you wrote regarding certs in the admin guide, > however as you already mentioned, in my point of view this certificate > stuff > is really a pain. We have lost multiple days to get it running together > with > LDAP, just because of the complexity of the whole configuration. And after > the upgrade to 1.7.0 we had again issues because of certs and the bug... > > Let me explain why we use wildcard certs. We have to use our company CA and > we have to manually insert the CSR on a website (with some additional > parameters) to get the certificate signed. If we have to do that for 20 > nodes or even more, this would be a huge work. Additionally our network > where the NiFi Nodes are, is a subnet secured by a firewall, so it's not > possible to connect from outside through the cluster port. If an attacker > is > inside the subnet and is able to create a NiFi Node who can join the > cluster > (with the certificate and the password for the keystore), then we would > anyway have bigger problems. But yes of course, wildcard certs are less > secure. > > *Two questions for you:* > > 1. We used the wildcard certs already in NiFi 1.5.0 in our lab, however we > would like to go live with 1.7.1 now. If we haven't seen any issues on NiFi > 1.5.0 with the wildcard certs, how likely would it be that we see some > issues on 1.7.1? > > 2. Somewhere I've read that in an optimal world (eg. with the NiFi TLS > Certkit) we should have a Cert with a unique DN and as well use the same DN > for the SAN per node. Would it be ok to have the following: > > 3-Node Cluster Environment: nifi-node-1, nifi-node-2, nifi-node-3 > > One Keystore Certificate for all NiFi nodes with the following attributes: > -> DN "CN=NiFi Apache"; > -> SAN = nifi-node-1, nifi-node-2, nifi-node-3 > > Background is the following, we are planning a loadbalancer in front of > NiFi > Webgui and I don't see any solution to get the whole thing work without the > procedure above. Today we use wildcard, with that we are good to go. But as > you already mentioned multiple times that wildcards are not supported we > are > looking for some alternatives. > > Thanks in advance > Josef > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >
