Conrad, For the site-to-site did you follow the instructions here [1]? Each node needs to be added as a user in order to make the connections.
[1] http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site <http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 20, 2016, at 7:36 AM, Conrad Crampton <conrad.cramp...@secdata.com> > wrote: > > Ok, > So I tried everything suggested so far to no avail unfortunately. > > So what I have done is to create all new certs etc. using the tookit. Updated > my existing authoriszed-users.xml to have to match the full cert > distinguished names CN=server.name, OU=NIFI etc. > > Recreated all my remote process groups to not reference the original NCM as > that still wouldn’t work – after a complete new install (upgrade). > > So now what I have is a six node cluster using original data/worker nodes and > they are part of the cluster – all appears to be working ie. I can log into > the UI (nice by the way ;-) on each server. There are no SSL handshake errors > etc. BUT the RPG (newly created) still don’t appear to be working. I am > getting > > 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0 > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623 > Unable to refresh Remote Group's peers due to Unable to communicate with > remote NiFi cluster in order to determine which nodes exist in the remote > cluster > 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0 > nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274 > Unable to refresh Remote Group's peers due to Unable to communicate with > remote NiFi cluster in order to determine which nodes exist in the remote > cluster > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0 > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1 Unable > to refresh Remote Group's peers due to Unable to communicate with remote NiFi > cluster in order to determine which nodes exist in the remote cluster > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0 > nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c > Unable to refresh Remote Group's peers due to Unable to communicate with > remote NiFi cluster in order to determine which nodes exist in the remote > cluster > 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0 > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef Unable > to refresh Remote Group's peers due to Unable to communicate with remote NiFi > cluster in order to determine which nodes exist in the remote cluster > 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0 > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037 > Unable to refresh Remote Group's peers due to Unable to communicate with > remote NiFi cluster in order to determine which nodes exist in the remote > cluster > 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0 > nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426 > Unable to refresh Remote Group's peers due to Unable to communicate with > remote NiFi cluster in order to determine which nodes exist in the remote > cluster > > I can telnet from server to server on both https (UI) port and S2S port. > I am really at a loss as to what to do now. > > Data is queuing up in my input processors with nowhere to go. > Do I have to do something radical here to get this working like stopping > everything, clearing out all the queues then starting up again??? I really > don’t want to do this obviously but I am getting nowhere on this – two days > of frustration with nothing to show for it. > > Any more suggestions please?? > Thanks for your patience. > Conrad > > > From: Andy LoPresto <alopre...@apache.org <mailto:alopre...@apache.org>> > Reply-To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Date: Wednesday, 19 October 2016 at 18:24 > To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups > > * PGP Signed by an unknown key > Hi Conrad, > > Bryan is correct that changing the certificates (and the encapsulating > keystores and truststores) will not affect any data held in the nodes. > > Regenerating everything using the TLS toolkit should hopefully not be too > challenging, but I am also curious as to why you are getting these handshake > exceptions now. As Bryan pointed out, adding the following line to > bootstrap.conf will provide substantial additional log output which should > help trace the issue. > > java.arg.15=-Djavax.net.debug=ssl,handshake > > You can also imitate the node connecting to the (previous) NCM via this > command: > > $ openssl s_client -connect <host:port> -debug -state -cert > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile > <path_to_your_CA_cert.pem> > > Where: > > <host:port> = the hostname and port of the “NCM” > <path_to_your_cert.pem> = the public key used to identify the “node” (can be > exported from the node keystore [1]) > <path_to_your_key.pem> = the private key used to identify the “node” (can be > exported from the node keystore via 2 step process) > <path_to_your_CA_cert.pem> = the public key used to sign the “NCM” > certificate (could be a 3rd party like Verisign or DigiCert, or an internal > organization CA if you have one) > > If you’ve already regenerated everything and it works, that’s fine. But if > you have the time to try and investigate the old certs, we are interested and > prepared to help. Thanks. > > [1] https://security.stackexchange.com/a/66865/16485 > <https://security.stackexchange.com/a/66865/16485> > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Oct 19, 2016, at 1:03 PM, Bryan Bende <bbe...@gmail.com > <mailto:bbe...@gmail.com>> wrote: > > That is definitely weird that it is only an issue on the node that used to be > the NCM. > Might be worth double checking the keystore and truststore of that one node > and make sure it has what you would expect, and also double check > nifi.properties compared to the others to see if anything seems different. > > Changing all of the keystores, truststores, etc should be fine from a data > perspective... > > If you decide to go that route it would probably be easiest to start back > over from a security perspective, meaning: > - Stop all the nodes and delete the users.xml and authorizations.xml from all > nodes > - Configure authorizers.xml with the appropriate initial admin (or legacy > file) and node identities based on the new certs > - Ensure authorizers.xml is the same on all nodes > - Then restart everything > > Alternatively, you might be able to manually add users for all of the new > certs before shutting everything down and give them the appropriate policies, > then restart everything, but requires more manual work to get everything > lined up. > > > On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton > <conrad.cramp...@secdata.com <mailto:conrad.cramp...@secdata.com>> wrote: > Hi, > As a plan for tomorrow – I have generated new keystores, truststores, client > certts etc. for all nodes in my cluster using the > > From: Bryan Bende <bbe...@gmail.com <mailto:bbe...@gmail.com>> > Reply-To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Date: Wednesday, 19 October 2016 at 15:33 > > To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups > > > Trying to think of things to check here... > > Does every node have nifi.remote.input.secure=true in nifi.properties and the > URL in the RPG is an https URL? > > On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton > <conrad.cramp...@secdata.com <mailto:conrad.cramp...@secdata.com>> wrote: > One other thing… > The RPGs have an unlocked padlock on them saying S2S is not secure. > Conrad > > From: Bryan Bende <bbe...@gmail.com <mailto:bbe...@gmail.com>> > Reply-To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Date: Wednesday, 19 October 2016 at 15:20 > To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups > > Ok that does seem like a TLS/SSL issue... > > Is this a single cluster doing site-to-site to itself? > > On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <joe.w...@gmail.com > <mailto:joe.w...@gmail.com>> wrote: > thanks conrad - did get it. Bryan is being more helpful that I so I > went silent :-) > > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton > <conrad.cramp...@secdata.com <mailto:conrad.cramp...@secdata.com>> wrote: > > Hi Joe, > > Yep, > > Tried removing the RPG that referenced the NCM and adding new one with > > one of the nifis as url. > > That sort of worked, but kept getting errors about the NCM not being > > available for the ports and therefore couldn’t actually enable the port I > > needed to for that RPG. > > Thanks > > Conrad > > > > (sending again as don’t know if the stupid header ‘spoofed’ is stopping > > getting though – apologies if already sent) > > > > On 19/10/2016, 14:12, "Joe Witt" <joe.w...@gmail.com > > <mailto:joe.w...@gmail.com>> wrote: > > > > Conrad, > > > > For s2s now you can just point at any of the nodes in the cluster. > > Have you tried changing the URL or removing and adding new RPG > > entries? > > > > Thanks > > Joe > > > > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton > > <conrad.cramp...@secdata.com <mailto:conrad.cramp...@secdata.com>> > > wrote: > > > Hi, > > > > > > I have finally taken the plunge to upgrade my cluster from 0.6.1 > > to 1.0.0. > > > > > > 6 nodes with a NCM. > > > > > > With the removal of NCM in 1.0.0 I believe I now have an issue > > where none of > > > my Remote Process Groups work as they previously did because they > > were > > > configured to connect to the NCM (as the RPG url) which now > > doesn’t exist. > > > > > > I have tried converting my NCM to a node but whilst I can get it > > running > > > (sort of) when I try and connect to the cluster I get something > > like this in > > > my logs… > > > > > > > > > > > > 2016-10-19 13:14:44,109 ERROR [main] > > o.a.nifi.controller.StandardFlowService > > > Failed to load flow from cluster due to: > > > org.apache.nifi.controller.UninheritableFlowException: Failed to > > connect > > > node to cluster because local flow is different than cluster flow. > > > > > > org.apache.nifi.controller.UninheritableFlowException: Failed to > > connect > > > node to cluster because local flow is different than cluster flow. > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) > > > [nifi-jetty-1.0.0.jar:1.0.0] > > > > > > at org.apache.nifi.NiFi.<init>(NiFi.java:152) > > > [nifi-runtime-1.0.0.jar:1.0.0] > > > > > > at org.apache.nifi.NiFi.main(NiFi.java:243) > > > [nifi-runtime-1.0.0.jar:1.0.0] > > > > > > Caused by: org.apache.nifi.controller.UninheritableFlowException: > > Proposed > > > Authorizer is not inheritable by the flow controller because of > > Authorizer > > > differences: Proposed Authorizations do not match current > > Authorizations > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > ... 4 common frames omitted > > > > > > 2016-10-19 13:14:44,414 ERROR [main] > > o.a.n.c.c.node.NodeClusterCoordinator > > > Event Reported for ncm-cm1.local:9090 -- Node disconnected from > > > cluster due to > > org.apache.nifi.controller.UninheritableFlowException: Failed > > > to connect node to cluster because local flow is different than > > cluster > > > flow. > > > > > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator] > > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread > > Thread[Shutdown > > > Cluster Coordinator,5,main]: java.lang.NullPointerException > > > > > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator] > > > org.apache.nifi.NiFi > > > > > > java.lang.NullPointerException: null > > > > > > at > > > > > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) > > > ~[na:1.8.0_51] > > > > > > at > > > > > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) > > > ~[na:1.8.0_51] > > > > > > at > > > > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570) > > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119) > > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51] > > > > > > 2016-10-19 13:14:44,448 WARN [main] > > o.a.n.c.l.e.CuratorLeaderElectionManager > > > Failed to close Leader Selector for Cluster Coordinator > > > > > > java.lang.IllegalStateException: Already closed or has not been > > started > > > > > > at > > > > > com.google.common.base.Preconditions.checkState(Preconditions.java:173) > > > ~[guava-18.0.jar:na] > > > > > > at > > > > > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270) > > > ~[curator-recipes-2.11.0.jar:na] > > > > > > at > > > > > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303) > > > [nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339) > > > [nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753) > > > [nifi-jetty-1.0.0.jar:1.0.0] > > > > > > at org.apache.nifi.NiFi.<init>(NiFi.java:152) > > > [nifi-runtime-1.0.0.jar:1.0.0] > > > > > > at org.apache.nifi.NiFi.main(NiFi.java:243) > > > [nifi-runtime-1.0.0.jar:1.0.0] > > > > > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener] > > > org.apache.nifi.io.socket.SocketListener Failed to communicate > > with Unknown > > > Host due to java.net.SocketException: Socket closed > > > > > > java.net.SocketException: Socket closed > > > > > > at java.net.PlainSocketImpl.socketAccept(Native Method) > > > ~[na:1.8.0_51] > > > > > > at > > > java.net > > <http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) > > > ~[na:1.8.0_51] > > > > > > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > > > ~[na:1.8.0_51] > > > > > > at > > > > > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) > > > ~[na:1.8.0_51] > > > > > > at > > > > > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112) > > > ~[nifi-socket-utils-1.0.0.jar:1.0.0] > > > > > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > > > > > > 2016-10-19 13:14:45,064 WARN [main] > > org.apache.nifi.web.server.JettyServer > > > Failed to start web server... shutting down. > > > > > > java.lang.Exception: Unable to load flow due to: > > java.io.IOException: > > > org.apache.nifi.controller.UninheritableFlowException: Failed to > > connect > > > node to cluster because local flow is different than cluster flow. > > > > > > at > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755) > > > ~[nifi-jetty-1.0.0.jar:1.0.0] > > > > > > at org.apache.nifi.NiFi.<init>(NiFi.java:152) > > > [nifi-runtime-1.0.0.jar:1.0.0] > > > > > > at org.apache.nifi.NiFi.main(NiFi.java:243) > > > [nifi-runtime-1.0.0.jar:1.0.0] > > > > > > Caused by: java.io.IOException: > > > org.apache.nifi.controller.UninheritableFlowException: Failed to > > connect > > > node to cluster because local flow is different than cluster flow. > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) > > > ~[nifi-jetty-1.0.0.jar:1.0.0] > > > > > > ... 2 common frames omitted > > > > > > Caused by: org.apache.nifi.controller.UninheritableFlowException: > > Failed to > > > connect node to cluster because local flow is different than > > cluster flow. > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > ... 3 common frames omitted > > > > > > Caused by: org.apache.nifi.controller.UninheritableFlowException: > > Proposed > > > Authorizer is not inheritable by the flow controller because of > > Authorizer > > > differences: Proposed Authorizations do not match current > > Authorizations > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > at > > > > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) > > > ~[nifi-framework-core-1.0.0.jar:1.0.0] > > > > > > ... 4 common frames omitted > > > > > > [root@ncm-cm1 logs]# > > > > > > > > > > > > I don’t know if the ‘Proposed Authorizer is not inheritable…’ > > exception is > > > part of the problem too. > > > > > > The docs weren’t very clear on whether (when upgrading and using > > the legacy > > > support of the authorized-user.xml path required the nodes to be > > also added > > > to the authorizers.xml. > > > > > > I did add them in the end as various attempts to get the cluster > > up and > > > running without them failed (as each server didn’t seem to have > > rights to do > > > anything. > > > > > > > > > > > > I have a lot of RPG in my work flows as I am ingesting many > > syslog data > > > sources and this was the recommended pattern to distribute the > > data > > > (listensyslog…run on primary, output to port (RPG), pick up in > > rest of data > > > flow), > > > > > > > > > > > > Any suggestions on where to start trying to get this working? > > > > > > I’ve tried creating a new RPG on one on the nifis and connecting > > the > > > syslog to that which sort of worked but then I have a bunch of > > other errors > > > when trying to enable the ports to do with not being able to > > connect to > > > (what was) the NCM. > > > > > > > > > > > > Thanks > > > > > > Conrad > > > > > > > > > > > > SecureData, combating cyber threats > > > > > > ________________________________ > > > > > > The information contained in this message or any of its > > attachments may be > > > privileged and confidential and intended for the exclusive use of > > the > > > intended recipient. If you are not the intended recipient any > > disclosure, > > > reproduction, distribution or other dissemination or use of this > > > communications is strictly prohibited. The views expressed in > > this email are > > > those of the individual and not necessarily of SecureData Europe > > Ltd. Any > > > prices quoted are only valid if followed up by a formal written > > quote. > > > > > > SecureData Europe Limited. Registered in England & Wales 04365896. > > > Registered Address: SecureData House, Hermitage Court, Hermitage > > Lane, > > > Maidstone, Kent, ME16 9NT > > > > > > ***This email originated outside SecureData*** > > > > Click > > https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== > > <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this > > email as spam. > > > > > > > > > > > > > * Unknown Key > * 0x2F7DEF69 >
signature.asc
Description: Message signed with OpenPGP using GPGMail
