Ok, I am not completely familiar with all the ins and outs of the site-to-site client code, but I know one place that it creates a connection is here:
https://github.com/apache/nifi/blob/a3586e04d9978e105cc5645e893dc6d77b79b86e/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/socket/EndpointConnectionPool.java#L439 The SSLContext that will be used in there comes from the FlowController which takes the SSL related properties from nifi.properties: https://github.com/apache/nifi/blob/1745c1274b274a994a92312054d8951ce1c499d0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L442 The part I am not sure about is whether or not there is a call somewhere before the EndpointConnectionPool, but you could at least start by seeing if it is hitting there. I can look a little more later. -Bryan On Fri, Sep 2, 2016 at 12:25 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > Bryan, > > > > In the log on the server side we see this message: > > > > INFO [NiFi Web Server-324] o.a.n.w.a.c.AccessDeniedExceptionMapper > anonymous does not have permission to access the requested resource. > Returning Unauthorized response. > > > > I forgot to mention, we tried adding a user named anonymous and granting > it access. When we did this Site-to-Site started working. Obviously that > is not a course of action we want to take, but it was a good exercise. > > > > When I debugged the X509AuthenticationFilter it found no certs, here is > the screenshot showing some of the stack/variables at the time. > > https://goo.gl/photos/938E5A8vsb7nQ6Kh8 > > > > What would be the equivalent source code for debugging the client to find > out why no cert is being sent? > > > > Thanks, > > Peter > > > > > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Friday, September 02, 2016 10:09 AM > > *To:* users@nifi.apache.org > *Subject:* Re: I need help configuring Site-to-Site in Secure Mode. > > > > Paul/Peter, > > > > Having Kerberos enabled should not have any impact, you can have Kerberos > or LDAP enabled, but if a certificate is provided that should always take > precedence. > > > > What do you see in the nifi-user.log on the second instance (the one where > the remote process group is pointing at)? > > > > If it found the incoming cert it should log something like: > > > > "Attempting request for <cert of first NiFi> %s %s (source ip: %s)" > "Authentication success for <cert of first NiFi>" > > > > If it didn't find one I think it should log: > > > "Rejecting access to web api: %s", ae.getMessage() > > > > It might be help to try debugging into the X509AuthenticationFilter: > > > > https://github.com/apache/nifi/blob/master/nifi-nar- > bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi- > web-security/src/main/java/org/apache/nifi/web/security/ > x509/X509AuthenticationFilter.java > > > > The base class it extends is: > > > > https://github.com/apache/nifi/blob/master/nifi-nar- > bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi- > web-security/src/main/java/org/apache/nifi/web/security/ > NiFiAuthenticationFilter.java > > > > If it goes into the X509 filter and returns null from > attemptAuthentication, we can at least narrow down to the fact that that > the client certificate is not in the request for some reason. > > > > It sounds like you might have already done this, but there is a line in > bootstrap.conf that you can uncomment to hook up a remote debugger: > > java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y, > suspend=n,address=8000 > > -Bryan > > > > > > > > On Fri, Sep 2, 2016 at 11:27 AM, Peter Wicks (pwicks) <pwi...@micron.com> > wrote: > > Matt, > > > > That was the case on our first go round, when we only had SSL certs. We > went back yesterday and got new certificates that support both Server Auth > and Client Auth and rebuilt our KeyStore. > > > > When I use keytool to look at my KeyStore I can see both of these on the > certificate: > > > > #6: ObjectId: 2.5.29.37 Criticality=false > > ExtendedKeyUsages [ > > * clientAuth* > > * serverAuth* > > ] > > > > Thanks, > > Peter > > > > *From:* Matthew Clarke [mailto:matt.clarke....@gmail.com] > *Sent:* Friday, September 02, 2016 9:23 AM > > > *To:* users@nifi.apache.org > *Subject:* Re: I need help configuring Site-to-Site in Secure Mode. > > > > Do the certs you created/obtained support being used for both client and > server auth. If they were created for server auth only, this could explain > your issue. NiFi instances need to act as a client and a server at times. > > > > Thanks, > > Matt > > > > On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) <pwi...@micron.com> > wrote: > > Bryan, > > > > We’ve fixed our certs, with no change to the outcome. > > > > We have username/password authentication enabled, via Kerberos, are there > issues having Kerberos enabled (username/password) and trying to do > site-to-site? When I try to initiate site-to-site with an instance of NiFI > configured for Kerberos all requests come through to the server as > anonymous because no challenge appears to be sent. We’ve debugged the code > and even deep down in NiFiUserUtils.getNiFiUser the request is already > marked as anonymous by the Spring framework. It appears to me that the > client has a cert, and is waiting for a challenge(?) from the server, and > the server is configured for Kerberos and it’s waiting for a ‘bearer’ token… > > > > We’ve debugged both client and server, the client sends the request and > gets back a 401 (Unauthorized). SSL verifies good. > > Server doesn’t appear to get any authorization information of any kind. > > > > Looking for further guidance/next steps. > > > > Thanks, > > Peter > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Thursday, September 01, 2016 9:44 AM > > > *To:* users@nifi.apache.org > *Subject:* Re: I need help configuring Site-to-Site in Secure Mode. > > > > Peter, > > > > Yes, by no means am I saying everyone should use the TLS toolkit. I was > just using that because many people are not familiar with how to create > certificates, and for people trying to follow a tutorial it is the easiest > option. > > > > In your case you definitely want to be using your CA. What you described > about not having a cert for client authentication definitely sounds like it > would be a problem. Let us know if everything works out after getting the > new certs. > > > > -Bryan > > > > > > On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) <pwi...@micron.com> > wrote: > > Bryan, > > > > Paul and I have been working on this, and I think our issue is related to > certificates. > > > > In your blog posting you used TLS-Toolkit in your example, but I think > that is unrealistic for many environments. For example, this also creates > the certificates for SSL right? But these will be self-signed and thus > untrusted by default in web browsers. In our environment we generated SSL > certificates from our CA and loaded them into the KeyStore. We then > extracted public keys for the SSL certs and put them in each of the Trust > Stores. This I think is where our main problem is… > > > > I’m making a few assumptions here, so feel free to correct me, but my > understanding is that when you use TLS-Toolkit it either creates multiple > certs (SSL & Client Auth), or it creates a cert that you are allowed to use > for both activities. In our case we ONLY have SSL certs, and the certs are > marked such that they aren’t allowed to be used for Client Authentication. > I believe this is the reason why our requests are showing up as > ‘Anonymous’, because there are no Client Authentication certificates in the > KeyStore, just SSL certs. > > > > I’ve asked our security team for Client Authentication certs for each > server, since it would be our preference to use our CA rather than having > TLS-Toolkit be its own CA. > > > > Thoughts? > > > > Thanks, > > Peter > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Thursday, September 01, 2016 9:26 AM > > > *To:* users@nifi.apache.org > *Subject:* Re: I need help configuring Site-to-Site in Secure Mode. > > > > Paul, > > > > Clustering is not a requirement for site-to-site... This sounds strange > since "anonymous" is used to represent a user when NiFi is not secured. > > > > Can you double-check all your configs and make sure you have the following > properties set... > > > > nifi.remote.input.secure=true > > nifi.web.https.host= > nifi.web.https.port= > > nifi.security.keystore= > nifi.security.keystoreType= > nifi.security.keystorePasswd= > nifi.security.keyPasswd= > nifi.security.truststore= > nifi.security.truststoreType= > nifi.security.truststorePasswd=<your password> > > > > After your question the other day I went through the steps of setting > secure site-to-site to make sure I knew what I was talking about :) > > > > I wrote up the steps here: http://bryanbende.com/ > development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site > > > > Thanks, > > > > Bryan > > > > On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) < > pagibea...@micron.com> wrote: > > Bryan, > > > > Thanks for the reply. After increasing the log level for Authentication I > saw the target NiFi instance used the account “anonymous” for the > Site-to-Site connection. After creating a policy for “anonymous”, I was > able to view the remote ports and connect to them. > > > > Obviously this is not ideal. We would prefer to make policies for remote > hosts/users rather than anonymous. > > > > We are using the same SSL Certificate for both Key Store and Trust Store > on a NiFi Instance. This is likely the cause of the “anonymous” user as it > doesn’t have a DN. We are working to correct this. > > > > However, after getting my work flow set up across NiFi instances I see > this error: > > > > Unable to refresh Remote Group's peers due to Unable to communicate with > remote NiFi cluster in order to determine which nodes exist in the remote > cluster > > > > Our NiFi servers are not set up for clustering. Is clustering required to > perform Site-to-Site? > > > > Thanks, > > Paul Gibeault > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Tuesday, August 30, 2016 5:09 PM > *To:* users@nifi.apache.org > *Subject:* Re: I need help configuring Site-to-Site in Secure Mode. > > > > Paul, > > > > It sounds like you probably have the certificates/truststores setup > correctly and just need to create the appropriate policies... > > > > Lets say you have nifi-1 with an Remote Process Group pointing at the URL > of nifi-2, and nifi-2 has an Input port to receive data. > > > > In nifi-2 there needs to be a user for the certificate of nifi-1, and then > in the global policies of nifi-2 (top right menu) there needs to be a > policy for "retrieve site-to-site details" with the nifi-1 user added to > the policy. I think this is what is causing the error message you are > seeing since nifi-1 is not authorized to query nifi-2 for site-to-site > information (available ports, etc). > > > > I believe you also need to create a policy on the Input Port on nifi-2... > select the input port and use the lock icon in the left palette and choose > "receive data over site-to-site" and add the user of nifi-1. This gives > nifi-1 access to the specific port. > > > > Let us know if that works. If so we should definitely look at updating > some of the documentation to explain this. > > > > Thanks, > > > > Bryan > > > > > > On Tue, Aug 30, 2016 at 6:28 PM, Paul Gibeault (pagibeault) < > pagibea...@micron.com> wrote: > > Hello, > > > > We have been attempting to set up Site-to-Site for NiFi in secure mode and > have not been successful. > > > > When I create a Remote Process Group, and enter the URL* > https://servername:8443/nifi I receive an error icon. The hover status > is “Unauthorized” > > ** - servername is the actual hostname running NiFi* > > > > Things I have tried without success: > > - Closely followed the instructions in the NiFi System Administrator’s > Guide > <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html> > > - Enabled SSL Security and Kerberos User Authentication (no self-signed > certs) > > - Imported public keys of remote NiFi servers into the local keystores for > each instance > > - Created policies on each instance to allow for full permissions to the > accounts in use. > > - Tried various combinations of Linux & Windows instances of NiFi. > > - Connected a Site-to-Site process group to itself > > - Used both 1.0.0-BETA and 1.1.0-SNAPSHOT NiFi versions > > > > There are no warnings or errors in the log files when I attempt to connect > a NiFi instance running on Linux to another instance on Linux. However, I > did see something when attempting to connect a NiFi instance running on > Windows to an instance running on Linux > > From log of NiFi on Windows: > > 2016-08-30 16:11:21,173 ERROR [Remote Process Group > dd2e3ac9-0156-1000-4543-5ba3d10c6130: https://servername:8443/nifi > Thread-1] o.a.n.remote.StandardRemoteProcessGroup org.apache.nifi.remote. > StandardRemoteProcessGroup$InitializationTask@19bde1f3 Failed to request > account: got unexpected response code of 404:Not Found > > > > From log of NiFi on Linux: > > nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] > o.a.n.w.s.NiFiAuthenticationFilter > Attempting request for (CN=pagibeault) GET https://servername:8443/nifi- > api/site-to-site (source ip: 137.201.48.150) > > nifi-user.log:2016-08-30 16:39:53,973 INFO [NiFi Web Server-465] > o.a.n.w.s.NiFiAuthenticationFilter > Authentication success for CN=pagibeault > > nifi-user.log:2016-08-30 16:39:53,974 INFO [NiFi Web Server-465] > o.a.n.w.a.c.AccessDeniedExceptionMapper CN=pagibeault does not have > permission to access the requested resource. Returning Forbidden response. > > > > Any guidance would be grand. > > > > Thanks, > > > > [image: http://collab.micron.com/corp/brand/SiteAssets/Micron.png] > <http://www.micron.com/> > > *Paul Gibeault* > Sr. Software Engineer, Big Data > Enterprise Analytics & Data > Micron Technology, Inc. > > *Office* (208) 363-3238 <%28208%29%20363-3238> > > > pagibea...@micron.com > [image: http://collab.micron.com/corp/brand/SiteAssets/LinkedIn.png] > linkedin.com/in/paulgibeault > <https://www.linkedin.com/in/linkedin.com/in/paulgibeault> > > > > > > > > > > > > > > >
