(adding grpc.io group back) On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca> wrote:
> Hi, > > My suggestion that the connection was falling back to insecure was not > evidence based, I'm still trying to wrap my head around how all this is > working. > Okay. > > The target address on the client side is using the xds:/// prefix. > > I've enabled trace level logging on the io.grpc.xds logger but I'm not > seeing any additional log messages, have I missed something? I'm using > slf4j and logback and have the SLF4JBridgeHandler installed. > The code uses Java util logging so you can just something like -Djava.util.logging.config.file=/logging.properties in your Java invocation command line and enable the logger for io.grpc.xds . > > The grpc-bootstrap.json file seems reasonable, though I don't know just > what it all means (I've attached the content). The three pem files > referenced in certificate_providers point to real files containing > apparently valid PEM content. > > You've mentioned a couple times enabling vs. disabling mTLS, are you > referring to some specific setting on the client and/or server, or in istio > somewhere? My understanding has been that with the Xds server and channel, > both will use mTLS unless I specifically set the mtls mode to DISABLE in a > PeerAuthentication resource, which I haven't done. I've experimented with > mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple > as not enabling mTLS somewhere? > In your first post you said "I got everything working without too much trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got the impression that you got everything working in plaintext (without enabling mTLS) and then you enabled mTLS via Istio which is when you saw connection problems. I was just referring to the same - you enable mTLS in Istio security policy. I suggest couple of additional tests to troubleshoot this: - instead of using Xds Channel and Server credentials use the Tls Channel and Server credentials with the same files provided by Istio (under /var/lib/istio/data). You can check out the doc at https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html and https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html . Note that even if you are using XDS for load balancing, service discovery you can use Tls credentials for security. - if you can try a Golang (or C++) example with your Istio setup and verify mTLS is working with those examples. Hope that helps. > > Thanks again, > > Wesley > > On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io < > grpc-io@googlegroups.com> wrote: > >> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote: >> >> Thanks for getting back to me, Sanjay. As far as I can tell, my client >> and server are both using the appropriate Xds credentials: >> The client code is at >> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26 >> Grpc.newChannelBuilder(targetUrl, >> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build() >> >> The server code is at >> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45 >> XdsServerBuilder.forPort(8443, >> XdsServerCredentials.create(InsecureServerCredentials.create())) >> >> The insecure credentials provided to both a fallback, and it looks like >> the sample you linked is doing the same thing. >> >> >> Yes, you are right - Xds credentials are being correctly used so that's >> not an issue. >> >> I'm not sure why, but I'm guessing that the secure connection is failing >> and it is falling back to insecure. >> >> >> No, that cannot be the case. As described here >> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api >> fallback credentials are *not* meant to be used when secure connection >> fails. If you suspect that's happening can you provide some evidence (logs >> etc)? >> >> >> Based on the example you linked, the only other requirement is that the >> GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by >> the istio sidecar; kubectl describe pod shows that both the client and >> server containers have two environment variables injected: >> GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true >> GRPC_XDS_BOOTSTRAP: >> /etc/istio/proxy/grpc-bootstrap.json >> >> >> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable) is >> required not just for security but overall XDS support. Based on your >> observations in the first email it looks like client and server are able to >> communicate based on xds configuration so that part must be working. Just >> to confirm your client did use the "xds:///" prefix to access your service, >> right? >> >> If you are certain that XDS is working for load balancing and plaintext >> communication but breaks only when mTLS is enabled it might be useful to >> enable debug logging on both client and server side (for the io.grpc.xds >> package and below). >> >> Also you can check the contents of /etc/istio/proxy/grpc-bootstrap.json esp. >> the certificate_providers part and confirm that the 3 files provided in the >> config have valid certificate material. Let me know if you need help >> debugging that part. >> >> >> >> There are only two warning lines being logged from both the client and >> the server: >> >> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown >> channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]' >> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown >> channel option >> 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' >> for channel '[id: 0xba433026]' >> >> >> I doubt this is an issue. Just to confirm you see these messages also >> when mTLS is not enabled, right? >> >> >> >> Do you know of anything else I might be missing that is required for a >> secure connection? >> >> Thanks, >> >> Wesley >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "grpc.io" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> grpc-io+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com >> <https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2BPad6jze2Rh5%3DgS%3DKk6i5hZ-nB9-HttExRSs4TAXyMCBNksrw%40mail.gmail.com.
