I looked at the logs and I can confirm that the client is not using mTLS because Istio didn't provide the right configuration. Let me explain:
>From your server-mtls-strict.log I see this "transportSocket": { "name": "envoy.transport_sockets.tls", "typedConfig": { "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext", "commonTlsContext": { "combinedValidationContext": { "defaultValidationContext": { }, "validationContextCertificateProviderInstance": { "instanceName": "default", "certificateName": "ROOTCA" } }, "tlsCertificateCertificateProviderInstance": { "instanceName": "default", "certificateName": "default" } }, "requireClientCertificate": true } }, "name": "inbound-mtls" In the Listener resource received (see timestamp 16:47:54.070) However I don't see a similar security configuration on the client side - which is a bit complex I admin. In the client-mtls-strict.log I see a Cluster resource response: { "versionInfo": "2023-05-26T16:47:41Z/749", "resources": [{ "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster", "name": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local", "type": "EDS", "edsClusterConfig": { "edsConfig": { "ads": { } }, "serviceName": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local" } }], "typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster", "nonce": "bd0fe5d1-9cb8-4342-a15f-d5d834dd3255", "controlPlane": { "identifier": "{\"Component\":\"istiod\",\"ID\":\"istiod-7fd9d6dd48-nf42k\",\"Info\":{\"version\":\"1.17.2\",\"revision\":\"3e857775086a061d12ee445f32a0b35ea17c8488\",\"golang_version\":\"go1.20.2\",\"status\":\"Clean\",\"tag\":\"1.17.2\"}}" } } which has no security/mTLS configuration. I would expect a config similar to the transportSocket snippet I described above. So the error is on the Istio side: whether a bug in Istio or a user-error I can't tell. I hope you are able to figure it out? On Friday, May 26, 2023 at 10:53:34 AM UTC-7 Wesley Hartford wrote: > I've attached a copy of the log files with xds logging set to trace for an > execution of the client and server with istio's mtls mode set to STRICT and > PERMISSIVE. My interpretation of these logs is: > > In PERMISSIVE mode, neither client nor server is trying to use any type of > TLS; they're both using plain text and can interact just fine. I was under > the impression that PERMISSIVE mode would use mTLS, but reading the istio > docs, it's a little ambiguous, so I guess this is a correct behaviour. > > In STRICT mode, it looks like the server is using the supplied TLS key > material to accept mTLS connections, but it doesn't look like the client is > making any attempt to use TLS. > > I'm working on your other suggestions, but thought I should update you > with the logs before spending too much time on them. > > On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare <sanjay...@google.com> > wrote: > >> (adding grpc.io group back) >> >> On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca> >> wrote: >> >>> Hi, >>> >>> My suggestion that the connection was falling back to insecure was not >>> evidence based, I'm still trying to wrap my head around how all this is >>> working. >>> >> >> Okay. >> >> >>> >>> The target address on the client side is using the xds:/// prefix. >>> >>> I've enabled trace level logging on the io.grpc.xds logger but I'm not >>> seeing any additional log messages, have I missed something? I'm using >>> slf4j and logback and have the SLF4JBridgeHandler installed. >>> >> >> The code uses Java util logging so you can just something like >> -Djava.util.logging.config.file=/logging.properties in your Java >> invocation command line and enable the logger for io.grpc.xds . >> >> >>> >>> The grpc-bootstrap.json file seems reasonable, though I don't know just >>> what it all means (I've attached the content). The three pem files >>> referenced in certificate_providers point to real files containing >>> apparently valid PEM content. >>> >>> You've mentioned a couple times enabling vs. disabling mTLS, are you >>> referring to some specific setting on the client and/or server, or in istio >>> somewhere? My understanding has been that with the Xds server and channel, >>> both will use mTLS unless I specifically set the mtls mode to DISABLE in a >>> PeerAuthentication resource, which I haven't done. I've experimented with >>> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple >>> as not enabling mTLS somewhere? >>> >> >> In your first post you said "I got everything working without too much >> trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got >> the impression that you got everything working in plaintext (without >> enabling mTLS) and then you enabled mTLS via Istio which is when you saw >> connection problems. I was just referring to the same - you enable mTLS in >> Istio security policy. >> >> I suggest couple of additional tests to troubleshoot this: >> >> - instead of using Xds Channel and Server credentials use the Tls Channel >> and Server credentials with the same files provided by Istio (under >> /var/lib/istio/data). You can check out the doc at >> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html >> and >> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html >> . Note that even if you are using XDS for load balancing, service discovery >> you can use Tls credentials for security. >> >> - if you can try a Golang (or C++) example with your Istio setup and >> verify mTLS is working with those examples. >> >> Hope that helps. >> >> >> >>> >>> Thanks again, >>> >>> Wesley >>> >>> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io < >>> grp...@googlegroups.com> wrote: >>> >>>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote: >>>> >>>> Thanks for getting back to me, Sanjay. As far as I can tell, my client >>>> and server are both using the appropriate Xds credentials: >>>> The client code is at >>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26 >>>> Grpc.newChannelBuilder(targetUrl, >>>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build() >>>> >>>> The server code is at >>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45 >>>> XdsServerBuilder.forPort(8443, >>>> XdsServerCredentials.create(InsecureServerCredentials.create())) >>>> >>>> The insecure credentials provided to both a fallback, and it looks like >>>> the sample you linked is doing the same thing. >>>> >>>> >>>> Yes, you are right - Xds credentials are being correctly used so that's >>>> not an issue. >>>> >>>> I'm not sure why, but I'm guessing that the secure connection is >>>> failing and it is falling back to insecure. >>>> >>>> >>>> No, that cannot be the case. As described here >>>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api >>>> >>>> fallback credentials are *not* meant to be used when secure connection >>>> fails. If you suspect that's happening can you provide some evidence (logs >>>> etc)? >>>> >>>> >>>> Based on the example you linked, the only other requirement is that the >>>> GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by >>>> the istio sidecar; kubectl describe pod shows that both the client and >>>> server containers have two environment variables injected: >>>> GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true >>>> GRPC_XDS_BOOTSTRAP: >>>> /etc/istio/proxy/grpc-bootstrap.json >>>> >>>> >>>> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable) >>>> is required not just for security but overall XDS support. Based on your >>>> observations in the first email it looks like client and server are able >>>> to >>>> communicate based on xds configuration so that part must be working. Just >>>> to confirm your client did use the "xds:///" prefix to access your >>>> service, >>>> right? >>>> >>>> If you are certain that XDS is working for load balancing and plaintext >>>> communication but breaks only when mTLS is enabled it might be useful to >>>> enable debug logging on both client and server side (for the io.grpc.xds >>>> package and below). >>>> >>>> Also you can check the contents of >>>> /etc/istio/proxy/grpc-bootstrap.json esp. the certificate_providers >>>> part and confirm that the 3 files provided in the config have valid >>>> certificate material. Let me know if you need help debugging that part. >>>> >>>> >>>> >>>> There are only two warning lines being logged from both the client and >>>> the server: >>>> >>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - >>>> Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]' >>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - >>>> Unknown channel option >>>> 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' >>>> >>>> for channel '[id: 0xba433026]' >>>> >>>> >>>> I doubt this is an issue. Just to confirm you see these messages also >>>> when mTLS is not enabled, right? >>>> >>>> >>>> >>>> Do you know of anything else I might be missing that is required for a >>>> secure connection? >>>> >>>> Thanks, >>>> >>>> Wesley >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "grpc.io" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> grpc-io+u...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/8e2dc818-f08c-4c77-ac3e-65c68c3fe6b6n%40googlegroups.com.