I've attached a copy of the log files with xds logging set to trace for an execution of the client and server with istio's mtls mode set to STRICT and PERMISSIVE. My interpretation of these logs is:
In PERMISSIVE mode, neither client nor server is trying to use any type of TLS; they're both using plain text and can interact just fine. I was under the impression that PERMISSIVE mode would use mTLS, but reading the istio docs, it's a little ambiguous, so I guess this is a correct behaviour. In STRICT mode, it looks like the server is using the supplied TLS key material to accept mTLS connections, but it doesn't look like the client is making any attempt to use TLS. I'm working on your other suggestions, but thought I should update you with the logs before spending too much time on them. On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare <sanjaypuj...@google.com> wrote: > (adding grpc.io group back) > > On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca> > wrote: > >> Hi, >> >> My suggestion that the connection was falling back to insecure was not >> evidence based, I'm still trying to wrap my head around how all this is >> working. >> > > Okay. > > >> >> The target address on the client side is using the xds:/// prefix. >> >> I've enabled trace level logging on the io.grpc.xds logger but I'm not >> seeing any additional log messages, have I missed something? I'm using >> slf4j and logback and have the SLF4JBridgeHandler installed. >> > > The code uses Java util logging so you can just something like > -Djava.util.logging.config.file=/logging.properties in your Java > invocation command line and enable the logger for io.grpc.xds . > > >> >> The grpc-bootstrap.json file seems reasonable, though I don't know just >> what it all means (I've attached the content). The three pem files >> referenced in certificate_providers point to real files containing >> apparently valid PEM content. >> >> You've mentioned a couple times enabling vs. disabling mTLS, are you >> referring to some specific setting on the client and/or server, or in istio >> somewhere? My understanding has been that with the Xds server and channel, >> both will use mTLS unless I specifically set the mtls mode to DISABLE in a >> PeerAuthentication resource, which I haven't done. I've experimented with >> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple >> as not enabling mTLS somewhere? >> > > In your first post you said "I got everything working without too much > trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got > the impression that you got everything working in plaintext (without > enabling mTLS) and then you enabled mTLS via Istio which is when you saw > connection problems. I was just referring to the same - you enable mTLS in > Istio security policy. > > I suggest couple of additional tests to troubleshoot this: > > - instead of using Xds Channel and Server credentials use the Tls Channel > and Server credentials with the same files provided by Istio (under > /var/lib/istio/data). You can check out the doc at > https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html > and > https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html > . Note that even if you are using XDS for load balancing, service discovery > you can use Tls credentials for security. > > - if you can try a Golang (or C++) example with your Istio setup and > verify mTLS is working with those examples. > > Hope that helps. > > > >> >> Thanks again, >> >> Wesley >> >> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io < >> grpc-io@googlegroups.com> wrote: >> >>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote: >>> >>> Thanks for getting back to me, Sanjay. As far as I can tell, my client >>> and server are both using the appropriate Xds credentials: >>> The client code is at >>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26 >>> Grpc.newChannelBuilder(targetUrl, >>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build() >>> >>> The server code is at >>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45 >>> XdsServerBuilder.forPort(8443, >>> XdsServerCredentials.create(InsecureServerCredentials.create())) >>> >>> The insecure credentials provided to both a fallback, and it looks like >>> the sample you linked is doing the same thing. >>> >>> >>> Yes, you are right - Xds credentials are being correctly used so that's >>> not an issue. >>> >>> I'm not sure why, but I'm guessing that the secure connection is failing >>> and it is falling back to insecure. >>> >>> >>> No, that cannot be the case. As described here >>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api >>> fallback credentials are *not* meant to be used when secure connection >>> fails. If you suspect that's happening can you provide some evidence (logs >>> etc)? >>> >>> >>> Based on the example you linked, the only other requirement is that the >>> GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by >>> the istio sidecar; kubectl describe pod shows that both the client and >>> server containers have two environment variables injected: >>> GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true >>> GRPC_XDS_BOOTSTRAP: >>> /etc/istio/proxy/grpc-bootstrap.json >>> >>> >>> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable) is >>> required not just for security but overall XDS support. Based on your >>> observations in the first email it looks like client and server are able to >>> communicate based on xds configuration so that part must be working. Just >>> to confirm your client did use the "xds:///" prefix to access your service, >>> right? >>> >>> If you are certain that XDS is working for load balancing and plaintext >>> communication but breaks only when mTLS is enabled it might be useful to >>> enable debug logging on both client and server side (for the io.grpc.xds >>> package and below). >>> >>> Also you can check the contents of /etc/istio/proxy/grpc-bootstrap.json esp. >>> the certificate_providers part and confirm that the 3 files provided in the >>> config have valid certificate material. Let me know if you need help >>> debugging that part. >>> >>> >>> >>> There are only two warning lines being logged from both the client and >>> the server: >>> >>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown >>> channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]' >>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown >>> channel option >>> 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' >>> for channel '[id: 0xba433026]' >>> >>> >>> I doubt this is an issue. Just to confirm you see these messages also >>> when mTLS is not enabled, right? >>> >>> >>> >>> Do you know of anything else I might be missing that is required for a >>> secure connection? >>> >>> Thanks, >>> >>> Wesley >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "grpc.io" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> grpc-io+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2B%2B-c5z%2BHRDSiZUMuU_vvvLwG%2BPjn96da3HRHp2SVX9fOZKqtQ%40mail.gmail.com.
<<attachment: xds-debug-logs.zip>>
