[grpc-io] Proxyless gRPC services in Istio mesh
I'm using the xDS support in grpc-java to build a sample project with proxyless client and server in Kotlin which participate in an Istio service mesh. My project is available here: https://github.com/wfhartford/kotlin-grpc-xds I got everything working without too much trouble, but I'm a little concerned that things might not be working as intended. What's working correctly: - Client and server can communicate, - Client requests are round-robin load-balanced across multiple service instances. What doesn't seem right: - A server interceptor reports that ServerCall.getSecurityLevel() returns NONE, - When I configure Istio to enforce STRICT mTLS via a namespace wide PeerAuthentication resource, the client's connection to the server fails with: io.grpc.StatusException: UNAVAILABLE: Connection timeout for priority outbound|8443||server.kotlin-grpc-xds.svc.cluster.local[child1] Is this the expected behavior, or have I missed something? Thanks for any insight you might have. Wesley -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/a516a4db-38e9-4e10-94f2-11187f5ec49bn%40googlegroups.com.
Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh
Thanks for getting back to me, Sanjay. As far as I can tell, my client and server are both using the appropriate Xds credentials: The client code is at https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26 Grpc.newChannelBuilder(targetUrl, XdsChannelCredentials.create(InsecureChannelCredentials.create())).build() The server code is at https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45 XdsServerBuilder.forPort(8443, XdsServerCredentials.create(InsecureServerCredentials.create())) The insecure credentials provided to both a fallback, and it looks like the sample you linked is doing the same thing. I'm not sure why, but I'm guessing that the secure connection is failing and it is falling back to insecure. Based on the example you linked, the only other requirement is that the GRPC_XDS_BOOTSTRAP environment variable is set, which is being done by the istio sidecar; kubectl describe pod shows that both the client and server containers have two environment variables injected: GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true GRPC_XDS_BOOTSTRAP: /etc/istio/proxy/grpc-bootstrap.json There are only two warning lines being logged from both the client and the server: 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]' 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap - Unknown channel option 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' for channel '[id: 0xba433026]' Do you know of anything else I might be missing that is required for a secure connection? Thanks, Wesley On Tue, May 23, 2023 at 11:10 PM 'sanjay...@google.com' via grpc.io < grpc-io@googlegroups.com> wrote: > On Wednesday, May 17, 2023 at 11:07:43 AM UTC-7 Wesley Hartford wrote: > > ... > What doesn't seem right: > >- A server interceptor reports that ServerCall.getSecurityLevel() >returns NONE, > > > Seems right when you are using InsecureChannelCredentials i.e. plaintext. > > > >- When I configure Istio to enforce STRICT mTLS via a namespace wide >PeerAuthentication resource, the client's connection to the server fails >with: io.grpc.StatusException: UNAVAILABLE: Connection timeout for >priority outbound|8443||server.kotlin-grpc-xds.svc.cluster.local[child1] > > > You will have to modify the client code to use XdsCredentials as described > in > https://github.com/grpc/grpc-java/tree/master/examples/example-xds#run-the-example-with-xds-credentials > . I am assuming the server is using XdsServerCredentials. > > > Is this the expected behavior, or have I missed something? > > Thanks for any insight you might have. > > Wesley > > -- > You received this message because you are subscribed to a topic in the > Google Groups "grpc.io" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > grpc-io+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/3e513ace-30e7-4a7d-8023-dde3a904be3cn%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/3e513ace-30e7-4a7d-8023-dde3a904be3cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2B%2B-c5w3J6ROX1NRmHFmhficeh-YRDe0Y1b6EUR%2B40hYDhEpjw%40mail.gmail.com.
Re: [grpc-io] Re: Proxyless gRPC services in Istio mesh
I've attached a copy of the log files with xds logging set to trace for an execution of the client and server with istio's mtls mode set to STRICT and PERMISSIVE. My interpretation of these logs is: In PERMISSIVE mode, neither client nor server is trying to use any type of TLS; they're both using plain text and can interact just fine. I was under the impression that PERMISSIVE mode would use mTLS, but reading the istio docs, it's a little ambiguous, so I guess this is a correct behaviour. In STRICT mode, it looks like the server is using the supplied TLS key material to accept mTLS connections, but it doesn't look like the client is making any attempt to use TLS. I'm working on your other suggestions, but thought I should update you with the logs before spending too much time on them. On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare wrote: > (adding grpc.io group back) > > On Wed, May 24, 2023 at 2:57 PM Wesley Hartford > wrote: > >> Hi, >> >> My suggestion that the connection was falling back to insecure was not >> evidence based, I'm still trying to wrap my head around how all this is >> working. >> > > Okay. > > >> >> The target address on the client side is using the xds:/// prefix. >> >> I've enabled trace level logging on the io.grpc.xds logger but I'm not >> seeing any additional log messages, have I missed something? I'm using >> slf4j and logback and have the SLF4JBridgeHandler installed. >> > > The code uses Java util logging so you can just something like > -Djava.util.logging.config.file=/logging.properties in your Java > invocation command line and enable the logger for io.grpc.xds . > > >> >> The grpc-bootstrap.json file seems reasonable, though I don't know just >> what it all means (I've attached the content). The three pem files >> referenced in certificate_providers point to real files containing >> apparently valid PEM content. >> >> You've mentioned a couple times enabling vs. disabling mTLS, are you >> referring to some specific setting on the client and/or server, or in istio >> somewhere? My understanding has been that with the Xds server and channel, >> both will use mTLS unless I specifically set the mtls mode to DISABLE in a >> PeerAuthentication resource, which I haven't done. I've experimented with >> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple >> as not enabling mTLS somewhere? >> > > In your first post you said "I got everything working without too much > trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got > the impression that you got everything working in plaintext (without > enabling mTLS) and then you enabled mTLS via Istio which is when you saw > connection problems. I was just referring to the same - you enable mTLS in > Istio security policy. > > I suggest couple of additional tests to troubleshoot this: > > - instead of using Xds Channel and Server credentials use the Tls Channel > and Server credentials with the same files provided by Istio (under > /var/lib/istio/data). You can check out the doc at > https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html > and > https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html > . Note that even if you are using XDS for load balancing, service discovery > you can use Tls credentials for security. > > - if you can try a Golang (or C++) example with your Istio setup and > verify mTLS is working with those examples. > > Hope that helps. > > > >> >> Thanks again, >> >> Wesley >> >> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io < >> grpc-io@googlegroups.com> wrote: >> >>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote: >>> >>> Thanks for getting back to me, Sanjay. As far as I can tell, my client >>> and server are both using the appropriate Xds credentials: >>> The client code is at >>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26 >>> Grpc.newChannelBuilder(targetUrl, >>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build() >>> >>> The server code is at >>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45 >>> XdsServerBuilder.forPort(8443, >>> XdsServerCredentials.create(InsecureServerCredent
