Hi,
I'm facing a problem, I thought it's mostly related to gRPC, but not 100%
sure. Need some help to find the right direction.
Here is my setup (in GKE):
service-A runs in one k8s pod (node.js) which uses gPRC client to connect
service-B (C++, gRpc server), and it uses a service name for the
connection. Both services are defined using NodePort type in k8s spec.
[Problem]
Everything works fine, *but sometimes when service-B restarts (crashes),
the connection doesn't work anymore, until I restart service-A manually, it
works again*. Yes, it is sometimes, other times, the connection still works
for some other times when service-B restarts.
Obviously, this issue won't allow me to sleep because we can't guarantee
that service B does not restart/crash.
I have a few thoughts, but can't tell which is more likely the issue.
1. Is it because gRpc client doesn't re-resolve the DNS name (service
name)?
2. Is it because the connection to the old pod is still not closed?
3. Did I miss any gRPC client option that handles this type of case?
4. Did I misuse the service type - instead of NodePort, I should use a
headless service?
Since there are many pieces, I still don't have a good picture to debug
this issue properly, your input will be very appreciated.
Reference:
1. gRpc client (this client is reused, makeClient is called when RPC
call has error)
const serviceAddr = 'nebula:9190';
const cred = nebula.grpc.credentials.createInsecure();
const options = {
'grpc.max_receive_message_length': MESSAGE_LENGTH,
'grpc.max_send_message_length': MESSAGE_LENGTH,
'grpc.min_reconnect_backoff_ms': MIN_RECONNECT_BACKOFF_MS,
'grpc.max_reconnect_backoff_ms': MAX_RECONNECT_BACKOFF_MS,
};
const makeClient = () => new nebula.V1Client(serviceAddr, cred, options);
1. K8s spec
apiVersion: v1
kind: Service
metadata:
name: nebula
spec:
type: NodePort
selector:
app: nebula-server
ports:
- port: 9190
name: server
targetPort: 9190
Any suggestion is appreciated. (I also posted this question in
StackOverflow
<https://stackoverflow.com/questions/76765013/how-to-root-cause-the-connection-issue-k8s-docker-grpc>
.)
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/050a9d6d-737c-41dc-8807-eeb9fb18cca5n%40googlegroups.com.
