Hi Community:
We are running skywalking 8.7.0 cluster with 8 nodes.
Recently, there are a lot of following ERRORs.
I thought maybe it's the server side consume capability problem. So I had
increase METRICS_L2_AGGREGATION_THREAD to 40, but it doesn't work.
Can you give me some leads about this problem?
2021-10-08 23:18:11,790 -
org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient - 224
[grpc-default-executor-18748] ERROR [] - DEADLINE_EXCEEDED: deadline exceeded
after 19.999959545s. [remote_addr=/12.11.0.105:11800]
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after
19.999959545s. [remote_addr=/12.11.0.105:11800]
at io.grpc.Status.asRuntimeException(Status.java:533)
~[grpc-api-1.32.1.jar:1.32.1]
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
[grpc-stub-1.32.1.jar:1.32.1]
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
[grpc-core-1.32.1.jar:1.32.1]
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
[grpc-core-1.32.1.jar:1.32.1]
at
io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:403)
[grpc-core-1.32.1.jar:1.32.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.32.1.jar:1.32.1]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.32.1.jar:1.32.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2021-10-08 23:18:11,934 -
org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient - 224
[grpc-default-executor-18734] ERROR [] - DEADLINE_EXCEEDED: deadline exceeded
after 19.999995159s. [remote_addr=/12.11.0.59:11800]
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after
19.999995159s. [remote_addr=/12.11.0.59:11800]
at io.grpc.Status.asRuntimeException(Status.java:533)
~[grpc-api-1.32.1.jar:1.32.1]
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
[grpc-stub-1.32.1.jar:1.32.1]
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
[grpc-core-1.32.1.jar:1.32.1]
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
[grpc-core-1.32.1.jar:1.32.1]
at
io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:403)
[grpc-core-1.32.1.jar:1.32.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.32.1.jar:1.32.1]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.32.1.jar:1.32.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Thank you very much!
