[GitHub] [dubbo] zewade opened a new issue #8297: Dubbo 2.6.3 can send request but no response in the product environment sometimes.

GitBox Wed, 14 Jul 2021 23:01:25 -0700


zewade opened a new issue #8297:
URL: https://github.com/apache/dubbo/issues/8297



   ### Environment
   
   * Dubbo version: 2.6.3
   * Operating System version: Linux
   * Java version: openjdk8u201
   
   
在我们的线上环境偶尔会发生A服务调用其他服务Dubbo接口超时的问题，从调用链系统看其他服务已经处理完成，但是调用方A服务接受不到响应，最终报超时异常，需要重启A服务才能恢复。在些期间A可以正常接收请求，进行数据库操作并返回响应，也没有特别大的延迟，资源占用也正常。求助排查的方法。
   
   第一种超时发生在DefaultFuture.java:136
   `2021-07-12 08:35:44,124 ERROR [xxxxyy-patient-provider] 
[DubboServerHandler-10.130.3.104:20882-thread-6] 
[c.f.plugin.transmission.EmrCodeUtils:115] - 
trace[b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1] 项目名：[classes!]
   com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method 
checkHosDeptPepList in the service 
cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService. Tried 3 times of 
the providers [10.131.4.190:20880, 10.128.3.66:20880, 10.129.6.128:20880] 
(3/16) from the registry zookeeper:2181 on the consumer 10.130.3.104 using the 
dubbo version 2.6.3. Last error is: com.alibaba.dubbo.rpc.RpcException: Invoke 
remote method timeout. method: checkHosDeptPepList, provider: 
dubbo://10.131.4.190:20880/cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dispatcher=all&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService&methods=getStartDate,listEveryMoe,BS50002ESB,checkHosDeptPepList,listGroupMoe,stopMoeForDoc,lineDrugsDetail,similarDietary,stopMoeForDocV1,undo,selectListMoePrintLog,getLastFee
 
Time2,stopMoeForDocV2,havingYryMoe,countDosageForOrdCode,deprecatedMoe,docListMoePrint,endMoeV1,getAdvices,getCateMethod,listMoeSpecialTreatment,listMoe,listEveryMoeForMem,listEveryMoeV2,test,listMoeV4,listOrderCycle,listMoePrint,inquireForMedical,similarSkinTest,listMoeByRescueBindNo,getEnterDate,countDosageForOrdCodeCheck,listEveryMoeForMemPage,listMoeByEnterDate,similarDietaryEcg,getLastFeeTime,aboutEffectiveNursing,searchPatSrescueOrders,endMoe,effectiveInspection&payload=83886080&pid=1&qos.enable=false&register.ip=10.130.3.104&remote.timestamp=1624626328873&revision=0.0.5-SNAPSHOT&side=consumer&timestamp=1625275654554&version=2.0.0,
 cause: Waiting server-side response timeout. start time: 2021-07-12 
08:35:32.122, end time: 2021-07-12 08:35:44.123, client elapsed: 0 ms, server 
elapsed: 12001 ms, timeout: 12000 ms, request: Request [id=3706895, 
version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation 
[methodName=checkHosDeptPepList, parameterTypes=[class co.faao.
 plugin.transmission.request.MessageHeader], 
arguments=[co.faao.plugin.transmission.request.MessageHeader@72fb7075], 
attachments={input=2570, 
path=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService, 
application=node-dubbo, 
uber-trace-id=b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1, 
interface=cc.xxxxx.xxxxyy.patient.api.service.IInpNursePatientService, 
version=2.0.0, timeout=12000}]], channel: /10.130.3.104:38668 -> 
/10.131.4.190:20880
        at 
com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:102)
        at 
com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:234)
        at 
com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
        at 
com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
        at 
com.alibaba.dubbo.common.bytecode.proxy17.checkHosDeptPepList(proxy17.java)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.busines.select.InpOtherSelectServiceImpl.getMoesByInpVisistIds(InpOtherSelectServiceImpl.java:287)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.getMoesByInpVisistIds(InpNursePatBusServiceImpl.java:1929)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.queryListMoeNurCheck(InpNursePatBusServiceImpl.java:1623)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.getPatNumsBySelect(InpNursePatBusServiceImpl.java:2337)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.homePageStatistics(InpNursePatBusServiceImpl.java:2128)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.dubbo.InpNursePatientServiceImpl.homePageStatistics(InpNursePatientServiceImpl.java:110)
        at 
com.alibaba.dubbo.common.bytecode.Wrapper58.invokeMethod(Wrapper58.java)
        at 
com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
        at 
com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:76)
        at 
com.alibaba.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:52)
        at 
com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
        at 
io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:62)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processProviderTrace(OpenTracingFilter.java:97)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:63)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.starter.dubbo.filter.UserFilter.invoke(UserFilter.java:39)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:73)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:104)
        at 
com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
        at 
com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:172)
        at 
com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
        at 
com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   Caused by: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. 
method: checkHosDeptPepList, provider: 
dubbo://10.131.4.190:20880/cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dispatcher=all&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService&methods=getStartDate,listEveryMoe,BS50002ESB,checkHosDeptPepList,listGroupMoe,stopMoeForDoc,lineDrugsDetail,similarDietary,stopMoeForDocV1,undo,selectListMoePrintLog,getLastFeeTime2,stopMoeForDocV2,havingYryMoe,countDosageForOrdCode,deprecatedMoe,docListMoePrint,endMoeV1,getAdvices,getCateMethod,listMoeSpecialTreatment,listMoe,listEveryMoeForMem,listEveryMoeV2,test,listMoeV4,listOrderCycle,listMoePrint,inquireForMedical,similarSkinTest,listMoeByRescueBindNo,getEnterDate,countDosageForOrdCodeCheck,listEveryMoeForMemPage,listMoe
 
ByEnterDate,similarDietaryEcg,getLastFeeTime,aboutEffectiveNursing,searchPatSrescueOrders,endMoe,effectiveInspection&payload=83886080&pid=1&qos.enable=false&register.ip=10.130.3.104&remote.timestamp=1624626328873&revision=0.0.5-SNAPSHOT&side=consumer&timestamp=1625275654554&version=2.0.0,
 cause: Waiting server-side response timeout. start time: 2021-07-12 
08:35:32.122, end time: 2021-07-12 08:35:44.123, client elapsed: 0 ms, server 
elapsed: 12001 ms, timeout: 12000 ms, request: Request [id=3706895, 
version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation 
[methodName=checkHosDeptPepList, parameterTypes=[class 
co.faao.plugin.transmission.request.MessageHeader], 
arguments=[co.faao.plugin.transmission.request.MessageHeader@72fb7075], 
attachments={input=2570, 
path=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService, 
application=node-dubbo, 
uber-trace-id=b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1, 
interface=cc.xxxxx.xxxxyy.patient.api.service.IInpNurseP
 atientService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:38668 
-> /10.131.4.190:20880
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:98)
        at 
com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:154)
        at 
co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processConsumerTrace(OpenTracingFilter.java:196)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:65)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:54)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:49)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:77)
        at 
com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
        at 
com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:78)
        ... 50 common frames omitted
   Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side 
response timeout. start time: 2021-07-12 08:35:32.122, end time: 2021-07-12 
08:35:44.123, client elapsed: 0 ms, server elapsed: 12001 ms, timeout: 12000 
ms, request: Request [id=3706895, version=2.0.2, twoway=true, event=false, 
broken=false, data=RpcInvocation [methodName=checkHosDeptPepList, 
parameterTypes=[class co.faao.plugin.transmission.request.MessageHeader], 
arguments=[co.faao.plugin.transmission.request.MessageHeader@72fb7075], 
attachments={input=2570, 
path=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService, 
application=node-dubbo, 
uber-trace-id=b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1, 
interface=cc.xxxxx.xxxxyy.patient.api.service.IInpNursePatientService, 
version=2.0.0, timeout=12000}]], channel: /10.130.3.104:38668 -> 
/10.131.4.190:20880
        at 
com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:136)
        at 
com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:112)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:95)
        ... 67 common frames omitted`
   
   第二种超时发生在DefaultFuture.java:220
   `2021-07-12 08:35:38,870 ERROR [xxxxyy-patient-provider] 
[DubboServerHandler-10.130.3.104:20882-thread-186] 
[c.f.plugin.transmission.EmrCodeUtils:115] - 
trace[37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b912a3:377906989b7f7603:1] 
项目名：[classes!]
   com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method 
searchAllDictByCodeList in the service 
cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService. Tried 3 times of the 
providers [10.131.4.168:20888, 10.129.4.18:20888, 10.129.6.108:20888] (3/12) 
from the registry zookeeper:2181 on the consumer 10.130.3.104 using the dubbo 
version 2.6.3. Last error is: com.alibaba.dubbo.rpc.RpcException: Invoke remote 
method timeout. method: searchAllDictByCodeList, provider: 
dubbo://10.129.4.18:20888/cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService&methods=searchAllDictByCodeList,searchStaffCodeByHrpCode,searchWarehouseDrugStockByDrugList,searchLikeRangeDictByConditions,searchMainDictByCompareCode,queryFeeItemByFeeType,searchAllergeSuorceByConditions,searchMainD
 
ictByCodeList,searchMainDictUnionByMoe,searchMainDictUnionByMoeV2,searchLikeMainDictByCodesByPages,queryBaseSubDict,queryBaseSubDictByCodes,searchMainDictUnionByAllMoe,searchMainDictByHerbal&pid=1&qos.enable=false&register.ip=10.130.3.104&remote.timestamp=1624008835759&revision=0.0.4-SNAPSHOT&side=consumer&timestamp=1625275650525&version=2.0.0,
 cause: Waiting server-side response timeout by scan timer. start time: 
2021-07-12 08:35:26.840, end time: 2021-07-12 08:35:38.866, client elapsed: 0 
ms, server elapsed: 12026 ms, timeout: 12000 ms, request: Request [id=3706836, 
version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation 
[methodName=searchAllDictByCodeList, parameterTypes=[class 
co.faao.plugin.transmission.request.MessageHeader], 
arguments=[co.faao.plugin.transmission.request.MessageHeader@454f904f], 
attachments={input=1019, 
path=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService, application=node-dubbo, 
uber-trace-id=37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b91
 2a3:377906989b7f7603:1, 
interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.ILingYiDubboService, 
version=2.0.0, timeout=12000}]], channel: /10.130.3.104:42286 -> 
/10.129.4.18:20888
        at 
com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:102)
        at 
com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:234)
        at 
com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
        at 
com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
        at 
com.alibaba.dubbo.common.bytecode.proxy1.searchAllDictByCodeList(proxy1.java)
        at 
cc.xxxxx.xxxxyy.patient.common.QueryDictNameUtil.getBaseDictCode(QueryDictNameUtil.java:82)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.busines.InpHostialRecServiceImpl.queryPatInfor(InpHostialRecServiceImpl.java:922)
        at 
cc.xxxxx.xxxxyy.patient.service.impl.dubbo.InputPatientServiceImpl.queryPatInfor(InputPatientServiceImpl.java:134)
        at 
com.alibaba.dubbo.common.bytecode.Wrapper21.invokeMethod(Wrapper21.java)
        at 
com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
        at 
com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:76)
        at 
com.alibaba.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:52)
        at 
com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
        at 
io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:62)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processProviderTrace(OpenTracingFilter.java:97)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:63)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.starter.dubbo.filter.UserFilter.invoke(UserFilter.java:39)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:73)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:104)
        at 
com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
        at 
com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:172)
        at 
com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
        at 
com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   Caused by: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. 
method: searchAllDictByCodeList, provider: 
dubbo://10.129.4.18:20888/cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService&methods=searchAllDictByCodeList,searchStaffCodeByHrpCode,searchWarehouseDrugStockByDrugList,searchLikeRangeDictByConditions,searchMainDictByCompareCode,queryFeeItemByFeeType,searchAllergeSuorceByConditions,searchMainDictByCodeList,searchMainDictUnionByMoe,searchMainDictUnionByMoeV2,searchLikeMainDictByCodesByPages,queryBaseSubDict,queryBaseSubDictByCodes,searchMainDictUnionByAllMoe,searchMainDictByHerbal&pid=1&qos.enable=false&register.ip=10.130.3.104&remote.timestamp=1624008835759&revision=0.0.4-SNAPSHOT&side=consumer&timestamp=1625275650525&version=2.0.0,
 ca
 use: Waiting server-side response timeout by scan timer. start time: 
2021-07-12 08:35:26.840, end time: 2021-07-12 08:35:38.866, client elapsed: 0 
ms, server elapsed: 12026 ms, timeout: 12000 ms, request: Request [id=3706836, 
version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation 
[methodName=searchAllDictByCodeList, parameterTypes=[class 
co.faao.plugin.transmission.request.MessageHeader], 
arguments=[co.faao.plugin.transmission.request.MessageHeader@454f904f], 
attachments={input=1019, 
path=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService, application=node-dubbo, 
uber-trace-id=37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b912a3:377906989b7f7603:1,
 interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.ILingYiDubboService, 
version=2.0.0, timeout=12000}]], channel: /10.130.3.104:42286 -> 
/10.129.4.18:20888
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:98)
        at 
com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:154)
        at 
co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processConsumerTrace(OpenTracingFilter.java:196)
        at 
co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:65)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:54)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:49)
        at 
com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
        at 
com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:77)
        at 
com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
        at 
com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:78)
        ... 47 common frames omitted
   Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side 
response timeout by scan timer. start time: 2021-07-12 08:35:26.840, end time: 
2021-07-12 08:35:38.866, client elapsed: 0 ms, server elapsed: 12026 ms, 
timeout: 12000 ms, request: Request [id=3706836, version=2.0.2, twoway=true, 
event=false, broken=false, data=RpcInvocation 
[methodName=searchAllDictByCodeList, parameterTypes=[class 
co.faao.plugin.transmission.request.MessageHeader], 
arguments=[co.faao.plugin.transmission.request.MessageHeader@454f904f], 
attachments={input=1019, 
path=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService, application=node-dubbo, 
uber-trace-id=37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b912a3:377906989b7f7603:1,
 interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.ILingYiDubboService, 
version=2.0.0, timeout=12000}]], channel: /10.130.3.104:42286 -> 
/10.129.4.18:20888
        at 
com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:220)
        at 
com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:139)
        at 
com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:112)
        at 
com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:95)
        ... 64 common frames omitted`
   
   在此期间zk也有断开重连发生
   `2021-07-12 08:34:22,405 WARN  [xxxxyy-patient-provider] 
[main-SendThread(zookeeper:2181)] [org.apache.zookeeper.ClientCnxn:1112] - 
trace[] Client session timed out, have not heard from server in 29209ms for 
sessionid 0x20b1457b40e018a
   2021-07-12 08:34:22,407 INFO  [xxxxyy-patient-provider] 
[main-SendThread(zookeeper:2181)] [org.apache.zookeeper.ClientCnxn:1160] - 
trace[] Client session timed out, have not heard from server in 29209ms for 
sessionid 0x20b1457b40e018a, closing socket connection and attempting reconnect
   2021-07-12 08:34:39,598 INFO  [xxxxyy-patient-provider] [main-EventThread] 
[o.a.c.f.state.ConnectionStateManager:228] - trace[] State change: SUSPENDED
   2021-07-12 08:34:45,301 ERROR [xxxxyy-patient-provider] 
[Curator-Framework-0] [org.apache.curator.ConnectionState:228] - trace[] 
Connection timed out for connection string (zookeeper:2181) and timeout (5000) 
/ elapsed (5678)
   org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = 
ConnectionLoss
        at 
org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
        at 
org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
        at 
org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
        at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835)
        at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
        at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
        at 
org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2021-07-12 08:34:46,242 INFO  [xxxxyy-patient-provider] [main-EventThread] 
[o.a.c.f.state.ConnectionStateManager:228] - trace[] State change: RECONNECTED`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[GitHub] [dubbo] zewade opened a new issue #8297: Dubbo 2.6.3 can send request but no response in the product environment sometimes.

Reply via email to