Hi Due to you keep failing in sending attachment, please submit issue at GitHub. We could discuss there too.
Sheng Wu Apache Skywalking, ShardingSphere, Zipkin > 在 2019年8月7日，上午11:12，吴延涛 <[email protected]> 写道： > > > FYI. > > > > > At 2019-08-07 10:56:47, "吴延涛" <[email protected]> wrote: > > Hi All, > > See the graphs in attachments. > For the first issue, I want to have a supplement. The root cause is > the visibility issue in multithread programming. > According to JMM(Java Memory Model), when one thread modify the buffer, the > other thread can't obtain the modified > result immediately without any synchronized measure. So I change the buffer > array to the JDK ArrayBlockingQueue, > use blocking queue to guarantee the visibility between the different threads. > In the code I use none blocking method，if you want to use blocking > way，you should change the offer method to put method. > > > > > At 2019-08-06 20:39:55, "Sheng Wu" <[email protected]> wrote: > >Hi > > > >Thanks for the feedback. We can't see the graphs you posted, could you send > >them in attachments? > > > >Sheng Wu 吴晟 > > > >Apache SkyWalking, Apache ShardingSphere(Incubating), Zipkin > >Twitter, wusheng1108 > > > > > >吴延涛 <[email protected]> 于2019年8月6日周二 下午8:31写道： > > > >> Hi, > >> Wu Sheng, I am a developer of Tongcheng. I find some performance > >> issue when I use Skywalking in production environment. > >> My Skywalking version is 6.0.0-GA. I find three issues： > >> 1.Skywalking UI will not obtain trace data after the OAP server runs > >> for a period of time and ES does't have data also. But reboot > >> the OAP server, SW UI can obatin the data. I dump the thread when the OAP > >> server does't work. After I analysis the dump file, > >> I find the GRPC's thread that put the trace data into buffer is in > >> sleeping state and the persistence worker which consume the buffer also > >> blocked。 > >> So I think the buffer has problem, because the buffer producer and the > >> buffer consumer is in two threads, when the producer save data > >> into buffer the consumer does't know the buffer has data, which is the > >> visibility issue in multithread。So I change the buffer array into > >> ArrayBlockingQueue like this: > >> After this change everything runs well, besides the performance > >> improved obviously. > >> 2.The GRPC server is using the JDK CachedThreadPool，the thread > >> number of which increase with requests，so when plenty of requests is coming > >> and OAP server > >> is not enough or ES is not enough，the GRPC thead number can reach to a big > >> number，then the OAP server crashed。So I change the code like this： > >> 3.We use consul as register center，the consul health check with GRPC > >> way is not stable，so I change the way to tcp： > >> > >> Above is the issue I encounter in production environment。 > >> > >> > >> BRs， > >> Wu Yantao（吴延涛） > >> > >> > >> > >> > >> > >> > > > > > >
