[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-533441251 Really, there is nothing I can do. There are plenty of users using the project in production environments. Some of them are supporting several TB traces per day. We can't provide the ElasticSearch service for you. ElasticSearch is a stable, powerful and widely used data platform, so you should be able to find the solution by yourself. The only thing I could say is, either you don't set your ElasticSearch right(we highlighted some in our FAQ, https://github.com/apache/skywalking/blob/master/docs/en/FAQ/ES-Server-FAQ.md), or your ElasticSearch cluster is too small. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-533438779 Read `ElasticsearchStorageTTL`, if you will find the delete wouldn't happen in every 5 mins. This is only the check running every 5 mins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-533435504 > yes i will investigate our ES, but what i think is the data keeper should be the lowest priority task compare to others, especially those to save metrics/trace or register service/serviceInstance into ES, so can the data keeper modified to delete older datas only in "Idle" time? Delete? The 6.4.0(6.3.0+) delete only happens in middle night, and once per day. You may read the delete logs, but in ElasticSearch implementation, it would execute like that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-533416580 If they are blocked, that is expected from design level, You need to find out why elasticsearch doesn't response. Check your ES server status and monitoring tool. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-531554608 In 6.4.0, we use our own thread pool. Because in some cases, too many grpc thread created. That is expected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-531535556 That is new to me. Could you find why? How to make grpc thread exit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-531534862 What do you mean thread destoried? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-531534825 They you have to look up your server env. Something blocks, I can't be sure why. Most likely, your storage crashed or facing performance issue, then, the grpc can't write data to memory cache, which could trigger this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task
wu-sheng commented on issue #3441: Grpc server thread pool is full, rejecting the task URL: https://github.com/apache/skywalking/issues/3441#issuecomment-529722735 We added thread size control to make sure you wouldn't OOM. But two many connections are not acceptable. You should not have so many connections. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
