| Gehel added a comment. |
In T192759#4176404, @Smalyshev wrote:Lots of threads in executor service seems ok, that's how the queries are served IIRC and this can require a number of them. Not sure about management - but it may be cheaper to leave some to hang around than to collect them all.
All depends on the definition of "lots" :) A spike of 480 thread creation, probably created to serve a single task is probably a bit too much. As far as I can see, the underlying thread pool is unbounded and caches threads for a minute. Looking at the number of threads graphs, it looks like there is a spike in the number of threads, no queuing is done, 100's of threads are created, and destroyed after a minute of not being used. Not a very efficient use of resources.
A lot of HTTP client threads are really suspect, I am not sure we have that many service calls, it's not the most frequently used feature afaik. Maybe we have bigger pools than necessary. I am also not sure how these pools work - are all queries get to use the same pool or there will be several pools? I imagine the right thing to do is to have common pool, and then 20 threads may be too much but not excessively so. But in that case, getting rid of the pool may be a mistake?
I'm adding some logging tot he MWAPIServiceCall, which might help us understand what's going on there.
The QueuedThreadPool backing the HttpClient seems to use the default min 8, max 200 threads, with an idle timeout of 1 minute. What is somewhat surprising is to find 20 of them in epollWait(). I'm not familiar enough with NIO, but this looks like a bunch of threads just waiting for IO, which might indicate a contention at that level.
No idea what QueuedThreadPool is either.
Looking at the threads that are actually doing something in that thread pool, it looks like the qtp... pool this is the Jetty HTTP request pool. That seems reasonable.
ChunkedRunningQuery is query processing thing, not sure either why they are locked but maybe just locking between updater and querying or something like that?
This definitely requires some more digging into. It looks like we have some lock contention, but I'm not sure how bad it is, which processes are competing and if we can do anything about it.
Cc: Smalyshev, Gehel, Aklapper, Lucas_Werkmeister_WMDE, Jonas, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
