| Smalyshev added a comment. |
Doing the full replay of the recorded traffic, I get 9658 executorService threads. Again, load and thread set keeps ramping up long after traffic source is removed. I suspect the tasks accumulate in jetty's processing queue and then the executor tries to get them processed even though they should long have timed out and the client has long disconnected.
Timeout detection also seems to be unreliable under load, as I get queries with 500+s runtimes, even though the should have been long cancelled.
The running query queue itself seems to be limited by 32 parallel queries, but this does not constrain the growth of executorService thread set. Eventually it gets to the max of 10k and all threads fail with java.lang.OutOfMemoryError: unable to create new native thread.
Cc: Aklapper, Gehel, Smalyshev, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
