Hi Walter,
looking at the code and doing a bit of searches around, it seems that
indeed it is prone to
run out of threads if the GC cannot reclaim the timer threads fast enough
(those are VM threads,
not response threads, so one should run out at 32k threads, not a few
hundreds...)
That said, I don't remember seeing this issue in the wild (or at least,
it's uncommon enough that I can't remember
about it), so I'm wondering if there is anything peculiar in your setup or
GC configuration
making it more likely to accumulate unreclaimed garbage?
About alternatives... hum... wondering if it could be replaced by usage of
a single ScheduledExecutorService
instead. Pull requests welcomed :-)
Hope this helps
Cheers
Andrea
On Fri, Sep 8, 2017 at 12:31 PM, Walter Stovall <walter.stov...@byers.com>
wrote:
> I’m load testing a 2.8 geoserver. Geoserver runs in tomcat on the backend
> of an apache web server processing requests thru AJP. I have a jmeter load
> test setup to send WMS GetMap requests (to get a JPEG image) for 40 users
> continually. The test was running for about 10 seconds and processing
> hundreds of requests and then failed with an Out of Memory exception. I
> doubled the heapsize from 1024MB to 2048MB and it ran for longer and then
> still reported Out of Memory.
>
>
>
> But the problem would appear to be threads and not memory…my exception
> reads:
>
> <?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE
> ServiceExceptionReport SYSTEM "https://svcdev1.byers.com:
> 443/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd">
> <ServiceExceptionReport version="1.1.1" > <ServiceException>
>
> java.lang.OutOfMemoryError: unable to create new native thread
>
> unable to create new native thread
>
> </ServiceException></ServiceExceptionReport>
>
>
>
> The code is apparently running out of threads, not memory. This is
> related to the timer threads that WMS creates to detect the rendering
> taking too long.
>
>
>
> My tomcat thread pool is sized to 50 threads and I’ve confirmed that only
> that many are created. But by doing a threaddump while the system is
> running under this jmeter load I see many more threads getting created
> which are simply the timer threads each request is creating. For example I
> might see 50 threads that are blocked on the Oracle database while there
> are 157 of the timer threads.
>
>
>
> It would appear that the timers get canceled but the timer thread itself
> does not find an opportunity to exit/die before more timer threads are
> started. Creating a Timer for every GetMap request would seem to be an
> inefficient way to attack the problem which invites this issue – there is
> apparently no Thread.join in scope to ensure these threads die.
>
>
>
> How can I get a more graceful failure under load? In general my policy
> with load is to size the tomcat thread pool so as to not have too many
> threads for reasonable throughput. Then if more requests are received
> apache will say the server is overloaded/retry. I don’t want various other
> exceptions that are supposed to indicate too much load but require
> judgement to that effect by non-technical users.
>
>
>
> Walter Stovall – Byers Engineering Company
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Geoserver-users mailing list
>
> Please make sure you read the following two resources before posting to
> this list:
> - Earning your support instead of buying it, but Ian Turton:
> http://www.ianturton.com/talks/foss4g.html#/
> - The GeoServer user list posting guidelines: http://geoserver.org/comm/
> userlist-guidelines.html
>
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
--
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this
list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
