Hi Chris,
the “recurring theme” does ring a bell - I guess you already know
https://www.mail-archive.com/user@commons.apache.org/msg10368.html
<https://www.mail-archive.com/user@commons.apache.org/msg10368.html> :-)
Do you have some Git repo to re-produce the problem?
Thanks in advance,
Siegfried Goeschl
> On 21 Jun 2017, at 17:48, Chris Gamache <cgama...@gmail.com> wrote:
>
> Ok so, it's hard to tell if this made much of a difference. There might be a
> few less messages? Hard to be sure.
>
> I'm still getting those messages, but I'm now suspecting Selenium is
> generating the majority of these messages. It is behaving badly when it comes
> to killing its spawned phantomjs processes, thereby leaving the executors
> open. Seems like a recurring theme, eh Siegfried? ;)
>
> Selenium is co-opting the Apache Commons Exec. It looks like it is using
> ExecuteWatchdog.INFINITE_TIMEOUT, so no help there. I see they are making a
> pretty good effort to kill the processes. I won't rule-out that I'm doing it
> wrong, but I'm pretty sure
>
> } finally {
> driver.quit();
> }
>
> is all I'm responsible for if I want Selenium to close phantomjs processes. I
> do have some room to upgrade Selenium and PhantomJS which might help. I
> didn't see any meaningful changes in Selenium that would lead me to believe
> that an upgrade would fix the issue. But, it's a good practice and any
> patches will have to be made at the head of the project anyway.
>
> Thanks for all the help!
>
> CG
>
>
> On Mon, Jun 19, 2017 at 4:11 PM, Siegfried Goeschl
> <siegfried.goes...@it20one.com <mailto:siegfried.goes...@it20one.com>> wrote:
> Staying tuned :-)
>
> > On 5 Jun 2017, at 13:59, Chris Gamache <cgama...@gmail.com
> > <mailto:cgama...@gmail.com>> wrote:
> >
> > Hi Siegfried,
> >
> > I have implemented the executor watchdogs and shutdown hooks. I will know
> > if our efforts have been fruitful when we make our first restart which I'm
> > sure will be soon. I promise I'll write back and let you know how it goes.
> >
> > Thank you vary much for checking in with me!
> >
> >> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl
> >> <siegfried.goes...@it20one.com <mailto:siegfried.goes...@it20one.com>>
> >> wrote:
> >>
> >> Hi Chris,
> >>
> >> any new findings from your side?
> >>
> >> Thanks in advance,
> >>
> >> Siegfried Goeschl
> >>
> >>> On 28 May 2017, at 21:41, Siegfried Goeschl
> >>> <siegfried.goes...@it20one.com <mailto:siegfried.goes...@it20one.com>>
> >>> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> there are couple of things to consider
> >>>
> >>> * You are using a PumpStreamHander but the STDERR is not consumed. Each
> >>> process has an internal buffer (size depends on the OS) and when the
> >>> buffer is full any write to STDERR is blocked
> >>> * That could happen if the process being executed actually writes some
> >>> error messages :-)
> >>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
> >>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
> >>>
> >>> Thanks in advance,
> >>>
> >>> Siegfried Goeschl
> >>>
> >>>
> >>>> On 27 May 2017, at 14:27, Chris Gamache <cgama...@gmail.com
> >>>> <mailto:cgama...@gmail.com>> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
> >>>>
> >>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
> >>>> threads and all of the memory in the form of Executors that I'm running
> >>>> but
> >>>> don't seem to be closing down ... When the server finally grinds to a
> >>>> halt
> >>>> I have to restart. When I do it looks like this at shutdown time:
> >>>>
> >>>> <snip>
> >>>>
> >>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> >>>> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> >>>> [Exec
> >>>> Default Executor] but has failed to stop it. This is very likely to
> >>>> create
> >>>> a memory leak. Stack trace of thread:
> >>>> java.lang.Object.wait(Native Method)
> >>>> java.lang.Object.wait(Object.java:502)
> >>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> >>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> >>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> >>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> >>>> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> >>>> [Exec
> >>>> Stream Pumper] but has failed to stop it. This is very likely to create a
> >>>> memory leak. Stack trace of thread:
> >>>> java.io.FileInputStream.readBytes(Native Method)
> >>>> java.io.FileInputStream.read(FileInputStream.java:255)
> >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
> >>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> </snip>
> >>>>
> >>>> And my thread dump is a mile long.
> >>>>
> >>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> >>>> relevant code. It is called from a regular method in a regular class,
> >>>> nothing fancy:
> >>>>
> >>>> CommandLine cmdLine = CommandLine.parse(command.toString());
> >>>> DefaultExecutor executor = new DefaultExecutor();
> >>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> >>>> executor.setStreamHandler(esh);
> >>>> executor.execute(cmdLine);
> >>>>
> >>>> `is` and `os` are passed in on the constructor. Their opens and closes
> >>>> are
> >>>> managed well and cleaned up on the outside of this class...
> >>>> Are there further steps I'm missing to ensure the threads I'm creating
> >>>> are
> >>>> getting shut down properly and the resources they are using are being
> >>>> returned?
> >>>>
> >>>> Any help is much appreciated.
> >>>
> >>
>
>
