Tom, Ok. I will fetch the code for 2.2 and work through the build process. I have filed an issue for this [1]. If someone can post a patch for that issue against 2.2, I will build the modified JARs and verify whether the problem is resolved by that change.
Thanks, Bryan PS: River 2.2 is listed as an "Unreleased Version" in Jira. [1] https://issues.apache.org/jira/browse/RIVER-403 > -----Original Message----- > From: Tom Hobbs [mailto:[email protected]] > Sent: Friday, January 13, 2012 7:35 AM > To: [email protected] > Subject: Re: RE: [Fwd: Re: DGC threads issue] > > The easiest way would be to the source distribution of River. > Unpack it somewhere. Modify the code and then run the ant > task called "build-all" (I think, whatever it's called it > default target for the build.xml). The build file should be > in the root of the unpacked source. > > I don't think Peter created a jira fir it because I've not > seen a notification email. I'm also unsure what code libel > he's referring to. > > Sent via mobile device, please forgive typos and spacing errors. > > On 13 Jan 2012 12:30, "Bryan Thompson" <[email protected]> wrote: > > > Peter, > > > > I would be happy to test out a fix on this. Can you point out the > > modified line and link me to any directions on how to rebuild the > > necessary jar(s)? > > > > Did you already file an issue for this? If not, I will > file an issue > > and then update it when I test out this fix. > > > > Thanks, > > Bryan > > > > > -----Original Message----- > > > From: Peter Firmstone [mailto:[email protected]] > > > Sent: Friday, January 13, 2012 3:21 AM > > > To: [email protected] > > > Cc: Bryan Thompson; [email protected] > > > Subject: Re: [Fwd: Re: DGC threads issue] > > > > > > Ok, found the problem, when I fixed River-142, I introduced a new > > > bug, after creating a new DGC thread, I didn't set the > running flag > > > to true, so each time, a new thread is created. > > > > > > Thankfully it's a very easy fix. > > > > > > Cheers, > > > > > > Peter. > > > > > > Peter Jones wrote: > > > > Peter, > > > > > > > > That internal Executor interface permits neither > > > synchronous execution in the calling thread ("this method itself > > > must not block") nor any small bound on the number of > Runnables that > > > can be executed concurrently ("execution of a given > action must not > > > be delayed indefinitely in order to complete execution of a > > > different action passed to a different invocation of this > method"). > > > In other words, concurrency-wise, it should be equivalent to "new > > > Thread(runnable, name)", or what the result of > > > j.u.c.Executors.newCachedThreadPool() would do. > Otherwise, callers > > > would not function correctly, because they expect to be > able to pass > > > Runnables that execute indefinitely, not just short-lived work > > > tasks. > > > > > > > > The real purpose of this internal Executor/ThreadPool was > > > to provide an internal alternative to "new Thread(...)" > that takes > > > care of certain things in common, like: > > > > > > > > - thread reuse instead of creation when reasonable > (idle timeout) > > > > - but still with threads (re)named for the task being executed > > > > - insulation of any thread creation permission requirement from > > > > context calling execute (NewThreadAction) > > > > - but instead requiring direct user of thread pool to be > > > trusted not > > > > to abuse that insulation, with permission requirement > for access > > > > (ThreadPoolPermission and GetThreadPoolAction) > > > > - common logging of uncaught exceptions > > > > > > > > -- Peter > > > > > > > > > > > > On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote: > > > > > > > > > > > >> ThreadPool implements the com.sun.jini.thread.Executor > interface. > > > >> > > > >> Because the interface states it should not block, I think > > > the calling thread should execute the task when the thread pool > > > becomes saturated, rather than continue to create new > threads as per > > > the current implementation. This will ensure that the task is > > > completed, the calling thread, if it uses a sequence of > tasks with > > > dependencies will have to ensure that it submits the > tasks in order. > > > Since only the caller knows the order, it makes sense for > this to be > > > the callers responsibility. As a result the executor > will honor the > > > non blocking contract. For that reason we'll use a zero length > > > queue, probably a SynchronousQueue. > > > >> > > > >> Regards, > > > >> > > > >> Peter. > > > >> > > > >> /** > > > >> * Executor is an abstraction for a thread factory or > > > thread pool for > > > >> * executing actions asynchronously. > > > >> * > > > >> * @author Sun Microsystems, Inc. > > > >> * > > > >> */ > > > >> public interface Executor { > > > >> > > > >> /** > > > >> * Executes the given Runnable action asynchronously in > > > some thread. > > > >> * > > > >> * The implementation may create a new thread to execute > > > the action, > > > >> * or it may execute the action in an existing thread. > > > >> * > > > >> * The execution of a given action must not be delayed > > > indefinitely > > > >> * in order to complete execution of a different action > > > passed to a > > > >> * different invocation of this method. In other words, the > > > >> * implementation must assume that there may be > > > arbitrary dependencies > > > >> * between actions passed to this method, so it needs to > > > be careful > > > >> * to avoid potential deadlock by delaying execution of > > > one action > > > >> * indefinitely until another completes. > > > >> * > > > >> * Also, this method itself must not block, because it > > > may be invoked > > > >> * by code that is serially processing data to produce > > > multiple such > > > >> * arbitrarily-dependent actions that need to be executed. > > > >> * > > > >> * @param runnable the Runnable action to execute > > > >> * > > > >> * @param name string to include in the name of the > > > thread used > > > >> * to execute the action > > > >> */ > > > >> void execute(Runnable runnable, String name); } > > > >> > > > >> Peter Firmstone wrote: > > > >> > > > >>> Thanks Brian, > > > >>> > > > >>> Looking at our implementation code, DGC uses an Executor > > > called ThreadPool, it's javadoc states: > > > >>> > > > >>> /** > > > >>> * ThreadPool is a simple thread pool implementation of > > > the Executor > > > >>> * interface. > > > >>> * > > > >>> * A new task is always given to an idle thread, if one is > > > available; > > > >>> * otherwise, a new thread is always created. There is no > > > >>> minimum > > > >>> * warm thread count, nor is there a maximum thread count > > > (tasks are > > > >>> * never queued unless there are sufficient idle threads to > > > >>> execute > > > >>> * them). > > > >>> * > > > >>> * New threads are created as daemon threads in the thread > > > group that > > > >>> * was passed to the ThreadPool instance's constructor. Each > > > >>> thread's > > > >>> * name is the prefix NewThreadAction.NAME_PREFIX > followed by the > > > >>> name > > > >>> * of the task it is currently executing, or "Idle" if it is > > > >>> currently > > > >>> * idle. > > > >>> > > > >>> ThreadPool predates Java 5, it looks like we can fix this > > > by using an Executor from Java 5, we can look at limiting > the number > > > of threads created based on available CPU's and a scaling > factor and > > > place the tasks in a BlockingQueue, so if the queue is filled, it > > > blocks. > > > >>> > > > >>> Can you report the issue as a Bug on Jira for me, I'll > > > fix this before the next release. > > > >>> > > > >>> Regards, > > > >>> > > > >>> Peter. > > > >>> > > > >>> Peter Firmstone wrote: > > > >>> > > > >>>> Hi Peter, > > > >>>> > > > >>>> I was wondering if you had any thoughts on this post > > > from Bryan on River users? > > > >>>> > > > >>>> Hope you don't mind me asking ;) > > > >>>> > > > >>>> Best Regards, > > > >>>> > > > >>>> Peter Firmstone. > > > >>>> > > > >>>> > > > > ------------------------------------------------------------------- > > > >>>> ----- > > > >>>> > > > >>>> Subject: > > > >>>> Re: DGC threads issue > > > >>>> From: > > > >>>> Tom Hobbs <[email protected]> > > > >>>> Date: > > > >>>> Thu, 12 Jan 2012 20:45:01 +0000 > > > >>>> To: > > > >>>> [email protected], [email protected] > > > >>>> > > > >>>> To: > > > >>>> [email protected], [email protected] > > > >>>> > > > >>>> > > > >>>> Hi Bryan, > > > >>>> > > > >>>> Sorry that no one got back to you about this. I'm > afraid that > > > >>>> I don't know the answer to your question, I've > copied the dev > > > >>>> list into this email in case someone who monitors that list > > > >>>> (but not this one) has any ideas. > > > >>>> > > > >>>> Best regards, > > > >>>> > > > >>>> Tom > > > >>>> > > > >>>> On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson > > > <[email protected]> wrote: > > > >>>> > > > >>>> > > > >>>>> Just to follow up on this thread myself. I modified > > > the pattern to return a "thick" future rather than a > proxy for the > > > future. This caused the RMI call to wait on the server until the > > > future was done and then sent back the outcome. This "fixed" the > > > DGC memory/thread leak by reducing the number of exported proxies > > > drammatically. > > > >>>>> > > > >>>>> In terms of best practices, is distributed DGC simply > > > not useful for exported objects with short life spans? > Can it only > > > be used with proxies for relatively long lived services? > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Bryan > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>> -----Original Message----- > > > >>>>>> From: Bryan Thompson > > > >>>>>> Sent: Tuesday, January 03, 2012 12:06 PM > > > >>>>>> To: [email protected] > > > >>>>>> Subject: DGC threads issue > > > >>>>>> > > > >>>>>> Hello, > > > >>>>>> > > > >>>>>> Background: > > > >>>>>> > > > >>>>>> I am seeing what would appear to be one DGC thread > > > allocated per > > > >>>>>> exported object. This is using River 2.2 and Sun JDK > > > 1.6.0_17. > > > >>>>>> Relevant configuration parameters are below. > > > >>>>>> > > > >>>>>> I am observing problems with the DGC threads not being > > > retired on > > > >>>>>> a timely basis. The exported objects are proxies > for Futures > > > >>>>>> which are being executed on the service. The code > pattern is > > > >>>>>> such that the proxied Future goes out of lexical > scope quite > > > >>>>>> quickly. E.g., rmiCallReturningProxyForFuture().get(). > > > >>>>>> > > > >>>>>> Under a modest load, a large number of such Futures > > > are exported > > > >>>>>> which results in a large number of long lived DGC > > > threads. This > > > >>>>>> turns into a problem for the JVM due to the stack > > > allocation per > > > >>>>>> thread. Presumably this is not good for other reasons as > > > >>>>>> well (e.g., scheduling). > > > >>>>>> > > > >>>>>> I have tried to override the leaseValue and checkInterval > > > >>>>>> defaults per the configuration options below. I > > > suspect that the > > > >>>>>> lease interval is somehow not being obeyed, which is > > > presumably a > > > >>>>>> problem on my end. However, I can verify that the > > > configuration > > > >>>>>> values are in fact showing up in > > > >>>>>> System.getProperties() for at least some of the > JVMs involved > > > >>>>>> (the one which drives the workload and the one that I am > > > >>>>>> monitoring with the large number of DGC lease threads). > > > >>>>>> > > > >>>>>> Some questions: > > > >>>>>> > > > >>>>>> Is this one-thread-per-exported proxy the expected > > > behavior when > > > >>>>>> DGC is requested for the exported object? > > > >>>>>> > > > >>>>>> The DGC lease checker threads appear to expire ~14 - > > > 15 minutes > > > >>>>>> after I terminate the process which was > originating the RMI > > > >>>>>> requests. This is close the sum of the default > > > leaseValue (10m) > > > >>>>>> and checkInterval (5m) parameters, but maybe there is > > > some other > > > >>>>>> timeout which is controlling this? If this is the sum > > > of those > > > >>>>>> parameters, why would the DGC lease threads live until > > > the sum of > > > >>>>>> those values? I thought that the lease would expire after > > > >>>>>> the leaseValue (10m default). > > > >>>>>> > > > >>>>>> Can the issue I am observing be caused by a low heap > > > pressure on > > > >>>>>> the JVM to which the RMI proxies were exported? If it > > > fails to > > > >>>>>> GC those proxies, even though they are reachable, > could that > > > >>>>>> cause DGC to continue to retain those proxies on the JVM > > > >>>>>> which exported them? > > > >>>>>> > > > >>>>>> Is there any way to configure DGC to use a thread pool > > > or to have > > > >>>>>> the leases managed by a single thread? > > > >>>>>> > > > >>>>>> Is it possible that there is an interaction with the > > > useNIO option? > > > >>>>>> > > > >>>>>> Relevant options that I am using include: > > > >>>>>> > > > >>>>>> -Dcom.sun.jini.jeri.tcp.useNIO=true > > > >>>>>> -Djava.rmi.dgc.leaseValue=30000 > > > >>>>>> -Dsun.rmi.dgc.checkInterval=15000 > > > >>>>>> -Dsun.rmi.transport.tcp.connectionPool=true > > > >>>>>> > > > >>>>>> Thanks in advance, > > > >>>>>> Bryan > > > >>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >>> > > > > > > > > > > > > > > > > > > >
