Brian,
First checkout the main trunk (I've only just committed the fix so make
sure it's updated, if you've checked it out already):
svn checkout http://svn.apache.org/repos/asf/river/jtsk/trunk river
Then create a build.properties file in trunk that contains something like the
following:
java.home=/usr/jdk/jdk1.6.0_25
river.home=/opt/src/river
log.config=/home/peter/logging.properties
debug=true
debuglevel=lines,source,vars
Then from the command line
$ ant all.clean all.build
If you want to run the qa tests:
$ ant qa.run
Cheers,
Peter.
Bryan Thompson wrote:
What about the following property? Is it still valid?
-Dsun.rmi.transport.tcp.connectionPool=true
Thanks,
Bryan
-----Original Message-----
From: Peter Firmstone [mailto:j...@zeus.net.au]
Sent: Friday, January 13, 2012 8:41 AM
To: Bryan Thompson
Cc: dev@river.apache.org; u...@river.apache.org
Subject: Re: [Fwd: Re: DGC threads issue]
Brian,
Do you have a Linux / Unix build environment?
River isn't building yet on Window's, FreeBSD, or OpenBSD,
known build environments that work are Ubuntu and Solaris.
If not, that's ok, well be able to get the necessary jar's off Hudson.
The DGC lease property values you need to set are:
com.sun.jini.jeri.dgc.leaseValue=30000
com.sun.jini.jeri.dgc.checkInterval=15000
RMI (called JRMP) is superseded code now, River doesn't use
that unless you configure your Exporter to be JrmpExporter,
but you're better off sticking with BasicJeriExporter.
Regards,
Peter.
Bryan Thompson wrote:
Peter,
I would be happy to test out a fix on this. Can you point
out the modified line and link me to any directions on how to
rebuild the necessary jar(s)?
Did you already file an issue for this? If not, I will
file an issue and then update it when I test out this fix.
Thanks,
Bryan
-----Original Message-----
From: Peter Firmstone [mailto:j...@zeus.net.au]
Sent: Friday, January 13, 2012 3:21 AM
To: dev@river.apache.org
Cc: Bryan Thompson; u...@river.apache.org
Subject: Re: [Fwd: Re: DGC threads issue]
Ok, found the problem, when I fixed River-142, I introduced a new
bug, after creating a new DGC thread, I didn't set the
running flag
to true, so each time, a new thread is created.
Thankfully it's a very easy fix.
Cheers,
Peter.
Peter Jones wrote:
Peter,
That internal Executor interface permits neither
synchronous execution in the calling thread ("this method
itself must
not block") nor any small bound on the number of Runnables
that can
be executed concurrently ("execution of a given action must not be
delayed indefinitely in order to complete execution of a different
action passed to a different invocation of this method").
In other
words, concurrency-wise, it should be equivalent to "new
Thread(runnable, name)", or what the result of
j.u.c.Executors.newCachedThreadPool() would do.
Otherwise, callers
would not function correctly, because they expect to be
able to pass
Runnables that execute indefinitely, not just short-lived
work tasks.
The real purpose of this internal Executor/ThreadPool was
to provide an internal alternative to "new Thread(...)" that takes
care of certain things in common, like:
- thread reuse instead of creation when reasonable (idle timeout)
- but still with threads (re)named for the task being executed
- insulation of any thread creation permission requirement from
context calling execute (NewThreadAction)
- but instead requiring direct user of thread pool to be
trusted not
to abuse that insulation, with permission requirement for access
(ThreadPoolPermission and GetThreadPoolAction)
- common logging of uncaught exceptions
-- Peter
On Jan 12, 2012, at 7:53 PM, Peter Firmstone wrote:
ThreadPool implements the com.sun.jini.thread.Executor interface.
Because the interface states it should not block, I think
the calling thread should execute the task when the thread pool
becomes saturated, rather than continue to create new
threads as per
the current implementation. This will ensure that the task is
completed, the calling thread, if it uses a sequence of tasks with
dependencies will have to ensure that it submits the tasks
in order.
Since only the caller knows the order, it makes sense for
this to be
the callers responsibility. As a result the executor will
honor the
non blocking contract. For that reason we'll use a zero length
queue, probably a SynchronousQueue.
Regards,
Peter.
/**
* Executor is an abstraction for a thread factory or
thread pool for
* executing actions asynchronously.
*
* @author Sun Microsystems, Inc.
*
*/
public interface Executor {
/**
* Executes the given Runnable action asynchronously in
some thread.
*
* The implementation may create a new thread to execute
the action,
* or it may execute the action in an existing thread.
*
* The execution of a given action must not be delayed
indefinitely
* in order to complete execution of a different action
passed to a
* different invocation of this method. In other words, the
* implementation must assume that there may be
arbitrary dependencies
* between actions passed to this method, so it needs to
be careful
* to avoid potential deadlock by delaying execution of
one action
* indefinitely until another completes.
*
* Also, this method itself must not block, because it
may be invoked
* by code that is serially processing data to produce
multiple such
* arbitrarily-dependent actions that need to be executed.
*
* @param runnable the Runnable action to execute
*
* @param name string to include in the name of the
thread used
* to execute the action
*/
void execute(Runnable runnable, String name); }
Peter Firmstone wrote:
Thanks Brian,
Looking at our implementation code, DGC uses an Executor
called ThreadPool, it's javadoc states:
/**
* ThreadPool is a simple thread pool implementation of
the Executor
* interface.
*
* A new task is always given to an idle thread, if one is
available;
* otherwise, a new thread is always created. There is
no minimum
* warm thread count, nor is there a maximum thread count
(tasks are
* never queued unless there are sufficient idle threads
to execute
* them).
*
* New threads are created as daemon threads in the thread
group that
* was passed to the ThreadPool instance's constructor. Each
thread's
* name is the prefix NewThreadAction.NAME_PREFIX
followed by the
name
* of the task it is currently executing, or "Idle" if it is
currently
* idle.
ThreadPool predates Java 5, it looks like we can fix this
by using an Executor from Java 5, we can look at limiting
the number
of threads created based on available CPU's and a scaling
factor and
place the tasks in a BlockingQueue, so if the queue is filled, it
blocks.
Can you report the issue as a Bug on Jira for me, I'll
fix this before the next release.
Regards,
Peter.
Peter Firmstone wrote:
Hi Peter,
I was wondering if you had any thoughts on this post
from Bryan on River users?
Hope you don't mind me asking ;)
Best Regards,
Peter Firmstone.
-------------------------------------------------------------------
-----
Subject:
Re: DGC threads issue
From:
Tom Hobbs <tvho...@googlemail.com>
Date:
Thu, 12 Jan 2012 20:45:01 +0000
To:
u...@river.apache.org, dev@river.apache.org
To:
u...@river.apache.org, dev@river.apache.org
Hi Bryan,
Sorry that no one got back to you about this. I'm
afraid that I
don't know the answer to your question, I've copied
the dev list
into this email in case someone who monitors that list
(but not
this one) has any ideas.
Best regards,
Tom
On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson
<br...@systap.com> wrote:
Just to follow up on this thread myself. I modified
the pattern to return a "thick" future rather than a proxy for the
future. This caused the RMI call to wait on the server until the
future was done and then sent back the outcome. This
"fixed" the DGC
memory/thread leak by reducing the number of exported proxies
drammatically.
In terms of best practices, is distributed DGC simply
not useful for exported objects with short life spans?
Can it only
be used with proxies for relatively long lived services?
Thanks,
Bryan
-----Original Message-----
From: Bryan Thompson
Sent: Tuesday, January 03, 2012 12:06 PM
To: u...@river.apache.org
Subject: DGC threads issue
Hello,
Background:
I am seeing what would appear to be one DGC thread
allocated per
exported object. This is using River 2.2 and Sun JDK
1.6.0_17.
Relevant configuration parameters are below.
I am observing problems with the DGC threads not being
retired on
a timely basis. The exported objects are proxies
for Futures
which are being executed on the service. The code
pattern is
such that the proxied Future goes out of lexical scope quite
quickly. E.g., rmiCallReturningProxyForFuture().get().
Under a modest load, a large number of such Futures
are exported
which results in a large number of long lived DGC
threads. This
turns into a problem for the JVM due to the stack
allocation per
thread. Presumably this is not good for other
reasons as well
(e.g., scheduling).
I have tried to override the leaseValue and checkInterval
defaults per the configuration options below. I
suspect that the
lease interval is somehow not being obeyed, which is
presumably a
problem on my end. However, I can verify that the
configuration
values are in fact showing up in
System.getProperties() for at least some of the JVMs
involved
(the one which drives the workload and the one that I am
monitoring with the large number of DGC lease threads).
Some questions:
Is this one-thread-per-exported proxy the expected
behavior when
DGC is requested for the exported object?
The DGC lease checker threads appear to expire ~14 -
15 minutes
after I terminate the process which was originating the RMI
requests. This is close the sum of the default
leaseValue (10m)
and checkInterval (5m) parameters, but maybe there is
some other
timeout which is controlling this? If this is the sum
of those
parameters, why would the DGC lease threads live until
the sum of
those values? I thought that the lease would expire
after the
leaseValue (10m default).
Can the issue I am observing be caused by a low heap
pressure on
the JVM to which the RMI proxies were exported? If it
fails to
GC those proxies, even though they are reachable, could that
cause DGC to continue to retain those proxies on the
JVM which
exported them?
Is there any way to configure DGC to use a thread pool
or to have
the leases managed by a single thread?
Is it possible that there is an interaction with the
useNIO option?
Relevant options that I am using include:
-Dcom.sun.jini.jeri.tcp.useNIO=true
-Djava.rmi.dgc.leaseValue=30000
-Dsun.rmi.dgc.checkInterval=15000
-Dsun.rmi.transport.tcp.connectionPool=true
Thanks in advance,
Bryan
