On 8/5/2012 1:21 PM, Adrian Crum wrote:
On 8/5/2012 11:02 AM, Adrian Crum wrote:
I just committed a bunch of changes to the Job Manager group of
classes. The changes help simplify the code and hopefully make the
Job Manager more robust. On the other hand, I might have broken
something. ;) I will monitor the mailing list for problems.
I believe the JobPoller settings in serviceengine.xml (the
<thread-pool> element) should be changed. I think min-threads should
be set to "2" and max-threads should be set to "5". Creating lots of
threads can hurt throughput because the JVM spends more time managing
them. I would be interested in hearing what others think.
Thinking about this more, there are some other things that need to be
fixed:
1. The JobPoller uses an unbounded queue. In a busy server, there is
the potential the queue will grow in size until it causes an
out-of-memory condition.
2. There is no accommodation for when a job cannot be added to the
queue - it is just lost. We could add a dequeue method to the Job
interface that will allow implementations to recover or reschedule the
job when it can't be added to the queue.
3. There is a JobPoller instance per delegator, and each instance
contains the number of threads configured in serviceengine.xml. With
the current max-threads setting of 15, a multi-tenant installation
with 100 tenants will create up to 1500 threads. (!!!) A smarter
strategy might be to have a single JobPoller instance that services
multiple JobManagers.
I fixed #1 and #2. I am considering working on #3, but I want some
feedback first.
A JobPoller instance is created for each delegator. So, in a
multi-tenant or multi-delegator scenario, multiple JobPollers will be
created - which means one job queue per delegator and (threads per
queue) threads per delegator. In a multi-server installation, things
are multiplied: (# of servers * # of delegators) job queues.
Fortunately, in that scenario we can disable the JobPoller on all but
one server.
So, we are left with the potential problem of too many queues/threads
being created on a multi-delegator or multi-tenant server. So, I think
there should be one JobPoller instance that services all delegators. At
each polling interval, the JobPoller gets a list of jobs from each
delegator (JobManager) - creating a list of lists. Then the JobPoller
creates a queue candidate list from the list of lists - using a
round-robin approach so each delegator gets an equal opportunity to
queue jobs. The JobPoller queues the candidate list, and any candidates
that don't fit in the queue are rescheduled. With this approach the
JobPoller can service any number of delegators without saturating the
server.
What do you think?
-Adrian
