Thanks. It gets even simpler, and therefore better.
With the statistics collection under a lock, I don't need to queue up
observations. I can just keep a running total and information about the
most recent observation. Each thread that adds an observation can do the
arithmetic to fold it into the running total.
Given that, I don't need any throttling beyond the basic not waiting for
the lock. If a thread wins the lock it might as well record its
observation. It could not do anything else, such as queue up a new
object or even make a decision about whether to record, much faster.
That also makes the thread manager's work simpler and faster. It just
needs to get the lock, add in a final observation, divide the
accumulated tastCount*time product sum by the time over which it was
collected, use that to adjust the number of threads, reset the
collection fields to start a new round, release the lock, and go back to
sleep.
Patricia
On 8/11/2011 3:58 AM, Peter Firmstone wrote:
I like this approach, it sounds elegant.
Cheers,
Peter.
Patricia Shanahan wrote:
I've thought of a significant simplification, based on the idea that I
only need samples of the task count.
I was going to try to use a concurrent queue and deal with observations
getting out of order. I just realized I can make it much cleaner and
simpler by using a ReentrantLock. The manager thread would use
lockInterruptibly, and wait indefinitely to get the lock if another
thread holds it.
A thread that has just made a change to the task count would use
tryLock(). If it gets the lock, it can add its report to a simple
LinkedList. If it fails to get the lock it would just carry on without
collecting a sample. Either the management thread is processing
collected data, or another thread is logging the task count at about the
same time. Either way it does not need to collect a sample.
This strategy automatically throttles the number of observations that
get collected for the manager to consider, though I may also want a way
to further limit the number of messages. However, that can be done more
simply, and with better information, inside code controlled by the lock.
Patricia
On 8/9/2011 12:36 PM, Patricia Shanahan wrote:
I'm investigating extended ThreadPoolExecutor to get an Executor that
bases the number of threads on the mean number of tasks over an
interval. If successful, that would flatten out sudden peaks in the task
load, using the queue as a buffer, but would make the number of threads
match a sustained load, so that there would be no unnecessary
bottleneck.
The current TaskManager allows specification of e.g. one thread per
three tasks, which does result in a lower rate of change but can leave
the number of threads a fraction of the number of tasks in a sustained
workload.
Here is a general outline of a design:
Methods that change the number of tasks would log the new number of
tasks and a timestamp on a concurrent queue. As a refinement, there may
be an option to use a ThreadLocal Random to log only with a specified
probability to bound the number of messages on the queue.
A thread (more about which thread later) would periodically drain the
queue into a local TreeSet, getting them sorted, and then calculate the
mean number of tasks and adjust the thread count accordingly.
The big design problem is which thread does this work. Here are the
options I'm considering:
1. Add it to the tasks, so that a ThreadPoolExecutor worker thread does
it. The downside is that all the existing worker threads could get busy
on long running tasks and tasks are building up in the queue. Nothing
would be done about it until a worker thread finishes its current task.
2. Do it from the execute method, in whatever thread is adding the task.
Again, we could get into situations in which nothing is done about a
situation that could be improved by adding threads. The effect would be
to make the execute method randomly slow from the point of view of its
caller.
3. Use a dedicated thread that sleeps until it is time to reconsider the
number of tasks, processes the queue of log messages, does any
adjustment, and then goes back to sleep. This is the most reliable in
terms of getting adjustments made when they are needed, but costs the
overhead of an extra thread that is just managing the thread pool.
Any views, alternatives?
Thanks,
Patricia
