*Adrian, I think the single JobPoller is a good idea and reduces the chance of too many JobPoller’s running on a machine.
We often setup multiple delegators to communicate with different databases. For example, our data warehouse is hosted on a separate server. These databases usually have a full ofbiz schema on them (jobsandbox table, etc). Here is how our data warehouse process works: The application has several ofbiz servers talking to a primary database. These servers contain all the user information for our application. When a person logs into the application they are redirected to a secondary ofbiz server that is used for running the application under heavy loads. The data is captured on the secondary server. A data warehouse process is scheduled to run every 5 mins on these secondary servers. The secondary servers have a delegator that talks to its local database and a delegator to talk to the data warehouse. With the new job poller changes would the poller pick up jobs from the data warehouse database since it has a delegator that points to that instance? For this example, we would need to make sure the job poller on the secondary server only serviced jobs from its local database (default delegator) and not the our configured olap delegator. Let me know if this will be possible with the new job poller or if you have any questions from my scenario. I realize this scenario is not the typical ofbiz e-commerce type of setup everyone is use to, but we use ofbiz to create lots of different types applications and have found it very flexible for creating just about any type of ERP application. Thanks for your work on the job poller. Brett * On Sun, Aug 12, 2012 at 5:36 AM, Adrian Crum < adrian.c...@sandglass-software.com> wrote: > On 8/5/2012 1:21 PM, Adrian Crum wrote: > >> On 8/5/2012 11:02 AM, Adrian Crum wrote: >> >>> I just committed a bunch of changes to the Job Manager group of classes. >>> The changes help simplify the code and hopefully make the Job Manager more >>> robust. On the other hand, I might have broken something. ;) I will monitor >>> the mailing list for problems. >>> >>> I believe the JobPoller settings in serviceengine.xml (the <thread-pool> >>> element) should be changed. I think min-threads should be set to "2" and >>> max-threads should be set to "5". Creating lots of threads can hurt >>> throughput because the JVM spends more time managing them. I would be >>> interested in hearing what others think. >>> >> >> Thinking about this more, there are some other things that need to be >> fixed: >> >> 1. The JobPoller uses an unbounded queue. In a busy server, there is the >> potential the queue will grow in size until it causes an out-of-memory >> condition. >> 2. There is no accommodation for when a job cannot be added to the queue >> - it is just lost. We could add a dequeue method to the Job interface that >> will allow implementations to recover or reschedule the job when it can't >> be added to the queue. >> 3. There is a JobPoller instance per delegator, and each instance >> contains the number of threads configured in serviceengine.xml. With the >> current max-threads setting of 15, a multi-tenant installation with 100 >> tenants will create up to 1500 threads. (!!!) A smarter strategy might be >> to have a single JobPoller instance that services multiple JobManagers. >> > > I fixed #1 and #2. I am considering working on #3, but I want some > feedback first. > > A JobPoller instance is created for each delegator. So, in a multi-tenant > or multi-delegator scenario, multiple JobPollers will be created - which > means one job queue per delegator and (threads per queue) threads per > delegator. In a multi-server installation, things are multiplied: (# of > servers * # of delegators) job queues. Fortunately, in that scenario we can > disable the JobPoller on all but one server. > > So, we are left with the potential problem of too many queues/threads > being created on a multi-delegator or multi-tenant server. So, I think > there should be one JobPoller instance that services all delegators. At > each polling interval, the JobPoller gets a list of jobs from each > delegator (JobManager) - creating a list of lists. Then the JobPoller > creates a queue candidate list from the list of lists - using a round-robin > approach so each delegator gets an equal opportunity to queue jobs. The > JobPoller queues the candidate list, and any candidates that don't fit in > the queue are rescheduled. With this approach the JobPoller can service any > number of delegators without saturating the server. > > What do you think? > > -Adrian > >
