On Sat, Nov 29, 2014 at 4:46 PM, Simon Hafner reactorm...@gmail.com wrote:
I've thrown some numbers at it (doubling each) and it's running at
comfortable 125 procs. However, at about 6.1k of 6.5k items, the procs
drop down to 30.
125 processes at 8 threads each = 1000 active pipelines. How
2014-11-30 7:25 GMT-06:00 Eddie Epstein eaepst...@gmail.com:
On Sat, Nov 29, 2014 at 4:46 PM, Simon Hafner reactorm...@gmail.com wrote:
I've thrown some numbers at it (doubling each) and it's running at
comfortable 125 procs. However, at about 6.1k of 6.5k items, the procs
drop down to 30.
Now you are hitting a limit configured in ducc.properties:
# Max number of work-item CASes for each job
ducc.threads.limit = 500
62 job process * 8 threads per process = 496 max concurrent work items.
This was put in to limit the memory required by the job driver. This value
can probably be
It is also possible that RM prediction has decided that additional
processes are not needed. It
appears that there were likely 64 work items dispatched, plus the 6
completed, leaving only
30 that were idle. If these work items appeared to be completing
quickly, the RM would decide
that
