Yes, each task manager offers a certain number of "task slots" to the JobManager. That is what you see in the web frontend also. Each TaskManager can offer a different number of slots, there is no restriction there.
Each slot takes an operator (or a pipeline of operators). So, if a task manager offers 4 slots, it can execute 4 parallel mappers, reducers, ... of the same type concurrently. A task manager that offers 8 slots can execute 8 concurrently, ... On Fri, Nov 28, 2014 at 5:40 PM, Stefano Bortoli <[email protected]> wrote: > sweet! so it does not have to be consistent across all nodes! then each > taskmanager will report to the job manager, that will understand the > distribution of the work accordingly. I guess. > > thanks a lot! > > 2014-11-28 17:12 GMT+01:00 Stephan Ewen <[email protected]>: > >> Sure that's possible. You need different configurations for each node, so >> instead of having one config directory shared across all nodes, give each >> node an individual config. >> >> The nodes that should do more work need to see a config with a larger JVM >> heap and more task slots, the nodes that should have fewer load need to >> have a config with smaller heap and fewer slots. >> >> On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <[email protected]> >> wrote: >> >>> Hi all, >>> >>> is it possible to configure a cluster with different number of task >>> manager per node? >>> >>> for example, I have a cluster where I am using a global index, and the >>> nodes where the global index in running should not be overloaded. However, >>> the other nodes could run more than 4 task manager per node. >>> >>> To have such features would be very helpful to fine tune processes on >>> small clusters. >>> >>> thanks a lot for your support! >>> >>> saluti, >>> Stefano >>> >> >> >
