Hi Pradeep,
Yes, if we partition an operator it will be placed in different
containers. Except for following 2 conditions:
1. If the partitioned operator is connected to an upstream operator by
CONTAINER_LOCAL stream
2. If the application requires more than MaxContainer(which is Max Int by
default, but can be placed by a property). Then all Physical Operators
after MaxContainer may get allocated in already assigned containers by
round robin. In this case, it may happen that partitions land on same
container due to max container resource constraint.
The code for this is here:
https://github.com/apache/incubator-apex-core/blob/master/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java#L350-L372
Thanks,
Isha
On Mon, Feb 8, 2016 at 12:27 AM, Pradeep A. Dalvi <[email protected]>
wrote:
> From my understanding, if we partition operator, they are being placed in
> separate containers. Such containers might get spawned on the same or
> different node.
> Is it always the case that partitions get placed in separate containers? Is
> this documented?
> Also it would be great, if someone could point me to code snippet to
> confirm the behavior?
>
> Thanks in advance,
> Pradeep A. Dalvi
>
