Hi all,
while reading the log from the join example (2 source vertexes and a sink vertex) I noticed the following:

2014-09-26 21:20:27,021 INFO [TaskSchedulerEventHandlerThread] org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Allocation *request* for task: attempt_1411734050933_0003_1_00_000000_0 with request: Capability[<memory:1024, vCores:1>]*Priority[2]* host: node02 rack: null
...
2014-09-26 21:20:28,237 INFO [DelayedContainerManager] org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Assigning container to task, container=*Container*: [ContainerId: container_1411734050933_0003_01_000002, NodeId: node02:34192, NodeHttpAddress: node02:8042, Resource: <memory:1024, vCores:1>, *Priority: 2*, Token: Token { kind: ContainerToken, service: 192.168.56.102:34192 }, ], task=attempt_1411734050933_0003_1_00_000000_0, containerHost=node02, localityMatchType=NodeLocal, matchedLocation=node02, honorLocalityFlags=true, reusedContainer=false, delayedContainers=1, containerResourceMemory=1024, containerResourceVCores=1

And something similar for the other "parent" vertex, nothing strange here. But this is about the "joiner" vertex:

2014-09-26 21:21:24,292 INFO [TaskSchedulerEventHandlerThread] org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Allocation *request* for task: attempt_1411734050933_0003_1_02_000000_0 with request: Capability[<memory:1024, vCores:1>]*Priority[4]* host: null rack: null
...
2014-09-26 21:21:24,318 INFO [DelayedContainerManager] org.apache.tez.dag.app.rm.YarnTaskSchedulerService: Assigning container to task, container=*Container*: [ContainerId: container_1411734050933_0003_01_000003, NodeId: node02:34192, NodeHttpAddress: node02:8042, Resource: <memory:1024, vCores:1>, *Priority: 2*, Token: Token { kind: ContainerToken, service: 192.168.56.102:34192 }, ], task=attempt_1411734050933_0003_1_02_000000_0, containerHost=node02, localityMatchType=NodeLocal, matchedLocation=node02, honorLocalityFlags=true, reusedContainer=true, delayedContainers=2, containerResourceMemory=1024, containerResourceVCores=1

Here the priority of the obtained container is still 2, but I was expecting to find the same priority of the request (4). So what is the priority of the obtained container, since it seems to be 2 regardless of the request? Is it used by Tez? How?

Another question I would like to ask is: I see the priority is calculated as /(vertexDistanceFromRoot + 1) * 2/, where vertexDistanceFromRoot is (I think) the distance from the vertex which got its input from a file, or at least not from another vertex. But I haven't been able to understand how this value is set, especially in case two (or more) branches converging in a common vertex X have not the same "depth"... in other words: what happens if X has two parents, one with priority 4 and one 6? Which will be its priority?

Thanks in advance

Fabio

Reply via email to