Thanks Rajesh, it was really that the problem! Actually... for a moment
I thought about those parameters, but I assumed they would have been
ignored during a session.
In my opinion, they should not be considered by the system while running
in session mode, and tez.am.container.session.delay-allocation-millis
should be the exact delay before releasing a container (at least when
tez.am.container.session.delay-allocation-millis >
tez.am.container.idle.release-timeout-min.millis)... Sure, this leads to
the risk of accumulating containers up to the upper limit of the
application/queue, if any. Or maybe devs could consider a warning if
this condition is met, to alert the user that that parameter is going to
be useless since containers will be released long before. How do you think?
Thanks for the help
Fabio
On 12/09/2014 11:11 AM, Rajesh Balamohan wrote:
>>>
2014-12-09 09:39:40,314 INFO
[ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerEventHandler]
rm.YarnTaskSchedulerService: TaskScheduler initialized with
configuration: maxRMHeartbeatInterval: 1000, containerReuseEnabled:
true, reuseRackLocal: true, reuseNonLocal: false,
localitySchedulingDelay: 250, idleContainerMinTimeout=5000,
idleContainerMaxTimeout=10000, sessionMinHeldContainers=0
>>>
Can you try the following settings instead?
tez.am.container.idle.release-timeout-min.millis=400000
tez.am.container.idle.release-timeout-max.millis=600000
60000 is setting to 10 minutes.
~Rajesh.B
On Tue, Dec 9, 2014 at 3:21 PM, Fabio <[email protected]
<mailto:[email protected]>> wrote:
Hi everyone,
I'm currently running Hive on Tez, especially I am testing the
session mode. I can actually submit different queries to the same
Tez AM, and that's ok. But for some reason containers are released
a very short time after the end of the assigned task, whenever no
new task is pending. In such a way there is no chance for
container reuse among different queries. I already tried to set
tez.am.container.session.delay-allocation-millis=-1 (and before
this, to 600000), but this behavior persists.
In the logs I see this two suspicious lines:
2014-12-09 09:44:23,035 INFO [DelayedContainerManager]
rm.YarnTaskSchedulerService: Releasing unused container:
container_1418090991482_0008_01_000002
and a few milliseconds after the container is stopped:
2014-12-09 09:44:23,274 INFO [TezChild] task.ContainerReporter:
Got TaskUpdate: 7439 ms after starting to poll. TaskInfo:
shouldDie: true
2014-12-09 09:44:23,276 INFO [main] task.TezChild: ContainerTask
returned shouldDie=true, Exiting
It seems to me that the container is really released as soon as it
is no more required (regardless of what could happen in the
future). Is it so? How can I solve this?
I attach the aggregated log and the swimlanes graph that highlight
this behavior.
Thanks guys
Fabio
--
~Rajesh.B
