Rajesh Balamohan created TEZ-1793:
-------------------------------------
Summary: Setting different container sizes for jobs in the same
tez session can cause cluster underutilization
Key: TEZ-1793
URL: https://issues.apache.org/jira/browse/TEZ-1793
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
set tez.am.container.idle.release-timeout-min.millis to 200000
set tez.am.container.idle.release-timeout-max.millis to 500000
set hive.tez.container.size to 4096 (similar to tez.task.resource.memory.mb)
- Ran a hive query (ie with container size 4096) in tez mode. In the small
cluster (with 720 GB total memory), this launched 180 containers.
- Reset container size to 8192 and rerun the query. Now, it launched 90
containers.
- Reset container size to 4096 and rerun the query. Now, it reuses the same 90
containers.
This could lead to cluster underutilization where a large job's settings can
cause adverse impact (e.g concurrency) on the subsequent small jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
