[
https://issues.apache.org/jira/browse/IMPALA-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556513#comment-17556513
]
ASF subversion and git services commented on IMPALA-10316:
----------------------------------------------------------
Commit 70568c80b3bb19e1945896d0a9492b8bc8f37164 in impala's branch
refs/heads/master from Laszlo Gaal
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=70568c80b ]
IMPALA-10316: Increase Yarn minimum container size for dataload
This is an attempt to get rod of IMPALA-10669 and friends, crashing Tez
containers during the loading of nested ORC data.
The usual error message logged for these failures is:
Container [pid=11530,containerID=container_1618776748992_0039_01_000003]
is running 2785280B beyond the 'PHYSICAL' memory limit.
Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB
virtual memory used. Killing container.
https://stackoverflow.com/a/43827548/143681 explains that the tunable
setting 'yarn.scheduler.minimum-allocation-mb' in yarn-site.xml sets
both the minimum memory size and the memory size increment for Yarn
containers
This patch is an attempt to work around the failure by forcibly setting
a minimum size for the Yarn containers used in dataload that is
significantly larger than the 1 GB size reported in the failure messages.
Tested by running the dataload phase successfully on the following
platform combinations:
- Ubuntu 16.04, m6i.8xlarge (128 GB RAM, Docker)
- Ubuntu 16.04, m5.12xlarge (192 GB RAM, Docker)
- Centos 7.4, m5.4xlarge (64 GB RAM)
- Centos 7.4, r5.4xlarge (128 GB RAM)
- Ubuntu 16.04, m6i.4xlarge (64 GB RAM)
Change-Id: I77e7c9e9fa3491c6e5652351869d3a4410bbb7b8
Reviewed-on: http://gerrit.cloudera.org:8080/18630
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Reviewed-by: Michael Smith <michael.sm...@cloudera.com>
Reviewed-by: Laszlo Gaal (Cloudera) <laszlo.g...@cloudera.com>
> load_nested.py failed due to out of memory during Jenkins GVO
> -------------------------------------------------------------
>
> Key: IMPALA-10316
> URL: https://issues.apache.org/jira/browse/IMPALA-10316
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Zoltán Borók-Nagy
> Assignee: Michael Smith
> Priority: Critical
> Labels: broken-build, flaky
>
> The following job failed due to out of memory:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12588] (please click
> on "Don't keep this build forever" once this issue is resolved)
> Relevant log lines:
> {noformat}
> 02:33:42 Loading nested orc data (logging to
> /home/ubuntu/Impala/logs/data_loading/load-nested.log)...
> 02:35:39 FAILED (Took: 1 min 57 sec)
> 02:35:39 '/home/ubuntu/Impala/testdata/bin/load_nested.py -t
> tpch_nested_orc_def -f orc/def' failed. Tail of log:
> 02:35:39 2020-11-11 02:35:06,225 INFO:load_nested[348]:Executing:
> 02:35:39
> 02:35:39 CREATE EXTERNAL TABLE supplier
> 02:35:39 STORED AS orc
> 02:35:39 TBLPROPERTIES('orc.compress' =
> 'ZLIB','external.table.purge'='TRUE')
> 02:35:39 AS SELECT * FROM tmp_supplier
> 02:35:39 Traceback (most recent call last):
> 02:35:39 File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 415,
> in <module>
> 02:35:39 load()
> 02:35:39 File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 349,
> in load
> 02:35:39 hive.execute(stmt)
> 02:35:39 File "/home/ubuntu/Impala/tests/comparison/db_connection.py", line
> 206, in execute
> 02:35:39 return self._cursor.execute(sql, *args, **kwargs)
> 02:35:39 File
> "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 331, in execute
> 02:35:39 self._wait_to_finish() # make execute synchronous
> 02:35:39 File
> "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 413, in _wait_to_finish
> 02:35:39 raise OperationalError(resp.errorMessage)
> 02:35:39 impala.error.OperationalError: Error while compiling statement:
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1605060173780_0039_2_00, diagnostics=[Task failed,
> taskId=task_1605060173780_0039_2_00_000000, diagnostics=[TaskAttempt 0
> failed, info=[Container container_1605060173780_0039_01_000002 finished with
> diagnostics set to [Container failed, exitCode=-104. [2020-11-11
> 02:35:11.768]Container
> [pid=16810,containerID=container_1605060173780_0039_01_000002] is running
> 7729152B beyond the 'PHYSICAL' memory limit. Current usage: 1.0 GB of 1 GB
> physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing
> container.{noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
