Hello Yongzhi Chen, Vihang Karajgaonkar, Sudhanshu Arora, Joe McDonnell, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13224
to look at the new patch set (#9).
Change subject: IMPALA-8369 (part 2): Hive 3: switch to Tez-on-YARN execution
......................................................................
IMPALA-8369 (part 2): Hive 3: switch to Tez-on-YARN execution
This switches away from Tez local mode to tez-on-YARN. After spending a
couple of days trying to debug issues with Tez local mode, it seemed
like it was just going to be too much of a lift.
This patch switches on the starting of a Yarn RM and NM when
USE_CDP_HIVE is enabled. It also switches to a new yarn-site.xml with a
minimized set of configurations, generated by the new python templating.
In order for everything to work properly I also had to update the Hadoop
dependency to come from CDP instead of CDH when using CDP Hive.
Otherwise, the classpath of the launched Tez containers had conflicting
versions of various Hadoop classes which caused tasks to fail.
I verified that this fixes concurrent query execution by running queries
in parallel in two beeline sessions. With local mode, these queries
would periodically fail due to various races (HIVE-21682). I'm also able
to get farther along in data loading.
Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/generate_xml_config.py
M bin/impala-config.sh
M bin/jenkins/critique-gerrit-review.py
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/test/resources/hive-site.xml.py
M shaded-deps/pom.xml
M testdata/cluster/admin
A testdata/cluster/node_templates/common/etc/hadoop/conf/capacity-scheduler.xml
A testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py
D testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.tmpl
13 files changed, 389 insertions(+), 173 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/13224/9
--
To view, visit http://gerrit.cloudera.org:8080/13224
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If96064f271582b2790a3cfb3d135f3834d46c41d
Gerrit-Change-Number: 13224
Gerrit-PatchSet: 9
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <sudhan...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
Gerrit-Reviewer: Yongzhi Chen <yc...@cloudera.com>
