Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5184: build fe against both Hive 1 & 2 APIs ......................................................................
IMPALA-5184: build fe against both Hive 1 & 2 APIs This adds a compatibility shim layer with Hive 1 and Hive 2 implementations. The version-specific code lives in fe/src/compat-hive-$IMPALA_HIVE_MAJOR_VERSION and common/thrift/hive-$IMPALA_HIVE_MAJOR_VERSION-api/ The shim adds wrapper methods to handle differing method signatures and and config variables that changed slightly. Some thrift classes were also moved from the 'cli' to 'rpc' package. We work around these by implementing subclasses with the same name in a different package for compatibility or by implementing shim methods that operate on the classes. We also need to change the package in the TCLIService.thrift, which is done with a search-and-replace. Also avoid the sticky config variable problem with some of the source paths by requiring an _OVERRIDE suffix on the variable to override it from the environment. Testing: Made sure that I could build Impala on master as normal, and also with the following config overrides in bin/impala-config-local.sh: export IMPALA_HADOOP_VERSION=3.0.0-alpha1-cdh6.x-SNAPSHOT export IMPALA_HBASE_VERSION=2.0.0-cdh6.x-SNAPSHOT export IMPALA_HIVE_VERSION=2.1.0-cdh6.x-SNAPSHOT export IMPALA_SENTRY_VERSION=1.5.1-cdh6.x-SNAPSHOT export IMPALA_PARQUET_VERSION=1.5.0-cdh6.x-SNAPSHOT I manually assembled the dependencies by copying the following files from the Hive 2 source and from a Hadoop 3 build: $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift $CDH_COMPONENTS_HOME/hadoop-3.0.0-alpha1-cdh6.x-SNAPSHOT/lib/native/* $CDH_COMPONENTS_HOME/hadoop-3.0.0-alpha1-cdh6.x-SNAPSHOT/include/hdfs.h Change-Id: Ifbc265281c04fe3136bc3c920dbac966742ce09a Reviewed-on: http://gerrit.cloudera.org:8080/5538 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt R common/thrift/hive-1-api/TCLIService.thrift M fe/pom.xml A fe/src/compat-hive-1/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java A fe/src/compat-hive-1/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java A fe/src/compat-hive-1/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java A fe/src/compat-hive-1/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java A fe/src/compat-hive-1/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java A fe/src/compat-hive-1/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java A fe/src/compat-hive-1/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java 27 files changed, 492 insertions(+), 85 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/5538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ifbc265281c04fe3136bc3c920dbac966742ce09a Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
