Vihang Karajgaonkar has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/13005 )
Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 ...................................................................... IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 This change adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.0. It moves the existing Metastoreshim class to a compat-hive-2 directory and adds a new Metastoreshim class under compat-hive-3 directory. These shim classes implement method which are different in hive-2 v/s hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_MAJOR_VERSION one of the two shims is added to as source using the fe/pom.xml build plugin. Additionally, in order to reduce the dependencies footprint of Hive in the front end code, this patch also introduces a new module called shaded-deps. This module using shade plugin to include only the source files from hive-exec which are need by the fe code. For hive-2 build path, no changes are done with respect to hive dependencies to minimize the risk of destabilizing the master branch on the default build option of using Hive-2. The different set of dependencies are activated using maven profiles. The activation of each profile is automatic based on the IMPALA_HIVE_MAJOR_VERSION. Testing: 1. Code compiles and runs against both HMS-3 and HMS-2 2. Ran full-suite of tests using the private jenkins job against HMS-2 3. Running full-tests against HMS-3 will need more work like supporting Tez in the mini-cluster (for dataloading) and HMS transaction support since HMS3 create transactional tables by default. THis will be on-going effort and test failures on Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21596. This hack will be removed when the patches are available in official CDP Hive builds. 2. Some of the existing tests rely on the fact the UDFs implement the UDF interface in Hive (UDFLength, UDFHour, UDFYear). These built-in hive functions have been moved to use GenericUDF interface in Hive 3. Impala currently only supports UDFExecutor. In order to have a full compatibility with all the functions in Hive 2.x we should support GenericUDFs too. That would be taken up as a separate patch. 3. Sentry dependencies bring a lot of transitive hive dependencies. The patch excludes such dependencies since they create problems while building against Hive-3. Since these hive-2 dependencies are already included when building against hive-2 this should not be a problem. Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 --- M CMakeLists.txt M README.md M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/set-classpath.sh M common/thrift/.gitignore M common/thrift/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml A fe/src/compat-hive-2/java/org/apache/impala/compat/HiveShim.java A fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveShim.java A fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java D fe/src/main/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/hive/executor/UdfExecutorTest.java M fe/src/test/java/org/apache/impala/testutil/EmbeddedMetastoreClientPool.java M impala-parent/pom.xml A shaded-deps/.gitignore A shaded-deps/CMakeLists.txt A shaded-deps/pom.xml M testdata/bin/run-hive-server.sh M tests/custom_cluster/test_permanent_udfs.py 34 files changed, 1,870 insertions(+), 449 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/13005/12 -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Sudhanshu Arora <sudhan...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
