Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14711 )
Change subject: IMPALA-8778: Support Apache Hudi Read Optimized Table ...................................................................... IMPALA-8778: Support Apache Hudi Read Optimized Table Hudi Read Optimized Table contains multiple versions of parquet files, in order to load the table correctly, Impala needs to recognize Hudi Read Optimized Table as a HdfsTable and load the latest version of the file using HoodieROTablePathFilter. Tests - Unit test for Hudi in FileMetadataLoader - Create table tests in functional_schema_template.sql - Query tests in hudi-parquet.test Change-Id: I65e146b347714df32fe968409ef2dde1f6a25cdf Reviewed-on: http://gerrit.cloudera.org:8080/14711 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/service/query-options-test.cc M bin/impala-config.sh M bin/rat_exclude_files.txt M common/thrift/CatalogObjects.thrift M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java A fe/src/main/java/org/apache/impala/util/HudiUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M impala-parent/pom.xml M testdata/bin/generate-schema-statements.py M testdata/data/README A testdata/data/hudi_parquet/.hoodie/20200210090610.clean A testdata/data/hudi_parquet/.hoodie/20200210090610.clean.inflight A testdata/data/hudi_parquet/.hoodie/20200210090610.clean.requested A testdata/data/hudi_parquet/.hoodie/20200210090610.commit A testdata/data/hudi_parquet/.hoodie/20200210090610.commit.requested A testdata/data/hudi_parquet/.hoodie/20200210090610.inflight A testdata/data/hudi_parquet/.hoodie/20200210090618.clean A testdata/data/hudi_parquet/.hoodie/20200210090618.clean.inflight A testdata/data/hudi_parquet/.hoodie/20200210090618.clean.requested A testdata/data/hudi_parquet/.hoodie/20200210090618.commit A testdata/data/hudi_parquet/.hoodie/20200210090618.commit.requested A testdata/data/hudi_parquet/.hoodie/20200210090618.inflight A testdata/data/hudi_parquet/.hoodie/hoodie.properties A testdata/data/hudi_parquet/year=2015/month=03/day=16/.hoodie_partition_metadata A testdata/data/hudi_parquet/year=2015/month=03/day=16/5f541af5-ca07-4329-ad8c-40fa9b353f35-0_1-70-118_20200210090610.parquet A testdata/data/hudi_parquet/year=2015/month=03/day=16/5f541af5-ca07-4329-ad8c-40fa9b353f35-0_2-103-391_20200210090618.parquet A testdata/data/hudi_parquet/year=2015/month=03/day=17/.hoodie_partition_metadata A testdata/data/hudi_parquet/year=2015/month=03/day=17/675e035d-c146-4658-9404-fe590e296d80-0_0-103-389_20200210090618.parquet A testdata/data/hudi_parquet/year=2015/month=03/day=17/675e035d-c146-4658-9404-fe590e296d80-0_0-70-117_20200210090610.parquet A testdata/data/hudi_parquet/year=2016/month=03/day=15/.hoodie_partition_metadata A testdata/data/hudi_parquet/year=2016/month=03/day=15/940359ee-cc79-4974-8a2a-5d133a81a3fd-0_1-103-390_20200210090618.parquet A testdata/data/hudi_parquet/year=2016/month=03/day=15/940359ee-cc79-4974-8a2a-5d133a81a3fd-0_2-70-119_20200210090610.parquet M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/hudi-parquet.test M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/query_test/test_scanners.py 44 files changed, 626 insertions(+), 41 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I65e146b347714df32fe968409ef2dde1f6a25cdf Gerrit-Change-Number: 14711 Gerrit-PatchSet: 25 Gerrit-Owner: Yanjia Gary Li <yanjia.gary...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Norbert Luksa <norbert.lu...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Yanjia Gary Li <yanjia.gary...@gmail.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
