[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 06:38:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. IMPALA-2: Impala can't resolve json tables created by Hive Impala was using wrong json serde string which was causing the subsequent errors. The error got resolved on correcting the json serde string. Now impala can resolve json tables created by hive and also hive can resolve json tables created in impala. End-to-end tests are also included in tests/metadata/test_hms_integration.py. Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Reviewed-on: http://gerrit.cloudera.org:8080/18263 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M tests/metadata/test_hms_integration.py 3 files changed, 22 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 10 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 Like IMPALA-8369, this patch adds a compatibility shim in fe so that Impala can interoperate with Hive 3.1.2. we need adds a new Metastoreshim class under compat-apache-hive-3 directory. These shim classes implement method which are different in cdp-hive-3 vs apache-hive-3 and are used by front end code. At the build time, based on the environment variable IMPALA_HIVE_DIST_TYPE one of the two shims is added to as source using the fe/pom.xml build plugin. Some codes that directly use Hive 4 APIs need to be ignored in compilation, eg. fe/src/main/java/org/apache/impala/catalog/metastore/. Use Maven profile to ignore some codes, profile will automatically activated based on the IMPALA_HIVE_DIST_TYPE. Testing: 1. Code compiles and runs against both HMS-3 and ASF-HMS-3 2. Ran full-suite of tests against HMS-3 3. Running full-tests against ASF-HMS-3 will need more work supporting Tez in the mini-cluster (for dataloading) and HMS transaction support. This will be on-going effort and test failures on ASF-Hive-3 will be fixed in additional sub-tasks. Notes: 1. Patch uses a custom build of Apache Hive to be deployed in mini-cluster. This build has the fixes for HIVE-21569, HIVE-20038. This hack will be added to the build script in additional sub-tasks. Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Reviewed-on: http://gerrit.cloudera.org:8080/17774 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/jenkins/build-all-flag-combinations.sh M bin/rat_exclude_files.txt M buildall.sh M fe/pom.xml A fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/metastore/CatalogHmsUtils.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServer.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java R fe/src/main/java/org/apache/impala/util/HiveMetadataFormatUtils.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M java/pom.xml A testdata/bin/patch_hive.sh A testdata/cluster/hive/README A testdata/cluster/hive/patch0-HIVE-21586.diff 27 files changed, 2,514 insertions(+), 1,030 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 18 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 17: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 17 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sun, 27 Feb 2022 06:36:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 20: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7882/ -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 20 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 04:43:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7884/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 01:56:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 01:56:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 01:56:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11112: Impala can't resolve json tables created by Hive
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18263 ) Change subject: IMPALA-2: Impala can't resolve json tables created by Hive .. Patch Set 8: > Patch Set 8: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7875/ The failure is due to a flaky test: IMPALA-10999 -- To view, visit http://gerrit.cloudera.org:8080/18263 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9cac55b58dca88d900db3256ceaa25c17d7864d5 Gerrit-Change-Number: 18263 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 01:56:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7883/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 17 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sun, 27 Feb 2022 01:49:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 17 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sun, 27 Feb 2022 01:49:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 16: Code-Review+2 Thanks for the great work! I have verified that this won't break downstream jobs in Cloudera. -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 16 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sun, 27 Feb 2022 01:48:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 20: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10233/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 20 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 00:42:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7882/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 20 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 00:23:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 20: (11 comments) http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py File tests/catalog_service/test_partition_catalog_updates.py: http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@25 PS20, Line 25: from CatalogService import CatalogService flake8: F401 'CatalogService.CatalogService' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@26 PS20, Line 26: from CatalogService.CatalogService import TGetFunctionsRequest flake8: F401 'CatalogService.CatalogService.TGetFunctionsRequest' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@27 PS20, Line 27: from ErrorCodes.ttypes import TErrorCode flake8: F401 'ErrorCodes.ttypes.TErrorCode' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@28 PS20, Line 28: from thrift.protocol import TBinaryProtocol flake8: F401 'thrift.protocol.TBinaryProtocol' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@31 PS20, Line 31: from tests.common.impala_cluster import ImpalaCluster flake8: F401 'tests.common.impala_cluster.ImpalaCluster' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@32 PS20, Line 32: from tests.common.impala_test_suite import ImpalaTestSuite flake8: F401 'tests.common.impala_test_suite.ImpalaTestSuite' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@33 PS20, Line 33: from tests.common.skip import SkipIfDockerizedCluster flake8: F401 'tests.common.skip.SkipIfDockerizedCluster' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@34 PS20, Line 34: from tests.common.test_dimensions import create_single_exec_option_dimension flake8: F401 'tests.common.test_dimensions.create_single_exec_option_dimension' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@35 PS20, Line 35: from tests.util.filesystem_utils import WAREHOUSE flake8: F401 'tests.util.filesystem_utils.WAREHOUSE' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@36 PS20, Line 36: from tests.util.thrift_util import create_transport flake8: F401 'tests.util.thrift_util.create_transport' imported but unused http://gerrit.cloudera.org:8080/#/c/18141/20/tests/catalog_service/test_partition_catalog_updates.py@88 PS20, Line 88: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 20 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sun, 27 Feb 2022 00:23:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Hello Qifan Chen, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18141 to look at the new patch set (#20). Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. IMPALA-10898: Add runtime IN-list filters for ORC tables ORC files have optional bloom filter indexes for each column. Since ORC-1.7.0, the C++ reader supports pushing down predicates to skip unreleated RowGroups. The pushed down predicates will be evaludated on file indexes (i.e. statistics and bloom filter indexes). Note that only EQUALS and IN-list predicates can leverage bloom filter indexes. Currently Impala has two kinds of runtime filters: bloom filter and min-max filter. Unfortunately they can't be converted into EQUALS or IN-list predicates. So they can't leverage the file level bloom filter indexes. This patch adds runtime IN-list filters for this purpose. Currently they are generated for the build side of a broadcast join. They will only be applied on ORC tables and be pushed down to the ORC reader(i.e. ORC lib). To avoid exploding the IN-list, if # of distinct values of the build side exceeds a threshold (default to 1024), we set the filter to ALWAYS_TRUE and clear its entry. The threshold can be configured by a new query option, RUNTIME_IN_LIST_FILTER_ENTRY_LIMIT. Evaluating runtime IN-list filters is much slower than evaluating runtime bloom filters due to the current simple implementation (i.e. std::unorder_set) and the lack of codegen. So we disable it at row level. For visibility, this patch addes two counters in the HdfsScanNode: - NumPushedDownPredicates - NumPushedDownRuntimeFilters They reflect the predicates and runtime filters that are pushed down to the ORC reader. Currently, runtime IN-list filters are disabled by default. This patch extends the query option, ENABLED_RUNTIME_FILTER_TYPES, to support a comma separated list of filter types. It defaults to be "BLOOM,MIN_MAX". Add "IN_LIST" in it to enable runtime IN-list filters. Ran perf tests on a 3 instances cluster on my desktop using TPC-DS with scale factor 20. It shows significant improvements in some queries: +---+-+++-++++---++-++ | Workload | Query | File Format| Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +---+-+++-++++---++-++ | TPCDS(20) | TPCDS-Q67A | orc / snap / block | 35.07 | 44.01 | I -20.32% | 0.38%| 1.38%| 10| I -25.69% | -3.58 | -45.33 | | TPCDS(20) | TPCDS-Q37 | orc / snap / block | 1.08 | 1.45| I -25.23% | 7.14%| 3.09%| 10| I -34.09% | -3.58 | -12.94 | | TPCDS(20) | TPCDS-Q70A | orc / snap / block | 6.30 | 8.60| I -26.81% | 5.24%| 4.21%| 10| I -36.67% | -3.58 | -14.88 | | TPCDS(20) | TPCDS-Q16 | orc / snap / block | 1.33 | 1.85| I -28.28% | 4.98%| 5.92%| 10| I -39.38% | -3.58 | -12.93 | | TPCDS(20) | TPCDS-Q18A | orc / snap / block | 5.70 | 8.06| I -29.25% | 3.00%| 4.12%| 10| I -40.30% | -3.58 | -19.95 | | TPCDS(20) | TPCDS-Q22A | orc / snap / block | 2.01 | 2.97| I -32.21% | 6.12%| 5.94%| 10| I -47.68% | -3.58 | -14.05 | | TPCDS(20) | TPCDS-Q77A | orc / snap / block | 8.49 | 12.44 | I -31.75% | 6.44%| 3.96%| 10| I -49.71% | -3.58 | -16.97 | | TPCDS(20) | TPCDS-Q75 | orc / snap / block | 7.76 | 12.27 | I -36.76% | 5.01%| 3.87%| 10| I -59.56% | -3.58 | -23.26 | | TPCDS(20) | TPCDS-Q21 | orc / snap / block | 0.71 | 1.27| I -44.26% | 4.56%| 4.24%| 10| I -77.31% | -3.58 | -28.31 | | TPCDS(20) | TPCDS-Q80A | orc / snap / block | 9.24 | 20.42 | I -54.77% | 4.03%| 3.82%| 10| I -123.12% | -3.58 | -40.90 | | TPCDS(20) | TPCDS-Q39-1 | orc / snap / block | 1.07 | 2.26| I -52.74% | * 23.83% * | 2.60%| 10| I -149.68% | -3.58 | -14.43 | | TPCDS(20) | TPCDS-Q39-2 | orc / snap / block | 1.00 | 2.33| I -56.95% | * 19.53% * | 2.07%| 10| I -151.89% | -3.58 | -20.81 | +---+-+++-++++---++-++ "Base Avg" is the avg of the original time. "Avg" is the current time. However, we also see some regressions due to the suboptimal implementation. The follow-up JIRAs will focus
[Impala-ASF-CR] IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17774 ) Change subject: IMPALA-10871: Add MetastoreShim to support Apache Hive 3.1.2 .. Patch Set 16: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17774 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f08db5f6da735ac431819063060941f0941f606 Gerrit-Change-Number: 17774 Gerrit-PatchSet: 16 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 26 Feb 2022 09:15:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 26 Feb 2022 08:19:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10049: Include RPC call id in slow RPC logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18243 ) Change subject: IMPALA-10049: Include RPC call_id in slow RPC logs .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7881/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18243 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fb5746fa0be575745a8e168405d43115c425389 Gerrit-Change-Number: 18243 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 26 Feb 2022 08:19:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10898: Add runtime IN-list filters for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18141 ) Change subject: IMPALA-10898: Add runtime IN-list filters for ORC tables .. Patch Set 19: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7879/ -- To view, visit http://gerrit.cloudera.org:8080/18141 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25080628233799aa0b6be18d5a832f1385414501 Gerrit-Change-Number: 18141 Gerrit-PatchSet: 19 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 26 Feb 2022 08:09:31 + Gerrit-HasComments: No
