[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9704/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 12 Sep 2023 05:09:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 7: Running an ARM build at https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/35/ to make sure I didn't break them. -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 04:12:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 7: Verified+1 Overriding failure due to https://issues.apache.org/jira/browse/IMPALA-12266. -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 04:09:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Michael Smith has removed a vote on this change. Change subject: IMPALA-12441: Simplify local toolchain development .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 04:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9703/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 04:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20474/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/20474/1/be/src/exprs/expr-test.cc@235 PS1, Line 235: executor_.reset( > line too long (95 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 04:07:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 12: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9702/ -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 12 Sep 2023 03:59:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20443 ) Change subject: IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache_ .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13982/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e3c10660e371498c2edc1eb8d235b7b8ca170c9 Gerrit-Change-Number: 20443 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 12 Sep 2023 02:06:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12444: Fix minimum parallelism bug in scan fragment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20475 ) Change subject: IMPALA-12444: Fix minimum parallelism bug in scan fragment .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394 Gerrit-Change-Number: 20475 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 12 Sep 2023 01:45:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache
Hello Quanlong Huang, Tamas Mate, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20443 to look at the new patch set (#2). Change subject: IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache_ .. IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache_ The local CatalogMetaProvider's cache_ need to doing some loading process with the default Concurrency level of 4 , when the table number is very big ,the loading process will need very much time, in that case the restart process will cost much time too, so we make this parameter configurable. Change-Id: I8e3c10660e371498c2edc1eb8d235b7b8ca170c9 --- M be/src/runtime/exec-env.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java 4 files changed, 12 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/20443/2 -- To view, visit http://gerrit.cloudera.org:8080/20443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e3c10660e371498c2edc1eb8d235b7b8ca170c9 Gerrit-Change-Number: 20443 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-12444: Fix minimum parallelism bug in scan fragment
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20475 Change subject: IMPALA-12444: Fix minimum parallelism bug in scan fragment .. IMPALA-12444: Fix minimum parallelism bug in scan fragment Scan fragment did not follow PROCESSING_COST_MIN_THREADS set by user even if total scan ranges allow to do so. This patch fix the issue by exposing ScanNode.maxScannerThreads_ to PlanFragment.adjustToMaxParallelism(). By using ScanNode.maxScannerThreads_ as an upper bound, ScanNode does not need to artificially lower ProcessingCost if maxScannerThreads_ is lower than minimum parallelism dictated by the original ProcessingCost. Thus, the synthetic ProcessingCost logic in ScanNode class is revised to only apply if input cardinality is unknown (-1). This patch also does the following adjustments: - Remove some dead codes in Frontend.java and PlanFragment.java. - Add sanity check such that PROCESSING_COST_MIN_THREADS <= MAX_FRAGMENT_INSTANCES_PER_NODE. - Tidy up test_query_cpu_count_divisor_default to reduce number of SET query. Testing: - Update test_query_cpu_count_divisor_default to ensure that PROCESSING_COST_MIN_THREADS is respected by scan fragment and error is returned if PROCESSING_COST_MIN_THREADS is greater than MAX_FRAGMENT_INSTANCES_PER_NODE. - Pass test_executor_groups.py. Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/CostingSegment.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 6 files changed, 61 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/20475/1 -- To view, visit http://gerrit.cloudera.org:8080/20475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I69e5a80146d4ac41de5ef406fc2bdceffe3ec394 Gerrit-Change-Number: 20475 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20425 ) Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 12 Sep 2023 01:04:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20473 )
Change subject: IMPALA-11553: Add event specific metrics in the table metrics
..
Patch Set 2:
(4 comments)
Thanks for adding these! I just take a quick look. Will look into this deeper.
http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG@22
PS2, Line 22:
Can we add more metrics at table level, e.g. events-skipped,
last-synced-event-id, events-consuming-delay-ms?
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@942
PS2, Line 942: latestEventTimeMs_.set(event.getEventTime());
I made a mistake that the unit is actually second, not milli second. Could you
also correct the variable name and the metric text appropriately? E.g.
latestEventTimeMs_ -> latestEventTimeSecs_
lastSyncedEventTimeMs_ -> lastSyncedEventTimeSecs_
"latest-event-time-ms" -> "latest-event-time-secs"
"last-synced-event-time-ms" -> "last-synced-event-time-secs"
"event-processing-delay-ms" -> "event-processing-delay-secs"
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1063
PS2, Line 1063: MILLISECONDS
TimeUnit.SECONDS ?
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1078
PS2, Line 1078: public long now() {
We use 'System.currentTimeMillis() / 1000' directly in many places. Maybe don't
worth a method for this, and it's ambigous what the unit is.
$ git grep 'System.currentTimeMillis() / 1000'
fe/src/main/java/org/apache/impala/catalog/Table.java:
msTbl.putToParameters(propertyKey, Long.toString(System.currentTimeMillis() /
1000));
fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java:
(int) (System.currentTimeMillis() / 1000),
fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java:
(int) (System.currentTimeMillis() / 1000), FunctionType.JAVA, resources);
fe/src/test/java/org/apache/impala/service/JdbcTest.java:
lastTimeSessionActive.add(System.currentTimeMillis() / 1000);
fe/src/test/java/org/apache/impala/service/JdbcTest.java: long now =
System.currentTimeMillis() / 1000;
--
To view, visit http://gerrit.cloudera.org:8080/20473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd
Gerrit-Change-Number: 20473
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Tue, 12 Sep 2023 00:42:05 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59
PS5, Line 59: ./buildall.py --multi --builder=${BUILDER} ${DISTRO_PARAM}
Would it make sense to have an environment variable / Jenkins parameter that
toggles whether to do the multi-arch build or a single arch build?
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Tue, 12 Sep 2023 00:23:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9700/ -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 00:14:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13980/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 00:08:46 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG@17 PS5, Line 17: Moves the publishing step to happen within buildx, because buildx does : not support loading multi-platform images in the local Docker registry. : This does mean that a build failure could result in publishing only some : of the images. These are not updated frequently and are usually preceded : by a test run, so this risk seems minimal. Can you update this to describe how we avoid this problem with the two passes? http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md File docker/README.md: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md@45 PS5, Line 45: acd Nit: and http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58 PS5, Line 58: build_cmd += ['--builder', args.builder] > Building everything in the same builder seems to require a fairly new versi Does that impact the x86 single-arch build? Or is that basically fine because it is using the default builder? -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 00:00:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13979/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 00:00:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 3: Code-Review+2 LGTM, thanks for finding this -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 23:52:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Hello Andrew Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20474 to look at the new patch set (#3). Change subject: IMPALA-12411: Fix data race in expr-test teardown .. IMPALA-12411: Fix data race in expr-test teardown TSAN ThreadSanitizer detected a data race during expr-test teardown when a query finishes after JvmMetricCache's destructor is called. expr-test can finish with a query still running because ImpaladQueryExecutor closes the previous query when you start the next query and relies on the destructor to close the last query. expr-test creates ImpaladQueryExecutor as a global as part of setting up an in-memory cluster (with Statestore and InProcessImpalaServer). So the last query isn't guaranteed to be closed until global destruction, which leads to a race with other global destruction (in this case with JvmMetricCache while updating QueryState on query completion). ImpaladQueryExecutor is only used in expr-test. Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c --- M be/src/exprs/expr-test.cc 1 file changed, 15 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/20474/3 -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Hello Andrew Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20474 to look at the new patch set (#2). Change subject: IMPALA-12411: Fix data race in expr-test teardown .. IMPALA-12411: Fix data race in expr-test teardown TSAN ThreadSanitizer detected a data race during expr-test teardown. It happens because ImpaladQueryExecutor closes the previous query when you start the next query and it relies on the destructor to close the last query. expr-test creates ImpaladQueryExecutor as a global as part of setting up an in-memory cluster (with Statestore and InProcessImpalaServer). So the last query isn't guaranteed to be closed until global destruction, which leads to a race with other global destruction (in this case with JvmMetricCache while updating QueryState on query completion). ImpaladQueryExecutor is only used in expr-test. Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c --- M be/src/exprs/expr-test.cc 1 file changed, 15 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/20474/2 -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20474 Change subject: IMPALA-12411: Fix data race in expr-test teardown .. IMPALA-12411: Fix data race in expr-test teardown TSAN ThreadSanitizer detected a data race during expr-test teardown. It happened because ImpaladQueryExecutor closes the previous query when you start the next query and it relies on the destructor to close the last query. expr-test creates ImpaladQueryExecutor as a global as part of setting up an in-memory cluster (with Statestore and InProcessImpalaServer). So the last query isn't guaranteed to be closed until global destruction, which leads to a race with other global destruction (in this case the JvmMetricCache, which is used when updating QueryState on query completion). This only impacts expr-test as that's the only place ImpaladQueryExecutor is used. Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c --- M be/src/exprs/expr-test.cc 1 file changed, 14 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/20474/1 -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-12411: Fix data race in expr-test teardown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20474 ) Change subject: IMPALA-12411: Fix data race in expr-test teardown .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20474/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/20474/1/be/src/exprs/expr-test.cc@235 PS1, Line 235: executor_.reset(new ImpaladQueryExecutor(FLAGS_hostname, impala_server->GetBeeswaxPort())); line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/20474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e289b292cb11154f1245e2b987cde0995b6243c Gerrit-Change-Number: 20474 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 23:34:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9702/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 11 Sep 2023 23:27:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 11 Sep 2023 23:27:32 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58 PS5, Line 58: build_cmd += ['--builder', args.builder] Building everything in the same builder seems to require a fairly new version of buildx. Docker 20 wasn't new enough; I'm using Docker 24 locally. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 23:23:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12390 (part 2): Enable some clang-tidy performance related checks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20445 ) Change subject: IMPALA-12390 (part 2): Enable some clang-tidy performance related checks .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13978/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20445 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibad8bd0f12aab92ad874f5a6b9ec922dce7f3190 Gerrit-Change-Number: 20445 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 23:20:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13977/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Sep 2023 23:00:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12390 (part 2): Enable some clang-tidy performance related checks
gsi...@cloudera.com has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/20445 ) Change subject: IMPALA-12390 (part 2): Enable some clang-tidy performance related checks .. IMPALA-12390 (part 2): Enable some clang-tidy performance related checks This enables the clang tidy performance check: performance-inefficient-string-concatenation "warning: string concatenation results in allocation of unnecessary temporary strings" Fix: Use StrCat() to concatenate multiple strings Testing: - Ran bin/run_clang_tidy.sh with the new checks - Ran GVO Change-Id: Ibad8bd0f12aab92ad874f5a6b9ec922dce7f3190 --- M .clang-tidy M be/src/exprs/expr-test.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/timestamp-test.cc M be/src/scheduling/cluster-membership-test-util.cc M be/src/util/runtime-profile.cc M bin/run_clang_tidy.sh 7 files changed, 93 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/20445/5 -- To view, visit http://gerrit.cloudera.org:8080/20445 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibad8bd0f12aab92ad874f5a6b9ec922dce7f3190 Gerrit-Change-Number: 20445 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13976/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Sep 2023 22:37:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#2). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 354 insertions(+), 149 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/2 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](asf-site) IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20403 ) Change subject: IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I93083b999ec404ca3f407977b8e930b85a53b695 Gerrit-Change-Number: 20403 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 22:33:54 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20403 ) Change subject: IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo .. IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js is also slow to download in China and it usually fails at last. This adds the file to the asf-site branch so the website can use a local copy of it. Tests: - Verified the pages locally. Change-Id: I93083b999ec404ca3f407977b8e930b85a53b695 Reviewed-on: http://gerrit.cloudera.org:8080/20403 Reviewed-by: Daniel Becker Tested-by: Quanlong Huang --- M bylaws.html M community.html M downloads.html M impala-docs.html M index.html A js/bootstrap-3.3.5.min.js D js/bootstrap.js D js/bootstrap.min.js M overview.html 9 files changed, 13 insertions(+), 2,296 deletions(-) Approvals: Daniel Becker: Looks good to me, approved Quanlong Huang: Verified -- To view, visit http://gerrit.cloudera.org:8080/20403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: I93083b999ec404ca3f407977b8e930b85a53b695 Gerrit-Change-Number: 20403 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [arm64] remove dependence on sse2neon
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/19954 )
Change subject: [arm64] remove dependence on sse2neon
..
Patch Set 6: Code-Review+1
(1 comment)
This is looking good to me, a couple minor things
Not your change, but if you don't mind, can you also remove the includes of
sse-util.h and cpu-util.h from be/src/runtime/string-value.inline.h? I was
grepping through to see what code used sse-util.h and found that, and the code
hasn't needed those includes for a while.
https://github.com/apache/impala/blob/master/be/src/runtime/string-value.inline.h#L25-L26
http://gerrit.cloudera.org:8080/#/c/19954/6/be/src/util/sse-util.h
File be/src/util/sse-util.h:
http://gerrit.cloudera.org:8080/#/c/19954/6/be/src/util/sse-util.h@179
PS6, Line 179: static inline int POPCNT_popcnt_u64(uint64_t x) {
If I understand the code correctly, I think we would also need to change this
code in bit-util.h:
https://github.com/apache/impala/blob/master/be/src/util/bit-util.h#L151-L163
Right now, it uses PopcountNoHw(x) for aarch64.
--
To view, visit http://gerrit.cloudera.org:8080/19954
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If3c78c877ef530fa9f35d36da523ad67ab34e5e7
Gerrit-Change-Number: 19954
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Mon, 11 Sep 2023 22:26:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13975/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 11 Sep 2023 22:17:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11068: Add query option to reduce scanner thread launch.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18126 ) Change subject: IMPALA-11068: Add query option to reduce scanner thread launch. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18126 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I03cadf1230eed00d69f2890c82476c6861e37466 Gerrit-Change-Number: 18126 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Sep 2023 22:15:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20473 Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 353 insertions(+), 149 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/1 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12385: Enable Periodic metrics by default
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20377 ) Change subject: IMPALA-12385: Enable Periodic metrics by default .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/20377/9/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/20377/9/tests/query_test/test_observability.py@615 PS9, Line 615: expected_strs = ["HostCpuIoWaitPercentage (50.000ms):", This test failed in the Jenkins run. -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Comment-Date: Mon, 11 Sep 2023 22:09:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12385: Enable Periodic metrics by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20377 ) Change subject: IMPALA-12385: Enable Periodic metrics by default .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9699/ -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Comment-Date: Mon, 11 Sep 2023 22:07:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 11: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 11 Sep 2023 21:53:52 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29 PS2, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204'] : : def main(): > Yeah, the normal path is fine, so it's only the --multi path that would be Done http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32 PS2, Line 32: nfig(leve > I agree that we don't really need Redhat 9 ARM at the moment, so even if we Ack -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:52:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 10: > Patch Set 10: > > > Patch Set 10: Verified-1 > > > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9690/ > > These is a failure that might due to this patch. Please take a look: > https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/321 > > custom_cluster/test_metastore_service.py:405: in > test_fallback_get_partitions_by_names > TestMetastoreService.part_tbl, valid_part_names, True) > custom_cluster/test_metastore_service.py:1207: in > __run_partitions_by_names_tests > "Internal error") > custom_cluster/test_metastore_service.py:1289: in > __get_parts_by_names_expect_exception > assert expected_exception_str in str(e) > E assert 'Internal error' in > "InvalidObjectException(message='hive.test_fallback_get_partitions_by_names_efbbe23c.table-does-not-exist > table not found')" > E+ where > "InvalidObjectException(message='hive.test_fallback_get_partitions_by_names_efbbe23c.table-does-not-exist > table not found')" = > str(InvalidObjectException(message='hive.test_fallback_get_partitions_by_names_efbbe23c.table-does-not-exist > table not found')) True. The test should now expect a 'InvalidObjectException' instead of 'Internal error' error message. -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 11 Sep 2023 21:52:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20420 to look at the new patch set (#11). Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. IMPALA-12231: Bump GBN to get HMS thrift API changes We need a couple of hive changes HIVE-27319 and HIVE-27337 for catalogD to work with latest HMS server to fix IMPALA-11768 and IMPALA-11939 respectively. Bump CDP_BUILD_NUMBER (GBN) to 44206393 Bump various CDP versiona numbers to be based on 7.2.18.0-273 TESTING: Exhaustive tests ran clean Added a couple of tests for IMPALA-11939 and IMPALA-11768 Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M java/shaded-deps/hive-exec/pom.xml M java/test-hive-udfs/pom.xml M tests/custom_cluster/test_events_custom_configs.py M tests/custom_cluster/test_metastore_service.py 7 files changed, 81 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/20420/11 -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12442: Remove execute serially from stress tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20395 ) Change subject: IMPALA-12442: Remove execute_serially from stress tests .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13973/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I49bfd745881da992815292d16e1a311ab1884abf Gerrit-Change-Number: 20395 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 21:54:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12442: Remove execute serially from stress tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20395 ) Change subject: IMPALA-12442: Remove execute_serially from stress tests .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13972/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I49bfd745881da992815292d16e1a311ab1884abf Gerrit-Change-Number: 20395 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 21:54:28 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20469 to look at the new patch set (#5). Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb --- M docker/README.md M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 7 files changed, 73 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/5 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11068: Add query option to reduce scanner thread launch.
Hello Kurt Deschler, Joe McDonnell, Csaba Ringhofer, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18126 to look at the new patch set (#2). Change subject: IMPALA-11068: Add query option to reduce scanner thread launch. .. IMPALA-11068: Add query option to reduce scanner thread launch. Under heavy decompression workload, Impala running with scanner thread parallelism (MT_DOP=0) can still hit OOM error due to launching too many threads too soon. We have logic in ScannerMemLimiter to limit the number of scanner threads by calculating the thread's memory requirement and estimating the memory growth rate of all threads. However, it does not prevent a scanner node from quickly launching many threads and immediately reaching the memtracker's spare capacity. Even after ScannerMemLimiter rejects a new thread launch, some existing threads might continue increasing their non-reserved memory for decompression work until finally the memory limit is exceeded. IMPALA-7096 adds hdfs_scanner_thread_max_estimated_bytes flag as a heuristic to count for non-reserved memory growth. Increasing this flag value can help reduce thread count, but might severely regress other queries that do not have heavy decompression characteristics. Similarly with lowering the NUM_SCANNER_THREADS query option. This patch adds one more query option as an alternative to mitigate OOM called HDFS_SCANNER_NON_RESERVED_BYTES. This flag intent to offer the same control as hdfs_scanner_thread_max_estimated_bytes, but as a query option such that tuning can be done at per query granulality. If this query option is not set, revert to use the value of hdfs_scanner_thread_max_estimated_bytes flag. Testing: - Add test case in TestScanMemLimit::test_hdfs_scanner_thread_mem_scaling. Change-Id: I03cadf1230eed00d69f2890c82476c6861e37466 --- M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M testdata/workloads/functional-query/queries/QueryTest/hdfs-scanner-thread-mem-scaling.test 8 files changed, 86 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18126/2 -- To view, visit http://gerrit.cloudera.org:8080/18126 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I03cadf1230eed00d69f2890c82476c6861e37466 Gerrit-Change-Number: 18126 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] [arm64] remove dependence on sse2neon
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19954 ) Change subject: [arm64] remove dependence on sse2neon .. Patch Set 6: Code-Review+1 Kudu changes match https://gerrit.cloudera.org/c/20374/. -- To view, visit http://gerrit.cloudera.org:8080/19954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3c78c877ef530fa9f35d36da523ad67ab34e5e7 Gerrit-Change-Number: 19954 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:39:00 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29 PS2, Line 29: # apt install binfmt-support qemu-user-static qemu-system-x86 : # then create a docker-container builder : # docker buildx create --use > I think the errors you get from Docker are pretty clear. These are only req Yeah, the normal path is fine, so it's only the --multi path that would be a problem. I noticed that we have a README.md file in this directory with some instructions for building the docker images. Maybe we could add a section there about what to install, how to get a builder, how to invoke buildall.py? http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32 PS2, Line 32: 'redhat8' > We could if we think we'll move CDW to it sometime soon. I was mostly tryin I agree that we don't really need Redhat 9 ARM at the moment, so even if we produce the docker images, we can probably skip the full build until later. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:34:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12442: Remove execute serially from stress tests
Hello Quanlong Huang, Laszlo Gaal, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20395 to look at the new patch set (#3). Change subject: IMPALA-12442: Remove execute_serially from stress tests .. IMPALA-12442: Remove execute_serially from stress tests Avoids labeling stress tests with execute_serially so they're only run once during run-all-tests. Previously stress tests would be run twice, once for 'execute_serially' and again for 'stress'. Documents the markers in pytest.ini. Change-Id: I49bfd745881da992815292d16e1a311ab1884abf --- M tests/metadata/test_recursive_listing.py M tests/pytest.ini M tests/run-tests.py M tests/stress/test_acid_stress.py M tests/stress/test_insert_stress.py 5 files changed, 4 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/20395/3 -- To view, visit http://gerrit.cloudera.org:8080/20395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I49bfd745881da992815292d16e1a311ab1884abf Gerrit-Change-Number: 20395 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12442: Remove execute serially from stress tests
Hello Quanlong Huang, Laszlo Gaal, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20395 to look at the new patch set (#2). Change subject: IMPALA-12442: Remove execute_serially from stress tests .. IMPALA-12442: Remove execute_serially from stress tests Avoids labeling stress tests with execute_serially so they're only run once during run-all-tests. Previously stress tests would be run twice, once for 'execute_serially' and again for 'stress'. Documents the markers in pytest.ini. Change-Id: I49bfd745881da992815292d16e1a311ab1884abf --- M tests/metadata/test_recursive_listing.py M tests/pytest.ini M tests/run-tests.py M tests/stress/test_acid_stress.py M tests/stress/test_insert_stress.py 5 files changed, 4 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/20395/2 -- To view, visit http://gerrit.cloudera.org:8080/20395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I49bfd745881da992815292d16e1a311ab1884abf Gerrit-Change-Number: 20395 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20425 ) Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/20425/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20425/5//COMMIT_MSG@29 PS5, Line 29: : This change allows implicit cast between matching types > Can you highlight that this change only affects char/varchar, as for other Done -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 21:26:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Hello Quanlong Huang, Daniel Becker, Peter Rozsa, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20425 to look at the new patch set (#6). Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. IMPALA-10086: Implicit cast comparing char and varchar Until IMPALA-7368, Impala allowed comparing char and varchar slots as in select * from functional.chars_tiny where cs = vc; IMPALA-7368 added DATE type support, and as part of that changed function call resolution to use a fit function based on the number of arguments that match the call types. Previously the comparison above would take the first matching function, which happened to be equality between STRING and STRING; CHAR and VARCHAR can both be implicitly cast to STRING, so this function worked. With the new function resolution, the best fit is equality between VARCHAR and VARCHAR, however implicit casting to VARCHAR(*) from CHAR wasn't allowed. The behavior before IMPALA-7368 was somewhat accidental; it depended on the order that builtin EQ functions are added via BinaryPredicate.initBuiltins -> Type.getSupportedTypes. Supported types happened to be ordered with STRING preceding VARCHAR and CHAR. The fit function makes sense and changing its behavior may have other consequences; it also makes sense that CHAR should be castable to VARCHAR. This change allows implicit cast between matching types. Functionally it only changes how we handle char/varchar comparison with wildcard char/varchar, because decimals are handled before checking for matching types and other type matching is the same as equals. It now allows casting to a compatible type when it is a char or varchar and the target type is a wildcard version of the same. Does not attempt to address differences from CHAR padding (IMPALA-1652). Testing: - Adds tests covering cast comparison and other implicit conversions. - Passed exhaustive test run. Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M tests/query_test/test_cast_with_format.py 3 files changed, 71 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/20425/6 -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20425 ) Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. Patch Set 5: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/20425/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20425/5//COMMIT_MSG@29 PS5, Line 29: : This change allows implicit cast between matching types Can you highlight that this change only affects char/varchar, as for other types matching is the same as equals, and decimals are handled separately? -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 21:19:32 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29 PS2, Line 29: # apt install binfmt-support qemu-user-static qemu-system-x86 : # then create a docker-container builder : # docker buildx create --use > Nit: Is there anything we can do to detect if these are appropriately speci I think the errors you get from Docker are pretty clear. These are only required if specifying '--multi', I'll update the comment to clarify that. I made sure most local development workflows would not require QEMU or a new builder, so these should only be needed if testing out multi-platform builds specifically. http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32 PS2, Line 32: 'redhat8' > Should we include redhat9? We could if we think we'll move CDW to it sometime soon. I was mostly trying to keep native-toolchain build varieties down to ones we would use. http://gerrit.cloudera.org:8080/#/c/20469/2/jenkins/build.sh File jenkins/build.sh: http://gerrit.cloudera.org:8080/#/c/20469/2/jenkins/build.sh@65 PS2, Line 65: popd > Something we could do to make sure we only push if all images succeeded is Done -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:12:04 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20469 to look at the new patch set (#3). Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb --- M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 6 files changed, 54 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/3 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20471 ) Change subject: IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28 .. IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28 In the change for IMPALA-11726, we added some usages of Guava's ImmutableMap.of() with >5 key/value pairs. This won't compile in older versions of Guava (like Guava 28) that only support up to 5 key/value pairs in ImmutableMap.of(). Being compatible with older versions of Guava makes it easier for Impala to match the Guava version of Hadoop, Hive, etc when compiling against older versions. This switches to using ImmutableMap.builder() to build the larger maps. Testing: - Built with Guava 28 and ran the tests Change-Id: Ic6d9c69fff749bcdb0887a6676574db9526f68e8 Reviewed-on: http://gerrit.cloudera.org:8080/20471 Reviewed-by: Michael Smith Tested-by: Michael Smith --- M fe/src/test/java/org/apache/impala/customcluster/LdapKerberosImpalaShellTest.java M fe/src/test/java/org/apache/impala/customcluster/LdapKerberosImpalaShellTestBase.java 2 files changed, 32 insertions(+), 32 deletions(-) Approvals: Michael Smith: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic6d9c69fff749bcdb0887a6676574db9526f68e8 Gerrit-Change-Number: 20471 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28
Michael Smith has removed a vote on this change. Change subject: IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28 .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ic6d9c69fff749bcdb0887a6676574db9526f68e8 Gerrit-Change-Number: 20471 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20471 ) Change subject: IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28 .. Patch Set 1: Verified+1 Code-Review+2 Makes sense to me. Jenkins failed on https://issues.apache.org/jira/browse/IMPALA-12266. -- To view, visit http://gerrit.cloudera.org:8080/20471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic6d9c69fff749bcdb0887a6676574db9526f68e8 Gerrit-Change-Number: 20471 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:01:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20471 ) Change subject: IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28 .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9698/ -- To view, visit http://gerrit.cloudera.org:8080/20471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic6d9c69fff749bcdb0887a6676574db9526f68e8 Gerrit-Change-Number: 20471 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Sep 2023 20:55:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20425 ) Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9701/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 20:42:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 19:49:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9700/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 19:49:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. IMPALA-12357: Skip scheduling bloom filter from full-build scan PK-FK join between a dimension table and a fact table is common occurrences in a query. Such join often does not involve any predicate filter in the dimension table. Thus, bloom filter value from this kind of dimension table scan (PK) will most likely to have all values from the fact table column (FK). It is ineffective to generate this filter because it is unlikely to reject any rows, especially if the bloom filter size is large and has high false positive probability (fpp) estimate. This patch skip scheduling bloom filter from join node that has this characteristics: 1. Build side is full table scan (has hard estimates). 2. The build scan does not have any predicate filter nor consume any runtime filter. 3. The join node is assumed to have PK-FK relationship. 4. The planned bloom filter has resulting fpp estimate higher than max_filter_error_rate_from_full_scan flag (default to 0.9). The fourth criteria is an additional control to eliminate based on fpp threshold because low fpp filter sometimes is still effective in eliminating rows (i.e., rows with NULL value). Non-bloom filters remain unchanged as they are relatively lighter to build and evaluate than bloom filter. Testing: - Add testcase in testBloomFilterAssignment - Pass core tests - Ran TPC-DS 3TB with following query options: * RUNTIME_FILTER_MIN_SIZE=8192 * RUNTIME_FILTER_MAX_SIZE=2097152 * MAX_NUM_RUNTIME_FILTERS=50 * RUNTIME_FILTER_WAIT_TIME_MS=1 19 out of 103 queries show reduction in number of runtime bloom filters without any notable performance regression. Change-Id: I494533bc06da84e606cbd1ae161908089a5e Reviewed-on: http://gerrit.cloudera.org:8080/20366 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test 12 files changed, 1,433 insertions(+), 37 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 19 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. Patch Set 18: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Sep 2023 19:17:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 6: Code-Review+2 This looks good to me -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 19:15:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13971/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 18:54:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: IMPALA-12441: Simplify local toolchain development .. Patch Set 5: (2 comments) > Patch Set 5: > > (2 comments) > > Please file a JIRA. Done. > > This simplifies the case where someone wants to > modify the toolchain. It can be checked out to the > appropriate hash, and then any future modifications > will take effect through buildall.sh. I think this > looks good. > > What this doesn't do is deal with this local > toolchain consistently if you switch branches. If > the IMPALA_TOOLCHAIN_COMMIT_HASH goes from x to y, > the toolchain checkout isn't touched and stays at x. > This is a use case we don't currently care about > much, so we could do that separately. Heading in > that direction gets us closer to Kudu's thirdparty > model where it just gets triggered and kept up to > date as part of the build. There are two conflict workflows here: - make local changes in native-toolchain that you want to test. There would be no remote branch to fetch and buildall would get in the way if it tried to switch to another commit hash. - switching branches, when you want to use a different toolchain commit. Most of the time if I switch branches I probably want to go back to using prebuilt toolchains, so you'd want to clear NATIVE_TOOLCHAIN_HOME and re-source impala-config.sh. I don't have a clear idea how to resolve those, so going to leave it alone. > > We also have a new toolchain since this upload. I believe I already resolved that conflict. http://gerrit.cloudera.org:8080/#/c/20267/5/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/20267/5/buildall.sh@433 PS5, Line 433: cd - > To make it explicit, should we use pushd/popd? Sure, this is the only place that used the 'cd -' pattern. http://gerrit.cloudera.org:8080/#/c/20267/5/buildall.sh@452 PS5, Line 452: cp $IMPALA_HOME/../hadoopAarch64NativeLibs/lib* $HADOOP_HOME/lib/native/ > A theory I have is that this can create problems for a running cluster. If Ack -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 18:28:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12441: Simplify local toolchain development
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20267 to look at the new patch set (#6). Change subject: IMPALA-12441: Simplify local toolchain development .. IMPALA-12441: Simplify local toolchain development If NATIVE_TOOLCHAIN_HOME is set, that will be used to provide the native toolchain instead of the default in IMPALA_TOOLCHAIN. Overrides IMPALA_TOOLCHAIN_PACKAGES_HOME and sets SKIP_TOOLCHAIN_BOOTSTRAP=true. Adds IMPALA_TOOLCHAIN_REPO, IMPALA_TOOLCHAIN_BRANCH, and IMPALA_TOOLCHAIN_COMMIT_HASH so everything is clear about what toolchain is used for this Impala commit. Also skips downloading Kudu if SKIP_TOOLCHAIN_BOOTSTRAP is true as Kudu is built from native-toolchain. Normalizes aarch64 logic, which skipped Kudu because it would always build native-toolchain locally. Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py M bin/impala-config.sh M buildall.sh 4 files changed, 40 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/20267/6 -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. IMPALA-12383: Fix SingleNodePlanner aggregation limits IMPALA-2581 added enforcement of the limit when adding entries to the grouping aggregation. It would stop adding new entries if the number of entries in the grouping aggregation was >= the limit. If the grouping aggregation never contains more entries than the limit, then it would not output more entries. However, this limit was not enforced exactly when adding. It would add a whole batch before checking the limit, so it can go past the limit. In practice the exchange in a distributed aggregation would enforce limits, so this would only show up when num_nodes=1. As a result, the following query incorrectly returns 16 rows, not 10: set num_nodes=1; select distinct l_orderkey from tpch.lineitem limit 10; One option is to be exact when adding items to the group aggregation, which would require testing the limit on each row (we don't know which are duplicates). This is awkward. Removing the limit on the output of the aggregation also is not really needed for the original change (stopping the children early once the limit is reached). Instead, we restore the limit on the output of the grouping agg (which is already known to work). Testing: - added a test case where we assert number of rows returned by an aggregation node (rather than an exchange or top-n). - restores definition of ALL_CLUSTER_SIZES and makes it simpler to enable for individual test suites. Filed IMPALA-12394 to generally re-enable testing with ALL_CLUSTER_SIZES. Enables ALL_CLUSTER_SIZES for aggregation tests. Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Reviewed-on: http://gerrit.cloudera.org:8080/20379 Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell --- M be/src/exec/aggregation-node-base.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/query_test/test_aggregation.py 6 files changed, 31 insertions(+), 22 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20425 ) Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13970/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 18:03:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12402: v1 change the term from consistency to concurrency and fix nits
Michael Smith has abandoned this change. ( http://gerrit.cloudera.org:8080/20467 ) Change subject: IMPALA-12402: v1 change the term from consistency to concurrency and fix nits .. Abandoned Merge into parent commit. -- To view, visit http://gerrit.cloudera.org:8080/20467 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I700a486aa3c1ff5ef27f2db636a59cb404ddbb8e Gerrit-Change-Number: 20467 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache
Michael Smith has abandoned this change. ( http://gerrit.cloudera.org:8080/20466 ) Change subject: IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache_ .. Abandoned Please use the Change-Id from changeset 20443. -- To view, visit http://gerrit.cloudera.org:8080/20466 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I9a8370e50db3f82234c3e0caae1d92522d054f91 Gerrit-Change-Number: 20466 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache
Michael Smith has abandoned this change. ( http://gerrit.cloudera.org:8080/20472 ) Change subject: IMPALA-12402: Add some configurations for CatalogdMetaProvider's cache_ .. Abandoned Please use the Change-Id from changeset 20443. -- To view, visit http://gerrit.cloudera.org:8080/20472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I9b6f6de7213eb0916c53ab7e03ba5f76796cef0c Gerrit-Change-Number: 20472 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12385: Enable Periodic metrics by default
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20377 ) Change subject: IMPALA-12385: Enable Periodic metrics by default .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 8 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Comment-Date: Mon, 11 Sep 2023 17:45:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12425: Upgrade json-smart to fix CVE
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20461 ) Change subject: IMPALA-12425: Upgrade json-smart to fix CVE .. IMPALA-12425: Upgrade json-smart to fix CVE Change-Id: Id0090da84e40cf79289ad13cce07127e64f08ad3 Reviewed-on: http://gerrit.cloudera.org:8080/20461 Tested-by: Impala Public Jenkins Reviewed-by: Laszlo Gaal --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Verified Laszlo Gaal: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id0090da84e40cf79289ad13cce07127e64f08ad3 Gerrit-Change-Number: 20461 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12385: Enable Periodic metrics by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20377 ) Change subject: IMPALA-12385: Enable Periodic metrics by default .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Comment-Date: Mon, 11 Sep 2023 17:45:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12385: Enable Periodic metrics by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20377 ) Change subject: IMPALA-12385: Enable Periodic metrics by default .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9699/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Comment-Date: Mon, 11 Sep 2023 17:45:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20425 ) Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/20425/4/fe/src/main/java/org/apache/impala/analysis/Expr.java File fe/src/main/java/org/apache/impala/analysis/Expr.java: http://gerrit.cloudera.org:8080/#/c/20425/4/fe/src/main/java/org/apache/impala/analysis/Expr.java@1576 PS4, Line 1576: ways returns a "real" (not wildcard) type. > This part is essentially the same as 'cast to the type both values can be a Done -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 17:37:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Hello Quanlong Huang, Daniel Becker, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20425 to look at the new patch set (#5). Change subject: IMPALA-10086: Implicit cast comparing char and varchar .. IMPALA-10086: Implicit cast comparing char and varchar Until IMPALA-7368, Impala allowed comparing char and varchar slots as in select * from functional.chars_tiny where cs = vc; IMPALA-7368 added DATE type support, and as part of that changed function call resolution to use a fit function based on the number of arguments that match the call types. Previously the comparison above would take the first matching function, which happened to be equality between STRING and STRING; CHAR and VARCHAR can both be implicitly cast to STRING, so this function worked. With the new function resolution, the best fit is equality between VARCHAR and VARCHAR, however implicit casting to VARCHAR(*) from CHAR wasn't allowed. The behavior before IMPALA-7368 was somewhat accidental; it depended on the order that builtin EQ functions are added via BinaryPredicate.initBuiltins -> Type.getSupportedTypes. Supported types happened to be ordered with STRING preceding VARCHAR and CHAR. The fit function makes sense and changing its behavior may have other consequences; it also makes sense that CHAR should be castable to VARCHAR. This change allows implicit cast between matching types. That includes equals, and adds casting to a compatible type when it is a char or varchar and the target type is a wildcard version of the same. Does not attempt to address differences from CHAR padding (IMPALA-1652). Testing: - Adds tests covering cast comparison and other implicit conversions. - Passed exhaustive test run. Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 --- M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M tests/query_test/test_cast_with_format.py 3 files changed, 71 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/20425/5 -- To view, visit http://gerrit.cloudera.org:8080/20425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436 Gerrit-Change-Number: 20425 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [tools] Simplify local toolchain development
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20267 ) Change subject: [tools] Simplify local toolchain development .. Patch Set 5: (2 comments) Please file a JIRA. This simplifies the case where someone wants to modify the toolchain. It can be checked out to the appropriate hash, and then any future modifications will take effect through buildall.sh. I think this looks good. What this doesn't do is deal with this local toolchain consistently if you switch branches. If the IMPALA_TOOLCHAIN_COMMIT_HASH goes from x to y, the toolchain checkout isn't touched and stays at x. This is a use case we don't currently care about much, so we could do that separately. Heading in that direction gets us closer to Kudu's thirdparty model where it just gets triggered and kept up to date as part of the build. We also have a new toolchain since this upload. http://gerrit.cloudera.org:8080/#/c/20267/5/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/20267/5/buildall.sh@433 PS5, Line 433: cd - To make it explicit, should we use pushd/popd? http://gerrit.cloudera.org:8080/#/c/20267/5/buildall.sh@452 PS5, Line 452: cp $IMPALA_HOME/../hadoopAarch64NativeLibs/lib* $HADOOP_HOME/lib/native/ A theory I have is that this can create problems for a running cluster. If something is running and occasionally accessing these native libraries, this creates a window where they are being overwritten/modified. Not your change, so we can think about this separately. -- To view, visit http://gerrit.cloudera.org:8080/20267 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a9e51b7f54c738d8cc01b32428ac88a344de376 Gerrit-Change-Number: 20267 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 17:28:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 16:59:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12390 (part 2): Enable some clang-tidy performance related checks
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20445 ) Change subject: IMPALA-12390 (part 2): Enable some clang-tidy performance related checks .. Patch Set 4: Now that https://gerrit.cloudera.org/#/c/20387/ has merged, please rebase on top of it. Then also do the .clang-tidy change that Michael mentioned to enable this by default. There is one usability thing that I would like to add to this change. Since we are mostly solving this using StrCat(), it would be nice for our clang-tidy output to mention it if performance-inefficient-string-concatenation is mentioned. I would preserve the original Clang message and append an Impala specific note saying that Impala code can also solve the issue using the StrCat() function from be/src/gutil/strings/strcat.h. One way to do that is to pipe output through this: sed '/performance-inefficient-string-concatenation/ s#$# (Impala-specific note: This can also be fixed using the StrCat() function from be/src/gutil/strings/strcat.h)#' It matches the performance-inefficient-string-concatenation lines and replaces the end of line with our message. We'll have to figure out the right part of bin/run_clang_tidy.sh to modify. -- To view, visit http://gerrit.cloudera.org:8080/20445 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibad8bd0f12aab92ad874f5a6b9ec922dce7f3190 Gerrit-Change-Number: 20445 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 16:58:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20471 ) Change subject: IMPALA-12432: Make LdapKerberosImpalaShellTest* work with Guava 28 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9698/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic6d9c69fff749bcdb0887a6676574db9526f68e8 Gerrit-Change-Number: 20471 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Sep 2023 16:31:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11284: Do non-optional rewrites for || and Between predicate
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/18581 )
Change subject: IMPALA-11284: Do non-optional rewrites for || and Between
predicate
..
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18581/9/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
File fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java:
http://gerrit.cloudera.org:8080/#/c/18581/9/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@439
PS9, Line 439: private static boolean
isRequireMandatoryRewrite(List operands) {
> In the version in PS7, 'equireMandatoryRewrite_' could be reset to false if
Looking again, I think requireMandatoryRewrite_ should not be reset like it did
in PS7.
Assignment in PS9 is the right one. It will enable mandatory rewrites for all
operands even if only one of them has CompoundVerticalBarExpr or
BetweenPredicate. The rewrites will become a no-op for operand that does not
require it.
--
To view, visit http://gerrit.cloudera.org:8080/18581
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I99b8b33bf6468d12b9e26f0a6e744feb7072619c
Gerrit-Change-Number: 18581
Gerrit-PatchSet: 9
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 11 Sep 2023 15:02:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9697/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Sep 2023 14:48:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 18 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Sep 2023 14:48:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20405 ) Change subject: IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13969/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 Gerrit-Change-Number: 20405 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Sep 2023 14:21:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
Noemi Pap-Takacs has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20405 ) Change subject: IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE .. IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE If an Iceberg table is frequently updated/written to in small batches, a lot of small files are created. This decreases read performance. Similarly, frequent row-level deletes contribute to this problem by creating delete files which have to be merged on read. So far INSERT OVERWRITE (rewriting the table with itself) has been used to compact Iceberg tables. However, it comes with some RESTRICTIONS: - The table should not have multiple partition specs/partition evolution. - The table should not contain complex types. The OPTIMIZE statement offers a new syntax and an Iceberg specific solution to this problem. This patch introduces the new syntax as an alias for INSERT OVERWRITE as a temporary solution. See IMPALA-12293 for further information. Note that executing OPTIMIZE TABLE requires ALL privileges. Testing: - negative tests - FE planner test - Ranger test - E2E tests Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java A fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/jflex/sql-scanner.flex M testdata/workloads/functional-planner/queries/PlannerTest/insert-sort-by-zorder.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M tests/query_test/test_iceberg.py 12 files changed, 353 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/20405/3 -- To view, visit http://gerrit.cloudera.org:8080/20405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 Gerrit-Change-Number: 20405 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20405 ) Change subject: IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20405/3/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/20405/3/tests/query_test/test_iceberg.py@1300 PS3, Line 1300: t flake8: E122 continuation line missing indentation or outdented -- To view, visit http://gerrit.cloudera.org:8080/20405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 Gerrit-Change-Number: 20405 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Sep 2023 13:55:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12430: Skip compression when sending row batches within same process
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20462 ) Change subject: IMPALA-12430: Skip compression when sending row batches within same process .. IMPALA-12430: Skip compression when sending row batches within same process LZ4 compression doesn't seem useful when the RowBatch is sent to a fragment instance within the same process instead of a remote host. After this change KrpcDataStreamSender skips compression for channels where the destination is in the same process. Other changes: - OutboundRowBatch is moved to a separate file to make the commonly included row-batch.h lighter. - TestObservability.test_global_exchange_counters had to be changed as skipping compression changed metric ExchangeScanRatio. Also added a sleep to the test query because it was flaky on my machine (it doesn't seem flaky in jenkins runs, probably my CPU is faster). See the Jira for more details on tasks that could be skipped in intra process RowBatch transfer. From these compression is both the most expensive and easiest to avoid. Note that it may also make sense to skip compression if the target is not the in same process but resides on the same host. This setup is not typical in production environment AFAIK and it would complicate testing compression as impalad processes often run on the same host during tests. For these reasons it seems better to only implement this if both the host and port are the same. TPCH benchmark shows significant improvement but it uses only 3 impalad processes so 1/3 of exchanges are affected - in bigger clusters the change should be much smaller. +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(42) | parquet / none / none | 3.59| -4.95% | 2.37 | -2.51% | +--+---+-++++ Change-Id: I7ea23fd1f0f10f72f3dbd8594f3def3ee190230a Reviewed-on: http://gerrit.cloudera.org:8080/20462 Tested-by: Impala Public Jenkins Reviewed-by: Daniel Becker --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h A be/src/runtime/outbound-row-batch.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M tests/query_test/test_observability.py 8 files changed, 128 insertions(+), 65 deletions(-) Approvals: Impala Public Jenkins: Verified Daniel Becker: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7ea23fd1f0f10f72f3dbd8594f3def3ee190230a Gerrit-Change-Number: 20462 Gerrit-PatchSet: 8 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler
[Impala-ASF-CR](asf-site) IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20403 ) Change subject: IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I93083b999ec404ca3f407977b8e930b85a53b695 Gerrit-Change-Number: 20403 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 13:01:34 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20403 ) Change subject: IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo .. Patch Set 1: > Patch Set 1: Code-Review+1 > > Thanks Quanlong! > Did we use the deleted files at all before? Thank Daniel! I can only see they are added in https://github.com/apache/impala/commit/5661bfcc6c037a273584828c6ae91c3b072a284d I can't find any usage on them. -- To view, visit http://gerrit.cloudera.org:8080/20403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I93083b999ec404ca3f407977b8e930b85a53b695 Gerrit-Change-Number: 20403 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 12:57:38 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20403 ) Change subject: IMPALA-12055: (Addendum) Use bootstrap.min.js from local repo .. Patch Set 1: Code-Review+1 Thanks Quanlong! Did we use the deleted files at all before? -- To view, visit http://gerrit.cloudera.org:8080/20403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I93083b999ec404ca3f407977b8e930b85a53b695 Gerrit-Change-Number: 20403 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Sep 2023 12:12:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12152: Add query option to wait for events sync up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20131 ) Change subject: IMPALA-12152: Add query option to wait for events sync up .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13968/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36ac941bb2c2217b09fcfa2eb567b011b38efa2a Gerrit-Change-Number: 20131 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 11 Sep 2023 11:29:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12371: Add better cardinality estimation for Iceberg V2 tables with deletes
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20460 ) Change subject: IMPALA-12371: Add better cardinality estimation for Iceberg V2 tables with deletes .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/20460/2/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java File fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java: http://gerrit.cloudera.org:8080/#/c/20460/2/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java@98 PS2, Line 98: is not zero > If there are input files to scan, leftCard will always be greater than 0, s Thanks, I like the idea about a precondition check and using 1 in the Math.math() call. http://gerrit.cloudera.org:8080/#/c/20460/2/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java@100 PS2, Line 100: leftCard > Yes. Thanks. It may be worth mentioning it either here or in the commit message. -- To view, visit http://gerrit.cloudera.org:8080/20460 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I988dc8d7e1074932c460b3702d3381341e5b23c5 Gerrit-Change-Number: 20460 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Sep 2023 11:19:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10086: Implicit cast comparing char and varchar
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/20425 )
Change subject: IMPALA-10086: Implicit cast comparing char and varchar
..
Patch Set 4:
(1 comment)
Thanks Michael.
http://gerrit.cloudera.org:8080/#/c/20425/4/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/20425/4/fe/src/main/java/org/apache/impala/analysis/Expr.java@1576
PS4, Line 1576: Requested cast must be to assignment-compatible type
This part is essentially the same as 'cast to the type both values can be
assigned to', so we could delete it and move the parenthesised part ('which
implies no loss of precision') to the sentence above.
--
To view, visit http://gerrit.cloudera.org:8080/20425
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib89d0a391bc8f2152ecd9151c8872a01ba19c436
Gerrit-Change-Number: 20425
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Mon, 11 Sep 2023 11:15:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11284: Do non-optional rewrites for || and Between predicate
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/18581 )
Change subject: IMPALA-11284: Do non-optional rewrites for || and Between
predicate
..
Patch Set 9:
(1 comment)
Thanks Riza. I just have a question, I can +2 after that if nobody else wants
to review it.
http://gerrit.cloudera.org:8080/#/c/18581/9/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
File fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java:
http://gerrit.cloudera.org:8080/#/c/18581/9/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java@439
PS9, Line 439: private static boolean
isRequireMandatoryRewrite(List operands) {
In the version in PS7, 'equireMandatoryRewrite_' could be reset to false if the
'resultExpr' of some operand didn't contain || or BETWEEN. In this version it
can't be reset. Was it a bug in the older version?
--
To view, visit http://gerrit.cloudera.org:8080/18581
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I99b8b33bf6468d12b9e26f0a6e744feb7072619c
Gerrit-Change-Number: 18581
Gerrit-PatchSet: 9
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 11 Sep 2023 11:10:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12152: Add query option to wait for events sync up
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20131 )
Change subject: IMPALA-12152: Add query option to wait for events sync up
..
Patch Set 13:
(10 comments)
Thanks for the review! I'll upload the next patch set when I get more cycles.
http://gerrit.cloudera.org:8080/#/c/20131/12//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20131/12//COMMIT_MSG@23
PS12, Line 23: Note that the current implementation waits for the latest event
id when
: the WaitForHmsEvent RPC is received at catalogd side. We can
improve it
: once HIVE-27499 is resolved, so we can efficiently detect
whether some
: given tables have unprocessed events and just wait for the
*largest* id
: of them. Tables that have no unprocessed events don't need to
block the
: query planning.
> Are there plans on how to implement this from Impala side?
Yeah, the current patch is a simple implementation without doing any analysis.
In the future we can invoke some FE logic to get the candidate table list, e.g.
by collectTableCandidates():
https://github.com/apache/impala/blob/715af9af42e86a3a1f31061ed2d41ed425accd0e/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java#L350
I planned to do this in a following patch soon. But didn't expect the review of
this initial patch took so long..
http://gerrit.cloudera.org:8080/#/c/20131/12/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/20131/12/be/src/catalog/catalog-server.cc@335
PS12, Line 335: }
> Please make sure you don't output sensitive data with ThriftDebugString().
Ack
http://gerrit.cloudera.org:8080/#/c/20131/12/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/20131/12/be/src/service/client-request-state.h@468
PS12, Line 468: SetEarlyErr
> The name could indicate that this should be used early in the life cycle of
Done
http://gerrit.cloudera.org:8080/#/c/20131/12/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/20131/12/be/src/service/impala-server.h@445
PS12, Line 445: RuntimeProfile::EventSequence* timeline);
> nit: catches up to?
Done
http://gerrit.cloudera.org:8080/#/c/20131/12/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:
http://gerrit.cloudera.org:8080/#/c/20131/12/common/thrift/CatalogService.thrift@691
PS12, Line 691: Status.TStatus status
> I see that it is a common pattern to use it like this, but why don't we add
There is an ancient TODO for this at L50. I think we can revisit and do it in a
separate JIRA.
http://gerrit.cloudera.org:8080/#/c/20131/12/common/thrift/CatalogService.thrift@751
PS12, Line 751: UpdateTableUsa
> High level question:
This is really a good idea! I like the point that it considers catalog server
changes in the background.
However, there is a trap that we also need to consider. When the last synced
event id reaches the target, the new table might not be collected in the
catalog update due to the skipping logic, i.e. if other concurrent DDLs are
holding the table lock, the catalog update collector thread will skip that
table after waiting for some time.
https://github.com/apache/impala/blob/f563cce6b8eed4519772af06d89edc6e374839d3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L1376
https://github.com/apache/impala/blob/f563cce6b8eed4519772af06d89edc6e374839d3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L1440
I then realize that the current patch also has this issue. When coordinator
receives the catalog update from statestore that reaches the desired catalog
version, it doesn't mean the tables that it's waiting for all arrive.
DDL/DML RPCs return the updated db/table metadata directly in response so they
don't have this issue. We need this new RPC to do the same. I'll address this
in the next patch set.
For query cancellation, currently DDL/DMLs also have the issue that when they
are in CREATED state, they can't be cancelled. I think we can address them
together by using separate thread to do the RPC, and do it in a separate JIRA.
http://gerrit.cloudera.org:8080/#/c/20131/12/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/20131/12/common/thrift/Query.thrift@668
PS12, Line 668: sync_hms_events_strict_mode
> I didn't find any test for this.
There is a test in
tests/custom_cluster/test_events_custom_configs.py::test_hms_event_sync_with_event_processing_disabled
https://gerrit.cloudera.org/c/20131/12/tests/custom_cluster/test_events_custom_configs.py#962
# Verify failures in strict mode
err = self.execute_query_expect_failure(
self.client, query,
{"sync_hms_events_wait_time_s": 60,
