[Impala-ASF-CR] WIP IMPALA-12156: Support High Availability for Statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: WIP IMPALA-12156: Support High Availability for Statestore .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13798/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Aug 2023 05:09:44 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12156: Support High Availability for Statestore
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20372 ) Change subject: WIP IMPALA-12156: Support High Availability for Statestore .. WIP IMPALA-12156: Support High Availability for Statestore To support statestore HA, we allow two statestored instances in an Active-Passive HA pair to be added to an Impala cluster. We add the preemptive behavior for statestored. When HA is enabled, the preemptive behavior allows the statestored with the higher priority to become active and the paired statestored becomes standby. The active statestored acts as the owner of Impala cluster and provides statestore service for the cluster members. To enable catalog HA for a cluster, two statestoreds in the HA pair and all subscribers must be started with starting flag "enable_statestored_ha". - Define new service for Statestore HA. - Negotiate role for HA with peer statestore instance on startup. - Create HA monitor thread: active statestored send heartbeat to standby statestored. standby statestored monitor peer's connection states - Standby statestored send heartbeat to subscribers with request for connection state between active statestore and subscribers. Standby statestored save the connection state as failure detecer. - When standby statestored loss connection with active statestore, check the connection states for active statestore, and take over active role if majority of subscribers lose connections with active statestore. - New active statestored send RPC notification to all subscribers for new active statestored and active catalogd elected by new active staetstored. - New active statestored start to send heartbeat to its peer when it receive handshake from its peer. - Subscriber register to two statestoreds. - Subscriber report connection state for in-active statestore. - Subscriber switch to new active statstore, refuse topic update from standby statestored. Testings: TODO Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/catalog/catalog-server.cc M be/src/common/global-flags.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admissiond-env.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-catalog.cc M be/src/statestore/statestore-subscriber-catalog.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore-test.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/statestore/statestored-main.cc M bin/start-impala-cluster.py M common/thrift/StatestoreService.thrift M common/thrift/metrics.json M tests/common/impala_cluster.py M tests/common/impala_service.py 21 files changed, 1,487 insertions(+), 87 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20372/2 -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] WIP IMPALA-12156: Support High Availability for Statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: WIP IMPALA-12156: Support High Availability for Statestore .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/20372/2/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/20372/2/bin/start-impala-cluster.py@355 PS2, Line 355: def statestored_service_name(i): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/20372/2/bin/start-impala-cluster.py@791 PS2, Line 791: D flake8: F602 dictionary key variable DEFAULT_STATESTORE_HA_SERVICE_PORT repeated with different values http://gerrit.cloudera.org:8080/#/c/20372/2/bin/start-impala-cluster.py@792 PS2, Line 792: D flake8: F602 dictionary key variable DEFAULT_STATESTORE_HA_SERVICE_PORT repeated with different values -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Aug 2023 04:45:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 6: Code-Review+2 Setting proper mem_limit_coordinators as the original value was too small for the testcase. -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Aug 2023 04:25:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Abhishek Rawat has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. IMPALA-10860: Allow setting mem_limit for coordinators Added support for MEM_LIMIT_COORDINATORS query option. This is similar to exisiting MEM_LIMIT_EXECUTORS, but applies to coordinators. There are cases where Planner generates inaccurate estimates for coordinator fragments and would be good to be able to set mem limit just for the coordinator, since a query's memory requirement on coordinator tends to be much lower compared to that on executors. If MEM_LIMIT is set, then MEM_LIMIT_COORDINATORS is ignored. Also updated the documentation for the new query option. Testing: - Added new custom cluster tests which validates MEM_LIMIT_COORDINATORS applies only on coordinator. The test also validates that both MEM_LIMIT_EXECUTORS and MEM_LIMIT_COORDINATORS can be set together. - Built docs and made sure that the new changes have proper formatting. Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c --- M be/src/scheduling/schedule-state.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M docs/topics/impala_mem_limit.xml M tests/custom_cluster/test_admission_controller.py 8 files changed, 90 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/20378/6 -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9611/ -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Aug 2023 03:50:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9610/ -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 22 Aug 2023 03:49:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20368 ) Change subject: IMPALA-12024: Add catalog profile for createTable .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13797/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa Gerrit-Change-Number: 20368 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 22 Aug 2023 03:19:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20368 )
Change subject: IMPALA-12024: Add catalog profile for createTable
..
Patch Set 3:
(12 comments)
> Patch Set 3:
>
> (12 comments)
>
> I think it is important to make the event label self explanatory and as clear
> as possible. Otherwise, an explanation should be documented somewhere for
> each label.
>
> Some general comments:
> - Replace HMS with Metastore for consistency.
> - Consider turning some duplicated event label into string constant.
> - Clarify the source of event. Is it Metastore vs CatalogD vs Filesystem vs
> other service (like Kudu master).
Thanks for the suggestions!
Refactored the patch to use string constants. I was intended to write the
labels differently by strings to get rid of multiple hits when searching the
code. But it makes sense to normalize them as constants so we don't need to
document them.
Also clarified the sources. BTW, use "Kudu" instead of "Kudu master" since we
treate Kudu as a system and can't distinguish what are created in Kudu master
and what in Kudu tablet servers.
I plan to add more tests but need to figure out a clean way of verifying the
EventSequence. Found an existing bug of the JSON profile by the way:
IMPALA-12391
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3523
PS3, Line 3523: catalogTimeline.markEvent("Created Metastore
table");
: } else {
: catalogTimeline.markEvent("Found table exists");
> Maybe reorganize this to 2 things: existence check and new table creation:
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3552
PS3, Line 3552: Created catalog table
> nit: "Cached table in catalogd"?
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3610
PS3, Line 3610: Created Metastore table
> nit: "Created table in Metastore"?
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3627
PS3, Line 3627: HMS
> nit: "Fetched table from Metastore"?
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3656
PS3, Line 3656: Created catalog table
> nit: "Cached table in catalog"?
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3702
PS3, Line 3702: HMS
> nit: Metastore
Yeah, this one is different than other fetches since it just use the createTime
and not caching other things.
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3786
PS3, Line 3786: Created iceberg table
> nit: created in where? HMS, filesystem, or CatalogD cache?
It depends on what iceberg catalog is used. Hard to say it's just HMS or
filesystem..
This label generally means the invocation on iceberg createTable API finished.
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3827
PS3, Line 3827: Created Metastore table
> nit: dulicated String. Change to constant?
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3834
PS3, Line 3834: Found table exists
> nit: dulicated String. Change to constant?
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@102
PS3, Line 102: Checked table exists
> nit: Checked table existence
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@342
PS3, Line 342: Checked table exists
> nit: Checked table existence
Done
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@369
PS3, Line 369: cols
> nit: columns
Done
--
To view, visit http://gerrit.cloudera.org:8080/20368
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa
Gerrit-Change-Number: 20368
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Tue, 22 Aug 2023 02:54:37 +
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Hello Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20368 to look at the new patch set (#4). Change subject: IMPALA-12024: Add catalog profile for createTable .. IMPALA-12024: Add catalog profile for createTable DDL/DMLs might spend a long time in the execution in catalogd. Currently, the profiles just have a counter of CatalogOpExecTimer. We need more items to show the details of how they are executed in catalogd. As the first step, this patch adds the profile timeline for how the createTable DDL is executed in Catalog Server. Also added more events in the existing "Query Timeline" for when the CatalogDdlRequest finished and when the catalog updates are applied. To implement this, a 'profile' field is added in TDdlExecResponse to carry the execution counters and timeline in catalogd. Currently, we just uses the timeline of it. We can add more counters in the future. Several methods add a timeline parameter to mark the progress in them. Timeline events are added after each RPC finished. Here is an example when HMS is hanging for 26s in a CTAS. I used gdb to attach to HMS as the JIRA description mentioned. In the timeline, we can see the time is spent in the first HMS RPC that fetching the current HMS event id: Catalog Server Operation: 26s560ms - Got metastoreDdlLock: 163.977us (163.977us) - Got Metastore client: 166.339us (2.362us) - Got current Metastore event id 8355270: 26s494ms (26s494ms) - Created table in Metastore: 26s558ms (63.507ms) - Fetched event batch from Metastore: 26s559ms (1.155ms) - Created table in catalog cache: 26s560ms (1.164ms) - DDL finished: 26s560ms (65.538us) Query Compilation: 164.257ms - Metadata of all 1 tables cached: 10.535ms (10.535ms) - Analysis finished: 118.324ms (107.788ms) - Authorization finished (noop): 118.489ms (164.626us) - Value transfer graph computed: 118.830ms (341.792us) - Single node plan created: 150.150ms (31.319ms) - Runtime filters computed: 150.254ms (103.529us) - Distributed plan created: 151.832ms (1.578ms) - Planning finished: 164.257ms (12.425ms) Query Timeline: 27s304ms - Query submitted: 129.658us (129.658us) - Planning finished: 170.224ms (170.095ms) - CatalogDdlRequest finished: 26s731ms (26s561ms) - Applied catalog updates from DDL: 26s740ms (8.752ms) - Submit for admission: 26s740ms (22.233us) - Completed admission: 26s740ms (286.295us) - Ready to start on 3 backends: 26s740ms (155.916us) - All 3 execution backends (3 fragment instances) started: 26s751ms (10.864ms) - Last row fetched: 26s920ms (168.226ms) - Released admission control resources: 26s920ms (27.635us) - DML data written: 26s920ms (126.369us) - Applied catalog updates from DDL: 26s985ms (65.354ms) - DML Metastore update finished: 26s985ms (30.343us) - Rows available: 26s985ms (27.050us) - Unregister query: 27s304ms (318.661ms) An example of creating a Kudu table: Catalog Server Operation: 1s730ms - Got Metastore client: 113.403us (113.403us) - Got current Metastore event id 8355276: 974.500us (861.097us) - Got Kudu client: 212.123ms (211.148ms) - Got kuduDdlLock: 212.128ms (4.680us) - Checked table existence in Kudu: 850.786ms (638.658ms) - Created table in Kudu: 1s623ms (772.379ms) - Got metastoreDdlLock: 1s623ms (397.305us) - Got Metastore client: 1s623ms (7.813us) - Checked table existence in Metastore: 1s648ms (25.154ms) - Created table in Metastore: 1s725ms (76.348ms) - Fetched event batch from Metastore: 1s728ms (3.004ms) - Created table in catalog cache: 1s730ms (2.024ms) - DDL finished: 1s730ms (84.448us) An example of creating an Iceberg table: Catalog Server Operation: 1s573ms - Got Metastore client: 141.799us (141.799us) - Checked table existence in Metastore: 2.957ms (2.815ms) - Got current Metastore event id 8355277: 3.669ms (712.475us) - Created table using Iceberg Catalog HIVE_CATALOG: 1s379ms (1s375ms) - Fetched event batch from Metastore: 1s381ms (2.188ms) - Created table in catalog cache: 1s382ms (1.556ms) - Set Iceberg table owner in Metastore: 1s573ms (190.262ms) - DDL finished: 1s573ms (59.176us) Tests: - Add e2e tests to verify the DDL timeline events exist in profile Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa --- M be/src/service/client-request-state.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M
[Impala-ASF-CR] IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20367 ) Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13796/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 22 Aug 2023 02:02:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20367 to look at the new patch set (#2). Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients .. IMPALA-10976: Sync db/table in catalogD to latest HMS event id for all DDLS from Impala clients The idea is that when any DDL operation is performed by Impala, it also syncs the db/table to its latest event ID as per HMS. This way updates to a db/table's are applied in the same order as they appear in the Notification log table in HMS which ensures consistency. Currently catalogD applies any updates received from Impala clients in-place. Instead it should perform an HMS operation first and then replay all the HMS events since the last synced event id. Implementation: when the enable_sync_to_latest_event_on_ddls flag is set to true, we fetch the latest snapshot of db/table and update it in the catalogD's cache. Once HIVE-27499 is implemented, we can update the snapshot only if there are any pending events. Currently, there is no efficient way to identify if there are pending events for a db/table. Set enable_sync_to_latest_event_on_ddls to true to use this feature. Testing: Added few tests in the MetaStoreEventProcessorForTest to verify this feature. Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 149 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/20367/2 -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 21 Aug 2023 23:49:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13795/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 23:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9611/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 23:40:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 23:40:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9610/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 23:22:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20386 Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir The default behavior for deleting files on Hadoop is to move them to a trash folder. The trash folder can be aged out, but Impala's developer environment sets the trash to live forever. This is a problem, because the trash contents will continue to accumulate. This changes HadoopFsCommandLineClient's delete_file_dir to use -skipTrash to avoid accumulating the trash for this case. It also modifies DelegatingHdfsClient to use HadoopFsCommandLineClient for delete_file_dir. The WebHDFS client doesn't have the option to skip trash. Testing: - Ran tests locally and examined the trash directory Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 --- M tests/util/hdfs_util.py 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/1 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. Patch Set 1: (1 comment) Great to see you again, Fredy :) http://gerrit.cloudera.org:8080/#/c/20380/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20380/1//COMMIT_MSG@19 PS1, Line 19: necesary nit: typo 'necessary' -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 21 Aug 2023 22:04:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13794/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 21:34:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13793/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 21:33:52 + Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20380/1/.devcontainer/Dockerfile File .devcontainer/Dockerfile: http://gerrit.cloudera.org:8080/#/c/20380/1/.devcontainer/Dockerfile@1 PS1, Line 1: FROM ubuntu:18.04 Could this be done on 20.04? That's where we're moving most testing. -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 21 Aug 2023 21:15:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 4: (2 comments) Used https://gerrit.cloudera.org/c/17821/18/testdata/workloads/targeted-perf/queries/aggregation.test to verify that FastLimitCheck still happens for DistributedPlanner. http://gerrit.cloudera.org:8080/#/c/20379/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java File fe/src/main/java/org/apache/impala/planner/AggregationNode.java: http://gerrit.cloudera.org:8080/#/c/20379/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@85 PS1, Line 85: endsMultiPhase_ = f > It's not. useStreamingPreagg is only set on the pre-aggregation step. Looked into this a bit more. Most of those paths don't use pushdown because they have conjuncts and/or aggregate expressions. Mainly seems to affect simple select distinct/group by queries. I added one other case to the test, but it wasn't failing before. http://gerrit.cloudera.org:8080/#/c/20379/3/tests/query_test/test_aggregation.py File tests/query_test/test_aggregation.py: http://gerrit.cloudera.org:8080/#/c/20379/3/tests/query_test/test_aggregation.py@405 PS3, Line 405: ( > flake8: E131 continuation line unaligned for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 21:14:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Hello Quanlong Huang, Qifan Chen, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20379 to look at the new patch set (#4). Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. IMPALA-12383: Fix SingleNodePlanner aggregation limits When IMPALA-2581 was implemented, it assumed all aggregation nodes would have a pre-aggregation step that limits could be pushed to. That's not the case when using SingleNodePlanner, such as when num_nodes=1. As a result, the following query would incorrectly return 16 rows, not 10: set num_nodes=1; select distinct l_orderkey from tpch.lineitem limit 10; This fix identifies all aggregation nodes that use pre-aggregation so we use fast_limit_check in only those cases. Testing: - added a test case where we assert number of rows returned by an aggregation node (rather than an exchange or top-n). - restores running with num_nodes=0 and num_nodes=1 for default test dimensions; IMPALA-561 was fixed ages ago. Disables runtime filter tests with num_nodes=1 due to finstance expectations. Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M tests/common/test_dimensions.py M tests/query_test/test_aggregation.py M tests/query_test/test_runtime_filters.py 6 files changed, 39 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/20379/4 -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Hello Quanlong Huang, Qifan Chen, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20379 to look at the new patch set (#3). Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. IMPALA-12383: Fix SingleNodePlanner aggregation limits When IMPALA-2581 was implemented, it assumed all aggregation nodes would have a pre-aggregation step that limits could be pushed to. That's not the case when using SingleNodePlanner, such as when num_nodes=1. As a result, the following query would incorrectly return 16 rows, not 10: set num_nodes=1; select distinct l_orderkey from tpch.lineitem limit 10; This fix identifies all aggregation nodes that use pre-aggregation so we use fast_limit_check in only those cases. Testing: - added a test case where we assert number of rows returned by an aggregation node (rather than an exchange or top-n). - restores running with num_nodes=0 and num_nodes=1 for default test dimensions; IMPALA-561 was fixed ages ago. Disables runtime filter tests with num_nodes=1 due to finstance expectations. Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M tests/common/test_dimensions.py M tests/query_test/test_aggregation.py M tests/query_test/test_runtime_filters.py 6 files changed, 39 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/20379/3 -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20379/3/tests/query_test/test_aggregation.py File tests/query_test/test_aggregation.py: http://gerrit.cloudera.org:8080/#/c/20379/3/tests/query_test/test_aggregation.py@405 PS3, Line 405: " flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 21:08:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 4: Code-Review+2 LGTM! -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 21:00:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/20378/3/be/src/service/query-options.h File be/src/service/query-options.h: http://gerrit.cloudera.org:8080/#/c/20378/3/be/src/service/query-options.h@308 PS3, Line 308: > nit: right align this backslash? Done -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 20:54:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Abhishek Rawat has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. IMPALA-10860: Allow setting mem_limit for coordinators Added support for MEM_LIMIT_COORDINATORS query option. This is similar to exisiting MEM_LIMIT_EXECUTORS, but applies to coordinators. There are cases where Planner generates inaccurate estimates for coordinator fragments and would be good to be able to set mem limit just for the coordinator, since a query's memory requirement on coordinator tends to be much lower compared to that on executors. If MEM_LIMIT is set, then MEM_LIMIT_COORDINATORS is ignored. Also updated the documentation for the new query option. Testing: - Added new custom cluster tests which validates MEM_LIMIT_COORDINATORS applies only on coordinator. The test also validates that both MEM_LIMIT_EXECUTORS and MEM_LIMIT_COORDINATORS can be set together. - Built docs and made sure that the new changes have proper formatting. Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c --- M be/src/scheduling/schedule-state.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M docs/topics/impala_mem_limit.xml M tests/custom_cluster/test_admission_controller.py 8 files changed, 90 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/20378/4 -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock Previous to this patch we tried to load table iceberg_lineitem_multiblock with HDFS block size 524288. This failed in builds that use HDFS erasure coding which requires block size at least 1048576. This patch increases the block size to 1048576. This also triggers the bug that was fixed by IMPALA-12327. But to have more tests with multiblock tables this patch also adds table iceberg_lineitem_sixblocks and few tests with different MT_DOP settings. Testing: * tested in build with HDFS EC Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Reviewed-on: http://gerrit.cloudera.org:8080/20359 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test 3 files changed, 50 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 21 Aug 2023 20:53:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13792/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 19:30:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/20379/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20379/1//COMMIT_MSG@12 PS1, Line 12: i > would incorrectly return Done http://gerrit.cloudera.org:8080/#/c/20379/1//COMMIT_MSG@11 PS1, Line 11: As a : result, > nit. As a result, Done http://gerrit.cloudera.org:8080/#/c/20379/1//COMMIT_MSG@17 PS1, Line 17: This fix i > nit. This fix identifies Done -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 19:05:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Hello Quanlong Huang, Qifan Chen, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20379 to look at the new patch set (#2). Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. IMPALA-12383: Fix SingleNodePlanner aggregation limits When IMPALA-2581 was implemented, it assumed all aggregation nodes would have a pre-aggregation step that limits could be pushed to. That's not the case when using SingleNodePlanner, such as when num_nodes=1. As a result, the following query would incorrectly return 16 rows, not 10: set num_nodes=1; select distinct l_orderkey from tpch.lineitem limit 10; This fix identifies all aggregation nodes that use pre-aggregation so we use fast_limit_check in only those cases. Testing: - added a test case where we assert number of rows returned by an aggregation node (rather than an exchange or top-n). - restores running with num_nodes=0 and num_nodes=1 for default test dimensions; IMPALA-561 was fixed ages ago. Disables runtime filter tests with num_nodes=1 due to finstance expectations. Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 --- M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M tests/common/test_dimensions.py M tests/query_test/test_aggregation.py M tests/query_test/test_runtime_filters.py 5 files changed, 30 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/20379/2 -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20379/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java File fe/src/main/java/org/apache/impala/planner/AggregationNode.java: http://gerrit.cloudera.org:8080/#/c/20379/1/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@85 PS1, Line 85: hasStreamingPreagg_ > I wonder if this new variable is logically equivalent to the following It's not. useStreamingPreagg is only set on the pre-aggregation step. All these flags are difficult to reason about though. I'm going to spend some time with them; there are 3 places we create aggregation nodes: - SingleNodePlanner#createAggregationPlan, which can add distinct and transpose phases - DistributedPlanner#createMergeAggregationFragment - DistributedPlanner#createPhase2DistinctAggregationFragment I think I need to understand how to generate all these and make sure we have test coverage. Maybe I can come up with a clearer way to describe the different use cases. -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 17:32:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20368 )
Change subject: IMPALA-12024: Add catalog profile for createTable
..
Patch Set 3:
(12 comments)
I think it is important to make the event label self explanatory and as clear
as possible. Otherwise, an explanation should be documented somewhere for each
label.
Some general comments:
- Replace HMS with Metastore for consistency.
- Consider turning some duplicated event label into string constant.
- Clarify the source of event. Is it Metastore vs CatalogD vs Filesystem vs
other service (like Kudu master).
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3523
PS3, Line 3523: catalogTimeline.markEvent("Created Metastore
table");
: } else {
: catalogTimeline.markEvent("Found table exists");
Maybe reorganize this to 2 things: existence check and new table creation:
- Checked table existence in Metastore: xx ms
- Created table in Metastore: xx ms
The first one is always printed. The second one only show up in new table
creation.
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3552
PS3, Line 3552: Created catalog table
nit: "Cached table in catalogd"?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3610
PS3, Line 3610: Created Metastore table
nit: "Created table in Metastore"?
This is referred twice in chis class. Should this be a constant instead?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3627
PS3, Line 3627: HMS
nit: "Fetched table from Metastore"?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3656
PS3, Line 3656: Created catalog table
nit: "Cached table in catalog"?
This is referred twice in chis class. Should this be a constant instead?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3702
PS3, Line 3702: HMS
nit: Metastore
Is this label intentionally made different from the other "HMS table fetched"
at line 3627?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3786
PS3, Line 3786: Created iceberg table
nit: created in where? HMS, filesystem, or CatalogD cache?
Why iceberg has its own specific label?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3827
PS3, Line 3827: Created Metastore table
nit: dulicated String. Change to constant?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3834
PS3, Line 3834: Found table exists
nit: dulicated String. Change to constant?
Found in CatalogD or Metastore?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@102
PS3, Line 102: Checked table exists
nit: Checked table existence
Exist in where? Kudu master, HMS, or CatalogD?
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@342
PS3, Line 342: Checked table exists
nit: Checked table existence
http://gerrit.cloudera.org:8080/#/c/20368/3/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@369
PS3, Line 369: cols
nit: columns
--
To view, visit http://gerrit.cloudera.org:8080/20368
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa
Gerrit-Change-Number: 20368
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Mon, 21 Aug 2023 17:18:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12274: Fix memory leak from unreleased local reference
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20175 ) Change subject: IMPALA-12274: Fix memory leak from unreleased local reference .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic93ea3270dcba3ad4903aa053cc283d4f700e948 Gerrit-Change-Number: 20175 Gerrit-PatchSet: 7 Gerrit-Owner: zhangqianqiong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: zhangqianqiong Gerrit-Comment-Date: Mon, 21 Aug 2023 16:36:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9609/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 21 Aug 2023 16:27:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20378/3/be/src/service/query-options.h File be/src/service/query-options.h: http://gerrit.cloudera.org:8080/#/c/20378/3/be/src/service/query-options.h@308 PS3, Line 308: \ nit: right align this backslash? -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 16:21:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9608/ -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 21 Aug 2023 16:24:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 16:01:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20378/2/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/20378/2/common/thrift/ImpalaService.thrift@839 PS2, Line 839: unspecified or a limit of 0 > nit: Unspecified or a limit of 0 or negative value Done -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 21 Aug 2023 15:37:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10860: Allow setting mem limit for coordinators
Abhishek Rawat has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20378 ) Change subject: IMPALA-10860: Allow setting mem_limit for coordinators .. IMPALA-10860: Allow setting mem_limit for coordinators Added support for MEM_LIMIT_COORDINATORS query option. This is similar to exisiting MEM_LIMIT_EXECUTORS, but applies to coordinators. There are cases where Planner generates inaccurate estimates for coordinator fragments and would be good to be able to set mem limit just for the coordinator, since a query's memory requirement on coordinator tends to be much lower compared to that on executors. If MEM_LIMIT is set, then MEM_LIMIT_COORDINATORS is ignored. Also updated the documentation for the new query option. Testing: - Added new custom cluster tests which validates MEM_LIMIT_COORDINATORS applies only on coordinator. The test also validates that both MEM_LIMIT_EXECUTORS and MEM_LIMIT_COORDINATORS can be set together. - Built docs and made sure that the new changes have proper formatting. Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c --- M be/src/scheduling/schedule-state.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M docs/topics/impala_mem_limit.xml M tests/custom_cluster/test_admission_controller.py 8 files changed, 90 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/20378/3 -- To view, visit http://gerrit.cloudera.org:8080/20378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dfc9a735e82dce2fd903bdaf6bc2e46e982ef8c Gerrit-Change-Number: 20378 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13791/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Aug 2023 15:08:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20010 )
Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying
..
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h
File be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h:
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h@115
PS3, Line 115:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc
File be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@39
PS3, Line 39:
iceberg_api_scan_timer_(ADD_TIMER(scan_node_->runtime_profile(),
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@186
PS3, Line 186: jobject column_field = env->CallObjectMethod(columns,
list_get_, slot_desc->col_pos());
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@245
PS3, Line 245: RETURN_IF_ERROR(MaterializeNextRow(env, struct_like_row,
tuple, state, row_batch->tuple_data_pool()));
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@249
PS3, Line 249: if
(ExecNode::EvalConjuncts(scan_node_->conjunct_evals().data(),
scan_node_->conjunct_evals().size(), tuple_row)) {
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@255
PS3, Line 255: Tuple::ClearNullBits(tuple,
tuple_desc_->null_bytes_offset(), tuple_desc_->num_null_bytes());
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@263
PS3, Line 263: Status IcebergMetadataTableScanner::MaterializeNextRow(JNIEnv*
env, jobject struct_like_row,
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@280
PS3, Line 280: RETURN_IF_ERROR(ReadStringValue(env, slot_desc,
struct_like_row, tuple, tuple_data_pool));
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/service/frontend.cc@110
PS3, Line 110: {"getCatalogTable",
"([B)Lorg/apache/impala/catalog/FeTable;", _catalog_table_id_},
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java:
http://gerrit.cloudera.org:8080/#/c/20010/3/fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java@40
PS3, Line 40: private final static Logger LOG =
LoggerFactory.getLogger(IcebergMetadataScanNode.class);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java@45
PS3, Line 45: protected IcebergMetadataScanNode(PlanNodeId id, List
conjuncts, TableRef tblRef) {
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20010
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7
Gerrit-Change-Number: 20010
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 21 Aug 2023 14:43:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Tamas Mate has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. IMPALA-11996: Scanner change for Iceberg metadtata querying This commit adds a scan node for querying Iceberg metadata tables. The method resembles to HBase table scans, the scan is executed with the help of the Iceberg API and through JNI. There is only one fragment created to query the Iceberg Metadata table which is supposed to be executed on the coordinator node that already has the metadata ready. This change will not cover struct column types these slots are se to NULL, it will be done in IMPALA-12205. Testing: - Added e2e tests for querying metadata tables - Updated planner tests Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 --- M be/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/iceberg-metadata/CMakeLists.txt A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h M be/src/scheduling/scheduler.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A testdata/data/iceberg_test/iceberg_test_metadata/data/3d481ed88b2941f0-ea338162_1109948289_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/944a2355e618932f-18f086b6_1283312202_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/d64ba652b36f2d76-d0752076_883987622_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-1f43b217940cc094-fedf5156_248998721_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-bc4b163507d1da6f-a691cf610002_511900219_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/metadata/38e5a1bd-5b7f-4eae-9362-16a2de3c575d-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/7c952273-c5d6-43d2-bb1b-72fbe1555f25-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/88a39285-529f-41a4-bd69-6d2560fac64e-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/ef78f8eb-b2b5-41df-a2a0-7944ac3c38da-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-7858675898458780516-1-ef78f8eb-b2b5-41df-a2a0-7944ac3c38da.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8283026816932323050-1-38e5a1bd-5b7f-4eae-9362-16a2de3c575d.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8491702501245661704-1-88a39285-529f-41a4-bd69-6d2560fac64e.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-9046920472784493998-1-7c952273-c5d6-43d2-bb1b-72fbe1555f25.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v5.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 46 files changed, 1,762 insertions(+), 156 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20010/3 -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer:
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExpr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20385 ) Change subject: IMPALA-12386: Fix clone constructor in CastExpr .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13790/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f Gerrit-Change-Number: 20385 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 21 Aug 2023 14:20:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20368 ) Change subject: IMPALA-12024: Add catalog profile for createTable .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13789/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa Gerrit-Change-Number: 20368 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 21 Aug 2023 14:11:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExpr
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/20385 )
Change subject: IMPALA-12386: Fix clone constructor in CastExpr
..
Patch Set 2:
(1 comment)
Thanks, just a small suggestion.
http://gerrit.cloudera.org:8080/#/c/20385/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20385/2//COMMIT_MSG@15
PS2, Line 15: 'insert into t01(a, b) values(null, "23"), ("21", null)'
Thinking about it, using the example from the test would be better as the
column names are more informative.
--
To view, visit http://gerrit.cloudera.org:8080/20385
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f
Gerrit-Change-Number: 20385
Gerrit-PatchSet: 2
Gerrit-Owner: Peter Rozsa
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
Gerrit-Comment-Date: Mon, 21 Aug 2023 14:10:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11957: Implement Regression functions: regr slope(), regr intercept() and regr r2()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19569 ) Change subject: IMPALA-11957: Implement Regression functions: regr_slope(), regr_intercept() and regr_r2() .. Patch Set 20: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9607/ -- To view, visit http://gerrit.cloudera.org:8080/19569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab6bd84ae3e0c02ec924c30183308123b951caa3 Gerrit-Change-Number: 19569 Gerrit-PatchSet: 20 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 13:59:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20376 ) Change subject: IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13788/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20376 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9c01129d3cdcaeb222ea910521704ce2305fd2e Gerrit-Change-Number: 20376 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 21 Aug 2023 13:56:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExpr
Peter Rozsa has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/20385 )
Change subject: IMPALA-12386: Fix clone constructor in CastExpr
..
IMPALA-12386: Fix clone constructor in CastExpr
This commit addresses an issue in the CastExpr class where the clone
constructor was not properly preserving compatibility settings. The
clone constructor assigned the default compatibility regardless of the
source expression, causing substitution errors later.
Example:
'insert into t01(a, b) values(null, "23"), ("21", null)'
Throws:
ERROR: IllegalStateException: Failed analysis after expr substitution.
CAUSED BY: IllegalStateException: cast STRING to INT
Tests:
- new test case added to insert-unsafe.test
Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f
---
M fe/src/main/java/org/apache/impala/analysis/CastExpr.java
M testdata/workloads/functional-query/queries/QueryTest/insert-unsafe.test
2 files changed, 5 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20385/2
--
To view, visit http://gerrit.cloudera.org:8080/20385
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f
Gerrit-Change-Number: 20385
Gerrit-PatchSet: 2
Gerrit-Owner: Peter Rozsa
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20368 ) Change subject: IMPALA-12024: Add catalog profile for createTable .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/20368/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20368/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@455 PS2, Line 455: createTable(ddlRequest.getCreate_table_params(), response, catalogTimeline, > line too long (94 > 90) Done http://gerrit.cloudera.org:8080/#/c/20368/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2072 PS2, Line 2072: private List getNextMetastoreEventsIfEnabled( > line too long (96 > 90) Done http://gerrit.cloudera.org:8080/#/c/20368/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4047 PS2, Line 4047: createKuduTable(tbl, createTableParams, wantMinimalResult, response, > line too long (92 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/20368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa Gerrit-Change-Number: 20368 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 21 Aug 2023 13:43:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12024: Add catalog profile for createTable
Hello Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20368 to look at the new patch set (#3). Change subject: IMPALA-12024: Add catalog profile for createTable .. IMPALA-12024: Add catalog profile for createTable DDL/DMLs might spend a long time in the execution in catalogd. Currently, the profiles just have a counter of CatalogOpExecTimer. We need more items to show the details of how they are executed in catalogd. As the first step, this patch adds the profile timeline for how the createTable DDL is executed in Catalog Server. Also added more events in the existing "Query Timeline" for when the CatalogDdlRequest finished and when the catalog updates are applied. To implement this, a 'profile' field is added in TDdlExecResponse to carry the execution counters and timeline in catalogd. Currently, we just uses the timeline of it. We can add more counters in the future. Several methods add a timeline parameter to mark the progress in them. Timeline events are added after each RPC finished. Here is an example when HMS is hanging for 26s in a CTAS. I used gdb to attach to HMS as the JIRA description mentioned. In the timeline, we can see the time is spent in the first HMS RPC that fetching the current HMS event id: Catalog Server Operation: 26s560ms - Got metastoreDdlLock: 163.977us (163.977us) - Got Metastore client: 166.339us (2.362us) - Got current Metastore event id: 26s494ms (26s494ms) - Created Metastore table: 26s558ms (63.507ms) - Fetched Metastore event batch: 26s559ms (1.155ms) - Created catalog table: 26s560ms (1.164ms) - DDL finished: 26s560ms (65.538us) Query Compilation: 164.257ms - Metadata of all 1 tables cached: 10.535ms (10.535ms) - Analysis finished: 118.324ms (107.788ms) - Authorization finished (noop): 118.489ms (164.626us) - Value transfer graph computed: 118.830ms (341.792us) - Single node plan created: 150.150ms (31.319ms) - Runtime filters computed: 150.254ms (103.529us) - Distributed plan created: 151.832ms (1.578ms) - Planning finished: 164.257ms (12.425ms) Query Timeline: 27s304ms - Query submitted: 129.658us (129.658us) - Planning finished: 170.224ms (170.095ms) - CatalogDdlRequest finished: 26s731ms (26s561ms) - Applied catalog updates from DDL: 26s740ms (8.752ms) - Submit for admission: 26s740ms (22.233us) - Completed admission: 26s740ms (286.295us) - Ready to start on 3 backends: 26s740ms (155.916us) - All 3 execution backends (3 fragment instances) started: 26s751ms (10.864ms) - Last row fetched: 26s920ms (168.226ms) - Released admission control resources: 26s920ms (27.635us) - DML data written: 26s920ms (126.369us) - Applied catalog updates from DDL: 26s985ms (65.354ms) - DML Metastore update finished: 26s985ms (30.343us) - Rows available: 26s985ms (27.050us) - Unregister query: 27s304ms (318.661ms) Tests: - Add e2e tests to verify the DDL timeline events exist in profile Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa --- M be/src/service/client-request-state.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M tests/query_test/test_observability.py 9 files changed, 193 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/20368/3 -- To view, visit http://gerrit.cloudera.org:8080/20368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa Gerrit-Change-Number: 20368 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/20376 ) Change subject: IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature .. Patch Set 2: (1 comment) > Patch Set 1: > > (1 comment) http://gerrit.cloudera.org:8080/#/c/20376/1/fe/src/main/java/org/apache/impala/analysis/NullLiteral.java File fe/src/main/java/org/apache/impala/analysis/NullLiteral.java: http://gerrit.cloudera.org:8080/#/c/20376/1/fe/src/main/java/org/apache/impala/analysis/NullLiteral.java@68 PS1, Line 68: > Do we also need to implement uncheckedCastTo with the compatibility argumen Thank you for noticing it, it's indeed causing a problem, I added the missing/changed overload. -- To view, visit http://gerrit.cloudera.org:8080/20376 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9c01129d3cdcaeb222ea910521704ce2305fd2e Gerrit-Change-Number: 20376 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 21 Aug 2023 13:30:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature
Peter Rozsa has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20376 ) Change subject: IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature .. IMPALA-12384: Restore NullLiteral's uncheckedCastTo function signature This change restores NullLiteral's uncheckedCastTo function's signature to preserve the external compatibility of the method and make it conform with changes regarding IMPALA-10173. Change-Id: Id9c01129d3cdcaeb222ea910521704ce2305fd2e --- M fe/src/main/java/org/apache/impala/analysis/NullLiteral.java 1 file changed, 8 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/20376/2 -- To view, visit http://gerrit.cloudera.org:8080/20376 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9c01129d3cdcaeb222ea910521704ce2305fd2e Gerrit-Change-Number: 20376 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9608/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 21 Aug 2023 12:07:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 21 Aug 2023 12:07:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12342: Erasure coding build fails on loading iceberg lineitem multiblock
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/20359 ) Change subject: IMPALA-12342: Erasure coding build fails on loading iceberg_lineitem_multiblock .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad15a335407c12578eb822bb1cb4450647502e50 Gerrit-Change-Number: 20359 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 21 Aug 2023 12:00:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExrp
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20385 ) Change subject: IMPALA-12386: Fix clone constructor in CastExrp .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20385/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20385/1//COMMIT_MSG@9 PS1, Line 9: This commit addresses an issue in the CastExpr class where the clone Also, could you briefly describe how the error here, like in the Jira? -- To view, visit http://gerrit.cloudera.org:8080/20385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f Gerrit-Change-Number: 20385 Gerrit-PatchSet: 1 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 11:28:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExrp
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20385 ) Change subject: IMPALA-12386: Fix clone constructor in CastExrp .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20385/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20385/1//COMMIT_MSG@11 PS1, Line 11: You could mention the added test. -- To view, visit http://gerrit.cloudera.org:8080/20385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f Gerrit-Change-Number: 20385 Gerrit-PatchSet: 1 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 09:42:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11957: Implement Regression functions: regr slope(), regr intercept() and regr r2()
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19569 ) Change subject: IMPALA-11957: Implement Regression functions: regr_slope(), regr_intercept() and regr_r2() .. Patch Set 19: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab6bd84ae3e0c02ec924c30183308123b951caa3 Gerrit-Change-Number: 19569 Gerrit-PatchSet: 19 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 09:37:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11957: Implement Regression functions: regr slope(), regr intercept() and regr r2()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19569 ) Change subject: IMPALA-11957: Implement Regression functions: regr_slope(), regr_intercept() and regr_r2() .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9607/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab6bd84ae3e0c02ec924c30183308123b951caa3 Gerrit-Change-Number: 19569 Gerrit-PatchSet: 20 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 09:38:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11957: Implement Regression functions: regr slope(), regr intercept() and regr r2()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19569 ) Change subject: IMPALA-11957: Implement Regression functions: regr_slope(), regr_intercept() and regr_r2() .. Patch Set 20: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab6bd84ae3e0c02ec924c30183308123b951caa3 Gerrit-Change-Number: 19569 Gerrit-PatchSet: 20 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 21 Aug 2023 09:38:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExrp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20385 ) Change subject: IMPALA-12386: Fix clone constructor in CastExrp .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13787/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f Gerrit-Change-Number: 20385 Gerrit-PatchSet: 1 Gerrit-Owner: Peter Rozsa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 08:22:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12386: Fix clone constructor in CastExrp
Peter Rozsa has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20385 Change subject: IMPALA-12386: Fix clone constructor in CastExrp .. IMPALA-12386: Fix clone constructor in CastExrp This commit addresses an issue in the CastExpr class where the clone constructor was not properly preserving compatibility settings. Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f --- M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M testdata/workloads/functional-query/queries/QueryTest/insert-unsafe.test 2 files changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20385/1 -- To view, visit http://gerrit.cloudera.org:8080/20385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iff64ce02539651fcb3a90db678f74467f582648f Gerrit-Change-Number: 20385 Gerrit-PatchSet: 1 Gerrit-Owner: Peter Rozsa
