[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This metric is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Reviewed-on: http://gerrit.cloudera.org:8080/20473 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M tests/custom_cluster/test_events_custom_configs.py M tests/webserver/test_web_pages.py 5 files changed, 76 insertions(+), 24 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 13 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 04 Jan 2024 18:54:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 04 Jan 2024 14:10:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10117/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 04 Jan 2024 14:10:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 11: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/20473/11/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/20473/11/tests/custom_cluster/test_events_custom_configs.py@996 PS11, Line 996: assert events_skipped <= events_skipped_after I think we can remove this check in the future since it's always true unless somehow the metric is reset to 0. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 04 Jan 2024 14:10:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 03 Jan 2024 23:38:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, Fang-Yu Rao, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#11). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This metric is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M tests/custom_cluster/test_events_custom_configs.py M tests/webserver/test_web_pages.py 5 files changed, 76 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/11 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: The patch failed the same test again: custom_cluster.test_events_custom_configs.TestEventProcessingCustomConfigs.test_self_events according to https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/1290/testReport/junit/custom_cluster.test_events_custom_configs/TestEventProcessingCustomConfigs/test_self_events/. It looks like this patch has never passed the dry run test. Maybe this specific end-to-end test has to be fixed too? -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 23:53:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10094/ -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 21:38:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: Saw the following in the failed Jenkins run at https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/1286/testReport/junit/custom_cluster.test_events_custom_configs/TestEventProcessingCustomConfigs/test_self_events/ (from https://jenkins.impala.io/job/gerrit-verify-dryrun/10092/console). Is this a known flaky test? custom_cluster/test_events_custom_configs.py:234: in test_self_events self.__run_self_events_test(unique_database, False) custom_cluster/test_events_custom_configs.py:751: in __run_self_events_test self.__exec_sql_and_check_selfevent_counter(stmt, use_impala) custom_cluster/test_events_custom_configs.py:996: in __exec_sql_and_check_selfevent_counter assert events_skipped == events_skipped_after E assert 69 == 70 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 18:31:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10094/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 17:05:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10092/ -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 06:48:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 9: Code-Review+2 LGTM. Carry Csaba's +1. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 02:07:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 02:07:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10092/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 02:07:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14820/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 00:33:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 9: (5 comments) http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG@22 PS2, Line 22: Tests: > I think the lag of event processing at table level will be helpful. It show Ack. Created https://issues.apache.org/jira/browse/IMPALA-12664 to track this. http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG@11 PS5, Line 11: identif > I see, so this is simply how Impala treats gauges and not something specifi Yeah, that's correct. http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG@11 PS8, Line 11: metric > typo Ack http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG@23 PS8, Line 23: - Manually verified the metrics on catalogD UI page when running some > Can we add a simple test in tests/webserver/test_web_pages.py to fetch the Ack http://gerrit.cloudera.org:8080/#/c/20473/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2529 PS8, Line 2529: return droppedPartitions_; > If we return here, we should bump the counter of EVENTS_SKIPPED_METRIC. Ack -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 22 Dec 2023 00:06:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#9). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This metric is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M tests/webserver/test_web_pages.py 4 files changed, 74 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/9 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG@22 PS2, Line 22: Tests: > last-synced-event-id on table metric makes sense to me. I think the lag of event processing at table level will be helpful. It shows how much a table is impacted when event processing is lagging behind. Users can check if some critical tables have stale metadata and trigger manual REFRESH/INVALIDATE on them. However, we don't have a light-weight HMS API to get the latest event time of a table. So measuring the lag at table level is expensive. "events-consuming-delay-ms" is a cheaper counter to reflect the lag. "events-skipped" at table level helps to find tables that don't need event processing. E.g. if events on a table are all skipped, we can disable HMS event sync on it. To get this, we also need the counter of total events at table level. BTW, db level metrics also help to find such dbs. I'm ok to add some of them in a separate JIRA if they require more work. http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG@23 PS8, Line 23: - Manually verified the metrics on catalogD UI page when running some Can we add a simple test in tests/webserver/test_web_pages.py to fetch the metrics of a table and verify the existence of these event metrics? http://gerrit.cloudera.org:8080/#/c/20473/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2529 PS8, Line 2529: return; If we return here, we should bump the counter of EVENTS_SKIPPED_METRIC. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 21 Dec 2023 07:58:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 8: Code-Review+1 (3 comments) http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG@11 PS5, Line 11: identif > I have attached the screenshots of the following table metrics: events-proc I see, so this is simply how Impala treats gauges and not something specific to this metric, right? http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/8//COMMIT_MSG@11 PS8, Line 11: mertic typo http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1745 PS5, Line 1745: ventProcessing > For rename operation, we accumulate and associate the process time with the ack -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 20 Dec 2023 10:02:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14760/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Sun, 17 Dec 2023 20:12:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 8: (2 comments) http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG@11 PS5, Line 11: identif > I still don't see where this was changed in the patch. The only metric I se I have attached the screenshots of the following table metrics: events-processing-duration (table-level), and events-consuming-delay (event-processor level) in the Jira. Please take a look and let me know if it looks good to you. http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1745 PS5, Line 1745: ventProcessing > This will also happen with rename, so the duration of rename won't be added For rename operation, we accumulate and associate the process time with the new table name. Ack on creating a new Jira for dropped events. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Sun, 17 Dec 2023 19:45:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#8). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 68 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/8 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 8 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 7: (2 comments) Thanks for the refactor! http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG@11 PS5, Line 11: identif > MetastoreEventsProcessor.java#1168 -- We are tracking the duration in secon I still don't see where this was changed in the patch. The only metric I sew with 1min/5min/15min is avgNumberOfEventsReceived As far as I remember the slowest event processing I saw in production logs was around 50 minutes (reloading a table with lot of partitions on a slow namenode) http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1745 PS5, Line 1745: t type"); > Unfortunately, since the table doesn't exist in cache we cannot see the dur This will also happen with rename, so the duration of rename won't be added to the metric of the new table? I don't think that this is critical, but a Jira could be created + some comments could be added about this. For dropped events there could be a separate metric that collects the sum of duration of all drop events. Generally a drop should be fast, but it takes metastoreDdlLock_, which can block on HMS RPCs. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 7 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 14 Dec 2023 09:45:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14686/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 7 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 13 Dec 2023 00:19:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14683/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 6 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 12 Dec 2023 23:46:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#7). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 70 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/7 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 7 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 6: (4 comments) http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG@11 PS5, Line 11: identif > Where are the durations set? Are there also larger durations tracked, e.g. MetastoreEventsProcessor.java#1168 -- We are tracking the duration in seconds. I'm wondering if the event processor would take an hour/day to consume an event. Seconds seem to be more realistic to me. http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1534 PS5, Line 1534: // event > It would be nice to reduce code duplication related to the timer. We seem t Ack http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1536 PS5, Line 1536: lf-event > getTableNoThrow() would let us skip the try/ catch block Ack http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1745 PS5, Line 1745: > What will happen with the timer after drop/rename events? Will we able to s Unfortunately, since the table doesn't exist in cache we cannot see the duration of event processing for a dropped table on the web UI -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 6 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 12 Dec 2023 23:18:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#6). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 66 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/6 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 6 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20473/5//COMMIT_MSG@11 PS5, Line 11: extended Where are the durations set? Are there also larger durations tracked, e.g. 1 hour, 1 day? http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1534 PS5, Line 1534: Timer.Context context = null; It would be nice to reduce code duplication related to the timer. We seem to do the same for each MetastoreTableEvent, so a shared function could have the timer block. E.g. MetastoreTableEvent could implement process() that calls processTableEvent(), and existing process() logicof subclasses could be done in processTableEvent() http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1536 PS5, Line 1536: getTable getTableNoThrow() would let us skip the try/ catch block http://gerrit.cloudera.org:8080/#/c/20473/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1745 PS5, Line 1745: DropTableEvent What will happen with the timer after drop/rename events? Will we able to see in the webui that some event processing time was spent on these tables after the table no longer exists? -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 5 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 12 Dec 2023 10:04:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14650/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 5 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 12 Dec 2023 00:20:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, k.venureddy2...@gmail.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#5). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 350 insertions(+), 149 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/5 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 5 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14564/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 01 Dec 2023 00:26:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, k.venureddy2...@gmail.com, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#4). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 350 insertions(+), 149 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/4 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9776/ -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 29 Sep 2023 02:01:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9776/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 28 Sep 2023 21:37:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14006/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 15 Sep 2023 00:54:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Sai Hemanth Gantasala has posted comments on this change. (
http://gerrit.cloudera.org:8080/20473 )
Change subject: IMPALA-11553: Add event specific metrics in the table metrics
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG@22
PS2, Line 22:
> Can we add more metrics at table level, e.g. events-skipped, last-synced-ev
last-synced-event-id on table metric makes sense to me.
I don't think adding "events-skipped", "events-consuming-delay-ms" on table
metrics adds much value. These metrics on event-processor summary page makes
much better sense to me. Let me know if see good motivation to add them on
table metrics.
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@942
PS2, Line 942: Preconditions.checkState(event.getEventId() ==
currentEventId);
> I made a mistake that the unit is actually second, not milli second. Could
Ack
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1063
PS2, Line 1063:
> TimeUnit.SECONDS ?
Ack
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1078
PS2, Line 1078: LOG.info("Time elapsed in processing event batch: {}",
> We use 'System.currentTimeMillis() / 1000' directly in many places. Maybe d
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd
Gerrit-Change-Number: 20473
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Fri, 15 Sep 2023 00:27:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#3). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 363 insertions(+), 161 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/3 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20473 )
Change subject: IMPALA-11553: Add event specific metrics in the table metrics
..
Patch Set 2:
(4 comments)
Thanks for adding these! I just take a quick look. Will look into this deeper.
http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20473/2//COMMIT_MSG@22
PS2, Line 22:
Can we add more metrics at table level, e.g. events-skipped,
last-synced-event-id, events-consuming-delay-ms?
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@942
PS2, Line 942: latestEventTimeMs_.set(event.getEventTime());
I made a mistake that the unit is actually second, not milli second. Could you
also correct the variable name and the metric text appropriately? E.g.
latestEventTimeMs_ -> latestEventTimeSecs_
lastSyncedEventTimeMs_ -> lastSyncedEventTimeSecs_
"latest-event-time-ms" -> "latest-event-time-secs"
"last-synced-event-time-ms" -> "last-synced-event-time-secs"
"event-processing-delay-ms" -> "event-processing-delay-secs"
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1063
PS2, Line 1063: MILLISECONDS
TimeUnit.SECONDS ?
http://gerrit.cloudera.org:8080/#/c/20473/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1078
PS2, Line 1078: public long now() {
We use 'System.currentTimeMillis() / 1000' directly in many places. Maybe don't
worth a method for this, and it's ambigous what the unit is.
$ git grep 'System.currentTimeMillis() / 1000'
fe/src/main/java/org/apache/impala/catalog/Table.java:
msTbl.putToParameters(propertyKey, Long.toString(System.currentTimeMillis() /
1000));
fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java:
(int) (System.currentTimeMillis() / 1000),
fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java:
(int) (System.currentTimeMillis() / 1000), FunctionType.JAVA, resources);
fe/src/test/java/org/apache/impala/service/JdbcTest.java:
lastTimeSessionActive.add(System.currentTimeMillis() / 1000);
fe/src/test/java/org/apache/impala/service/JdbcTest.java: long now =
System.currentTimeMillis() / 1000;
--
To view, visit http://gerrit.cloudera.org:8080/20473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd
Gerrit-Change-Number: 20473
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Tue, 12 Sep 2023 00:42:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13977/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Sep 2023 23:00:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20473 ) Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13976/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 11 Sep 2023 22:37:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20473 to look at the new patch set (#2). Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 354 insertions(+), 149 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/2 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 2 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11553: Add event specific metrics in the table metrics
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20473 Change subject: IMPALA-11553: Add event specific metrics in the table metrics .. IMPALA-11553: Add event specific metrics in the table metrics This patch adds an event specific metric "avg-events-process-duration" at the table level metrics. This metric is also extended to last 1min, 5mins, 15mins duration. This mertic is useful to identify the average events processed duration on the table. This is helpful to identify if a particular table is causing event procssor lagging and as a temporary workaround, event processing can be disabled on that table. Another metric is also added in the event processor summary page, "events-consuming-delay-ms", is the time difference in milliseconds of the event created in the metastore and event processed by event processor. This is another useful metric to gauge how the event processor is lagging. Tests: - Manually verified the metrics on catalogD UI page when running some hive workloads. Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd --- M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 3 files changed, 353 insertions(+), 149 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20473/1 -- To view, visit http://gerrit.cloudera.org:8080/20473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2428029361e610a0fcd8ed11be2ab771f03b00dd Gerrit-Change-Number: 20473 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala
