[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 26: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 26 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 09 Nov 2023 02:32:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display along with additional disk and network metrics. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. The zoom out button is disabled, when further zoom out is not possible. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Test cases have been for the additional disk, network and fragment level memory metrics parsing functions. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Reviewed-on: http://gerrit.cloudera.org:8080/20355 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl M www/scripts/query_timeline/chart_commons.js M www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js M www/scripts/query_timeline/global_members.js M www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/tests/query_timeline/fragment_metrics_diagram.test.js M www/scripts/tests/query_timeline/host_utilization_diagram.test.js 12 files changed, 671 insertions(+), 29 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 27 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 26: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9906/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 26 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 22:04:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 26: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 26 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 22:04:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/20355/25/www/scripts/query_timeline/host_utilization_diagram.js File www/scripts/query_timeline/host_utilization_diagram.js: http://gerrit.cloudera.org:8080/#/c/20355/25/www/scripts/query_timeline/host_utilization_diagram.js@59 PS25, Line 59: var min_num_samples = 3 > Apologies for any miscommunication. But, 'array_values_start_index' has no Ah, OK. Please disregards this comment. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 17:43:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 17:33:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: (1 comment) http://gerrit.cloudera.org:8080/#/c/20355/25/www/scripts/query_timeline/host_utilization_diagram.js File www/scripts/query_timeline/host_utilization_diagram.js: http://gerrit.cloudera.org:8080/#/c/20355/25/www/scripts/query_timeline/host_utilization_diagram.js@59 PS25, Line 59: var min_num_samples = 3 > nit: assign with array_values_start_index + 1 for consistency? Apologies for any miscommunication. But, 'array_values_start_index' has no relation with 'min_num_samples'. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 17:32:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/20355/25/www/scripts/query_timeline/host_utilization_diagram.js File www/scripts/query_timeline/host_utilization_diagram.js: http://gerrit.cloudera.org:8080/#/c/20355/25/www/scripts/query_timeline/host_utilization_diagram.js@59 PS25, Line 59: var min_num_samples = 3 nit: assign with array_values_start_index + 1 for consistency? var min_num_samples = array_values_start_index + 1; -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 15:47:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14370/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 08:19:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: Thank you for the comments. I have also explained the reason for the constant's value as a comment. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 07:53:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 25: (4 comments) http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/chart_commons.js File www/scripts/query_timeline/chart_commons.js: http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/chart_commons.js@80 PS24, Line 80: > nit: nit: define a constant for this assignment. Done http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js File www/scripts/query_timeline/host_utilization_diagram.js: http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js@252 PS24, Line 252: > nit: define a constant for this. Done http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js@266 PS24, Line 266: e a > nit: I see several loops start from index 2 like this one. Is there any sig Done. I have also added a comment to define this constant. http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js@271 PS24, Line 271: > nit: define a constant for this assignment. Done -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 Nov 2023 07:51:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display along with additional disk and network metrics. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. The zoom out button is disabled, when further zoom out is not possible. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Test cases have been for the additional disk, network and fragment level memory metrics parsing functions. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl M www/scripts/query_timeline/chart_commons.js M www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js M www/scripts/query_timeline/global_members.js M www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/tests/query_timeline/fragment_metrics_diagram.test.js M www/scripts/tests/query_timeline/host_utilization_diagram.test.js 12 files changed, 671 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/25 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 25 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 24:
(5 comments)
Compared to patch set 13 and it looks OK to me. I just have 1 nit.
http://gerrit.cloudera.org:8080/#/c/20355/13/bin/run-all-tests.sh
File bin/run-all-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/13/bin/run-all-tests.sh@64
PS13, Line 64: # Run JS tests
: : ${JS_TEST:=false
> Done
Done
http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/chart_commons.js
File www/scripts/query_timeline/chart_commons.js:
http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/chart_commons.js@80
PS24, Line 80: = 2
nit: nit: define a constant for this assignment.
http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js
File www/scripts/query_timeline/host_utilization_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js@252
PS24, Line 252: <= 2
nit: define a constant for this.
http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js@266
PS24, Line 266: = 2
nit: I see several loops start from index 2 like this one. Is there any
significance on this (ie., good samples are expected to have at least 3 data
points)?
If yes, can you define a constant somewhere and use that constant for this kind
of assignment?
http://gerrit.cloudera.org:8080/#/c/20355/24/www/scripts/query_timeline/host_utilization_diagram.js@271
PS24, Line 271: = 2
nit: define a constant for this assignment.
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 24
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 07 Nov 2023 21:17:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 24: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 24 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 04 Nov 2023 00:15:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 24: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14316/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 24 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 03 Nov 2023 09:01:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display along with additional disk and network metrics. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. The zoom out button is disabled, when further zoom out is not possible. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Test cases have been for the additional disk, network and fragment level memory metrics parsing functions. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl M www/scripts/query_timeline/chart_commons.js M www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js M www/scripts/query_timeline/global_members.js M www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/tests/query_timeline/fragment_metrics_diagram.test.js M www/scripts/tests/query_timeline/host_utilization_diagram.test.js 12 files changed, 657 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/24 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 24 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 23: (1 comment) tested in Google Chrome and Firefox on ubuntu 20.02. Zoom-out button was not enabled sometimes. For example, click Zoom-in button once without resizing windows, the Zoom-out button is not enabled. But the button is enabled after resizing windows. http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js File www/scripts/query_timeline/fragment_metrics_diagram.js: http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@185 PS22, Line 185: > Sorry, is there a mistake here? previous line is a function call with parameters which cannot fit in one line so this line need to indent. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 23 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 02 Nov 2023 17:02:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14305/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 23 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 02 Nov 2023 13:45:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 23:
(6 comments)
> (6 comments)
>
> Once the charts and timeline fit in current windows, user cannot
> zoom-in further when clicking Zoomin button. Should we grep out the
> Zoomin button to indict no further zoomin.
Thank you for the comments. The buttons are now disabled, when no further
zoom-out is possible.
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_diagram.js
File www/scripts/query_timeline/fragment_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_diagram.js@588
PS22, Line 588: {
> nit: break as two lines
Done
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js
File www/scripts/query_timeline/fragment_metrics_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@21
PS22, Line 21:
> nit: two more spaces
Done
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@140
PS22, Line 140: .fil
> nit: move "+ 3)" to previous line
Done
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@185
PS22, Line 185:
> nit: 4 more spaces
Sorry, is there a mistake here?
I believe the 2 more lines are part of the same statement.
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@193
PS22, Line 193:
> nit: two more spaces
Done
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/host_utilization_diagram.js
File www/scripts/query_timeline/host_utilization_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/host_utilization_diagram.js@168
PS22, Line 168:
> nit: two more spaces
Done
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 23
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 02 Nov 2023 13:19:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display along with additional disk and network metrics. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. The zoom out button is disabled, when further zoom out is not possible. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Test cases have been for the additional disk, network and fragment level memory metrics parsing functions. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl M www/scripts/query_timeline/chart_commons.js M www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js M www/scripts/query_timeline/global_members.js M www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/tests/query_timeline/fragment_metrics_diagram.test.js M www/scripts/tests/query_timeline/host_utilization_diagram.test.js 12 files changed, 656 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/23 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 23 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 22:
(6 comments)
Once the charts and timeline fit in current windows, user cannot zoom-in
further when clicking Zoomin button. Should we grep out the Zoomin button to
indict no further zoomin.
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_diagram.js
File www/scripts/query_timeline/fragment_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_diagram.js@588
PS22, Line 588: {}
nit: break as two lines
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js
File www/scripts/query_timeline/fragment_metrics_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@21
PS22, Line 21:
nit: two more spaces
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@140
PS22, Line 140: + 3)
nit: move "+ 3)" to previous line
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@185
PS22, Line 185:
nit: 4 more spaces
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/fragment_metrics_diagram.js@193
PS22, Line 193:
nit: two more spaces
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/host_utilization_diagram.js
File www/scripts/query_timeline/host_utilization_diagram.js:
http://gerrit.cloudera.org:8080/#/c/20355/22/www/scripts/query_timeline/host_utilization_diagram.js@168
PS22, Line 168:
nit: two more spaces
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 22
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 01 Nov 2023 22:54:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 22: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14284/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 22 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 30 Oct 2023 14:21:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display along with additional disk and network metrics. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Test cases have been for the additional disk, network and fragment level memory metrics parsing functions. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl M www/scripts/query_timeline/chart_commons.js M www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js M www/scripts/query_timeline/global_members.js M www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/tests/query_timeline/fragment_metrics_diagram.test.js M www/scripts/tests/query_timeline/host_utilization_diagram.test.js 12 files changed, 635 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/22 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 22 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14198/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 21 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 18 Oct 2023 14:31:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display along with additional disk and network metrics. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Test cases have been for the additional disk, network and fragment level memory metrics parsing functions. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl M www/scripts/query_timeline/chart_commons.js M www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js M www/scripts/query_timeline/global_members.js M www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/tests/query_timeline/fragment_metrics_diagram.test.js M www/scripts/tests/query_timeline/host_utilization_diagram.test.js 12 files changed, 635 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/21 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 21 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 20: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 20 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 10 Oct 2023 17:19:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display and divides the query timeline script into multiple modules. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/rat_exclude_files.txt A www/icons/drag_handle.png A www/icons/horizontal_zoomin.png A www/icons/horizontal_zoomout.png M www/query_timeline.tmpl A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js M www/scripts/util.js 12 files changed, 1,621 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/20 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 20 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 19: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14157/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 19 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 09 Oct 2023 13:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 19: > We need to do something different for the zooming and scrolling as > it's hard to figure out how to control it, especially with a track > pad. The grey bar seems out of place and is confusing. I think it > would be better to have buttons to control the zoom. Panning > should work via click and drag or buttons. Now, I have added buttons for zooming in and out. And I have replaced the grey bar with a drag handle. This handle has extended length as to allow for resizing when zoomed in. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 19 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 09 Oct 2023 13:03:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display and divides the query timeline script into multiple modules. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit by dragging the resize bar's handle. Along with mouse wheel events, the diagrams can be horizontally stretched by the help of buttons with horizontal zoom icons at the top of the page. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- A www/icons/arrowhead_down.png A www/icons/arrowhead_up.png A www/icons/drag_handle.png A www/icons/zoomin.png A www/icons/zoomout.png M www/query_timeline.tmpl A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js M www/scripts/util.js 13 files changed, 1,620 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/19 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 19 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 18: We need to do something different for the zooming and scrolling as it's hard to figure out how to control it, especially with a track pad. The grey bar seems out of place and is confusing. I think it would be better to have buttons to control the zoom. Panning should work via click and drag or buttons. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 18 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Oct 2023 16:17:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14144/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 18 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 05 Oct 2023 17:29:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 18: (12 comments) Now, I have split the test changes into a separate patch as required. http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@8 PS17, Line 8: > Add: Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@9 PS17, Line 9: This patch adds fragment-level metrics to the WebUI query timeline > Limit line length to 73 characters in commit message. Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@10 PS17, Line 10: nd divides the query time > fragment's row in the phase Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@25 PS17, Line 25: - MemoryUsage > Do you mean they will be updated for a running query? Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@35 PS17, Line 35: A warning is displayed on clicking a fragment with less number of samples > Say that RESOURCE_TRACE_RATIO must be set to enable periodic metrics and re Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@48 PS17, Line 48: : > are displayed Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@59 PS17, Line 59: rolling. > What is preventing automated testing? I meant automated tests have been written in order to reduce manual testing. http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@59 PS17, Line 59: horizontal scrolling. > We may want to split the test changes into a separate patch. Done http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@76 PS17, Line 76: > Unclear what this is relevant to. Is this part of the testing infrastructur It was mentioned to address that this comment had been resolved... I'm a bit concern that look up on time_series_counters indices is straight by the index rather than by checking the name first (time_series_counters[i + offset_counters].name). There is no guarantee that the order will be the same all or there will be no new counter that will change the order in the future. Thus, we have name field in TTimeSeriesCounter. Rather than offset_counters, maybe should pass the exact counter name you're targeting (say, HostCpuSysPercentage) and look for counter with matching name. http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl File www/query_timeline.tmpl: http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@471 PS2, Line 471: > First comment was relating to the position of the charts meaning make the f Done http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@778 PS2, Line 778: > The x-axis for the periodic metrics and the phase diagram have exactly the Done http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1018 PS2, Line 1018: > Makes sense. So we can only show 2 scales or can there be other sets of lab Yes, only 2 scales/units that match either y1 or y2 axis can be shown. Other labels can be shown by scaling, but they would not relate to either axes. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 18 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 05 Oct 2023 17:01:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline This patch adds fragment-level metrics to the WebUI query timeline display and divides the query timeline script into multiple modules. The fragment's plan nodes are enlarged with an animated transition on hovering over the fragment's row in query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile during a running query. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option must be set for providing periodic metrics within the profile. This allows the following time series counters to be displayed on the query timeline. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js M www/scripts/util.js 8 files changed, 1,576 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/18 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 18 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 17: (13 comments) http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@8 PS17, Line 8: Add: This patch adds fragment-level metrics to the WebUI query timeline display.. Also mention splitting up the timeline logic into multiple files. http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@9 PS17, Line 9: The fragment's plan nodes are enlarged with an animated transition on hovering Limit line length to 73 characters in commit message. In vim: :set textwidth=73 gggqG http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@10 PS17, Line 10: query timeline's fragment fragment's row in the phase http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@25 PS17, Line 25: Once a fragment's metrics are displayed, they are updated as they Do you mean they will be updated for a running query? http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@35 PS17, Line 35: RESOURCE_TRACE_RATIO query option provides the utilization values to be Say that RESOURCE_TRACE_RATIO must be set to enable periodic metrics and remove the rest of the description. What matters for this change is what metrics are displayed. http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@48 PS17, Line 48: are being : displayed are displayed http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@59 PS17, Line 59: To help with manually testing the query timeline web page, We may want to split the test changes into a separate patch. http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@59 PS17, Line 59: manually What is preventing automated testing? http://gerrit.cloudera.org:8080/#/c/20355/17//COMMIT_MSG@76 PS17, Line 76: As the order and name of counters may change within the profile, Unclear what this is relevant to. Is this part of the testing infrastructure? http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl File www/query_timeline.tmpl: http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@65 PS2, Line 65: > I can assign a trigger such as clicking on the timeticks footer or any othe I think it's ok the way it is. http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@471 PS2, Line 471: > Yes, it would be very helpful to compare multiple fragment metrics together First comment was relating to the position of the charts meaning make the fragment metrics appear below the periodic metrics. http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@778 PS2, Line 778: > The tics/scaling for the fragment metrics are not aligned properly now. The x-axis for the periodic metrics and the phase diagram have exactly the same max value and align perfectly. However, the fragment metrics has a different max value and does not align. http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1018 PS2, Line 1018: > As the unit used to measure the fragment metrics is different compared to t Makes sense. So we can only show 2 scales or can there be other sets of labels? -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 17 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 05 Oct 2023 14:07:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14131/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 17 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 03 Oct 2023 17:58:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/impala-config.sh M bin/run-all-tests.sh A tests/run-js-tests.sh M www/query_timeline.tmpl A www/scripts/.gitignore A www/scripts/package.json A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js M www/scripts/util.js 17 files changed, 2,287 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/17 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 17 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 17:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/17/tests/run-js-tests.sh
File tests/run-js-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/17/tests/run-js-tests.sh@35
PS17, Line 35: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 17
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 03 Oct 2023 17:30:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14084/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 16 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 26 Sep 2023 12:50:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M bin/impala-config.sh M bin/run-all-tests.sh A tests/run-js-tests.sh M www/query_timeline.tmpl A www/scripts/.gitignore A www/scripts/package.json A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js M www/scripts/util.js 17 files changed, 2,253 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/16 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 16 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 16:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/16/tests/run-js-tests.sh
File tests/run-js-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/16/tests/run-js-tests.sh@35
PS16, Line 35: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 16
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 26 Sep 2023 12:24:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 15:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20355/13/tests/run-js-tests.sh
File tests/run-js-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/13/tests/run-js-tests.sh@1
PS13, Line 1: #!/bin/sh
:
: # Licensed to the Apache Software Foundation (ASF) under one
: # or more contributor license agreements. See the NOTICE file
: # distributed with this work for additional information
: # regarding copyright ownership. The ASF licenses this file
: # to you under the Apache License, Version 2.0 (the
: # "License"); you may not use this file except in compliance
: # with the License. You may obtain a copy of the License at
: #
: # http://www.apache.org/licenses/LICENSE-2.0
: #
: # Unless required by applicable law or agreed to in writing,
: # software distributed under the License is distributed on an
: # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
: # KIND, either express or implied. See the License for the
: # specific language governing permissions and limitations
: # under the License.
For our sanity, always include "set -euo pipefail" at the top of shell scripts.
This causes the shell script to fail if any command fails or if there is an
unbound variable.
We have an add-on that generates a JUnitXML file if this script fails. This can
speed up finding the issue when we see a failure in this area. Please add these
lines from bin/run-backend-tests.sh:
https://github.com/apache/impala/blob/master/bin/run-backend-tests.sh#L21-L22
http://gerrit.cloudera.org:8080/#/c/20355/13/tests/run-js-tests.sh@20
PS13, Line 20: NODEJS_LIB_PATH="${IMPALA_HOME}/lib/node"
I don't like our "lib" top level directory, and I intend to remove it at some
point. I don't want to add new things there. I think a location in
$IMPALA_TOOLCHAIN is a better fit.
Also, a good practice is to assume that we will be moving to a new version
later, so including the version is useful. i.e. :
NODEJS_LIB_PATH="${IMPALA_TOOLCHAIN}/node-${NODEJS_VERSION}"
If we do that, then when people are moving back and forth between different
branches with different versions, things work appropriately.
http://gerrit.cloudera.org:8080/#/c/20355/13/tests/run-js-tests.sh@49
PS13, Line 49: $NPM test
We would like output from the tests to end up in ${IMPALA_LOGS_DIR}. For
example, we have directories under there for be_tests, ee_tests,
custom_cluster_tests, etc. We have a list of all directories here in
bin/impala-config.sh:
https://github.com/apache/impala/blob/master/bin/impala-config.sh#L844-L863
Is there any way to get this command to produce JUnitXML? We have our Jenkins
jobs set up to look for JUnitXML and display the results on the Jenkins job
page. This helps a lot when finding failed tests.
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 25 Sep 2023 22:29:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14078/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 15 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 25 Sep 2023 20:19:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M .gitignore M bin/run-all-tests.sh A tests/run-js-tests.sh M www/query_timeline.tmpl A www/scripts/.gitignore A www/scripts/package.json A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js M www/scripts/util.js 17 files changed, 2,250 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/15 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 15 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/15/tests/run-js-tests.sh
File tests/run-js-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/15/tests/run-js-tests.sh@31
PS15, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 15
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 25 Sep 2023 19:52:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14077/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 14 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 25 Sep 2023 19:19:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 14:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/14/tests/run-js-tests.sh
File tests/run-js-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/14/tests/run-js-tests.sh@31
PS14, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 14
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 25 Sep 2023 18:52:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 14:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/20355/13/bin/run-all-tests.sh
File bin/run-all-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/13/bin/run-all-tests.sh@64
PS13, Line 64: # Run JS tests
: : ${JS_TEST:=${BE_
> I'm not sure if we should enable this by default now, since this is a new e
Done
http://gerrit.cloudera.org:8080/#/c/20355/13/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/13/www/query_timeline.tmpl@144
PS13, Line 144: import {renderTimingDiagram, collectFragmentEventsFromProfile}
from
: "./www/scripts/query_timeline/fragment_diagram.js";
: import {collectUtilizationFromProfile,
toogleUtilizationVisibility} from
: "/www/scripts/query_timeline/host_utilization_diagram.js";
: import {collectFragmentMetricsFromProfile} from
: "/www/scripts/query_timeline/fragment_metrics_diagram.js";
: import {profile, set_profile, diagram_width, set_diagram_width,
border_stroke_width,
: resizeHorizontalAll, maxts} from
"/www/scripts/query_timeline/global_members.js";
> nit: newbie JS question: why these JS scripts imported this way instead of
As we are using 'import' and 'export' statements in the other JS files, which
are part of ES6 modules and ECMA standard, we cannot use the usual
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M .gitignore M bin/run-all-tests.sh A tests/run-js-tests.sh M www/query_timeline.tmpl A www/scripts/.gitignore A www/scripts/package.json A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js M www/scripts/util.js 17 files changed, 2,254 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/14 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 14 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 13:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/20355/13/bin/run-all-tests.sh
File bin/run-all-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/13/bin/run-all-tests.sh@64
PS13, Line 64: # Run JS tests
: : ${JS_TEST:=true}
I'm not sure if we should enable this by default now, since this is a new
environment variable for jenkins.impala.io. Maybe enable this only if BE test
enabled is safer?
: ${JS_TEST:=$BE_TEST}
I'll defer this to others to review.
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@503
PS3, Line 503:
:
:
:
:
> I have broken it down to margins, borders, offset and other easily recogniz
Done
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@938
PS3, Line 938:
> I have now fixed this issue as mentioned above with array mappings.
Done
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@944
PS3, Line 944:
> Thank you. I will add comments here, and some other places.
Done
http://gerrit.cloudera.org:8080/#/c/20355/13/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/13/www/query_timeline.tmpl@144
PS13, Line 144: import {renderTimingDiagram, collectFragmentEventsFromProfile}
from
: "./www/scripts/query_timeline/fragment_diagram.js";
: import {collectUtilizationFromProfile,
toogleUtilizationVisibility} from
: "/www/scripts/query_timeline/host_utilization_diagram.js";
: import {collectFragmentMetricsFromProfile} from
: "/www/scripts/query_timeline/fragment_metrics_diagram.js";
: import {profile, set_profile, diagram_width, set_diagram_width,
border_stroke_width,
: resizeHorizontalAll, maxts} from
"/www/scripts/query_timeline/global_members.js";
nit: newbie JS question: why these JS scripts imported this way instead of
using HTML
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14047/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 13 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 21 Sep 2023 14:06:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M .gitignore M bin/run-all-tests.sh A tests/run-js-tests.sh M www/query_timeline.tmpl A www/scripts/.gitignore A www/scripts/package.json A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js M www/scripts/util.js 17 files changed, 2,235 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/13 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 13 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/13/tests/run-js-tests.sh
File tests/run-js-tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/13/tests/run-js-tests.sh@31
PS13, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 13
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 21 Sep 2023 13:39:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14041/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 12 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 20 Sep 2023 17:00:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/12/www/scripts/query_timeline/run_js_tests.sh
File www/scripts/query_timeline/run_js_tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/12/www/scripts/query_timeline/run_js_tests.sh@31
PS12, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 12
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 20 Sep 2023 16:35:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/.gitignore A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js A www/scripts/query_timeline/package.json A www/scripts/query_timeline/run_js_tests.sh M www/scripts/util.js 15 files changed, 2,230 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/12 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 12 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14009/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 11 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 15 Sep 2023 12:44:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. These charts can also be resized until a maximum and minmum limit through the resize bar. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumulate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/.gitignore A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js A www/scripts/query_timeline/package.json A www/scripts/query_timeline/run_js_tests.sh M www/scripts/util.js 15 files changed, 2,075 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/11 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 11 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/11/www/scripts/query_timeline/run_js_tests.sh
File www/scripts/query_timeline/run_js_tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/11/www/scripts/query_timeline/run_js_tests.sh@31
PS11, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 11
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 15 Sep 2023 12:17:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14004/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 10 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 14 Sep 2023 20:37:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 10: Both the fragment metrics and host utilization charts can now be resized through the bar provided on top of them. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 10 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 14 Sep 2023 20:11:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. Thread usage is being shown on the additional Y-axis. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. These counters are available within the profile with following names. - MemoryUsage - ThreadUsage Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to resize and close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumalate values by parsing the profile. As the order and name of counters may change within the profile, to ensure appropriate mappings between plotted readings, initially the profile is traversed to find the correct mapping. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/.gitignore A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js A www/scripts/query_timeline/package.json A www/scripts/query_timeline/run_js_tests.sh M www/scripts/util.js 15 files changed, 2,073 insertions(+), 730 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/10 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 10 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/10/www/scripts/query_timeline/run_js_tests.sh
File www/scripts/query_timeline/run_js_tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/10/www/scripts/query_timeline/run_js_tests.sh@31
PS10, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 10
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 14 Sep 2023 20:10:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13985/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 9 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 12 Sep 2023 12:35:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumalate values by parsing the profile. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/.gitignore A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js A www/scripts/query_timeline/package.json A www/scripts/query_timeline/run_js_tests.sh M www/scripts/util.js 15 files changed, 1,937 insertions(+), 726 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/9 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 9 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20355/9/www/scripts/query_timeline/run_js_tests.sh
File www/scripts/query_timeline/run_js_tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/9/www/scripts/query_timeline/run_js_tests.sh@31
PS9, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 9
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 12 Sep 2023 12:09:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13955/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 8 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 08 Sep 2023 13:21:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20355/8/www/scripts/query_timeline/run_js_tests.sh
File www/scripts/query_timeline/run_js_tests.sh:
http://gerrit.cloudera.org:8080/#/c/20355/8/www/scripts/query_timeline/run_js_tests.sh@31
PS8, Line 31: curl
https://nodejs.org/dist/${NODEJS_VERSION}/node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
-O
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/20355/8/www/scripts/query_timeline/run_js_tests.sh@39
PS8, Line 39: rm -rf node-${NODEJS_VERSION}-${NODEJS_DISTRO}.tar.xz
node-${NODEJS_VERSION}-${NODEJS_DISTRO}/
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 8
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 08 Sep 2023 12:58:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. To help with manually testing the query timeline web page, the tests have been integrated using JEST testing framework based on nodejs. 'jest-environment-jsdom' supports rendering DOM elements and testing them without the requirement of a browser. 'run_js_tests.sh' runs JEST test suits, after it fetches nodejs binaries and related nodejs packages, if they are not present locally. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported as ES6 modules. Unit tests have been written for functions that accumalate values by parsing the profile. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/.gitignore A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/chart_commons.test.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_diagram.test.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.test.js A www/scripts/query_timeline/global_dom.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js A www/scripts/query_timeline/host_utilization_diagram.test.js A www/scripts/query_timeline/package.json A www/scripts/query_timeline/run_js_tests.sh M www/scripts/util.js 15 files changed, 1,899 insertions(+), 726 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/8 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 8 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13900/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 7 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Aug 2023 17:32:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. For implementing unit and integration tests, the script has been divided into multiple properly functioning modules, that are imported into the webpage. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl A www/scripts/query_timeline/chart_commons.js A www/scripts/query_timeline/fragment_diagram.js A www/scripts/query_timeline/fragment_metrics_diagram.js A www/scripts/query_timeline/global_members.js A www/scripts/query_timeline/host_utilization_diagram.js M www/scripts/util.js 7 files changed, 1,221 insertions(+), 726 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/7 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 7 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13856/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 15:08:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl M www/scripts/util.js 2 files changed, 590 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/5 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13838/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 24 Aug 2023 23:44:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 4: It might be useful to provide the ability to vertically resize the popup charts for better responsiveness. Before introducing further such features and complexities, I was planning to integrate a JS testing framework to ensure things are working properly. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 24 Aug 2023 23:21:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 4: I have mentioned the comments and problems arising from displaying aggregated fragment metrics and periodic metrics together. But, instead of increasing the visual complexity of periodic metrics further. As it was suggested to me, it might be better to align the gridline with tooltips across multiple popup charts, which can be closed or opened for better accessebility. Also, comparing multiple fragment's memory metrics and thread usage together in the same chart might prove helpful. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 24 Aug 2023 23:20:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 4: For small values of 'periodic_counter_update_period_ms', the cursors across the timeline were not displaying tooltips in accordance with hovering due to small floating point differences. As there was no API from the c3js library available, I looked through the library's source code and have fixed this issue. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 24 Aug 2023 23:20:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl M www/scripts/util.js 2 files changed, 580 insertions(+), 154 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/4 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 4 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl File www/query_timeline.tmpl: http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@65 PS2, Line 65: > Yes. I was planning to do it in the subsequent patch set and have done it. It's a little strange that you can close the periodic chart and there is no way to get it back except re-load but it won't hurt to leave it this way. http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@471 PS2, Line 471: - (fragment_id_selected == undefined? 0 : getFragmentMetricsHeight()) > Probably better to show fragment metric after periodic metrics so the perio Are you planning to address this? http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@778 PS2, Line 778: width : diagram_width The tics/scaling for the fragment metrics are not aligned properly now. http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1018 PS2, Line 1018: clearTimesamples(sampled_utilization_timeseries, max_samples_utilization); > Can we aggregate the fragment metrics into the periodic metrics too? Are you planning to address this? http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1106 PS2, Line 1106: timestamp_gridline = get_svg_line(stroke_fill_colors.black, e.pageX, 0, e.pageX, > can we make this so the cursors on all charts move together? This seems to be implemented now. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 23 Aug 2023 21:23:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@966
PS1, Line 966: if (max_samples_utilization.available <= 2) {
: var utilization_samples_message = `Warning: Not enough
samples for
: CPU utilization plot. Please decrease the value of
starting flag
: variable periodic_counter_update_period_ms
to
: increase the granularity of CPU utilization plot.`;
: try {
: host_utilization_chart =
host_utilization_chart.destroy();
: } catch (e) {
: host_utilization_chart = null;
: }
: displayWarning(host_utilization_diagram,
utilization_samples_message);
: utilization_metrics_parse_successful = true;
: return;
: }
: cpu
> Thanks, I have separated common the walk part.
Done
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@1001
PS1, Line 1001: clearTimeseriesValues(acc_usage, max_samples_utilization);
: });
: read_write_metrics_aggregate.forEach(function (acc_usage) {
: clearTimeseriesValues(acc_usage, max_samples_utilization);
: });
: clearTimesamples(sampled_utilization_timeseries,
max_samples_utilization);
: prev_utilization_num_samples =
profile.child_profiles[1].summary_stats_counters[0]
: .num_of_samples;
: utilization_metrics_parse_successful = true;
: } catch (e) {
: u
> Yes. It's supposed to normalize the samples.
Done
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@503
PS3, Line 503: var display_height = Math.max(diagram_min_height,
Math.min(window.innerHeight -
: timing_diagram.offsetTop - 50 -
(utilization_metrics_parse_successful
: && host_utilization_visible? getUtilizationHeight() +
diagram_controls_height : 0)
: - (fragment_metrics_parse_successful &&
fragment_id_selected != undefined?
: getFragmentMetricsHeight() + diagram_controls_height : 0)
, rownum * row_height));
Please break this down using local variable so the expression can be shorter.
var display_height = Math.max(diagram_min_height, Math.min(var_a, var,b));
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@938
PS3, Line 938: time_series_counters[i + offset_counters]
I'm a bit concern that look up on time_series_counters indices is straight by
the index rather than by checking the name first (time_series_counters[i +
offset_counters].name). There is no guarantee that the order will be the same
all or there will be no new counter that will change the order in the future.
Thus, we have name field in TTimeSeriesCounter.
Rather than offset_counters, maybe should pass the exact counter name you're
targeting (say, HostCpuSysPercentage) and look for counter with matching name.
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@944
PS3, Line 944: function collectUtilizationFromProfile(ignored_arg) {
This function is quite long. I'd suggest to add whitespace in between and put
comment on what each section does for better readability.
For example, add whitespace before line 955, 961, 980, 990, 996, 1000, 1006.
Even better if a section can be taken out as a function.
http://gerrit.cloudera.org:8080/#/c/20355/3/www/query_timeline.tmpl@1026
PS3, Line 1026: profile.child_profiles[2]
What is the name of TRuntimeProfileNode profile and profile.child_profiles[2]?
Can you document it as comment?
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 22 Aug 2023 19:35:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13804/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 3 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 22 Aug 2023 14:49:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 3:
(9 comments)
The gridlines and tooltips in all the charts are shown while hovering. Both the
charts contain a close button. Also, the code's redundancy has been reduced.
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@966
PS1, Line 966: if (max_samples_utilization.available <= 2) {
: var utilization_samples_message = `Warning: Not enough
samples for
: CPU utilization plot. Please decrease the value of
starting flag
: variable periodic_counter_update_period_ms
to
: increase the granularity of CPU utilization plot.`;
: try {
: host_utilization_chart =
host_utilization_chart.destroy();
: } catch (e) {
: host_utilization_chart = null;
: }
: displayWarning(host_utilization_diagram,
utilization_samples_message);
: utilization_metrics_parse_successful = true;
: return;
: }
: cpu
> It will be nice to to separate the profile walk part (extracting the sample
Thanks, I have separated common the walk part.
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@1001
PS1, Line 1001: clearTimeseriesValues(acc_usage, max_samples_utilization);
: });
: read_write_metrics_aggregate.forEach(function (acc_usage) {
: clearTimeseriesValues(acc_usage, max_samples_utilization);
: });
: clearTimesamples(sampled_utilization_timeseries,
max_samples_utilization);
: prev_utilization_num_samples =
profile.child_profiles[1].summary_stats_counters[0]
: .num_of_samples;
: utilization_metrics_parse_successful = true;
: } catch (e) {
: u
> Is this code meant to normalize the data samples?
Yes. It's supposed to normalize the samples.
I have created a function for it now.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@65
PS2, Line 65:Download :
> Need a way to hide the fragment chart too. Maybe clocking on the fragment c
Yes. I was planning to do it in the subsequent patch set and have done it.
Now, the fragment chart and utilization chart both contain a close button.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@469
PS2, Line 469: for (var cpn = 1; cpn < fp.child_profiles.length;
++cpn) {
> Timing height bound should not go below 200:
I have set the minimum height for all the diagrams as 200.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@670
PS2, Line 670: });
> Change to:
Done
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@674
PS2, Line 674: }
> Math.min(window.innerHeight * 0.3, 200);
Done
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@695
PS2, Line 695:
> This should not be B units for Memory, not B/s rates.
Done
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@766
PS2, Line 766: format : function (x) { return x.toFixed(decimals); }
> These should be mapped by label rather than position.
Done
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1151
PS2, Line 1151:
> Zooming out in the browser (WITH ctrl-'-') then back in (ctrl-'+') is not c
After providing the horizontal zoom option, only the height of charts and
diagrams is being resized according to the zoom.
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 22 Aug 2023 14:24:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl M www/scripts/util.js 2 files changed, 564 insertions(+), 155 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/3 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 3 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 2:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@65
PS2, Line 65:
Need a way to hide the fragment chart too. Maybe clocking on the fragment chart
to add an X to the corner of the fragment chart to close it.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@469
PS2, Line 469: var display_height = Math.min(window.innerHeight -
timing_diagram.offsetTop - 50
Timing height bound should not go below 200:
var display_height = Math.min(Math.max(200, window.innerHeight - timing_diag
- (utilization_metrics_parse_successful? getUtilizationHeight() : 0)
- (fragment_metrics_toogled? getFragmentMetricsHeight() : 0))
, rownum * row_height);
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@471
PS2, Line 471: - (fragment_id_selected == undefined? 0 :
getFragmentMetricsHeight())
Probably better to show fragment metric after periodic metrics so the periodic
metric don't move. Also would be nice if you can display multiple fragment
metrics at the same time.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@670
PS2, Line 670: return window.innerHeight * 0.3;
Change to:
Math.min(window.innerHeight * 0.3, 200);
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@674
PS2, Line 674: return window.innerHeight * 0.2;
Math.min(window.innerHeight * 0.3, 200);
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@695
PS2, Line 695: var units = ['B/s', 'KB/s', 'MB/s', 'GB/s', 'TB/s', 'PB/s'];
This should not be B units for Memory, not B/s rates.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@766
PS2, Line 766: axis_mappings[fragment_metrics_counter_names[0]] = 'y';
These should be mapped by label rather than position.
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1018
PS2, Line 1018: clearTimesamples(sampled_utilization_timeseries,
max_samples_utilization);
Can we aggregate the fragment metrics into the periodic metrics too?
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1106
PS2, Line 1106: timestamp_gridline = get_svg_line(stroke_fill_colors.black,
e.pageX, 0, e.pageX,
can we make this so the cursors on all charts move together?
http://gerrit.cloudera.org:8080/#/c/20355/2/www/query_timeline.tmpl@1151
PS2, Line 1151: window.addEventListener('resize', function(event) {
Zooming out in the browser (WITH ctrl-'-') then back in (ctrl-'+') is not
currently restoring the original layout.
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 2
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Surya Hebbar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 17 Aug 2023 18:07:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13763/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 16 Aug 2023 14:13:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. The thread usage is being shown on the additional Y-axis. A temporary warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl 1 file changed, 428 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/2 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 1:
(2 comments)
As you add more functionalities to this script, you may move some functions to
separate script files to keep file size in reasonable size.
You also need to add some unit-test. Otherwise we have to run tests for each
function manually for each version of patch. There is no testing framework for
Javascript in Impala. You need to bring in one testing framework.
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@499
PS1, Line 499: try { parentElement.removeChild(childElement); }
: catch(e) {}
nit: break to multiple lines
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@629
PS1, Line 629: fragment_metrics_chart.load({
: unload : true
: });
nit: don't need to break to multiple lines
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 1
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 15 Aug 2023 21:44:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20355 )
Change subject: IMPALA-12364: Display memory, disk and network metrics in
webUI's query timeline
..
Patch Set 1:
(4 comments)
I have only these few comments for now. Will do another pass.
ps: we add more JS code recently. Probably it is time to experiment having some
codestyle checker like eslint in Impala codebase? I wish I can do something
like this to see potential JS issues:
npx eslint www/query_timeline.tmpl
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl
File www/query_timeline.tmpl:
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@701
PS1, Line 701: { val = val / 1e3; ++current_unit; }
nit: better make it multiline since there are more than 1 statement.
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@886
PS1, Line 886: if
(profile.child_profiles[1].summary_stats_counters[0].num_of_samples !=
: prev_utilization_num_samples) {
nit: This can be simplified by returning early:
if (profile.child_profiles[1].summary_stats_counters[0].num_of_samples ==
prev_utilization_num_samples) {
utilization_metrics_parse_successful = true;
return;
}
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@966
PS1, Line 966: var profile_fragments =
profile.child_profiles[2].child_profiles;
: try {
: profile_fragments.forEach(function (fragment_profile) {
: if(fragment_profile.profile_name == fragment_id) {
: selected_fragment_profile = fragment_profile;
: throw new Error();
: }
: });
: } catch(e) {}
: selected_fragment_profile.child_profiles.forEach(function
(fragment_node) {
: for (var i = 0; i < fragment_metrics_counter_names.length;
++i) {
: accumulateTimeseriesValues(fragment_metrics_aggregate[i],
: fragment_node.time_series_counters[i],
max_samples_fragment_metrics);
: }
: });
It will be nice to to separate the profile walk part (extracting the sample) as
their own function. Same comment for similar code in
collectUtilizationFromProfile().
http://gerrit.cloudera.org:8080/#/c/20355/1/www/query_timeline.tmpl@1001
PS1, Line 1001: var avg_period = max_samples_fragment_metrics.collected *
max_samples_fragment_metrics.period
: / max_samples_fragment_metrics.available;
: avg_period = avg_period / 1000;
: for (var k = 0; k <=
max_samples_fragment_metrics.available; k++) {
: sampled_fragment_metrics_timeseries[k + 1] = (k *
avg_period);
: }
: if (maxts / 1e9 >
sampled_utilization_timeseries[max_samples_fragment_metrics.available + 1]) {
:
sampled_fragment_metrics_timeseries[max_samples_fragment_metrics.available + 1]
= maxts / 1e9;
: } else {
: maxts =
sampled_fragment_metrics_timeseries[max_samples_fragment_metrics.available + 1]
* 1e9;
: }
Is this code meant to normalize the data samples?
There is similar/duplicated code in collectUtilizationFromProfile(). Maybe
extract this out as a shared function between them?
--
To view, visit http://gerrit.cloudera.org:8080/20355
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f
Gerrit-Change-Number: 20355
Gerrit-PatchSet: 1
Gerrit-Owner: Surya Hebbar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 15 Aug 2023 17:27:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13743/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 14 Aug 2023 17:03:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20355 Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. The thread usage is being shown on the additional Y-axis. A temporary warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl 1 file changed, 391 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/1 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar
