[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/13806/11/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/13806/11/tests/webserver/test_web_pages.py@765 PS11, Line 765: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 07 Feb 2020 14:37:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. IMPALA-6663: Expose current DDL metrics on WebUI This change adds a new '/operations' tab to the Catalog WebUI. On this page the users can see two summary tables, the first report is based on how many catalog operations are running and which tables are involved. The second report shows the number of catalog operations on table level. A new 'monitor' package is created to collect the monitoring classes. Each DDL/FinalizeDML/ResetMetada request get a class with common base class. The output of these classes are summarized and can be accessed with a new JNI call. A screenshot of the new '/operations' page can be found in the Jira. Testing: Unit tests were added to test the counters and a web server test to verify the availability of the page. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java M tests/webserver/test_web_pages.py A www/catalog_operations.tmpl 22 files changed, 1,260 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/11 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. IMPALA-6663: Expose current DDL metrics on WebUI This change adds a new '/operations' tab to the Catalog WebUI. On this page the users can see two summary tables, the first report is based on how many catalog operations are running and which tables are involved. The second report shows the number of catalog operations on table level. A new 'monitor' package is created to collect the monitoring classes. Each DDL/FinalizeDML/ResetMetada request get a class with common base class. The output of these classes are summarized and can be accessed with a new JNI call. A screenshot of the new '/operations' page can be found in the Jira. Testing: Unit tests were added to test the counters and a web server test to verify the availability of the page. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java M tests/webserver/test_web_pages.py A www/catalog_operations.tmpl 22 files changed, 1,261 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/12 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 12 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/13806/12/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/13806/12/tests/webserver/test_web_pages.py@766 PS12, Line 766: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 12 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 07 Feb 2020 14:46:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. IMPALA-6663: Expose current DDL metrics on WebUI This change adds a new '/operations' tab to the Catalog WebUI. On this page the users can see two summary tables, the first report is based on how many catalog operations are running and which tables are involved. The second report shows the number of catalog operations on table level. A new 'monitor' package is created to collect the monitoring classes. Each DDL/FinalizeDML/ResetMetada request get a class with common base class. The output of these classes are summarized and can be accessed with a new JNI call. A screenshot of the new '/operations' page can be found in the Jira. Testing: Unit tests were added to test the counters and a web server test to verify the availability of the page. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java M tests/webserver/test_web_pages.py A www/catalog_operations.tmpl 22 files changed, 1,260 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/13 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 13 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5153/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 11 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 07 Feb 2020 15:23:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5154/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 12 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 07 Feb 2020 15:32:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5156/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 13 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 07 Feb 2020 16:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
..
Patch Set 13:
(4 comments)
Only found a few nits, other then that LGTM
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java@34
PS13, Line 34: super();
I think it's implicitly called by Java.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@39
PS13, Line 39: AtomicLong
nit: since everything is in 'synchronized' I think a simple Long is enough
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@77
PS13, Line 77: summarizTable.javaes
typos
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java@65
PS13, Line 65: Optional tTableName =
: req.table_name != null ? Optional.of(req.table_name) :
Optional.empty();
: if (req.is_refresh) {
: decrementCounter(ResetMetadataType.REFRESH.toString(),
getTableName(tTableName));
: } else if (tTableName.isPresent()) {
: decrementCounter(
: ResetMetadataType.INVALIDATE_METADATA.toString(),
getTableName(tTableName));
: } else {
:
decrementCounter(ResetMetadataType.INVALIDATE_METADATA_GLOBAL.toString(),
: getTableName(tTableName));
nit: this is mostly duplicated from increment. Seems like it wouldn't be too
much effort to refactor the duplicated parts into a helper method.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Norbert Luksa
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 11 Feb 2020 13:33:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 13: Code-Review+1 (2 comments) Left a small nit, but lgtm. http://gerrit.cloudera.org:8080/#/c/13806/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13806/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@427 PS8, Line 427: case DROP_FUNCTION: > It is possible that I misunderstood the concerns here, but I will try to an I see, I misunderstood something, thanks for clarifying. http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java File fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java: http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java@34 PS13, Line 34: private final String TEST_UPDATE_SQL = "UPDATE table SET c3 = " : + "'test';"; nit: this string could be written without concatenation. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 13 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 13 Feb 2020 11:41:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Tamas Mate has posted comments on this change. (
http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663: Expose current DDL metrics on WebUI
..
Patch Set 13:
(5 comments)
Thank you for the review, additionally to the comments there was a test failure
due to COMMENT_ON requests, which was fixed as well.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java
File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java@34
PS13, Line 34: super();
> I think it's implicitly called by Java.
Done
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@39
PS13, Line 39: AtomicLong
> nit: since everything is in 'synchronized' I think a simple Long is enough
Right, moved from AtomicLong to MutableLong.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java@77
PS13, Line 77: summarizTable.javaes
> typos
Done
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java@65
PS13, Line 65: Optional tTableName =
: req.table_name != null ? Optional.of(req.table_name) :
Optional.empty();
: if (req.is_refresh) {
: decrementCounter(ResetMetadataType.REFRESH.toString(),
getTableName(tTableName));
: } else if (tTableName.isPresent()) {
: decrementCounter(
: ResetMetadataType.INVALIDATE_METADATA.toString(),
getTableName(tTableName));
: } else {
:
decrementCounter(ResetMetadataType.INVALIDATE_METADATA_GLOBAL.toString(),
: getTableName(tTableName));
> nit: this is mostly duplicated from increment. Seems like it wouldn't be to
Refactored, created a new method.
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java
File
fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java:
http://gerrit.cloudera.org:8080/#/c/13806/13/fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java@34
PS13, Line 34: private final String TEST_UPDATE_SQL = "UPDATE table SET c3 = "
: + "'test';";
> nit: this string could be written without concatenation.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 13
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Norbert Luksa
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 25 Feb 2020 14:07:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. IMPALA-6663: Expose current DDL metrics on WebUI This change adds a new '/operations' tab to the Catalog WebUI. On this page the users can see two summary tables, the first report is based on how many catalog operations are running and which tables are involved. The second report shows the number of catalog operations on table level. A new 'monitor' package is created to collect the monitoring classes. Each DDL/FinalizeDML/ResetMetada request get a class with common base class. The output of these classes are summarized and can be accessed with a new JNI call. A screenshot of the new '/operations' page can be found in the Jira. Testing: Unit tests were added to test the counters and a web server test to verify the availability of the page. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java M tests/webserver/test_web_pages.py A www/catalog_operations.tmpl 22 files changed, 1,216 insertions(+), 158 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/14 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 14 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5328/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 14 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 25 Feb 2020 14:57:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 14: Code-Review+2 Thanks for applying the changes! -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 14 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Feb 2020 13:47:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 15 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Feb 2020 13:47:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5421/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 15 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Feb 2020 13:47:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 15: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5421/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 15 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Feb 2020 18:15:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. IMPALA-6663: Expose current DDL metrics on WebUI This change adds a new '/operations' tab to the Catalog WebUI. On this page the users can see two summary tables, the first report is based on how many catalog operations are running and which tables are involved. The second report shows the number of catalog operations on table level. A new 'monitor' package is created to collect the monitoring classes. Each DDL/FinalizeDML/ResetMetada request get a class with common base class. The output of these classes are summarized and can be accessed with a new JNI call. A screenshot of the new '/operations' page can be found in the Jira. Testing: Unit tests were added to test the counters and a web server test to verify the availability of the page. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java M tests/webserver/test_web_pages.py A www/catalog_operations.tmpl 22 files changed, 1,202 insertions(+), 158 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/16 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 16 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 17 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 03 Mar 2020 12:54:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 16 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 03 Mar 2020 12:54:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5445/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 17 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 03 Mar 2020 12:54:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5404/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 16 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 03 Mar 2020 13:35:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. IMPALA-6663: Expose current DDL metrics on WebUI This change adds a new '/operations' tab to the Catalog WebUI. On this page the users can see two summary tables, the first report is based on how many catalog operations are running and which tables are involved. The second report shows the number of catalog operations on table level. A new 'monitor' package is created to collect the monitoring classes. Each DDL/FinalizeDML/ResetMetada request get a class with common base class. The output of these classes are summarized and can be accessed with a new JNI call. A screenshot of the new '/operations' page can be found in the Jira. Testing: Unit tests were added to test the counters and a web server test to verify the availability of the page. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Reviewed-on: http://gerrit.cloudera.org:8080/13806 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogDdlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogDdlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounterTest.java A fe/src/test/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounterTest.java M tests/webserver/test_web_pages.py A www/catalog_operations.tmpl 22 files changed, 1,202 insertions(+), 158 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 18 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6663: Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663: Expose current DDL metrics on WebUI .. Patch Set 17: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 17 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 03 Mar 2020 17:46:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13806 Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. IMPALA-6663 Expose current DDL metrics on WebUI This change adds two new tables to the Catalog WebUI's '/catalog' page. These tables are exposing the number of running catalog operations, which helps to better understand the catalog load and reason when the catalog is overloaded. A new 'monitor' package is created to collect the monitoring sub-classes. The new counters are called just after JNI methods are invoked, so it can notice the exceptions. The existing thrift structure is extended with the new counters, these values are unwrapped and displayed on the WebUI. Before/after pictures of the /catalog page can be found in the Jira. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M www/catalog.tmpl 11 files changed, 360 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/3 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13806/3/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/13806/3/fe/src/main/java/org/apache/impala/catalog/Table.java@186 PS3, Line 186: CatalogUsageMonitor.INSTANCE.getCatalogTableMetrics().updateFrequentlyAccessedTables(this); line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/13806/3/fe/src/main/java/org/apache/impala/catalog/Table.java@193 PS3, Line 193: CatalogUsageMonitor.INSTANCE.getCatalogTableMetrics().updateHighFileCountTables(this); line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Jul 2019 13:52:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3837/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Jul 2019 14:32:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. IMPALA-6663 Expose current DDL metrics on WebUI This change adds two new tables to the Catalog WebUI's '/catalog' page. These tables are exposing the number of running catalog operations, which helps to better understand the catalog load and reason when the catalog is overloaded. A new 'monitor' package is created to collect the monitoring sub-classes. The new counters are called just after JNI methods are invoked, so it can notice the exceptions. The existing thrift structure is extended with the new counters, these values are unwrapped and displayed on the WebUI. Before/after pictures of the '/catalog' page can be found in the Jira. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M www/catalog.tmpl 11 files changed, 362 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/4 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3846/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Jul 2019 07:23:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Jul 2019 07:24:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4587/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Jul 2019 09:03:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4588/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Jul 2019 12:43:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4588/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Jul 2019 14:21:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4597/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 6 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jul 2019 09:22:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4597/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 6 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jul 2019 11:00:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4598/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 6 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jul 2019 15:19:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 6: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4598/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 6 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 15 Jul 2019 16:55:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. IMPALA-6663 Expose current DDL metrics on WebUI This change adds two new tables to the Catalog WebUI's '/catalog' page. These tables are exposing the number of running catalog operations, which helps to better understand the catalog load and reason when the catalog is overloaded. A new 'monitor' package is created to collect the monitoring sub-classes. The new counters are called just after JNI methods are invoked, so it can notice the exceptions. The existing thrift structure is extended with the new counters, these values are unwrapped and displayed on the WebUI. Before/after pictures of the '/catalog' page can be found in the Jira. Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a --- M be/src/catalog/catalog-server.cc M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java R fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M www/catalog.tmpl 11 files changed, 367 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/13806/7 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4646/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 15:16:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 15:57:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4646/ -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Jul 2019 21:48:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4655/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Jul 2019 04:53:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 25 Jul 2019 11:12:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: The first build today failed due to a maven build problem. The previous builds failed because the CatalogOpRequestCounter.incrementDdlTypeCounter was not threadsafe, it is fixed now. Please let me know your thoughts on this change. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 25 Jul 2019 11:41:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: (2 comments) > (4 comments) > > I think this in general is a very useful addition. Can you also > specify the motivation of why we need to track the total number of > DDLs instead of tracking them on a per table level. Certain > metadata operations like invalidate metadata as indeed global and > cannot be tracked at table. However, I think it useful to track how > many refresh or invalidates were called on a given table and > finding top-n such tables. Similarly for DMLs, instead of a getting > a total count of INSERTS, would it be more useful to get top-N > tables which are inserted into? Took a second pass and realized that you are indeed tracking currently running DDL ops. The question in such a case is how do we determine if seeing 3 resetMetadata operations on 3 small tables is better or worse than 1 resetMetadata operation on a large table. Can you please comment on how should the end user interpret these values to determine the load on the catalog? http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java: http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@70 PS7, Line 70: decrementDdlTypeCounter > Is decrement ever needed? same for the other counters? you may disregard this comment since I thought you were only tracking the global count of operations instead of running ones. http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl File www/catalog.tmpl: http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl@202 PS7, Line 202: Running > Is this user-visible text? Running seems to suggest these operations are cu I think I misunderstood the patch a little bit. Looks like we are indeed tracking the running operations. You may discard this comment. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 13 Aug 2019 17:32:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: (4 comments) I think this in general is a very useful addition. Can you also specify the motivation of why we need to track the total number of DDLs instead of tracking them on a per table level. Certain metadata operations like invalidate metadata as indeed global and cannot be tracked at table. However, I think it useful to track how many refresh or invalidates were called on a given table and finding top-n such tables. Similarly for DMLs, instead of a getting a total count of INSERTS, would it be more useful to get top-N tables which are inserted into? http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java: http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@41 PS7, Line 41: private AtomicLong resetMetadataCounter_; : : private AtomicLong dmlOperationCounter_; : : private AtomicLong syncDdlOperationCounter_; Can you add a one line comment on each of these counters as to what do they represent? Also, would be great if you could specify what operations constitute a DML in this context. http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@70 PS7, Line 70: decrementDdlTypeCounter Is decrement ever needed? same for the other counters? http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@109 PS7, Line 109: getResetMetadataCounter By returning the AtomicLong object this class is leaking the private field and it is subject to modification by other classes. A more common pattern is to return its value (return type long). http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl File www/catalog.tmpl: http://gerrit.cloudera.org:8080/#/c/13806/7/www/catalog.tmpl@202 PS7, Line 202: Running Is this user-visible text? Running seems to suggest these operations are currently being run. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 13 Aug 2019 17:24:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: Hi Vihang, thank your for the review. These metrics could be useful to identify if the CatalogD starts to become the bottleneck, for example when the metadata operations start piling up due to IMPALA-6671. Although, this change will not help to identify which tables are causing problem, it could give the user an idea on how busy/stuck the CatalogD. To have further understanding on the load I believe we would need historical data. Displaying the table names could be a useful addition, will add a third column to this patch. What do you think? -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 14 Aug 2019 07:24:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: (1 comment) I have started adding the table names to this change, will need some additional time to finish. http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java: http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@41 PS7, Line 41: private AtomicLong resetMetadataCounter_; : : private AtomicLong dmlOperationCounter_; : : private AtomicLong syncDdlOperationCounter_; > Can you add a one line comment on each of these counters as to what do they DML in this context is when an INSERT/CTAS is being finalized, it is triggered here: https://github.com/apache/impala/blob/branch-3.3.0/be/src/service/client-request-state.cc#L1071 The current name is not very descriptive. -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 21 Aug 2019 16:20:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13806 ) Change subject: IMPALA-6663 Expose current DDL metrics on WebUI .. Patch Set 7: (1 comment) > (1 comment) > > I have started adding the table names to this change, will need > some additional time to finish. I will be on vacation for a month. I would not be able to take a look at this patch until I come back. May ask Bharath to take a look? http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java@190 PS7, Line 190: incrementResetMetadataCounter there are quite a few variations of reset metadata (global invalidate, refresh table, refresh functions, refresh authorization). I think it will be useful to have them track them individually -- To view, visit http://gerrit.cloudera.org:8080/13806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a Gerrit-Change-Number: 13806 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 28 Aug 2019 20:08:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6663 Expose current DDL metrics on WebUI
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/13806 )
Change subject: IMPALA-6663 Expose current DDL metrics on WebUI
..
Patch Set 7:
(6 comments)
Found a couple of nits, other than that lgtm. It will be even more useful when
you add the table names to the metrics.
http://gerrit.cloudera.org:8080/#/c/13806/7/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/13806/7/be/src/catalog/catalog-server.cc@545
PS7, Line 545: int i = 0; i < catalog_usage_result.ddl_type_counter.size(); ++i
nit: range-based for loop can be used instead:
for (const auto& ddl_type_counter : catalog_usage_result.ddl_type_counter) {
..
}
http://gerrit.cloudera.org:8080/#/c/13806/7/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/13806/7/common/thrift/JniCatalog.thrift@718
PS7, Line 718: dll
typo: ddl
http://gerrit.cloudera.org:8080/#/c/13806/7/common/thrift/JniCatalog.thrift@720
PS7, Line 720:
nit: unnecessary blank line
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOpRequestCounter.java@62
PS7, Line 62: if (isSyncDdl) {
: syncDdlOperationCounter_.incrementAndGet();
: }
nit: this and the other 'if (isSyncDdl)' statements in this file would fit in a
single line.
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogTableMetrics.java@26
PS7, Line 26: import com.google.common.base.Function;
it's also imported at L20.
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13806/7/fe/src/main/java/org/apache/impala/service/JniCatalog.java@95
PS7, Line 95: catalogUsageMonitor
nit: maybe you could create a reference to the CatalogOpRequestCounter object
since you only use that later.
This way all the increments and decrements will fit in a single line.
--
To view, visit http://gerrit.cloudera.org:8080/13806
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0ed76f134bad6d3b3d4dce132365a53a01e9512a
Gerrit-Change-Number: 13806
Gerrit-PatchSet: 7
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 22 Oct 2019 12:35:02 +
Gerrit-HasComments: Yes
