[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches There are several endpoints in WebUI that can dump a query profile: /query_profile, /query_profile_encoded, /query_profile_plain_text, /query_profile_json. The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the ClientRequestState. This could block client requests in fetching query results. To help identify this issue, this patch adds warning logs when such profile dumping requests run slow and the query is still in-flight. Also adds a profile counter, GetInFlightProfileTimeStats, for the summary stats of this time. Dumping the profiles after the query is archived (e.g. closed) won't be tracked. Logs for slow http responses are also added. The thresholds are defined by two new flags, slow_profile_dump_warning_threshold_ms, and slow_http_response_warning_threshold_ms. Note that dumping the profile in-flight won't always block the query, e.g. if there are no client fetch requests or if the coordinator fragment is idle waiting for executor fragment instances. So a long time shown in GetInFlightProfileTimeStats doesn't mean it's hitting the issue. To better identify this issue, this patch adds another profile counter, ClientFetchLockWaitTimer, as the cumulative time client fetch requests waiting for locks. Also fixes false positive logs for complaining invalid query handles. Such logs are added in GetQueryHandle() when the query is not found in the active query map, but it could still exist in the query log. This removes the logs in GetQueryHandle() and lets the callers decide whether to log the error. Tests: - Added e2e test - Ran CORE tests Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Reviewed-on: http://gerrit.cloudera.org:8080/21412 Reviewed-by: Impala Public Jenkins Tested-by: Michael Smith --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/util/webserver.cc M tests/query_test/test_observability.py 8 files changed, 101 insertions(+), 13 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Michael Smith: Verified -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 5: Verified+1 Ran into https://issues.apache.org/jira/browse/IMPALA-12266. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 May 2024 18:32:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Michael Smith has removed a vote on this change. Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10664/ -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 May 2024 18:28:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10664/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 May 2024 13:17:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 May 2024 13:17:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 May 2024 12:54:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16210/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 May 2024 03:06:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/21412 )
Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests
blocking client fetches
..
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/client-request-state.h@506
PS3, Line 506: void AddClientFetchLockWaitTime(int64_t lock_wait_time_ns) {
> another review uses "AddFetchLockWaitTime" for similar purpose. https://ger
Oops, just realized that patch.
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-beeswax-server.cc@352
PS3, Line 352: VLOG(1) << "Error in get_log, could not get query hand
> Is this supposed to be a long term log? It could contain more information,
Done
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-hs2-server.cc@1099
PS3, Line 1099: VLOG(1) << "Error in GetLog, could not get query handle:
" << status.GetDetail();
> Same as in impala-beeswax-server.cc
Done
--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 23 May 2024 02:42:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Hello Kurt Deschler, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21412 to look at the new patch set (#4). Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches There are several endpoints in WebUI that can dump a query profile: /query_profile, /query_profile_encoded, /query_profile_plain_text, /query_profile_json. The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the ClientRequestState. This could block client requests in fetching query results. To help identify this issue, this patch adds warning logs when such profile dumping requests run slow and the query is still in-flight. Also adds a profile counter, GetInFlightProfileTimeStats, for the summary stats of this time. Dumping the profiles after the query is archived (e.g. closed) won't be tracked. Logs for slow http responses are also added. The thresholds are defined by two new flags, slow_profile_dump_warning_threshold_ms, and slow_http_response_warning_threshold_ms. Note that dumping the profile in-flight won't always block the query, e.g. if there are no client fetch requests or if the coordinator fragment is idle waiting for executor fragment instances. So a long time shown in GetInFlightProfileTimeStats doesn't mean it's hitting the issue. To better identify this issue, this patch adds another profile counter, ClientFetchLockWaitTimer, as the cumulative time client fetch requests waiting for locks. Also fixes false positive logs for complaining invalid query handles. Such logs are added in GetQueryHandle() when the query is not found in the active query map, but it could still exist in the query log. This removes the logs in GetQueryHandle() and lets the callers decide whether to log the error. Tests: - Added e2e test - Ran CORE tests Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/util/webserver.cc M tests/query_test/test_observability.py 8 files changed, 101 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/21412/4 -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/21412 )
Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests
blocking client fetches
..
Patch Set 3: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/client-request-state.h@506
PS3, Line 506: void UpdateClientFetchLockWaitTime(int64_t lock_wait_time_ns) {
another review uses "AddFetchLockWaitTime" for similar purpose.
https://gerrit.cloudera.org/#/c/20850/16/be/src/service/client-request-state.h
"add" seems clearer to me in this case than "update"
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-beeswax-server.cc@352
PS3, Line 352: VLOG(1) << "Error in get_log: " << status.GetDetail();
Is this supposed to be a long term log? It could contain more information, e.g.
"Error in get_log, could not get query handle:"
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/21412/3/be/src/service/impala-hs2-server.cc@1099
PS3, Line 1099: VLOG(1) << "Error in GetLog: " << status.GetDetail();
Same as in impala-beeswax-server.cc
--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Tue, 21 May 2024 11:03:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16182/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 17 May 2024 20:00:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 3: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/21412/2/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/21412/2/be/src/service/client-request-state.h@498 PS2, Line 498: > Done. Implicitly inline them by defining them here (inside the declaration Done http://gerrit.cloudera.org:8080/#/c/21412/2/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/21412/2/tests/query_test/test_observability.py@974 PS2, Line 974: impalad = cluster.get_first_impalad() > Didn't dig deeper into this. We can use handle.get_handle().id here. Done -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 17 May 2024 17:18:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21412/2/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/21412/2/be/src/service/client-request-state.h@498 PS2, Line 498: > nit: we usually include a comment describing the function. Might be nice to Done. Implicitly inline them by defining them here (inside the declaration of the class). http://gerrit.cloudera.org:8080/#/c/21412/2/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/21412/2/tests/query_test/test_observability.py@974 PS2, Line 974: impalad = cluster.get_first_impalad() > Doesn't handle.query_id have the query ID? Not sure why you're parsing the Didn't dig deeper into this. We can use handle.get_handle().id here. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 17 May 2024 07:19:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Hello Kurt Deschler, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21412 to look at the new patch set (#3). Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches There are several endpoints in WebUI that can dump a query profile: /query_profile, /query_profile_encoded, /query_profile_plain_text, /query_profile_json. The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the ClientRequestState. This could block client requests in fetching query results. To help identify this issue, this patch adds warning logs when such profile dumping requests run slow and the query is still in-flight. Also adds a profile counter, GetInFlightProfileTimeStats, for the summary stats of this time. Dumping the profiles after the query is archived (e.g. closed) won't be tracked. Logs for slow http responses are also added. The thresholds are defined by two new flags, slow_profile_dump_warning_threshold_ms, and slow_http_response_warning_threshold_ms. Note that dumping the profile in-flight won't always block the query, e.g. if there are no client fetch requests or if the coordinator fragment is idle waiting for executor fragment instances. So a long time shown in GetInFlightProfileTimeStats doesn't mean it's hitting the issue. To better identify this issue, this patch adds another profile counter, ClientFetchLockWaitTimer, as the cumulative time client fetch requests waiting for locks. Also fixes false positive logs for complaining invalid query handles. Such logs are added in GetQueryHandle() when the query is not found in the active query map, but it could still exist in the query log. This removes the logs in GetQueryHandle() and lets the callers decide whether to log the error. Tests: - Added e2e test - Ran CORE tests Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/util/webserver.cc M tests/query_test/test_observability.py 8 files changed, 101 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/21412/3 -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/21412/2/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/21412/2/be/src/service/client-request-state.h@498 PS2, Line 498: void UpdateGetInFlightProfileTime(int64_t elapsed_time_ns); nit: we usually include a comment describing the function. Might be nice to reference the visible metric name it updates. Also might make sense as inline functions since they're so simple. http://gerrit.cloudera.org:8080/#/c/21412/2/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/21412/2/tests/query_test/test_observability.py@974 PS2, Line 974: query_id = query_id_search.group(1) Doesn't handle.query_id have the query ID? Not sure why you're parsing the profile. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 14 May 2024 21:16:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16132/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 13:44:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21412 to look at the new patch set (#2). Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches There are several endpoints in WebUI that can dump a query profile: /query_profile, /query_profile_encoded, /query_profile_plain_text, /query_profile_json. The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the ClientRequestState. This could block client requests in fetching query results. To help identify this issue, this patch adds warning logs when such profile dumping requests run slow and the query is still in-flight. Also adds a profile counter, GetInFlightProfileTimeStats, for the summary stats of this time. Dumping the profiles after the query is archived (e.g. closed) won't be tracked. Logs for slow http responses are also added. The thresholds are defined by two new flags, slow_profile_dump_warning_threshold_ms, and slow_http_response_warning_threshold_ms. Note that dumping the profile in-flight won't always block the query, e.g. if there are no client fetch requests or if the coordinator fragment is idle waiting for executor fragment instances. So a long time shown in GetInFlightProfileTimeStats doesn't mean it's hitting the issue. To better identify this issue, this patch adds another profile counter, ClientFetchLockWaitTimer, as the cumulative time client fetch requests waiting for locks. Also fixes false positive logs for complaining invalid query handles. Such logs are added in GetQueryHandle() when the query is not found in the active query map, but it could still exist in the query log. This removes the logs in GetQueryHandle() and lets the callers decide whether to log the error. Tests: - Added e2e test - Ran CORE tests Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/util/webserver.cc M tests/query_test/test_observability.py 8 files changed, 103 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/21412/2 -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/21412 )
Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests
blocking client fetches
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972:
> flake8: W605 invalid escape sequence '\('
Done
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972: )
> flake8: W605 invalid escape sequence '\)'
Done
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@985
PS1, Line 985: i
> flake8: F841 local variable 'page' is assigned to but never used
Done
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: (
> flake8: W605 invalid escape sequence '\d'
Done
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: )
> flake8: W605 invalid escape sequence '\)'
Done
--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 08 May 2024 13:21:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21412 ) Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16130/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 May 2024 12:19:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21412 Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches .. IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches There are several endpoints in WebUI that can dump a query profile: /query_profile, /query_profile_encoded, /query_profile_plain_text, /query_profile_json. The HTTP handler thread goes into ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the ClientRequestState. This could block client requests in fetching query results. To help identify this issue, this patch adds warning logs when such profile dumping requests run slow and the query is still in-flight. Also adds a profile counter, GetInFlightProfileTimeStats, for the summary stats of this time. Dumping the profiles after the query is archived (e.g. closed) won't be tracked. Logs for slow http responses are also added. The thresholds are defined by two new flags, slow_profile_dump_warning_threshold_ms, and slow_http_response_warning_threshold_ms. Note that dumping the profile in-flight won't always block the query, e.g. if there are no client fetch requests or if the coordinator fragment is idle waiting for executor fragment instances. So a long time shown in GetInFlightProfileTimeStats doesn't mean it's hitting the issue. To better identify this issue, this patch adds another profile counter, ClientFetchLockWaitTimer, as the cumulative time client fetch requests waiting for locks. Also fixes false positive logs for complaining invalid query handles. Such logs are added in GetQueryHandle() when the query is not found in the active query map, but it could still exist in the query log. This removes the logs in GetQueryHandle() and lets the callers decide whether to log the error. Tests: - Added e2e test - Ran CORE tests Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/util/webserver.cc M tests/query_test/test_observability.py 8 files changed, 103 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/21412/1 -- To view, visit http://gerrit.cloudera.org:8080/21412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc Gerrit-Change-Number: 21412 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/21412 )
Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests
blocking client fetches
..
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972: \
flake8: W605 invalid escape sequence '\('
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972: \
flake8: W605 invalid escape sequence '\)'
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@985
PS1, Line 985: p
flake8: F841 local variable 'page' is assigned to but never used
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: \
flake8: W605 invalid escape sequence '\d'
http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: \
flake8: W605 invalid escape sequence '\)'
--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 08 May 2024 11:56:39 +
Gerrit-HasComments: Yes
