Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
ajkh88 commented on PR #7000: URL: https://github.com/apache/hbase/pull/7000#issuecomment-2935256579 In addition to the tests in this PR, I have deployed this to a test cluster and ran a load test against it. Everything is functioning as expected -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
ajkh88 commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2121245360
##
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long
actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long
estimateSize) {
return calculateReadCapacityUnit(actualSize) -
calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double
requestsPerSecond,
Review Comment:
In this context, `handler usage time` refers to the amount of time (in
milliseconds) a handler thread will use.
We estimate the number of milliseconds we expect the thread to take to
complete the request and deduct that amount from the quota. If there are enough
milliseconds left in the quota, we proceed by taking that amount from the quota
and continuing with the request. Otherwise, we throw a throttling exception.
When the request is complete, we calculate the actual amount of time used in
the `close()` method and adjust the quota by either adding back time or further
subtracting it, as appropriate.
Does this make things clearer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Apache9 commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2119198564
##
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long
actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long
estimateSize) {
return calculateReadCapacityUnit(actualSize) -
calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double
requestsPerSecond,
Review Comment:
Mind explain a bit more about the algorithm here? I do not fully understand
what does 'handler usage time' mean...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Apache-HBase commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-2893146732
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s):
--brief-report-file --spotbugs-strict-precheck --author-ignore-list
--blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
_ Prechecks _ |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 14s | | master passed |
| +1 :green_heart: | compile | 1m 51s | | master passed |
| +1 :green_heart: | javadoc | 0m 54s | | master passed |
| +1 :green_heart: | shadedjars | 5m 57s | | branch has no errors when
building our shaded downstream artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 51s | | the patch passed |
| +1 :green_heart: | javac | 1m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed |
| +1 :green_heart: | shadedjars | 5m 57s | | patch has no errors when
building our shaded downstream artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hbase-protocol-shaded in the
patch passed. |
| +1 :green_heart: | unit | 1m 34s | | hbase-client in the patch
passed. |
| +1 :green_heart: | unit | 210m 35s | | hbase-server in the patch
passed. |
| | | 242m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/7000 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 07b6ca684b0f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May
23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 549e24450d8d2e0273ed38c0d247c306a2972ba9 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/testReport/
|
| Max. process+thread count | 4882 (vs. ulimit of 3) |
| modules | C: hbase-protocol-shaded hbase-client hbase-server U: . |
| Console output |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/console
|
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Apache-HBase commented on PR #7000:
URL: https://github.com/apache/hbase/pull/7000#issuecomment-2892801095
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | hbaseanti | 0m 0s | | Patch does not have any
anti-patterns. |
_ master Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 3m 15s | | master passed |
| +1 :green_heart: | compile | 4m 30s | | master passed |
| +1 :green_heart: | checkstyle | 0m 58s | | master passed |
| +1 :green_heart: | spotbugs | 4m 35s | | master passed |
| +1 :green_heart: | spotless | 0m 46s | | branch has no errors when
running spotless:check. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 4m 24s | | the patch passed |
| +1 :green_heart: | cc | 4m 24s | | the patch passed |
| +1 :green_heart: | javac | 4m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | spotbugs | 4m 48s | | the patch passed |
| +1 :green_heart: | hadoopcheck | 12m 2s | | Patch does not cause any
errors with Hadoop 3.3.6 3.4.0. |
| +1 :green_heart: | hbaseprotoc | 1m 34s | | the patch passed |
| +1 :green_heart: | spotless | 0m 44s | | patch has no errors when
running spotless:check. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 50m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/artifact/yetus-general-check/output/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/7000 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell
detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat
hbaseprotoc |
| uname | Linux 16300a5b6f0b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May
23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 549e24450d8d2e0273ed38c0d247c306a2972ba9 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 86 (vs. ulimit of 3) |
| modules | C: hbase-protocol-shaded hbase-client hbase-server U: . |
| Console output |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7000/1/console
|
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
Copilot commented on code in PR #7000:
URL: https://github.com/apache/hbase/pull/7000#discussion_r2095818581
##
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java:
##
@@ -288,4 +309,25 @@ private long calculateWriteCapacityUnitDiff(final long
actualSize, final long es
private long calculateReadCapacityUnitDiff(final long actualSize, final long
estimateSize) {
return calculateReadCapacityUnit(actualSize) -
calculateReadCapacityUnit(estimateSize);
}
+
+ private long calculateHandlerUsageTimeEstimate(final double
requestsPerSecond,
+final int numHandlerThreads) {
+if (requestsPerSecond <= numHandlerThreads) {
+ // If less than 1 request per second per handler thread, then we use the
number of handler
+ // threads as a baseline to avoid incorrect estimations when the number
of requests is very
+ // low.
+ return numHandlerThreads;
+} else {
+ double requestsPerMillisecond = Math.ceil(requestsPerSecond / 1000);
Review Comment:
[nitpick] Consider adding documentation or comments to clarify the rationale
for using Math.ceil(requestsPerSecond/1000) for estimating per-request handler
usage time, ensuring it aligns with the intended behavior under low request
rates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29231 - Thread handler usage throttling [hbase]
rmdmattingly commented on PR #7000: URL: https://github.com/apache/hbase/pull/7000#issuecomment-2891118070 lucky number 7000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
