(hbase) branch branch-3 updated: HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2) (#5592)
This is an automated email from the ASF dual-hosted git repository.
meszibalu pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 64eea86dbd1 HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput
(Part 2) (#5592)
64eea86dbd1 is described below
commit 64eea86dbd1aacf1f2e48b987a24ea9e5576439a
Author: Andor Molnár
AuthorDate: Fri Dec 22 10:07:15 2023 +0100
HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2) (#5592)
Signed-off-by: Tak Lon (Stephen) Wu
Signed-off-by: Wellington Chevreuil
Signed-off-by: Balazs Meszaros
---
.../hbase/io/asyncfs/TestFanOutOneBlockAsyncDFSOutput.java | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git
a/hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestFanOutOneBlockAsyncDFSOutput.java
b/hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestFanOutOneBlockAsyncDFSOutput.java
index 3c385283103..68b8bfa3d9f 100644
---
a/hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestFanOutOneBlockAsyncDFSOutput.java
+++
b/hbase-asyncfs/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestFanOutOneBlockAsyncDFSOutput.java
@@ -52,10 +52,12 @@ import org.apache.hadoop.ipc.RemoteException;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
+import org.junit.FixMethodOrder;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.rules.TestName;
+import org.junit.runners.MethodSorters;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -65,6 +67,7 @@ import
org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup;
import org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoopGroup;
import
org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel;
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
@Category({ MiscTests.class, MediumTests.class })
public class TestFanOutOneBlockAsyncDFSOutput extends AsyncFSTestBase {
@@ -142,8 +145,12 @@ public class TestFanOutOneBlockAsyncDFSOutput extends
AsyncFSTestBase {
writeAndVerify(FS, f, out);
}
+ /**
+ * Test method has been renamed to be the first in NAME_ASCENDING. It's an
ugly hack to avoid
+ * flakiness.
+ */
@Test
- public void testRecover() throws IOException, InterruptedException,
ExecutionException {
+ public void test0Recover() throws IOException, InterruptedException,
ExecutionException {
Path f = new Path("/" + name.getMethodName());
EventLoop eventLoop = EVENT_LOOP_GROUP.next();
FanOutOneBlockAsyncDFSOutput out =
FanOutOneBlockAsyncDFSOutputHelper.createOutput(FS, f, true,
(hbase) branch branch-3 updated: HBASE-28254 Flaky test: TestTableShell (#5590)
This is an automated email from the ASF dual-hosted git repository.
meszibalu pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 2ce1109884a HBASE-28254 Flaky test: TestTableShell (#5590)
2ce1109884a is described below
commit 2ce1109884a5eb8d9fde1ad5f516af3145dfb6ca
Author: Andor Molnár
AuthorDate: Tue Dec 19 21:43:29 2023 +0100
HBASE-28254 Flaky test: TestTableShell (#5590)
Add some delay (1/1000s) between HBase shell commands in Ruby to avoid two
commands ending up with the same version.
Signed-off-by: Wellington Ramos Chevreuil
Signed-off-by: Nihal Jain
Signed-off-by: Balazs Meszaros
---
hbase-shell/src/test/ruby/hbase/table_test.rb | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/hbase-shell/src/test/ruby/hbase/table_test.rb
b/hbase-shell/src/test/ruby/hbase/table_test.rb
index 05b80725efc..8ed6f663dbd 100644
--- a/hbase-shell/src/test/ruby/hbase/table_test.rb
+++ b/hbase-shell/src/test/ruby/hbase/table_test.rb
@@ -681,11 +681,15 @@ module Hbase
create_test_table(@test_name_raw)
@test_table = table(@test_name_raw)
- # Instert test data
+ # Insert test data
@test_table.put(1, "x:a", 1)
+ sleep(1.0/1000.0)
@test_table.put(2, "x:raw1", 11)
+ sleep(1.0/1000.0)
@test_table.put(2, "x:raw1", 11)
+ sleep(1.0/1000.0)
@test_table.put(2, "x:raw1", 11)
+ sleep(1.0/1000.0)
@test_table.put(2, "x:raw1", 11)
args = {}
(hbase) branch branch-3 updated: HBASE-28345 Close HBase connection on exit from HBase Shell (#5665)
This is an automated email from the ASF dual-hosted git repository. meszibalu pushed a commit to branch branch-3 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-3 by this push: new 3a81784a06e HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) 3a81784a06e is described below commit 3a81784a06e86d29fe6cba858d6dad4188defed9 Author: Istvan Toth AuthorDate: Wed Feb 7 13:06:11 2024 +0100 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) Signed-off-by: Wellington Chevreuil Signed-off-by: Balazs Meszaros --- hbase-shell/src/main/ruby/shell.rb | 5 + 1 file changed, 5 insertions(+) diff --git a/hbase-shell/src/main/ruby/shell.rb b/hbase-shell/src/main/ruby/shell.rb index 414ab9d2bd5..39fbd2ccba1 100644 --- a/hbase-shell/src/main/ruby/shell.rb +++ b/hbase-shell/src/main/ruby/shell.rb @@ -108,6 +108,11 @@ module Shell # exit the interactive shell and save that this # happend via a call to exit def exit(ret = 0) + # Non-deamon Netty threadpool in ZK ClientCnxnSocketNetty cannot be shut down otherwise + begin +hbase.shutdown + rescue Exception + end @exit_code = ret IRB.irb_exit(IRB.CurrentContext.irb, ret) end
(hbase) branch branch-2.6 updated: HBASE-27800: Add support for default user quotas (#5666)
This is an automated email from the ASF dual-hosted git repository.
bbeaudreault pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new 240dfa0a5eb HBASE-27800: Add support for default user quotas (#5666)
240dfa0a5eb is described below
commit 240dfa0a5eb621ca4c65acde47a76df59b6c9d65
Author: Ray Mattingly
AuthorDate: Wed Feb 7 09:23:17 2024 -0500
HBASE-27800: Add support for default user quotas (#5666)
Signed-off-by: Bryan Beaudreault
---
.../org/apache/hadoop/hbase/quotas/QuotaCache.java | 3 +-
.../org/apache/hadoop/hbase/quotas/QuotaUtil.java | 61 -
.../hadoop/hbase/quotas/TestDefaultQuota.java | 138 +
3 files changed, 200 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
index 0a57b9fd8f8..67b2aecc544 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
@@ -137,7 +137,8 @@ public class QuotaCache implements Stoppable {
* @return the quota info associated to specified user
*/
public UserQuotaState getUserQuotaState(final UserGroupInformation ugi) {
-return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
UserQuotaState::new,
+return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
+ () ->
QuotaUtil.buildDefaultUserQuotaState(rsServices.getConfiguration()),
this::triggerCacheRefresh);
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
index b307c5d7dab..8e7bce8b1e7 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
@@ -22,6 +22,7 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.DoNotRetryIOException;
@@ -48,7 +49,9 @@ import org.apache.yetus.audience.InterfaceStability;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.TimeUnit;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos;
import
org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.QuotaScope;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Quotas;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Throttle;
@@ -72,6 +75,26 @@ public class QuotaUtil extends QuotaTableUtil {
// the default one write capacity unit is 1024 bytes (1KB)
public static final long DEFAULT_WRITE_CAPACITY_UNIT = 1024;
+ /*
+ * The below defaults, if configured, will be applied to otherwise
unthrottled users. For example,
+ * set `hbase.quota.default.user.machine.read.size` to `1048576` in your
hbase-site.xml to ensure
+ * that any given user may not query more than 1mb per second from any given
machine, unless
+ * explicitly permitted by a persisted quota. All of these defaults use
TimeUnit.SECONDS and
+ * QuotaScope.MACHINE.
+ */
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_NUM =
+"hbase.quota.default.user.machine.read.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_SIZE =
+"hbase.quota.default.user.machine.read.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_NUM =
+"hbase.quota.default.user.machine.request.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_SIZE =
+"hbase.quota.default.user.machine.request.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_NUM =
+"hbase.quota.default.user.machine.write.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_SIZE =
+"hbase.quota.default.user.machine.write.size";
+
/** Table descriptor for Quota internal table */
public static final HTableDescriptor QUOTA_TABLE_DESC = new
HTableDescriptor(QUOTA_TABLE_NAME);
static {
@@ -283,10 +306,14 @@ public class QuotaUtil extends QuotaTableUtil {
assert isUserRowKey(key);
String user = getUserFromRowKey(key);
+ if (results[i].isEmpty()) {
+userQuotas.put(user,
buildDefaultUserQuotaState(connection.getConfiguration()));
+continue;
+ }
+
final UserQuotaState quotaInfo = new UserQuotaState(nowTs);
userQuotas.put(user, quotaInfo);
- if (results[i].isEmpty()) continue;
assert Bytes.equals(key, results[i].getRow());
try {
@@ -320,6
(hbase) branch branch-2 updated: HBASE-27800: Add support for default user quotas (#5666)
This is an automated email from the ASF dual-hosted git repository.
bbeaudreault pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new 60627328dc7 HBASE-27800: Add support for default user quotas (#5666)
60627328dc7 is described below
commit 60627328dc79eb4ce6c24e0ebc399a1c9e9ee2a6
Author: Ray Mattingly
AuthorDate: Wed Feb 7 09:23:17 2024 -0500
HBASE-27800: Add support for default user quotas (#5666)
Signed-off-by: Bryan Beaudreault
---
.../org/apache/hadoop/hbase/quotas/QuotaCache.java | 3 +-
.../org/apache/hadoop/hbase/quotas/QuotaUtil.java | 61 -
.../hadoop/hbase/quotas/TestDefaultQuota.java | 138 +
3 files changed, 200 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
index 0a57b9fd8f8..67b2aecc544 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
@@ -137,7 +137,8 @@ public class QuotaCache implements Stoppable {
* @return the quota info associated to specified user
*/
public UserQuotaState getUserQuotaState(final UserGroupInformation ugi) {
-return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
UserQuotaState::new,
+return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
+ () ->
QuotaUtil.buildDefaultUserQuotaState(rsServices.getConfiguration()),
this::triggerCacheRefresh);
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
index b307c5d7dab..8e7bce8b1e7 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
@@ -22,6 +22,7 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.DoNotRetryIOException;
@@ -48,7 +49,9 @@ import org.apache.yetus.audience.InterfaceStability;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.TimeUnit;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos;
import
org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.QuotaScope;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Quotas;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Throttle;
@@ -72,6 +75,26 @@ public class QuotaUtil extends QuotaTableUtil {
// the default one write capacity unit is 1024 bytes (1KB)
public static final long DEFAULT_WRITE_CAPACITY_UNIT = 1024;
+ /*
+ * The below defaults, if configured, will be applied to otherwise
unthrottled users. For example,
+ * set `hbase.quota.default.user.machine.read.size` to `1048576` in your
hbase-site.xml to ensure
+ * that any given user may not query more than 1mb per second from any given
machine, unless
+ * explicitly permitted by a persisted quota. All of these defaults use
TimeUnit.SECONDS and
+ * QuotaScope.MACHINE.
+ */
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_NUM =
+"hbase.quota.default.user.machine.read.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_SIZE =
+"hbase.quota.default.user.machine.read.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_NUM =
+"hbase.quota.default.user.machine.request.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_SIZE =
+"hbase.quota.default.user.machine.request.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_NUM =
+"hbase.quota.default.user.machine.write.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_SIZE =
+"hbase.quota.default.user.machine.write.size";
+
/** Table descriptor for Quota internal table */
public static final HTableDescriptor QUOTA_TABLE_DESC = new
HTableDescriptor(QUOTA_TABLE_NAME);
static {
@@ -283,10 +306,14 @@ public class QuotaUtil extends QuotaTableUtil {
assert isUserRowKey(key);
String user = getUserFromRowKey(key);
+ if (results[i].isEmpty()) {
+userQuotas.put(user,
buildDefaultUserQuotaState(connection.getConfiguration()));
+continue;
+ }
+
final UserQuotaState quotaInfo = new UserQuotaState(nowTs);
userQuotas.put(user, quotaInfo);
- if (results[i].isEmpty()) continue;
assert Bytes.equals(key, results[i].getRow());
try {
@@ -320,6
(hbase-site) branch asf-site updated: INFRA-10751 Empty commit
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hbase-site.git The following commit(s) were added to refs/heads/asf-site by this push: new ce1ffc71bcb INFRA-10751 Empty commit ce1ffc71bcb is described below commit ce1ffc71bcbe4a860bdc90d312cc80d85cff450d Author: jenkins AuthorDate: Wed Feb 7 14:42:55 2024 + INFRA-10751 Empty commit
(hbase) branch branch-3 updated: HBASE-27800: Add support for default user quotas (#5666)
This is an automated email from the ASF dual-hosted git repository.
bbeaudreault pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 7f531f38c0b HBASE-27800: Add support for default user quotas (#5666)
7f531f38c0b is described below
commit 7f531f38c0b64ed91307107f9c426e7051e677d0
Author: Ray Mattingly
AuthorDate: Wed Feb 7 09:23:17 2024 -0500
HBASE-27800: Add support for default user quotas (#5666)
Signed-off-by: Bryan Beaudreault
---
.../org/apache/hadoop/hbase/quotas/QuotaCache.java | 3 +-
.../org/apache/hadoop/hbase/quotas/QuotaUtil.java | 61 -
.../hadoop/hbase/quotas/TestDefaultQuota.java | 138 +
3 files changed, 200 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
index 0a57b9fd8f8..67b2aecc544 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
@@ -137,7 +137,8 @@ public class QuotaCache implements Stoppable {
* @return the quota info associated to specified user
*/
public UserQuotaState getUserQuotaState(final UserGroupInformation ugi) {
-return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
UserQuotaState::new,
+return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
+ () ->
QuotaUtil.buildDefaultUserQuotaState(rsServices.getConfiguration()),
this::triggerCacheRefresh);
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
index f9fa1a95c0c..2e51a8f7561 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
@@ -22,6 +22,7 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.DoNotRetryIOException;
@@ -49,7 +50,9 @@ import org.apache.yetus.audience.InterfaceStability;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.TimeUnit;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos;
import
org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.QuotaScope;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Quotas;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Throttle;
@@ -73,6 +76,26 @@ public class QuotaUtil extends QuotaTableUtil {
// the default one write capacity unit is 1024 bytes (1KB)
public static final long DEFAULT_WRITE_CAPACITY_UNIT = 1024;
+ /*
+ * The below defaults, if configured, will be applied to otherwise
unthrottled users. For example,
+ * set `hbase.quota.default.user.machine.read.size` to `1048576` in your
hbase-site.xml to ensure
+ * that any given user may not query more than 1mb per second from any given
machine, unless
+ * explicitly permitted by a persisted quota. All of these defaults use
TimeUnit.SECONDS and
+ * QuotaScope.MACHINE.
+ */
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_NUM =
+"hbase.quota.default.user.machine.read.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_SIZE =
+"hbase.quota.default.user.machine.read.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_NUM =
+"hbase.quota.default.user.machine.request.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_SIZE =
+"hbase.quota.default.user.machine.request.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_NUM =
+"hbase.quota.default.user.machine.write.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_SIZE =
+"hbase.quota.default.user.machine.write.size";
+
/** Table descriptor for Quota internal table */
public static final TableDescriptor QUOTA_TABLE_DESC =
TableDescriptorBuilder.newBuilder(QUOTA_TABLE_NAME)
@@ -284,10 +307,14 @@ public class QuotaUtil extends QuotaTableUtil {
assert isUserRowKey(key);
String user = getUserFromRowKey(key);
+ if (results[i].isEmpty()) {
+userQuotas.put(user,
buildDefaultUserQuotaState(connection.getConfiguration()));
+continue;
+ }
+
final UserQuotaState quotaInfo = new UserQuotaState(nowTs);
userQuotas.put(user, quotaInfo);
- if (results[i].isEmpty()) continue;
assert Bytes.equals(key, results[i].getRow());
try {
@@ -321,6
(hbase) branch master updated: HBASE-27800: Add support for default user quotas (#5666)
This is an automated email from the ASF dual-hosted git repository.
bbeaudreault pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 275d928a7d4 HBASE-27800: Add support for default user quotas (#5666)
275d928a7d4 is described below
commit 275d928a7d48aebc2172d32294bac40cc9460d11
Author: Ray Mattingly
AuthorDate: Wed Feb 7 09:23:17 2024 -0500
HBASE-27800: Add support for default user quotas (#5666)
Signed-off-by: Bryan Beaudreault
---
.../org/apache/hadoop/hbase/quotas/QuotaCache.java | 3 +-
.../org/apache/hadoop/hbase/quotas/QuotaUtil.java | 61 -
.../hadoop/hbase/quotas/TestDefaultQuota.java | 138 +
3 files changed, 200 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
index 0a57b9fd8f8..67b2aecc544 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaCache.java
@@ -137,7 +137,8 @@ public class QuotaCache implements Stoppable {
* @return the quota info associated to specified user
*/
public UserQuotaState getUserQuotaState(final UserGroupInformation ugi) {
-return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
UserQuotaState::new,
+return computeIfAbsent(userQuotaCache, getQuotaUserName(ugi),
+ () ->
QuotaUtil.buildDefaultUserQuotaState(rsServices.getConfiguration()),
this::triggerCacheRefresh);
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
index f9fa1a95c0c..2e51a8f7561 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
@@ -22,6 +22,7 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.DoNotRetryIOException;
@@ -49,7 +50,9 @@ import org.apache.yetus.audience.InterfaceStability;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos.TimeUnit;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos;
import
org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.QuotaScope;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Quotas;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.Throttle;
@@ -73,6 +76,26 @@ public class QuotaUtil extends QuotaTableUtil {
// the default one write capacity unit is 1024 bytes (1KB)
public static final long DEFAULT_WRITE_CAPACITY_UNIT = 1024;
+ /*
+ * The below defaults, if configured, will be applied to otherwise
unthrottled users. For example,
+ * set `hbase.quota.default.user.machine.read.size` to `1048576` in your
hbase-site.xml to ensure
+ * that any given user may not query more than 1mb per second from any given
machine, unless
+ * explicitly permitted by a persisted quota. All of these defaults use
TimeUnit.SECONDS and
+ * QuotaScope.MACHINE.
+ */
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_NUM =
+"hbase.quota.default.user.machine.read.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_READ_SIZE =
+"hbase.quota.default.user.machine.read.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_NUM =
+"hbase.quota.default.user.machine.request.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_SIZE =
+"hbase.quota.default.user.machine.request.size";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_NUM =
+"hbase.quota.default.user.machine.write.num";
+ public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_SIZE =
+"hbase.quota.default.user.machine.write.size";
+
/** Table descriptor for Quota internal table */
public static final TableDescriptor QUOTA_TABLE_DESC =
TableDescriptorBuilder.newBuilder(QUOTA_TABLE_NAME)
@@ -284,10 +307,14 @@ public class QuotaUtil extends QuotaTableUtil {
assert isUserRowKey(key);
String user = getUserFromRowKey(key);
+ if (results[i].isEmpty()) {
+userQuotas.put(user,
buildDefaultUserQuotaState(connection.getConfiguration()));
+continue;
+ }
+
final UserQuotaState quotaInfo = new UserQuotaState(nowTs);
userQuotas.put(user, quotaInfo);
- if (results[i].isEmpty()) continue;
assert Bytes.equals(key, results[i].getRow());
try {
@@ -321,6
(hbase) branch branch-2.4 updated: HBASE-28345 Close HBase connection on exit from HBase Shell (#5665)
This is an automated email from the ASF dual-hosted git repository. meszibalu pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new a04843d8b0e HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) a04843d8b0e is described below commit a04843d8b0ee2c5b8ad713b098a4c57bf19f7bb7 Author: Istvan Toth AuthorDate: Wed Feb 7 13:06:11 2024 +0100 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) Signed-off-by: Wellington Chevreuil Signed-off-by: Balazs Meszaros --- hbase-shell/src/main/ruby/shell.rb | 5 + 1 file changed, 5 insertions(+) diff --git a/hbase-shell/src/main/ruby/shell.rb b/hbase-shell/src/main/ruby/shell.rb index d2a90d35a06..da9f27fe848 100644 --- a/hbase-shell/src/main/ruby/shell.rb +++ b/hbase-shell/src/main/ruby/shell.rb @@ -108,6 +108,11 @@ module Shell # exit the interactive shell and save that this # happend via a call to exit def exit(ret = 0) + # Non-deamon Netty threadpool in ZK ClientCnxnSocketNetty cannot be shut down otherwise + begin +hbase.shutdown + rescue Exception + end @exit_code = ret IRB.irb_exit(IRB.CurrentContext.irb, ret) end
(hbase) branch branch-2.5 updated: HBASE-28345 Close HBase connection on exit from HBase Shell (#5665)
This is an automated email from the ASF dual-hosted git repository. meszibalu pushed a commit to branch branch-2.5 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.5 by this push: new 37f24858d93 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) 37f24858d93 is described below commit 37f24858d93a5469d8595a078f1172f502a86f40 Author: Istvan Toth AuthorDate: Wed Feb 7 13:06:11 2024 +0100 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) Signed-off-by: Wellington Chevreuil Signed-off-by: Balazs Meszaros --- hbase-shell/src/main/ruby/shell.rb | 5 + 1 file changed, 5 insertions(+) diff --git a/hbase-shell/src/main/ruby/shell.rb b/hbase-shell/src/main/ruby/shell.rb index f6b3efd945f..7aa7813e241 100644 --- a/hbase-shell/src/main/ruby/shell.rb +++ b/hbase-shell/src/main/ruby/shell.rb @@ -108,6 +108,11 @@ module Shell # exit the interactive shell and save that this # happend via a call to exit def exit(ret = 0) + # Non-deamon Netty threadpool in ZK ClientCnxnSocketNetty cannot be shut down otherwise + begin +hbase.shutdown + rescue Exception + end @exit_code = ret IRB.irb_exit(IRB.CurrentContext.irb, ret) end
(hbase) branch branch-2.6 updated: HBASE-28345 Close HBase connection on exit from HBase Shell (#5665)
This is an automated email from the ASF dual-hosted git repository. meszibalu pushed a commit to branch branch-2.6 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.6 by this push: new d65844ba419 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) d65844ba419 is described below commit d65844ba419b414fc644c71c5ce78635fae97f00 Author: Istvan Toth AuthorDate: Wed Feb 7 13:06:11 2024 +0100 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) Signed-off-by: Wellington Chevreuil Signed-off-by: Balazs Meszaros --- hbase-shell/src/main/ruby/shell.rb | 5 + 1 file changed, 5 insertions(+) diff --git a/hbase-shell/src/main/ruby/shell.rb b/hbase-shell/src/main/ruby/shell.rb index 3d95aa1623b..e7ae41200f4 100644 --- a/hbase-shell/src/main/ruby/shell.rb +++ b/hbase-shell/src/main/ruby/shell.rb @@ -108,6 +108,11 @@ module Shell # exit the interactive shell and save that this # happend via a call to exit def exit(ret = 0) + # Non-deamon Netty threadpool in ZK ClientCnxnSocketNetty cannot be shut down otherwise + begin +hbase.shutdown + rescue Exception + end @exit_code = ret IRB.irb_exit(IRB.CurrentContext.irb, ret) end
(hbase) branch branch-2 updated: HBASE-28345 Close HBase connection on exit from HBase Shell (#5665)
This is an automated email from the ASF dual-hosted git repository. meszibalu pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2 by this push: new 5735b83c647 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) 5735b83c647 is described below commit 5735b83c64726bc5797dd175f06dce9dbd714232 Author: Istvan Toth AuthorDate: Wed Feb 7 13:06:11 2024 +0100 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) Signed-off-by: Wellington Chevreuil Signed-off-by: Balazs Meszaros --- hbase-shell/src/main/ruby/shell.rb | 5 + 1 file changed, 5 insertions(+) diff --git a/hbase-shell/src/main/ruby/shell.rb b/hbase-shell/src/main/ruby/shell.rb index 3d95aa1623b..e7ae41200f4 100644 --- a/hbase-shell/src/main/ruby/shell.rb +++ b/hbase-shell/src/main/ruby/shell.rb @@ -108,6 +108,11 @@ module Shell # exit the interactive shell and save that this # happend via a call to exit def exit(ret = 0) + # Non-deamon Netty threadpool in ZK ClientCnxnSocketNetty cannot be shut down otherwise + begin +hbase.shutdown + rescue Exception + end @exit_code = ret IRB.irb_exit(IRB.CurrentContext.irb, ret) end
(hbase) branch master updated (afbf2adba25 -> 36562937468)
This is an automated email from the ASF dual-hosted git repository. meszibalu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git from afbf2adba25 HBASE-27687 Enhance quotas to consume blockBytesScanned rather than response size (#5654) add 36562937468 HBASE-28345 Close HBase connection on exit from HBase Shell (#5665) No new revisions were added by this update. Summary of changes: hbase-shell/src/main/ruby/shell.rb | 5 + 1 file changed, 5 insertions(+)
