Re: [PR] HBASE-29669 Implement basic row cache [hbase]
wchevreuil merged PR #7901: URL: https://github.com/apache/hbase/pull/7901 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3327218771
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -110,16 +229,67 @@ boolean tryGetFromCache(RowCacheKey key, Get get,
List results) {
}
results.addAll(row.getCells());
-// TODO: implement update of metrics
return true;
}
- void populateCache(List results, RowCacheKey key) {
-// TODO: implement with barrier to avoid cache read during mutation
-try {
- rowCacheStrategy.cacheRow(key, new RowCells(results));
-} catch (CloneNotSupportedException ignored) {
- // Not able to cache row cells, ignore
-}
+ void populateCache(HRegion region, List results, RowCacheKey key) {
Review Comment:
Following up — also switched `createRegionLevelBarrier`,
`removeRegionLevelBarrier`, and `getRegionLevelBarrier` to take `String
encodedRegionName`.
Commit:
https://github.com/apache/hbase/pull/7901/commits/361c8e8e934fcb3f773185a25529402fdeeb12e3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3327148185
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -2003,6 +2014,22 @@ public Pair> call()
throws IOException {
}
}
+ private void evictRowCache() {
+boolean evictOnClose =
getReadOnlyConfiguration().getBoolean(ROW_CACHE_EVICT_ON_CLOSE_KEY,
+ ROW_CACHE_EVICT_ON_CLOSE_DEFAULT);
+
+if (!evictOnClose) {
+ return;
+}
+
+if (!(rsServices instanceof HRegionServer regionServer)) {
+ return;
+}
+
+RowCache rowCache =
regionServer.getRSRpcServices().getServer().getRowCache();
+rowCache.evictRowsByRegion(this);
Review Comment:
Done — switched to `String encodedRegionName` to avoid the cyclic dependency
on `HRegion`.
Commit:
https://github.com/apache/hbase/pull/7901/commits/8ff2fd82a68f2799fff800bb9bd16b6468eeecbc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on PR #7901: URL: https://github.com/apache/hbase/pull/7901#issuecomment-4580221278 @wchevreuil Done — dropped the dev-support and .github/workflows commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
wchevreuil commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3319037711
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -2003,6 +2014,22 @@ public Pair> call()
throws IOException {
}
}
+ private void evictRowCache() {
+boolean evictOnClose =
getReadOnlyConfiguration().getBoolean(ROW_CACHE_EVICT_ON_CLOSE_KEY,
+ ROW_CACHE_EVICT_ON_CLOSE_DEFAULT);
+
+if (!evictOnClose) {
+ return;
+}
+
+if (!(rsServices instanceof HRegionServer regionServer)) {
+ return;
+}
+
+RowCache rowCache =
regionServer.getRSRpcServices().getServer().getRowCache();
+rowCache.evictRowsByRegion(this);
Review Comment:
Can we avoid cyclic dependency by changing evictRowsByRegion signature to
accept the encodedRegionName instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3168018154
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -3435,6 +3473,15 @@ private void updateDeleteLatestVersionTimestamp(Cell
cell, Get get, int count, b
@Override
public void put(Put put) throws IOException {
TraceUtil.trace(() -> {
+ // Put with TTL is not allowed on tables with row cache enabled, because
cached rows cannot
+ // track TTL expiration
+ if (isRowCacheEnabled) {
+if (put.getTTL() != Long.MAX_VALUE) {
+ throw new DoNotRetryIOException(
+"Tables with row cache enabled do not allow setting TTL on Puts");
+}
+ }
Review Comment:
Good point — turned out to be feasible. Done.
Quick verification: I was initially worried that `Get` results don't carry
TTL tags (the design doc assumed this), but that's actually a *client-side*
property — the RPC codec strips tags. **Server-side** cells delivered to
`RowCache` preserve their TTL tag (carried forward by
`TagUtil.carryForwardTTLTag` during mutation, the same tag
`ScanQueryMatcher.isCellTTLExpired` reads). Confirmed with a probe test against
a live region.
Implementation:
- `RowCells` now precomputes the earliest TTL expiration time across its
cells during construction. Cells without a TTL tag yield `Long.MAX_VALUE`, so
`RowCells.isExpired(now)` is an `O(1)` `long` comparison on every cache hit —
no per-cell tag iteration on the hot path.
- `RowCache.tryGetFromCache`: if the row is expired, evict and fall back to
the storage read path.
- `RowCache.cache`: skip caching when results are empty (otherwise an
evicted-then-empty row would be re-cached as an empty entry).
- `HRegion.put`: removed the guard that rejected Puts with TTL.
CF-level TTL still disables the row cache via `canCacheRow`'s `isDefaultTtl`
check; that policy is unchanged for now.
Commit:
https://github.com/apache/hbase/pull/7901/commits/d8fef38bfc02906f0314397a588371be43fa6bf2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3167826124
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -110,16 +229,67 @@ boolean tryGetFromCache(RowCacheKey key, Get get,
List results) {
}
results.addAll(row.getCells());
-// TODO: implement update of metrics
return true;
}
- void populateCache(List results, RowCacheKey key) {
-// TODO: implement with barrier to avoid cache read during mutation
-try {
- rowCacheStrategy.cacheRow(key, new RowCells(results));
-} catch (CloneNotSupportedException ignored) {
- // Not able to cache row cells, ignore
-}
+ void populateCache(HRegion region, List results, RowCacheKey key) {
Review Comment:
Done.
- Renamed `populateCache` to `cache`.
- Changed `regionLevelBarrierMap` key type from `HRegion` to the encoded
region name (`String`). The encoded name is the canonical region identifier
already used elsewhere (e.g., `RowCacheKey.isSameRegion`). As a result, `cache`
no longer needs an `HRegion` parameter; it derives the encoded region name from
the `RowCacheKey`.
I kept the external signatures of `create/remove/getRegionLevelBarrier`
taking `HRegion` to keep the caller's intent explicit; only the internal map
key type changes. Let me know if you'd prefer those to take `String` as well.
Commit:
https://github.com/apache/hbase/pull/7901/commits/e985135e33694b93a8e210a48cdb4836fa95c305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3167738038
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -63,8 +85,8 @@ R execute(RowOperation operation) throws IOException {
RowCache(Configuration conf) {
enabledByConf =
conf.getFloat(HConstants.ROW_CACHE_SIZE_KEY,
HConstants.ROW_CACHE_SIZE_DEFAULT) > 0;
-// TODO: implement row cache
-rowCacheStrategy = null;
+// Currently we only support TinyLfu implementation
+rowCacheStrategy = new
TinyLfuRowCacheStrategy(MemorySizeUtil.getRowCacheSize(conf));
Review Comment:
Done. Introduced `row.cache.strategy.class` config key so a custom
`RowCacheStrategy` can be plugged in via configuration; default remains
`TinyLfuRowCacheStrategy`. This follows the same convention used by `MemStore`
and `RegionSplitPolicy` (`Configuration`-arg constructor +
`ReflectionUtils.newInstance`).
Commit:
https://github.com/apache/hbase/pull/7901/commits/6739f062accbde26671deaee01b59cfc1fb3d693
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
wchevreuil commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3148130295
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -63,8 +85,8 @@ R execute(RowOperation operation) throws IOException {
RowCache(Configuration conf) {
enabledByConf =
conf.getFloat(HConstants.ROW_CACHE_SIZE_KEY,
HConstants.ROW_CACHE_SIZE_DEFAULT) > 0;
-// TODO: implement row cache
-rowCacheStrategy = null;
+// Currently we only support TinyLfu implementation
+rowCacheStrategy = new
TinyLfuRowCacheStrategy(MemorySizeUtil.getRowCacheSize(conf));
Review Comment:
Should be made pluggable.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -3435,6 +3473,15 @@ private void updateDeleteLatestVersionTimestamp(Cell
cell, Get get, int count, b
@Override
public void put(Put put) throws IOException {
TraceUtil.trace(() -> {
+ // Put with TTL is not allowed on tables with row cache enabled, because
cached rows cannot
+ // track TTL expiration
+ if (isRowCacheEnabled) {
+if (put.getTTL() != Long.MAX_VALUE) {
+ throw new DoNotRetryIOException(
+"Tables with row cache enabled do not allow setting TTL on Puts");
+}
+ }
Review Comment:
Can't we simply implement TTL expiration within the RowCache.getRow logic?
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -110,16 +229,67 @@ boolean tryGetFromCache(RowCacheKey key, Get get,
List results) {
}
results.addAll(row.getCells());
-// TODO: implement update of metrics
return true;
}
- void populateCache(List results, RowCacheKey key) {
-// TODO: implement with barrier to avoid cache read during mutation
-try {
- rowCacheStrategy.cacheRow(key, new RowCells(results));
-} catch (CloneNotSupportedException ignored) {
- // Not able to cache row cells, ignore
-}
+ void populateCache(HRegion region, List results, RowCacheKey key) {
Review Comment:
nit: just call this method "cache".
Do we really need the regionLevelBarrierMap key type to be HRegion, or could
it be just the String encoded region name? That way, we don't need an extra
HRegion parameter in this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3037590591
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -110,16 +229,67 @@ boolean tryGetFromCache(RowCacheKey key, Get get,
List results) {
}
results.addAll(row.getCells());
-// TODO: implement update of metrics
return true;
}
- void populateCache(List results, RowCacheKey key) {
-// TODO: implement with barrier to avoid cache read during mutation
-try {
- rowCacheStrategy.cacheRow(key, new RowCells(results));
-} catch (CloneNotSupportedException ignored) {
- // Not able to cache row cells, ignore
-}
+ void populateCache(HRegion region, List results, RowCacheKey key) {
+// The row cache is populated only when no region level barriers remain
+regionLevelBarrierMap.computeIfAbsent(region, t -> {
+ // The row cache is populated only when no row level barriers remain
+ rowLevelBarrierMap.computeIfAbsent(key, k -> {
+try {
+ rowCacheStrategy.cacheRow(key, new RowCells(results));
+} catch (CloneNotSupportedException ignored) {
+ // Not able to cache row cells, ignore
+}
+return null;
+ });
+ return null;
+});
+ }
+
+ void createRegionLevelBarrier(HRegion region) {
+regionLevelBarrierMap.computeIfAbsent(region, k -> new
AtomicInteger(0)).incrementAndGet();
+ }
+
+ void increaseRowCacheSeqNum(HRegion region) {
+region.increaseRowCacheSeqNum();
+ }
+
+ void removeTableLevelBarrier(HRegion region) {
+regionLevelBarrierMap.computeIfPresent(region, (k, counter) -> {
+ int remaining = counter.decrementAndGet();
+ return (remaining <= 0) ? null : counter;
+});
+ }
Review Comment:
Done, renamed to removeRegionLevelBarrier. (95de81b)
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -946,14 +949,19 @@ public HRegion(final HRegionFileSystem fs, final WAL wal,
final Configuration co
this.isRowCacheEnabled = checkRowCacheConfig();
}
- private boolean checkRowCacheConfig() {
+ boolean checkRowCacheConfig() {
Boolean fromDescriptor = htableDescriptor.getRowCacheEnabled();
// The setting from TableDescriptor has higher priority than the global
configuration
return fromDescriptor != null
? fromDescriptor
: conf.getBoolean(HConstants.ROW_CACHE_ENABLED_KEY,
HConstants.ROW_CACHE_ENABLED_DEFAULT);
}
+ // For testing only
+ void setRowCache(RowCache rowCache) {
+this.rowCache = rowCache;
+ }
Review Comment:
Done, added @RestrictedApi annotation. (267f4ef)
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java:
##
@@ -69,6 +69,7 @@ class MetricsRegionServerWrapperImpl implements
MetricsRegionServerWrapper {
private BlockCache l2Cache = null;
private MobFileCache mobFileCache;
private CacheStats cacheStats;
+ private final RowCache rowCache;
Review Comment:
Done, moved rowCache field next to MobFileCache. (782dd75)
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java:
##
@@ -99,6 +100,8 @@ public MetricsRegionServerWrapperImpl(final HRegionServer
regionServer) {
this.regionServer = regionServer;
initBlockCache();
initMobFileCache();
+RSRpcServices rsRpcServices = this.regionServer.getRSRpcServices();
+this.rowCache = rsRpcServices == null ? null :
rsRpcServices.getServer().getRowCache();
Review Comment:
Done, extracted initRowCache() method. (782dd75)
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRowCache.java:
##
@@ -0,0 +1,547 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.HFILE_BLOCK_CACHE_SIZE_KEY;
+import static org.apache.hadoop.hbase.HConstants.ROW_CACHE_SIZE_KEY;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_EVICTED_ROW_COUNT;
+import static
org.apache.hado
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r3037590591
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -110,16 +229,67 @@ boolean tryGetFromCache(RowCacheKey key, Get get,
List results) {
}
results.addAll(row.getCells());
-// TODO: implement update of metrics
return true;
}
- void populateCache(List results, RowCacheKey key) {
-// TODO: implement with barrier to avoid cache read during mutation
-try {
- rowCacheStrategy.cacheRow(key, new RowCells(results));
-} catch (CloneNotSupportedException ignored) {
- // Not able to cache row cells, ignore
-}
+ void populateCache(HRegion region, List results, RowCacheKey key) {
+// The row cache is populated only when no region level barriers remain
+regionLevelBarrierMap.computeIfAbsent(region, t -> {
+ // The row cache is populated only when no row level barriers remain
+ rowLevelBarrierMap.computeIfAbsent(key, k -> {
+try {
+ rowCacheStrategy.cacheRow(key, new RowCells(results));
+} catch (CloneNotSupportedException ignored) {
+ // Not able to cache row cells, ignore
+}
+return null;
+ });
+ return null;
+});
+ }
+
+ void createRegionLevelBarrier(HRegion region) {
+regionLevelBarrierMap.computeIfAbsent(region, k -> new
AtomicInteger(0)).incrementAndGet();
+ }
+
+ void increaseRowCacheSeqNum(HRegion region) {
+region.increaseRowCacheSeqNum();
+ }
+
+ void removeTableLevelBarrier(HRegion region) {
+regionLevelBarrierMap.computeIfPresent(region, (k, counter) -> {
+ int remaining = counter.decrementAndGet();
+ return (remaining <= 0) ? null : counter;
+});
+ }
Review Comment:
Done, renamed to removeRegionLevelBarrier.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -946,14 +949,19 @@ public HRegion(final HRegionFileSystem fs, final WAL wal,
final Configuration co
this.isRowCacheEnabled = checkRowCacheConfig();
}
- private boolean checkRowCacheConfig() {
+ boolean checkRowCacheConfig() {
Boolean fromDescriptor = htableDescriptor.getRowCacheEnabled();
// The setting from TableDescriptor has higher priority than the global
configuration
return fromDescriptor != null
? fromDescriptor
: conf.getBoolean(HConstants.ROW_CACHE_ENABLED_KEY,
HConstants.ROW_CACHE_ENABLED_DEFAULT);
}
+ // For testing only
+ void setRowCache(RowCache rowCache) {
+this.rowCache = rowCache;
+ }
Review Comment:
Done, added @RestrictedApi annotation.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java:
##
@@ -69,6 +69,7 @@ class MetricsRegionServerWrapperImpl implements
MetricsRegionServerWrapper {
private BlockCache l2Cache = null;
private MobFileCache mobFileCache;
private CacheStats cacheStats;
+ private final RowCache rowCache;
Review Comment:
Done, moved rowCache field next to MobFileCache.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java:
##
@@ -99,6 +100,8 @@ public MetricsRegionServerWrapperImpl(final HRegionServer
regionServer) {
this.regionServer = regionServer;
initBlockCache();
initMobFileCache();
+RSRpcServices rsRpcServices = this.regionServer.getRSRpcServices();
+this.rowCache = rsRpcServices == null ? null :
rsRpcServices.getServer().getRowCache();
Review Comment:
Done, extracted initRowCache() method.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java:
##
@@ -2347,8 +2347,21 @@ public BulkLoadHFileResponse bulkLoadHFile(final
RpcController controller,
return bulkLoadHFileInternal(request);
}
-// TODO: implement row cache logic for bulk load
-return bulkLoadHFileInternal(request);
+RowCache rowCache = region.getRegionServerServices().getRowCache();
+
+// Since bulkload modifies the store files, the row cache should be
disabled until the bulkload
+// is finished.
+rowCache.createRegionLevelBarrier(region);
+try {
+ // We do not invalidate the entire row cache directly, as it contains a
large number of
+ // entries and takes a long time. Instead, we increment rowCacheSeqNum,
which is used when
+ // constructing a RowCacheKey, thereby making the existing row cache
entries stale.
+ rowCache.increaseRowCacheSeqNum(region);
+ return bulkLoadHFileInternal(request);
+} finally {
+ // The row cache for the region has been enabled again
+ rowCache.removeTableLevelBarrier(region);
Review Comment:
Done, renamed to removeRegionLevelBarrier.
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRowCache.java:
##
@@ -0,0 +1,547 @@
+/*
+ * L
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on PR #7901: URL: https://github.com/apache/hbase/pull/7901#issuecomment-4186614775 @liuxiaocs7 Sorry, I just noticed your comments. I’ll take a look and follow up early next week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2938006853
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRowCache.java:
##
@@ -0,0 +1,547 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.HFILE_BLOCK_CACHE_SIZE_KEY;
+import static org.apache.hadoop.hbase.HConstants.ROW_CACHE_SIZE_KEY;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_EVICTED_ROW_COUNT;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_HIT_COUNT;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_MISS_COUNT;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CompatibilityFactory;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.SingleProcessHBaseCluster;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Append;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Row;
+import org.apache.hadoop.hbase.client.RowMutations;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.test.MetricsAssertHelper;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRowCache {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestRowCache.class);
Review Comment:
For new tests, it is best to use JUnit5 directly, since existing unit tests
are currently being migrated to JUnit5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2937996331
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowCache.java:
##
@@ -110,16 +229,67 @@ boolean tryGetFromCache(RowCacheKey key, Get get,
List results) {
}
results.addAll(row.getCells());
-// TODO: implement update of metrics
return true;
}
- void populateCache(List results, RowCacheKey key) {
-// TODO: implement with barrier to avoid cache read during mutation
-try {
- rowCacheStrategy.cacheRow(key, new RowCells(results));
-} catch (CloneNotSupportedException ignored) {
- // Not able to cache row cells, ignore
-}
+ void populateCache(HRegion region, List results, RowCacheKey key) {
+// The row cache is populated only when no region level barriers remain
+regionLevelBarrierMap.computeIfAbsent(region, t -> {
+ // The row cache is populated only when no row level barriers remain
+ rowLevelBarrierMap.computeIfAbsent(key, k -> {
+try {
+ rowCacheStrategy.cacheRow(key, new RowCells(results));
+} catch (CloneNotSupportedException ignored) {
+ // Not able to cache row cells, ignore
+}
+return null;
+ });
+ return null;
+});
+ }
+
+ void createRegionLevelBarrier(HRegion region) {
+regionLevelBarrierMap.computeIfAbsent(region, k -> new
AtomicInteger(0)).incrementAndGet();
+ }
+
+ void increaseRowCacheSeqNum(HRegion region) {
+region.increaseRowCacheSeqNum();
+ }
+
+ void removeTableLevelBarrier(HRegion region) {
+regionLevelBarrierMap.computeIfPresent(region, (k, counter) -> {
+ int remaining = counter.decrementAndGet();
+ return (remaining <= 0) ? null : counter;
+});
+ }
Review Comment:
this method should be renamed by `removeRegionLevelBarrier`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2938006853
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRowCache.java:
##
@@ -0,0 +1,547 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.HFILE_BLOCK_CACHE_SIZE_KEY;
+import static org.apache.hadoop.hbase.HConstants.ROW_CACHE_SIZE_KEY;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_EVICTED_ROW_COUNT;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_HIT_COUNT;
+import static
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSource.ROW_CACHE_MISS_COUNT;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CompatibilityFactory;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.SingleProcessHBaseCluster;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Append;
+import org.apache.hadoop.hbase.client.CheckAndMutate;
+import org.apache.hadoop.hbase.client.CheckAndMutateResult;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Row;
+import org.apache.hadoop.hbase.client.RowMutations;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.test.MetricsAssertHelper;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({ RegionServerTests.class, MediumTests.class })
+public class TestRowCache {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestRowCache.class);
Review Comment:
For new tests, it is best to use JUnit 5 directly, as existing unit tests
are currently being migrated to JUnit 5.
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRowCache.java:
##
@@ -0,0 +1,547 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+pa
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2938004540
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java:
##
@@ -2347,8 +2347,21 @@ public BulkLoadHFileResponse bulkLoadHFile(final
RpcController controller,
return bulkLoadHFileInternal(request);
}
-// TODO: implement row cache logic for bulk load
-return bulkLoadHFileInternal(request);
+RowCache rowCache = region.getRegionServerServices().getRowCache();
+
+// Since bulkload modifies the store files, the row cache should be
disabled until the bulkload
+// is finished.
+rowCache.createRegionLevelBarrier(region);
+try {
+ // We do not invalidate the entire row cache directly, as it contains a
large number of
+ // entries and takes a long time. Instead, we increment rowCacheSeqNum,
which is used when
+ // constructing a RowCacheKey, thereby making the existing row cache
entries stale.
+ rowCache.increaseRowCacheSeqNum(region);
+ return bulkLoadHFileInternal(request);
+} finally {
+ // The row cache for the region has been enabled again
+ rowCache.removeTableLevelBarrier(region);
Review Comment:
ditto `region` level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2938003101
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java:
##
@@ -99,6 +100,8 @@ public MetricsRegionServerWrapperImpl(final HRegionServer
regionServer) {
this.regionServer = regionServer;
initBlockCache();
initMobFileCache();
+RSRpcServices rsRpcServices = this.regionServer.getRSRpcServices();
+this.rowCache = rsRpcServices == null ? null :
rsRpcServices.getServer().getRowCache();
Review Comment:
better to use a method `initRowCache()` here, just like blockcache and
mobfilecache does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2938000437
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java:
##
@@ -69,6 +69,7 @@ class MetricsRegionServerWrapperImpl implements
MetricsRegionServerWrapper {
private BlockCache l2Cache = null;
private MobFileCache mobFileCache;
private CacheStats cacheStats;
+ private final RowCache rowCache;
Review Comment:
better move to L71 under `MobFileCache`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on code in PR #7901:
URL: https://github.com/apache/hbase/pull/7901#discussion_r2937998377
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##
@@ -946,14 +949,19 @@ public HRegion(final HRegionFileSystem fs, final WAL wal,
final Configuration co
this.isRowCacheEnabled = checkRowCacheConfig();
}
- private boolean checkRowCacheConfig() {
+ boolean checkRowCacheConfig() {
Boolean fromDescriptor = htableDescriptor.getRowCacheEnabled();
// The setting from TableDescriptor has higher priority than the global
configuration
return fromDescriptor != null
? fromDescriptor
: conf.getBoolean(HConstants.ROW_CACHE_ENABLED_KEY,
HConstants.ROW_CACHE_ENABLED_DEFAULT);
}
+ // For testing only
+ void setRowCache(RowCache rowCache) {
+this.rowCache = rowCache;
+ }
Review Comment:
For testing only, we could use `@RestrictedApi` annotation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
liuxiaocs7 commented on PR #7901: URL: https://github.com/apache/hbase/pull/7901#issuecomment-4064866117 Hi, @EungsopYoo CI has been migrated from Jenkins to GitHub Actions, and the pre-commit GitHub Jenkins job should have been disabled, more details: https://lists.apache.org/thread/z6o5hhsd9goh5j7fcl4bnwzsktlwlwl4 Some previously created feature branches also have this issue; perhaps we could give it a try, such as: https://lists.apache.org/thread/hrl7dktkrb8zwwbv62my68lvtbplv7sr Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] HBASE-29669 Implement basic row cache [hbase]
EungsopYoo commented on PR #7901: URL: https://github.com/apache/hbase/pull/7901#issuecomment-4037273317 @wchevreuil I have completed the implementation of the basic row cache. Please review it. By the way, it seems like CI is not working — do you know why? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
