[jira] [Created] (HBASE-28041) Rebase HBASE-27389 branch with master
Wellington Chevreuil created HBASE-28041: Summary: Rebase HBASE-27389 branch with master Key: HBASE-28041 URL: https://issues.apache.org/jira/browse/HBASE-28041 Project: HBase Issue Type: Sub-task Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28041) Rebase HBASE-27389 branch with master and fix conflicts
[ https://issues.apache.org/jira/browse/HBASE-28041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-28041. -- Resolution: Fixed Rebased and fixed conflicts. > Rebase HBASE-27389 branch with master and fix conflicts > --- > > Key: HBASE-28041 > URL: https://issues.apache.org/jira/browse/HBASE-28041 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Time for new releases?
Bump. If the release managers are not available, I could help make the next 2.5.x release. Thanks. 张铎(Duo Zhang) 于2023年8月15日周二 11:34写道： > > HBase 2.5.5 was released at 6.14, it is time for 2.5.6? > > For 2.4.18, I prefer we make it after we release 2.6.0. > > Thanks.
[jira] [Resolved] (HBASE-28004) Persistent cache map can get corrupt if crash happens midway through the write
[ https://issues.apache.org/jira/browse/HBASE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-28004. -- Resolution: Fixed > Persistent cache map can get corrupt if crash happens midway through the write > -- > > Key: HBASE-28004 > URL: https://issues.apache.org/jira/browse/HBASE-28004 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.6.0, 3.0.0-alpha-4, 4.0.0-alpha-1 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1 > > > HBASE-27686 added a background thread for periodically saving the cache index > map, together with a list of completed cached files so that we can recover > the cache state in case of crash or restart. Problem is that the cache index > can become few GB large (a sample case with 1.6TB of used bucket cache would > map to between 8GB to 10GB indexes), and these writes take few seconds to > complete, causing any RS crash very likely to leave a corrupt index file that > can't be recovered when the RS starts again. Worse, since we store the list > of cached files on a separate file, this also leads to cache inconsistencies, > with files in the list of cached files never cached once the RS is restarted, > even though we have no cache index for those and every read ends up going to > the FS. > This task aims to refactor the cache persistent as follows: > 1) Write both the list of completely cached files and the cache indexes in a > single file, so that we can have this synced atomically; > 2) When writing the persistent cache file, use a temp name first, then once > the write is successfully finished, rename it to the actual name. This way, > if crash happens whilst the persistent cache is still being written, the temp > file would be corrupt, but we could still recover from the last successful > sync, and we would only lose the caching ops since the last sync. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28010) Connection attributes can become corrupted on the server side
[ https://issues.apache.org/jira/browse/HBASE-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-28010. --- Fix Version/s: 3.0.0-beta-1 Resolution: Fixed > Connection attributes can become corrupted on the server side > - > > Key: HBASE-28010 > URL: https://issues.apache.org/jira/browse/HBASE-28010 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Ray Mattingly >Priority: Blocker > Fix For: 2.6.0, 3.0.0-beta-1 > > > In ServerRpcConnection.processOneRpc, it calls processConnectionHeader and > then immediately calls callCleanupIfNeeded. The parsing of the ByteBuff into > the ConnectionHeader does not copy the bytes. We keep a reference to > ConnectionHeader for later use, but since the underlying ByteBuff gets > released in callCleanupIfNeeded, later requests can override the memory > locations that the ConnectionHeader points at. > The unit tests we added dont catch this possibly because they don't send > enough requests to corrupt the buffers. It happens pretty quickly in a > deployed cluster. > We need to copy the List from the ConnectionHeader into a Map > before the buffer is released. This probably means we should remove > getConnectionHeader from the RpcCall interface and instead add > getConnectionAttributes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Branching for 2.6 code line (branch-2.6)
We resolved HBASE-28010, so we should be good to go there now. On Sun, Aug 20, 2023 at 7:22 AM Bryan Beaudreault wrote: > I made > https://issues.apache.org/jira/browse/HBASE-28010 a blocker so we don’t > release a broken new feature. We will have it fixed early this week. > > On Fri, Aug 18, 2023 at 11:08 AM Andrew Purtell > wrote: > >> Let’s do it. We can use the new branch to stabilize for release. >> >> I am back from vacation in two weeks and can then lend time and AWS >> resources for testing (and presumably fixing). >> >> > On Aug 18, 2023, at 10:53 AM, 张铎 wrote: >> > >> > HBASE-27947 has been resolved. >> > >> > So I think now we are good to cut branch-2.6? >> > >> > Another good news is we are also close to fix the WAL value >> > compression issue in HBASE-28028. >> > >> > Thanks. >> > >> > Bryan Beaudreault 于2023年6月23日周五 02:47写道： >> >> >> >> Thanks! >> >> >> >> We're looking into one other emergent issue that we uncovered during >> the >> >> rollout of server side TLS on RegionServers. It seems >> nettyDirectMemory has >> >> increased substantially when under load with it enabled. Details in >> >> https://issues.apache.org/jira/browse/HBASE-27947. >> >> >> >> >> >>> On Thu, Jun 22, 2023 at 12:02 PM 张铎(Duo Zhang) > > >> >>> wrote: >> >>> >> >>> PR is ready >> >>> >> >>> https://github.com/apache/hbase/pull/5305 >> >>> >> >>> PTAL. >> >>> >> >>> Thanks. >> >>> >> >>> 张铎(Duo Zhang) 于2023年6月22日周四 21:40写道： >> >> Ah, missed your last comment on HBASE-27782. >> >> Let me take a look. >> >> Netty has some rules about how the exceptions are passed through the >> pipeline(especially the order, forward or backward...) but honestly I >> always forget it just a day later after I finished the code... >> >> Bryan Beaudreault 于2023年6月17日周六 00:43写道： >> > >> > In terms of TLS: >> > >> > - All of our clients (many thousands) in production are using the >> > NettyRpcConnection with TLS enabled. However, these clients are >> >>> currently >> > connecting to the RegionServer/HMaster through an haproxy process >> >>> local to >> > each server which handles SSL termination. So not quite end-to-end >> yet. >> > - On the server side, most of our QA environment (a thousand >> >>> regionservers >> > and ~200 hmasters) are running it. So these are accepting TLS from >> >>> clients >> > and using TLS for intra-cluster communication. >> > >> > The migration is tricky for us due to the scale and the fact that we >> >>> need >> > to migrate off haproxy at the same time. Hopefully we should have >> some >> >>> of >> > production running end-to-end TLS within the next month or so. >> > >> > From what we've seen in QA so far, there have not been any major >> >>> issues. We >> > also couldn't discern any performance issues in testing, though we >> were >> > comparing against our legacy haproxy setup and can't really compare >> >>> against >> > kerberos. >> > >> > One outstanding issue is >> >>> https://issues.apache.org/jira/browse/HBASE-27782, >> > which we still see periodically. It doesn't seem to cause actual >> >>> issues, >> > since the RpcClient still handles it gracefully, but it does cause >> >>> noise >> > and may have implications. >> > >> > On Fri, Jun 16, 2023 at 11:41 AM 张铎(Duo Zhang) < >> palomino...@gmail.com> >> > wrote: >> > >> >> So any updates here? >> >> >> >> Do we have any good news about the TLS usage in production so we >> can >> >> move forward on release 2.6.x? >> >> >> >> Thanks. >> >> >> >> Andrew Purtell 于2023年4月7日周五 09:37写道： >> >>> >> >>> Agreed, that sounds like a good plan. >> >>> >> >>> On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang) < >> >>> palomino...@gmail.com> >> >> wrote: >> >>> >> I think we could follow the old pattern when we cut a new release >> >> branch. >> That is, after the new release branch is cut and the new minor >> >>> release >> >> is >> out, we will do a final release of the oldest release line and >> >>> then >> >> mark it >> as EOL. >> >> So here, I think once we cut branch-2.6 and release 2.6.0, we >> >>> can do a >> final release for 2.4.x and mark 2.4.x as EOL. >> >> Thanks. >> >> Bryan Beaudreault 于2023年3月27日周一 >> >>> 09:57写道： >> >> > Primary development on hbase-backup and TLS is complete. There >> >>> are a >> couple >> > minor things I may want to add to TLS in the future, such as >> >> pluggable >> cert >> > verification. But those are not needed for initial release IMO. >> > >> > We are almost ready integrating hbase-backup in production. >> >>> We’ve >> >> fixed a >> > few minor things (all committed) but otherwise it’s worked >> >>> well so >> >> f
[jira] [Created] (HBASE-28042) Snapshot corruptions due to non-atomic rename within same filesystem
Viraj Jasani created HBASE-28042:
Summary: Snapshot corruptions due to non-atomic rename within same
filesystem
Key: HBASE-28042
URL: https://issues.apache.org/jira/browse/HBASE-28042
Project: HBase
Issue Type: Bug
Reporter: Viraj Jasani
Assignee: Viraj Jasani
Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
Sequence of events that can lead to snapshot corruptions:
# Create snapshot using admin command
# Active master triggers async snapshot creation
# If the snapshot operation doesn't complete within 5 min, client gets
exception
{code:java}
org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Snapshot
'T1_1691888405683_1691888440827_1' wasn't completed in expectedTime:60 ms
{code}
# Client initiates snapshot deletion after this error
# In the snapshot completion/commit phase, the files are moved from tmp to
final dir.
# Snapshot delete and snapshot commit operations can cause corruption by
leaving incomplete metadata:
* [Snapshot commit] create
"/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo"
* [Snapshot delete from client] delete
"/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/.snapshotinfo"
* [Snapshot commit] create
"/hbase/.hbase-snapshot/T1_1691888405683_1691888440827_1/data-manifest"
The changes introduced by HBASE-21098 performs atomic rename for hbase 1 but
not for hbase 2
{code:java}
public static void completeSnapshot(Path snapshotDir, Path workingDir,
FileSystem fs,
FileSystem workingDirFs, final Configuration conf)
throws SnapshotCreationException, IOException {
LOG.debug(
"Sentinel is done, just moving the snapshot from " + workingDir + " to "
+ snapshotDir);
URI workingURI = workingDirFs.getUri();
URI rootURI = fs.getUri();
if (
(!workingURI.getScheme().equals(rootURI.getScheme()) ||
workingURI.getAuthority() == null
|| !workingURI.getAuthority().equals(rootURI.getAuthority())
|| workingURI.getUserInfo() == null //always true for hdfs://{cluster}
|| !workingURI.getUserInfo().equals(rootURI.getUserInfo())
|| !fs.rename(workingDir, snapshotDir)) //this condition isn't even
evaluated due to short circuit above
&& !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true,
true, conf) // non-atomic rename operation
) {
throw new SnapshotCreationException("Failed to copy working directory(" +
workingDir
+ ") to completed directory(" + snapshotDir + ").");
}
} {code}
whereas for hbase 1
{code:java}
// check UGI/userInfo
if (workingURI.getUserInfo() == null && rootURI.getUserInfo() != null) {
return true;
}
if (workingURI.getUserInfo() != null &&
!workingURI.getUserInfo().equals(rootURI.getUserInfo())) {
return true;
}
{code}
this causes shouldSkipRenameSnapshotDirectories() to return false if workingURI
and rootURI share the same filesystem, which would always lead to atomic rename:
{code:java}
if ((shouldSkipRenameSnapshotDirectories(workingURI, rootURI)
|| !fs.rename(workingDir, snapshotDir))
&& !FileUtil.copy(workingDirFs, workingDir, fs, snapshotDir, true, true,
conf)) {
throw new SnapshotCreationException("Failed to copy working directory(" +
workingDir
+ ") to completed directory(" + snapshotDir + ").");
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Time for new releases?
Yes I think it is about time too. I’m back from vacation next week and can do a release then unless you want to do it Duo. > On Aug 23, 2023, at 9:49 AM, 张铎 wrote: > > Bump. > > If the release managers are not available, I could help make the next > 2.5.x release. > > Thanks. > > 张铎(Duo Zhang) 于2023年8月15日周二 11:34写道： >> >> HBase 2.5.5 was released at 6.14, it is time for 2.5.6? >> >> For 2.4.18, I prefer we make it after we release 2.6.0. >> >> Thanks.
Re: Time for new releases?
Thanks Andrew. I think releasing next week is OK, no hurry, enjoy your vacation! Andrew Purtell 于2023年8月24日周四 06:39写道： > > Yes I think it is about time too. > > I’m back from vacation next week and can do a release then unless you want > to do it Duo. > > > On Aug 23, 2023, at 9:49 AM, 张铎 wrote: > > > > Bump. > > > > If the release managers are not available, I could help make the next > > 2.5.x release. > > > > Thanks. > > > > 张铎(Duo Zhang) 于2023年8月15日周二 11:34写道： > >> > >> HBase 2.5.5 was released at 6.14, it is time for 2.5.6? > >> > >> For 2.4.18, I prefer we make it after we release 2.6.0. > >> > >> Thanks.
Re: Branching for 2.6 code line (branch-2.6)
Let's get HBASE-28028 in too. Since Andrew is still on vacation, let me find another committer to help review the PR. Thanks. Bryan Beaudreault 于2023年8月24日周四 02:23写道： > > We resolved HBASE-28010, so we should be good to go there now. > > On Sun, Aug 20, 2023 at 7:22 AM Bryan Beaudreault > wrote: > > > I made > > https://issues.apache.org/jira/browse/HBASE-28010 a blocker so we don’t > > release a broken new feature. We will have it fixed early this week. > > > > On Fri, Aug 18, 2023 at 11:08 AM Andrew Purtell > > wrote: > > > >> Let’s do it. We can use the new branch to stabilize for release. > >> > >> I am back from vacation in two weeks and can then lend time and AWS > >> resources for testing (and presumably fixing). > >> > >> > On Aug 18, 2023, at 10:53 AM, 张铎 wrote: > >> > > >> > HBASE-27947 has been resolved. > >> > > >> > So I think now we are good to cut branch-2.6? > >> > > >> > Another good news is we are also close to fix the WAL value > >> > compression issue in HBASE-28028. > >> > > >> > Thanks. > >> > > >> > Bryan Beaudreault 于2023年6月23日周五 02:47写道： > >> >> > >> >> Thanks! > >> >> > >> >> We're looking into one other emergent issue that we uncovered during > >> the > >> >> rollout of server side TLS on RegionServers. It seems > >> nettyDirectMemory has > >> >> increased substantially when under load with it enabled. Details in > >> >> https://issues.apache.org/jira/browse/HBASE-27947. > >> >> > >> >> > >> >>> On Thu, Jun 22, 2023 at 12:02 PM 张铎(Duo Zhang) >> > > >> >>> wrote: > >> >>> > >> >>> PR is ready > >> >>> > >> >>> https://github.com/apache/hbase/pull/5305 > >> >>> > >> >>> PTAL. > >> >>> > >> >>> Thanks. > >> >>> > >> >>> 张铎(Duo Zhang) 于2023年6月22日周四 21:40写道： > >> > >> Ah, missed your last comment on HBASE-27782. > >> > >> Let me take a look. > >> > >> Netty has some rules about how the exceptions are passed through the > >> pipeline(especially the order, forward or backward...) but honestly I > >> always forget it just a day later after I finished the code... > >> > >> Bryan Beaudreault 于2023年6月17日周六 00:43写道： > >> > > >> > In terms of TLS: > >> > > >> > - All of our clients (many thousands) in production are using the > >> > NettyRpcConnection with TLS enabled. However, these clients are > >> >>> currently > >> > connecting to the RegionServer/HMaster through an haproxy process > >> >>> local to > >> > each server which handles SSL termination. So not quite end-to-end > >> yet. > >> > - On the server side, most of our QA environment (a thousand > >> >>> regionservers > >> > and ~200 hmasters) are running it. So these are accepting TLS from > >> >>> clients > >> > and using TLS for intra-cluster communication. > >> > > >> > The migration is tricky for us due to the scale and the fact that we > >> >>> need > >> > to migrate off haproxy at the same time. Hopefully we should have > >> some > >> >>> of > >> > production running end-to-end TLS within the next month or so. > >> > > >> > From what we've seen in QA so far, there have not been any major > >> >>> issues. We > >> > also couldn't discern any performance issues in testing, though we > >> were > >> > comparing against our legacy haproxy setup and can't really compare > >> >>> against > >> > kerberos. > >> > > >> > One outstanding issue is > >> >>> https://issues.apache.org/jira/browse/HBASE-27782, > >> > which we still see periodically. It doesn't seem to cause actual > >> >>> issues, > >> > since the RpcClient still handles it gracefully, but it does cause > >> >>> noise > >> > and may have implications. > >> > > >> > On Fri, Jun 16, 2023 at 11:41 AM 张铎(Duo Zhang) < > >> palomino...@gmail.com> > >> > wrote: > >> > > >> >> So any updates here? > >> >> > >> >> Do we have any good news about the TLS usage in production so we > >> can > >> >> move forward on release 2.6.x? > >> >> > >> >> Thanks. > >> >> > >> >> Andrew Purtell 于2023年4月7日周五 09:37写道： > >> >>> > >> >>> Agreed, that sounds like a good plan. > >> >>> > >> >>> On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang) < > >> >>> palomino...@gmail.com> > >> >> wrote: > >> >>> > >> I think we could follow the old pattern when we cut a new release > >> >> branch. > >> That is, after the new release branch is cut and the new minor > >> >>> release > >> >> is > >> out, we will do a final release of the oldest release line and > >> >>> then > >> >> mark it > >> as EOL. > >> > >> So here, I think once we cut branch-2.6 and release 2.6.0, we > >> >>> can do a > >> final release for 2.4.x and mark 2.4.x as EOL. > >> > >> Thanks. > >> > >> Bryan Beaudreault 于2023年3月27日周一 > >> >>> 09:57写道： > >> > >> > Primary development on hbase-backup and TLS
[jira] [Created] (HBASE-28043) Reduce seeks from beginning of block in StoreFileScanner.seekToPreviousRow
Becker Ewing created HBASE-28043: Summary: Reduce seeks from beginning of block in StoreFileScanner.seekToPreviousRow Key: HBASE-28043 URL: https://issues.apache.org/jira/browse/HBASE-28043 Project: HBase Issue Type: Improvement Reporter: Becker Ewing Attachments: Current_SeekToPreviousRowBehavior.png, Proposed_SeekToPreviousRowBehavior.png Currently, for non-RIV1 DBE encodings, each call to [StoreFileScanner.seekToPreviousRow|https://github.com/apache/hbase/blob/89ca7f4ade84c84a246281c71898543b6161c099/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java#L493-L506] (a common operation in reverse scans) results in two seeks: # Seek from the beginning of the block to before the given row to find the prior row # Seek from the beginning of the block to the first cell of the prior row So if there are "N" rows in a block, a reverse scan through each row results in seeking past 2(N-1)! rows. This is a particularly expensive operation for tall tables that have many rows in a block. By introducing a state variable "previousRow" to StoreFileScanner, I believe that we could modify the seeking algorithm to be: # Seek from the beginning of the block to before the given row to find the prior row # Seek from the beginning of the block to before the row that is before the row that was just seeked to (i.e. 2 rows back). _Save_ this as a hint for where the prior row is in "previousRow" # Reseek from "previousRow" (2 rows back from start) to 1 row back from start (to the actual previousRow) Then the rest of the calls where a "previousRow" is present, you just need to seek to the beginning of the block once instead of twice, i.e. # seek from the beginning of the block to right before the beginning of your "previousRow" marker. Save this as the new "previousRow" marker # Reseek to the next row (i.e. your previous "previousRow" marker) See the attached diagrams for the current and proposed behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
