Re: Meta replicas in LoadBalance mode broken in 2.4?
It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta. Thanks, Huaxiang On 2022/03/26 04:03:15 Andrew Purtell wrote: > Just to be clear when I say "it seems pointless to have meta replicas which > do not actually receive updates (by default)", what I should have said is > 'timely updates', because a long delay in updating meta might as well be a > missed update. > > On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell wrote: > > > > "Async WAL replication for META is added as a new feature in 2.4.0. It > > is still under active development. Use with caution. Set > > hbase.region.replica.replication.catalog.enabled to enable async WAL > > Replication for META region replicas. It is off by default." > > > > Do we still need this warning? > > > > Should hbase.region.replica.replication.catalog.enabled have a default of > > 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems > > pointless to have meta replicas which do not actually receive updates (by > > default). > > > > > > On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun > > wrote: > > > >> Hi Andor, > >> > >>I get what you are saying. The HFile refreshing is the old way for > >> replica regions to refresh hfiles periodically, default is 5 minutes. In > >> this itbll case, we need to have the wal replication enabled for meta > >> replica. Please check out, > >> > >> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0. > >> Basically, you need to set > >> "hbase.region.replica.replication.catalog.enabled" to true in the > >> configuration and rerun itbll. Otherwise, all meta changes at the primary > >> meta region wont be updated at the replica meta regions and it will result > >> in itbll failures. > >> > >>Hope this helps, > >> > >>Huaxiang > >> > >> > >> On 2022/03/25 13:46:42 Andor Molnar wrote: > >> > Hi Huaxiang, > >> > > >> > We use 2.4.6 for the tests. > >> > > >> > I run itbll with the following command: > >> > > >> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList > >> generator 15 100 /tmp/hbase-itbll > >> > > >> > for the generator step and essentially jobs have failed. We can see the > >> meta request are spanning out to replicas, but writes start failing after > >> this due to the stale cache which is not getting updated. > >> > > >> > Would you please tell me more about ‘hfile refresh’ and how to > >> configure it? > >> > > >> > Thanks, > >> > Andor > >> > > >> > > >> > > >> > > >> > > On 2022. Mar 24., at 17:43, Huaxiang Sun > >> wrote: > >> > > > >> > > Hi Andor, > >> > > > >> > > Which 2.4 release do you test in your lab? We use this feature at > >> production cluster with 2.4.5. > >> > > At server side, we use hfile refresh instead of wal replication. I > >> used to run itbll for each release with this feature enabled. How did you > >> find the errors, did itbll fail? > >> > > > >> > > Regards, > >> > > Huaxiang > >> > > >> > > >> > > > > > > -- > > Best regards, > > Andrew > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > It's what we’ve earned > > Welcome, apocalypse, what’s taken you so long? > > Bring us the fitting end that we’ve been counting on > >- A23, Welcome, Apocalypse > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on >- A23, Welcome, Apocalypse >
[jira] [Created] (HBASE-26891) Make MetricsConnection scope configurable
Bryan Beaudreault created HBASE-26891: - Summary: Make MetricsConnection scope configurable Key: HBASE-26891 URL: https://issues.apache.org/jira/browse/HBASE-26891 Project: HBase Issue Type: Improvement Reporter: Bryan Beaudreault Assignee: Bryan Beaudreault Currently MetricsConnection scope is just the connection.toString(), which is typically the default toString() implementation for an object. This is not very useful for identifying different connections in processes that might connect to multiple clusters or maintain different connections to the same cluster. We can add a new config param "hbase.client.metrics.scope" which defaults to connection.toString(). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26890) Make the WAL interface async so sync replication can be built up on the WAL interface
Duo Zhang created HBASE-26890: - Summary: Make the WAL interface async so sync replication can be built up on the WAL interface Key: HBASE-26890 URL: https://issues.apache.org/jira/browse/HBASE-26890 Project: HBase Issue Type: Improvement Reporter: Duo Zhang Instead of hacking into the WAL implementation. This could make the implementation more general if later we want to change the WAL implementation. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] On setting fix versions with both 2.5.0 and 2.6.0
I think we have consensus here that we should set fix version to 2.5.0 only if the patch has been committed to both branch-2 and branch-2.5. And I think the solution to cleanup the incorrect fix versions also works. Thanks Nick. And thanks all for chiming in. Andrew Purtell 于2022年3月22日周二 01:36写道： > I think the sanest policy is — lowest released first —. So if released in > 2.4.11, then that’s all we need, for .0 versions. > > 2.5.0 changelog will be based on that of 2.4.11 or later. 2.6.0 changelog > will be based on 2.5.0 or later. It all rolls up. > > When committing to releasing code lines we also need a fix version for > each minor it will appear in. > > So for 3.0.0, this code line is already releasing alphas. You should > update fix versions to latest 3.0.0-alphaX when committing a change to > master branch. > > > > > > On Mar 21, 2022, at 10:30 AM, Nick Dimiduk wrote: > > > > So for example, if I apply a patch to master, branch-2, branch-2.5, and > > branch-2.4, I should only set its fixFor version to 2.4.x and 2.5.0, > > because 2.6.0 is assumed by 2.5.0 and 3.0.0 is assumed by 2.6.0 ? > > > >> On Mon, Mar 21, 2022 at 3:59 PM Andrew Purtell < > andrew.purt...@gmail.com> > >> wrote: > >> > >> I have used JQL searches and git log inspection previously but will be > >> trying out our git-jira-release-audit tool this time. After populating > the > >> local DB from git and JIRA for the three respective versions, SQL query > >> against the generated SQLite file should be convenient. > >> > >>> > On Mar 21, 2022, at 3:53 AM, Nick Dimiduk > wrote: > >>> > >>> I have also been doing as Andrew, whether it is the right thing or > not. > >> I > >>> think Duo is correct in that until 2.5.0, a commit to both branch-2.5 > and > >>> branch-2 should only be marked as 2.5.0. This should be easy enough to > >> fix > >>> -- once 2.5.0 is released, we can search Jira for all issues with > >>> fixVersion in (2.5.0, 2.6.0) and remove the 2.6.0 label. > >>> > >>> If you are starting the 2.5.0 release notes from the most recent 2.4.x > >>> release, then a similar cleanup can be done for 2.5.0, by searching for > >> all > >>> issues in 2.4.x that also have 2.5.0 fixVersion, and removing 2.5.0. I > am > >>> not sure what our release automation supports in this regard. > >>> > On Sun, Mar 20, 2022 at 6:40 PM Andrew Purtell < > >> andrew.purt...@gmail.com> > wrote: > > Per the earlier discussion, to be comprehensive, there is also this > >> rule: > > 4. If a change is in branch-2.4, branch-2.5, and branch-2: fix > version = > 2.4.x, the version the change was released in. (Fix versions 2.5.0 and > 2.6.0 will be dropped for already released changes.) > > The 2.5.0 change log will be based on that of the most recent 2.4.x > release. 2.4.0’s change log was based on that of the most recent 2.3.x > >> at > the time. > > Hopefully this clears up any concerns. When 2.5.0RC0 is put to a vote, > >> fix > versions will be tidy. It will also good for RC reviewers to check my > >> work > on the change log per usual. > > > On Mar 20, 2022, at 10:20 AM, Andrew Purtell < > andrew.purt...@gmail.com > >>> > wrote: > > > > I have been setting both but don’t claim that is correct. > > > > This is partly my fault for pushing back 2.5.0 for about six months > since cutting the branch. Partly this was waiting for always the next > >> thing > to squeeze in, and partly life and work obligations getting in the > way. > > > > I am actually ready personally to release 2.5 now. I would like to > >> pause > commits to branch-2.5 and cut a release after test stabilization. > >> However > some tasks are not quite done. There is the SFT backport merge PR > that I > would like approvals on so I can merge it and there are the two issues > opened for post log4j2 merge issues, the one with the shell logging > zookeeper noise at startup, and the one for the test failures due to > >> CNFE. > These should be resolved. Then we can cut 2.5.0RC0. > > > > When cutting the RC I will clean up fix versions. I did this before > at > 2.4.0RC0. There was some discussion on dev@ at the time you can refer > >> to > in the archive. We have an audit tool for the purpose plus manual > inspection of the git history when necessary. This is how I would > adjust > fix versions, according to consensus at the last time: > > > > 1. If in branch-2.5 only: fix version = 2.5.0 > > > > 2. If in branch-2.5 and branch-2: fix version = 2.5.0 > > > > 3. If in branch-2 only: fix version = 2.6.0. > > > > There is no need for anyone to do anything special until 2.5.0RC0. It > would be helpful if you decide to do some work following the above > >> rules to > tidy fix versions, but not necessary, or you can continue to set both > or > either. > > > >>>
[jira] [Created] (HBASE-26889) nightly yetus tests don't reflect failure when required environment variables are missing.
Sean Busbey created HBASE-26889:
---
Summary: nightly yetus tests don't reflect failure when required
environment variables are missing.
Key: HBASE-26889
URL: https://issues.apache.org/jira/browse/HBASE-26889
Project: HBase
Issue Type: Bug
Components: test
Reporter: Sean Busbey
the nightly yetus wrapper shows an error when we're missing some needed
environment variables, but the stage shows success still.
e.g
{code:java}
[2022-03-24T22:35:39.450Z] [ERROR] Required environment variable 'DEBUG' is not
set.
[2022-03-24T22:35:39.451Z] [ERROR] Required environment variable
'USE_YETUS_PRERELEASE' is not set.
[2022-03-24T22:35:39.451Z] [ERROR] Please set the required environment
variables before invoking. If this error is on Jenkins, then please file a
JIRA about the error. {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26888) Update branch-specific nightly test handling to allow for earlier release line feature branches
Sean Busbey created HBASE-26888:
---
Summary: Update branch-specific nightly test handling to allow for
earlier release line feature branches
Key: HBASE-26888
URL: https://issues.apache.org/jira/browse/HBASE-26888
Project: HBase
Issue Type: Task
Components: integration tests, test
Reporter: Sean Busbey
due to limitations in ASF infra our branch protection rules prohibit force
pushes or deletions on branches that start with "master" or "branch-".
Our current nightly tests assume that feature branches specific to branch-2 or
branch-1 will start with the release branch name rather than the feature Jira
key. However, doing so would mean we need to file INFRA jiras to clean up
afterwards.
We can't change the limitations of branch protection rules, so we should update
the nightly tests to allow for feature branches named like
{{{}HBASE-X-branch-2{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26887) nightly integration test claims success while showing error parsing shell output
Sean Busbey created HBASE-26887:
---
Summary: nightly integration test claims success while showing
error parsing shell output
Key: HBASE-26887
URL: https://issues.apache.org/jira/browse/HBASE-26887
Project: HBase
Issue Type: Bug
Components: integration tests, shell
Reporter: Sean Busbey
current master branch claims that the integration test is succeeding, but
looking at the detailed output shows a failure to parse the shell output:
{code:java}
Starting up Hadoop
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
Verifying configs
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-integration/hadoop-3/hbase-conf/core-site.xml:
valid
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-integration/hadoop-3/hbase-conf/hbase-site.xml:
valid
OK
Listing HDFS contents
Starting up HBase
running master, logging to
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/hbase-install/bin/../logs/hbase-jenkins-master-jenkins-hbase12.out
retry waiting for hbase to come up.
Setting up table 'test:example' with 1,000 regions
writing out example TSV to example.tsv
uploading example.tsv to HDFS
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Importing TSV via shaded client artifact for HBase - MapReduce integration.
Verifying row count from import.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh:
line 418: [: hbase:002:0> : integer expression expected
Hadoop client jars not given; getting them from 'hadoop classpath' for the
example.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Building shaded client example.
Running shaded client example. It'll fetch the set of regions, round-trip them
to a file in HDFS, then write them one-per-row into the test table.
Checking on results of example program.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Verifying row count from example.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh:
line 531: [: hbase:002:0> : integer expression expected
ERROR: Only found hbase:002:0> rows.
Shutting down HBase {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
