Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
+1 on the log4j2 2.17.0 -Stephen On Sat, Dec 18, 2021 at 10:14 PM 张铎(Duo Zhang) wrote: > Let's also update log4j2 to 2.17.0 for hbase-oeprator-tools? > > Thanks. > > 张铎(Duo Zhang) 于2021年12月18日周六 17:07写道： > > > +1 (binding) > > > > Checked sigs and sums: Matched > > Rat check: Passed > > Built from src: Succeeded > > Run UTs: Passed > > CHANGES and RELEASENOTES: Missed two issues, for generating these two > > files and change version in pom. This is not an actual code problem so I > > always do not want to sink an RC due to the problems on these two files. > In > > the future hbase-operator-tools should consider removing these two files > > from git too. > > Log4j2: Checked the lib directory, the log4j2 version is 2.16.0, which is > > good. > > Run it against a hbase cluster: Use it with the cluster deployed by me > for > > testing HBASE-26233, the default command failed because of can > > not recognize the 'hdfs' protocol. I needed to manually prepend > > 'INTERNAL_CLASSPATH=true' when executing the command, like this > > INTERNAL_CLASSPATH=true ./bin/hbase hbck -j > > > /home/sa/disk1/hbase-operator-tools-1.2.0/hbase-hbck2/hbase-hbck2-1.2.0.jar > > extraRegionsInMeta default:IntegrationTestRegionReplicaReplication > > But this is not a problem of hbase-operator-tools, it is because we do > > not shade hadoop-hdfs in our shaded jar. And it is strange that why hbck > > uses hbase-shaded-client instead of hbase-shaded-mapreduce. > > The stacktrace > > 17:04:41.024 [main] ERROR org.apache.hbase.HBCK2 - Error on checking > extra > > regions: > > org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for > > scheme "hdfs" > > at > > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hbase.HBCKFsUtils.getRootDir(HBCKFsUtils.java:106) > > ~[hbase-hbck2-1.2.0.jar:1.2.0] > > at > > > org.apache.hbase.FsRegionsMetaRecoverer.(FsRegionsMetaRecoverer.java:66) > > ~[hbase-hbck2-1.2.0.jar:1.2.0] > > at org.apache.hbase.HBCK2.extraRegionsInMeta(HBCK2.java:268) > > [hbase-hbck2-1.2.0.jar:1.2.0] > > at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:1020) > > [hbase-hbck2-1.2.0.jar:1.2.0] > > at org.apache.hbase.HBCK2.run(HBCK2.java:830) > [hbase-hbck2-1.2.0.jar:1.2.0] > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > > [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) > > [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > > at org.apache.hbase.HBCK2.main(HBCK2.java:1145) > > [hbase-hbck2-1.2.0.jar:1.2.0 > > > > 张铎(Duo Zhang) 于2021年12月18日周六 16:36写道： > > > >> The readme of the project says we have two tools, one is HBCK2, the > other > >> is TableReporter, but in the binary I can only see HBCK2... > >> > >> But anyway, checking the previous release, we did not include > >> TableReporter either, so not this release's fault... > >> > >> Josh Elser 于2021年12月17日周五 23:03写道： > >> > >>> +1 (binding) > >>> > >>> Thanks for putting this together, Guangxu! > >>> > >>> * xsums/sigs are great > >>> * RAT check passes on src release > >>> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8) > >>> * Can build from source > >>> * CHANGES and RELEASENOTES look fine at a glance > >>> * Public key published in KEYS > >>> * Verified log4j2.16 is in the binary release (both as a jar and shaded > >>> inside hbase-hbck) > >>> > >>> - Josh > >>> > >>> On 12/14/21 10:32 PM, Guangxu Cheng wrote: > >>> > Please vote on this Apache hbase operator tools release candidate, > >>> > hbase-operator-tools-1.2.0RC0 > >>> > > >>> > The VOTE will remain open for at least 72 hours. > >>> > > >>> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0 > >>> > [ ] -1 Do not release this package because ... > >>> > > >>> > The tag to be voted on is 1.2.0RC0: > >>> > > >>> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 > >>> > > >>> > This tag currently points to git reference > >>> > > >>> >76d68624cebb66ec0e509b0a4c0d96445a601685 > >>> > > >>> > The release files, including signatures, digests, as well as > CHANGES.md > >>> > and RELEASENOTES.md included
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
https://github.com/apache/hbase/pull/3965
Andrew Purtell 于2021年12月19日周日 13:51写道：
> Sure, we are on the same page about this RC.
>
> > On Dec 18, 2021, at 9:46 PM, 张铎 wrote:
> >
> > I think we are on the same page that we should upgrade to the newest
> log4j2
> > version since the final release has not been published yet.
> >
> > But on log4j1, in our community we have discussed this before when there
> is
> > a CVE for it. You can view this page
> >
> > https://logging.apache.org/log4j/1.2/
> >
> > And even for the recent CVE, log4j1 is also affected, as listed on the
> page
> > you provided.
> >
> > Log4j 1.x mitigation
> >
> > Log4j 1.x does not have Lookups so the risk is lower. Applications using
> > Log4j 1.x are only vulnerable to this attack when they use JNDI in their
> > configuration. A separate CVE (CVE-2021-4104) has been filed for this
> > vulnerability. To mitigate: Audit your logging configuration to ensure it
> > has no JMSAppender configured. Log4j 1.x configurations without
> JMSAppender
> > are not impacted by this vulnerability.
> >
> > It is as you said in the first paragraph, log4j1 has a special CVE for
> it,
> > and it will never be fixed. We need to say that ‘yes it is affected but
> > only if you bla bla’, not good for end users right?
> >
> > So I still stand my point that, stay on log4j1 is not a good choice, it
> is
> > not because we have already done the work, it is our duty to keep our
> users
> > safe from security problem.
> >
> > And on the Hadoop part, it is also me that trying to upgrade to log4j2.
> But
> > as you know Hadoop is actually constructed by several projects, it is
> > really not easy to do this work, like what we have done in HBase.
> >
> > Anyway, let me prepare a new RC.
> >
> > Thanks.
> > Andrew Purtell 于2021年12月19日 周日09:12写道：
> >
> >> As to your first point, I think it is a simple consideration: A user’s
> >> security department or compliance regulator will ask: “Does this version
> >> include log4j with a known CVE?” Why would we provide a release where
> they
> >> have to answer “yes” when we can provide them a release where they can
> >> answer “no”? Based on todays knowledge. (And yes I am aware that a user
> can
> >> manually upgrade the jar versions in place after unpacking the tarballs.
> >> Nonetheless.)
> >>
> >> I disagree that there was a real need to upgrade log4j because 1.x was
> EOL
> >> but I won’t argue that staying with old dependencies is automatically
> good.
> >> It’s done, anyway. The main point I would like to make here is should a
> >> good alternative emerge from this mess I am going to look at replacing
> >> log4j 2 with it. Or, if log4j decides to accept the inevitable and
> remove
> >> the dangerous substitution/rewrite feature then that would be fine too.
> >>
> On Dec 18, 2021, at 4:42 PM, 张铎 wrote:
> >>>
> >>> After 2.15.0, all the problems require you manually put some special
> >>> markers in the pattern layout in your configuration file, so it is
> >> already
> >>> less hurt, we do not have something like %m{lookup} in the pattern
> layout
> >>> by default.
> >>>
> >>> Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to
> the
> >>> newest version.
> >>>
> >>> But stay on log4j1 should not be considered as a solution. Log4j1 is
> >>> already dead long ago and it has several CVEs where no one wants to fix
> >>> them, and our statement was just ‘do not use the feature’. That’s why
> we
> >>> want to migrate to log4j2. Every project may have CVEs, so I think
> >> whether
> >>> there are still enough people who are still maintaining the project is
> >> the
> >>> most important thing. Log4j2 is already the most active logging
> >> framework,
> >>> another option is logback, but there were no releases for nearly 4
> years
> >>> before 2021…
> >>>
> >>> Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
> >>>
> >>> Andrew Purtell 于2021年12月19日 周日05:25写道：
> >>>
> Apologies, I managed to hit the send button before finishing. My veto
> >> can
> be cured by upgrading Log4J to ** 2.17.0 ** . See
> https://logging.apache.org/log4j/2.x/security.html.
>
> > On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell
> > wrote:
> >
> > -1 (binding)
> >
> > The Log4J issues are not fixed by 2.15.
> >
> > I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
> know
> > they have plans to upgrade. It does not seem advisable to use Log4J 2
> >> at
> > all actually. Another option that does not include such a dangerous
> > reference/rewrite mechanism might be preferable.
> >
> >> On Sat, Dec 18, 2021 at 12:02 PM Josh Elser
> >> wrote:
> >
> >> +1 (binding)
> >>
> >> * Xsums/sigs good
> >> * Can build from source
> >> * Log4j 2.15 is included (more on this in the below)
> >> * log4j2.formatMsgNoLookups=true is set (multiple times per process,
> >> but
> >> properly set)
> >> *
[jira] [Created] (HBASE-26607) Put up 3.0.0-alpha-2RC2
Duo Zhang created HBASE-26607: - Summary: Put up 3.0.0-alpha-2RC2 Key: HBASE-26607 URL: https://issues.apache.org/jira/browse/HBASE-26607 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26569) Put up 3.0.0-alpha-2RC1
[ https://issues.apache.org/jira/browse/HBASE-26569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26569. --- Resolution: Fixed Still need to upgrade log4j2. > Put up 3.0.0-alpha-2RC1 > --- > > Key: HBASE-26569 > URL: https://issues.apache.org/jira/browse/HBASE-26569 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
Let's also update log4j2 to 2.17.0 for hbase-oeprator-tools? Thanks. 张铎(Duo Zhang) 于2021年12月18日周六 17:07写道： > +1 (binding) > > Checked sigs and sums: Matched > Rat check: Passed > Built from src: Succeeded > Run UTs: Passed > CHANGES and RELEASENOTES: Missed two issues, for generating these two > files and change version in pom. This is not an actual code problem so I > always do not want to sink an RC due to the problems on these two files. In > the future hbase-operator-tools should consider removing these two files > from git too. > Log4j2: Checked the lib directory, the log4j2 version is 2.16.0, which is > good. > Run it against a hbase cluster: Use it with the cluster deployed by me for > testing HBASE-26233, the default command failed because of can > not recognize the 'hdfs' protocol. I needed to manually prepend > 'INTERNAL_CLASSPATH=true' when executing the command, like this > INTERNAL_CLASSPATH=true ./bin/hbase hbck -j > /home/sa/disk1/hbase-operator-tools-1.2.0/hbase-hbck2/hbase-hbck2-1.2.0.jar > extraRegionsInMeta default:IntegrationTestRegionReplicaReplication > But this is not a problem of hbase-operator-tools, it is because we do > not shade hadoop-hdfs in our shaded jar. And it is strange that why hbck > uses hbase-shaded-client instead of hbase-shaded-mapreduce. > The stacktrace > 17:04:41.024 [main] ERROR org.apache.hbase.HBCK2 - Error on checking extra > regions: > org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for > scheme "hdfs" > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hbase.HBCKFsUtils.getRootDir(HBCKFsUtils.java:106) > ~[hbase-hbck2-1.2.0.jar:1.2.0] > at > org.apache.hbase.FsRegionsMetaRecoverer.(FsRegionsMetaRecoverer.java:66) > ~[hbase-hbck2-1.2.0.jar:1.2.0] > at org.apache.hbase.HBCK2.extraRegionsInMeta(HBCK2.java:268) > [hbase-hbck2-1.2.0.jar:1.2.0] > at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:1020) > [hbase-hbck2-1.2.0.jar:1.2.0] > at org.apache.hbase.HBCK2.run(HBCK2.java:830) [hbase-hbck2-1.2.0.jar:1.2.0] > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) > [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] > at org.apache.hbase.HBCK2.main(HBCK2.java:1145) > [hbase-hbck2-1.2.0.jar:1.2.0 > > 张铎(Duo Zhang) 于2021年12月18日周六 16:36写道： > >> The readme of the project says we have two tools, one is HBCK2, the other >> is TableReporter, but in the binary I can only see HBCK2... >> >> But anyway, checking the previous release, we did not include >> TableReporter either, so not this release's fault... >> >> Josh Elser 于2021年12月17日周五 23:03写道： >> >>> +1 (binding) >>> >>> Thanks for putting this together, Guangxu! >>> >>> * xsums/sigs are great >>> * RAT check passes on src release >>> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8) >>> * Can build from source >>> * CHANGES and RELEASENOTES look fine at a glance >>> * Public key published in KEYS >>> * Verified log4j2.16 is in the binary release (both as a jar and shaded >>> inside hbase-hbck) >>> >>> - Josh >>> >>> On 12/14/21 10:32 PM, Guangxu Cheng wrote: >>> > Please vote on this Apache hbase operator tools release candidate, >>> > hbase-operator-tools-1.2.0RC0 >>> > >>> > The VOTE will remain open for at least 72 hours. >>> > >>> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0 >>> > [ ] -1 Do not release this package because ... >>> > >>> > The tag to be voted on is 1.2.0RC0: >>> > >>> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 >>> > >>> > This tag currently points to git reference >>> > >>> >76d68624cebb66ec0e509b0a4c0d96445a601685 >>> > >>> > The release files, including signatures, digests, as well as CHANGES.md >>> > and RELEASENOTES.md included in this RC can be found at: >>> > >>> > >>> > >>> https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/ >>> > >>> > Maven artifacts are available in a staging repository at: >>> > >>> > >>> https://repository.apache.org/content/repositories/orgapachehbase-1474/ >>> > Artifacts were signed with the
[jira] [Created] (HBASE-26606) Upgrade log4j2 to 2.17.0
Duo Zhang created HBASE-26606: - Summary: Upgrade log4j2 to 2.17.0 Key: HBASE-26606 URL: https://issues.apache.org/jira/browse/HBASE-26606 Project: HBase Issue Type: Task Components: logging, security Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0-alpha-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
Sure, we are on the same page about this RC.
> On Dec 18, 2021, at 9:46 PM, 张铎 wrote:
>
> I think we are on the same page that we should upgrade to the newest log4j2
> version since the final release has not been published yet.
>
> But on log4j1, in our community we have discussed this before when there is
> a CVE for it. You can view this page
>
> https://logging.apache.org/log4j/1.2/
>
> And even for the recent CVE, log4j1 is also affected, as listed on the page
> you provided.
>
> Log4j 1.x mitigation
>
> Log4j 1.x does not have Lookups so the risk is lower. Applications using
> Log4j 1.x are only vulnerable to this attack when they use JNDI in their
> configuration. A separate CVE (CVE-2021-4104) has been filed for this
> vulnerability. To mitigate: Audit your logging configuration to ensure it
> has no JMSAppender configured. Log4j 1.x configurations without JMSAppender
> are not impacted by this vulnerability.
>
> It is as you said in the first paragraph, log4j1 has a special CVE for it,
> and it will never be fixed. We need to say that ‘yes it is affected but
> only if you bla bla’, not good for end users right?
>
> So I still stand my point that, stay on log4j1 is not a good choice, it is
> not because we have already done the work, it is our duty to keep our users
> safe from security problem.
>
> And on the Hadoop part, it is also me that trying to upgrade to log4j2. But
> as you know Hadoop is actually constructed by several projects, it is
> really not easy to do this work, like what we have done in HBase.
>
> Anyway, let me prepare a new RC.
>
> Thanks.
> Andrew Purtell 于2021年12月19日 周日09:12写道：
>
>> As to your first point, I think it is a simple consideration: A user’s
>> security department or compliance regulator will ask: “Does this version
>> include log4j with a known CVE?” Why would we provide a release where they
>> have to answer “yes” when we can provide them a release where they can
>> answer “no”? Based on todays knowledge. (And yes I am aware that a user can
>> manually upgrade the jar versions in place after unpacking the tarballs.
>> Nonetheless.)
>>
>> I disagree that there was a real need to upgrade log4j because 1.x was EOL
>> but I won’t argue that staying with old dependencies is automatically good.
>> It’s done, anyway. The main point I would like to make here is should a
>> good alternative emerge from this mess I am going to look at replacing
>> log4j 2 with it. Or, if log4j decides to accept the inevitable and remove
>> the dangerous substitution/rewrite feature then that would be fine too.
>>
On Dec 18, 2021, at 4:42 PM, 张铎 wrote:
>>>
>>> After 2.15.0, all the problems require you manually put some special
>>> markers in the pattern layout in your configuration file, so it is
>> already
>>> less hurt, we do not have something like %m{lookup} in the pattern layout
>>> by default.
>>>
>>> Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
>>> newest version.
>>>
>>> But stay on log4j1 should not be considered as a solution. Log4j1 is
>>> already dead long ago and it has several CVEs where no one wants to fix
>>> them, and our statement was just ‘do not use the feature’. That’s why we
>>> want to migrate to log4j2. Every project may have CVEs, so I think
>> whether
>>> there are still enough people who are still maintaining the project is
>> the
>>> most important thing. Log4j2 is already the most active logging
>> framework,
>>> another option is logback, but there were no releases for nearly 4 years
>>> before 2021…
>>>
>>> Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
>>>
>>> Andrew Purtell 于2021年12月19日 周日05:25写道：
>>>
Apologies, I managed to hit the send button before finishing. My veto
>> can
be cured by upgrading Log4J to ** 2.17.0 ** . See
https://logging.apache.org/log4j/2.x/security.html.
> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell
> wrote:
>
> -1 (binding)
>
> The Log4J issues are not fixed by 2.15.
>
> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
know
> they have plans to upgrade. It does not seem advisable to use Log4J 2
>> at
> all actually. Another option that does not include such a dangerous
> reference/rewrite mechanism might be preferable.
>
>> On Sat, Dec 18, 2021 at 12:02 PM Josh Elser
>> wrote:
>
>> +1 (binding)
>>
>> * Xsums/sigs good
>> * Can build from source
>> * Log4j 2.15 is included (more on this in the below)
>> * log4j2.formatMsgNoLookups=true is set (multiple times per process,
>> but
>> properly set)
>> * hbase-config.sh issue is fixed over rc1
>>
>> Best as I've been able to keep up, it seems like we should already
>> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
>> that 2.16 may have issues still. It's my opinion that the changes we
>> have here in rc2 are a massive improvement
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
I think we are on the same page that we should upgrade to the newest log4j2
version since the final release has not been published yet.
But on log4j1, in our community we have discussed this before when there is
a CVE for it. You can view this page
https://logging.apache.org/log4j/1.2/
And even for the recent CVE, log4j1 is also affected, as listed on the page
you provided.
Log4j 1.x mitigation
Log4j 1.x does not have Lookups so the risk is lower. Applications using
Log4j 1.x are only vulnerable to this attack when they use JNDI in their
configuration. A separate CVE (CVE-2021-4104) has been filed for this
vulnerability. To mitigate: Audit your logging configuration to ensure it
has no JMSAppender configured. Log4j 1.x configurations without JMSAppender
are not impacted by this vulnerability.
It is as you said in the first paragraph, log4j1 has a special CVE for it,
and it will never be fixed. We need to say that ‘yes it is affected but
only if you bla bla’, not good for end users right?
So I still stand my point that, stay on log4j1 is not a good choice, it is
not because we have already done the work, it is our duty to keep our users
safe from security problem.
And on the Hadoop part, it is also me that trying to upgrade to log4j2. But
as you know Hadoop is actually constructed by several projects, it is
really not easy to do this work, like what we have done in HBase.
Anyway, let me prepare a new RC.
Thanks.
Andrew Purtell 于2021年12月19日 周日09:12写道：
> As to your first point, I think it is a simple consideration: A user’s
> security department or compliance regulator will ask: “Does this version
> include log4j with a known CVE?” Why would we provide a release where they
> have to answer “yes” when we can provide them a release where they can
> answer “no”? Based on todays knowledge. (And yes I am aware that a user can
> manually upgrade the jar versions in place after unpacking the tarballs.
> Nonetheless.)
>
> I disagree that there was a real need to upgrade log4j because 1.x was EOL
> but I won’t argue that staying with old dependencies is automatically good.
> It’s done, anyway. The main point I would like to make here is should a
> good alternative emerge from this mess I am going to look at replacing
> log4j 2 with it. Or, if log4j decides to accept the inevitable and remove
> the dangerous substitution/rewrite feature then that would be fine too.
>
> > On Dec 18, 2021, at 4:42 PM, 张铎 wrote:
> >
> > After 2.15.0, all the problems require you manually put some special
> > markers in the pattern layout in your configuration file, so it is
> already
> > less hurt, we do not have something like %m{lookup} in the pattern layout
> > by default.
> >
> > Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
> > newest version.
> >
> > But stay on log4j1 should not be considered as a solution. Log4j1 is
> > already dead long ago and it has several CVEs where no one wants to fix
> > them, and our statement was just ‘do not use the feature’. That’s why we
> > want to migrate to log4j2. Every project may have CVEs, so I think
> whether
> > there are still enough people who are still maintaining the project is
> the
> > most important thing. Log4j2 is already the most active logging
> framework,
> > another option is logback, but there were no releases for nearly 4 years
> > before 2021…
> >
> > Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
> >
> > Andrew Purtell 于2021年12月19日 周日05:25写道：
> >
> >> Apologies, I managed to hit the send button before finishing. My veto
> can
> >> be cured by upgrading Log4J to ** 2.17.0 ** . See
> >> https://logging.apache.org/log4j/2.x/security.html.
> >>
> >>> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell
> >>> wrote:
> >>>
> >>> -1 (binding)
> >>>
> >>> The Log4J issues are not fixed by 2.15.
> >>>
> >>> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
> >> know
> >>> they have plans to upgrade. It does not seem advisable to use Log4J 2
> at
> >>> all actually. Another option that does not include such a dangerous
> >>> reference/rewrite mechanism might be preferable.
> >>>
> On Sat, Dec 18, 2021 at 12:02 PM Josh Elser
> wrote:
> >>>
> +1 (binding)
>
> * Xsums/sigs good
> * Can build from source
> * Log4j 2.15 is included (more on this in the below)
> * log4j2.formatMsgNoLookups=true is set (multiple times per process,
> but
> properly set)
> * hbase-config.sh issue is fixed over rc1
>
> Best as I've been able to keep up, it seems like we should already
> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
> that 2.16 may have issues still. It's my opinion that the changes we
> have here in rc2 are a massive improvement over before. I think this
> is
> fine; I just wanted to acknowledge that we may still need to update
> again real soon.
>
> Thanks for your release manager work, Duo!
>
> On
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
As to your first point, I think it is a simple consideration: A user’s security
department or compliance regulator will ask: “Does this version include log4j
with a known CVE?” Why would we provide a release where they have to answer
“yes” when we can provide them a release where they can answer “no”? Based on
todays knowledge. (And yes I am aware that a user can manually upgrade the jar
versions in place after unpacking the tarballs. Nonetheless.)
I disagree that there was a real need to upgrade log4j because 1.x was EOL but
I won’t argue that staying with old dependencies is automatically good. It’s
done, anyway. The main point I would like to make here is should a good
alternative emerge from this mess I am going to look at replacing log4j 2 with
it. Or, if log4j decides to accept the inevitable and remove the dangerous
substitution/rewrite feature then that would be fine too.
> On Dec 18, 2021, at 4:42 PM, 张铎 wrote:
>
> After 2.15.0, all the problems require you manually put some special
> markers in the pattern layout in your configuration file, so it is already
> less hurt, we do not have something like %m{lookup} in the pattern layout
> by default.
>
> Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
> newest version.
>
> But stay on log4j1 should not be considered as a solution. Log4j1 is
> already dead long ago and it has several CVEs where no one wants to fix
> them, and our statement was just ‘do not use the feature’. That’s why we
> want to migrate to log4j2. Every project may have CVEs, so I think whether
> there are still enough people who are still maintaining the project is the
> most important thing. Log4j2 is already the most active logging framework,
> another option is logback, but there were no releases for nearly 4 years
> before 2021…
>
> Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
>
> Andrew Purtell 于2021年12月19日 周日05:25写道：
>
>> Apologies, I managed to hit the send button before finishing. My veto can
>> be cured by upgrading Log4J to ** 2.17.0 ** . See
>> https://logging.apache.org/log4j/2.x/security.html.
>>
>>> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell
>>> wrote:
>>>
>>> -1 (binding)
>>>
>>> The Log4J issues are not fixed by 2.15.
>>>
>>> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
>> know
>>> they have plans to upgrade. It does not seem advisable to use Log4J 2 at
>>> all actually. Another option that does not include such a dangerous
>>> reference/rewrite mechanism might be preferable.
>>>
On Sat, Dec 18, 2021 at 12:02 PM Josh Elser wrote:
>>>
+1 (binding)
* Xsums/sigs good
* Can build from source
* Log4j 2.15 is included (more on this in the below)
* log4j2.formatMsgNoLookups=true is set (multiple times per process, but
properly set)
* hbase-config.sh issue is fixed over rc1
Best as I've been able to keep up, it seems like we should already
upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
that 2.16 may have issues still. It's my opinion that the changes we
have here in rc2 are a massive improvement over before. I think this is
fine; I just wanted to acknowledge that we may still need to update
again real soon.
Thanks for your release manager work, Duo!
On 12/14/21 9:06 AM, Duo Zhang wrote:
> Please vote on this Apache hbase release candidate,
> hbase-3.0.0-alpha-2RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 3.0.0-alpha-2RC1:
>
> https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
>
> This tag currently points to git reference
>
> a3ff8e4c812eefab6ad32af45ca449a1395a6510
>
> The release files, including signatures, digests, as well as
>> CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
> https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
>
> Maven artifacts are available in a staging repository at:
>
>
https://repository.apache.org/content/repositories/orgapachehbase-1473/
>
> Artifacts were signed with the 9AD2AE49 key which can be found in:
>
> https://downloads.apache.org/hbase/KEYS
>
> 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
line.
> HBase 3.0.0 includes the following big feature/changes:
> Synchronous Replication
> OpenTelemetry Tracing
> Distributed MOB Compaction
> Backup and Restore
> Move RSGroup balancer to core
> Reimplement sync client on async client
> CPEPs on shaded proto
> Move the logging framework from log4j to log4j2
>
> 3.0.0-alpha-2 contains a critical security fix for addressing the
>> log4j2
>
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
After 2.15.0, all the problems require you manually put some special
markers in the pattern layout in your configuration file, so it is already
less hurt, we do not have something like %m{lookup} in the pattern layout
by default.
Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
newest version.
But stay on log4j1 should not be considered as a solution. Log4j1 is
already dead long ago and it has several CVEs where no one wants to fix
them, and our statement was just ‘do not use the feature’. That’s why we
want to migrate to log4j2. Every project may have CVEs, so I think whether
there are still enough people who are still maintaining the project is the
most important thing. Log4j2 is already the most active logging framework,
another option is logback, but there were no releases for nearly 4 years
before 2021…
Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
Andrew Purtell 于2021年12月19日 周日05:25写道：
> Apologies, I managed to hit the send button before finishing. My veto can
> be cured by upgrading Log4J to ** 2.17.0 ** . See
> https://logging.apache.org/log4j/2.x/security.html.
>
> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell
> wrote:
>
> > -1 (binding)
> >
> > The Log4J issues are not fixed by 2.15.
> >
> > I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
> know
> > they have plans to upgrade. It does not seem advisable to use Log4J 2 at
> > all actually. Another option that does not include such a dangerous
> > reference/rewrite mechanism might be preferable.
> >
> > On Sat, Dec 18, 2021 at 12:02 PM Josh Elser wrote:
> >
> >> +1 (binding)
> >>
> >> * Xsums/sigs good
> >> * Can build from source
> >> * Log4j 2.15 is included (more on this in the below)
> >> * log4j2.formatMsgNoLookups=true is set (multiple times per process, but
> >> properly set)
> >> * hbase-config.sh issue is fixed over rc1
> >>
> >> Best as I've been able to keep up, it seems like we should already
> >> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
> >> that 2.16 may have issues still. It's my opinion that the changes we
> >> have here in rc2 are a massive improvement over before. I think this is
> >> fine; I just wanted to acknowledge that we may still need to update
> >> again real soon.
> >>
> >> Thanks for your release manager work, Duo!
> >>
> >> On 12/14/21 9:06 AM, Duo Zhang wrote:
> >> > Please vote on this Apache hbase release candidate,
> >> > hbase-3.0.0-alpha-2RC1
> >> >
> >> > The VOTE will remain open for at least 72 hours.
> >> >
> >> > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> >> > [ ] -1 Do not release this package because ...
> >> >
> >> > The tag to be voted on is 3.0.0-alpha-2RC1:
> >> >
> >> >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
> >> >
> >> > This tag currently points to git reference
> >> >
> >> >a3ff8e4c812eefab6ad32af45ca449a1395a6510
> >> >
> >> > The release files, including signatures, digests, as well as
> CHANGES.md
> >> > and RELEASENOTES.md included in this RC can be found at:
> >> >
> >> >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
> >> >
> >> > Maven artifacts are available in a staging repository at:
> >> >
> >> >
> >> https://repository.apache.org/content/repositories/orgapachehbase-1473/
> >> >
> >> > Artifacts were signed with the 9AD2AE49 key which can be found in:
> >> >
> >> >https://downloads.apache.org/hbase/KEYS
> >> >
> >> > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
> >> line.
> >> > HBase 3.0.0 includes the following big feature/changes:
> >> >Synchronous Replication
> >> >OpenTelemetry Tracing
> >> >Distributed MOB Compaction
> >> >Backup and Restore
> >> >Move RSGroup balancer to core
> >> >Reimplement sync client on async client
> >> >CPEPs on shaded proto
> >> >Move the logging framework from log4j to log4j2
> >> >
> >> > 3.0.0-alpha-2 contains a critical security fix for addressing the
> log4j2
> >> > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
> >> > to 3.0.0-alpha-2 ASAP.
> >> >
> >> > Notice that this is not a production ready release. It is used to let
> >> our
> >> > users try and test the new major release, to get feedback before the
> >> final
> >> > GA release is out.
> >> > So please do NOT use it in production. Just try it and report back
> >> > everything you find unusual.
> >> >
> >> > And this time we will not include CHANGES.md and RELEASENOTE.md
> >> > in our source code, you can find it on the download site. For getting
> >> these
> >> > two files for old releases, please go to
> >> >
> >> >https://archive.apache.org/dist/hbase/
> >> >
> >> > To learn more about Apache hbase, please see
> >> >
> >> >http://hbase.apache.org/
> >> >
> >> > Thanks,
> >> > Your HBase Release Manager
> >> >
> >>
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> >
[jira] [Created] (HBASE-26605) TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths
Josh Elser created HBASE-26605:
--
Summary: TestHStore#testRefreshStoreFiles broken due to
unqualified and qualified paths
Key: HBASE-26605
URL: https://issues.apache.org/jira/browse/HBASE-26605
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 2.4.9
Reporter: Josh Elser
Assignee: Josh Elser
Was looking at a failures of this method where
{noformat}
[ERROR] org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles
Time elapsed: 4.2 s <<< ERROR!
java.util.NoSuchElementException
at
org.apache.hbase.thirdparty.com.google.common.collect.AbstractIndexedListIterator.next(AbstractIndexedListIterator.java:75)
at
org.apache.hadoop.hbase.regionserver.TestHStore.closeCompactedFile(TestHStore.java:962)
at
org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:1000)
{noformat}
This was on a branch where I had some HBASE-26067 changes backported, so I
thought the problem _was_ those changes. After a bit of digging, I believe the
test case itself is "broken" (the test passes, but for the wrong reasons).
This test methods adds some files to a Store (via memstore flush or direct
addition of a file) and eventually tries to get the first file which is
candidate to be removed. The test {*}never compacted any files{*}. This was the
first sign that the test itself was wrong.
After lots of comparison with the HBASE-26067 logging to compare against, I
found that the Store was listing a file which was created by the memstore flush
as a file to retain AND a file to remove. Second warning. Upon closer
inspection, I finally noticed that one of the files was qualified with the
filesystem URI and the other was not.
{noformat}
2021-12-18 16:57:10,903 INFO [Time-limited test] regionserver.HStore(675):
toBeAddedFiles=[file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/dd8a430b391546d8b9bdc39bb77d447b,
file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73],
toBeRemovedFiles=[/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73]
{noformat}
{{d4c5442b772c43fd9ebdfed1a11c0e73}} how are we both adding and removing this
file! Turns out, this is because one of them is "/..." and the other is
"file:/...". Either the problem is in TestHStore in how it is creating/adding
these files behind the scenes or we should be qualifying the Path inside of
StoreFileInfo with the filesystem that we're using.
I remember too vividly the problems when trying to separate the rootdir and
waldir from each other and am cautious against adding something to
StoreFileInfo to {{{}fs.qualifyPath(p){}}}. Need to look some more, but will
get a patch up to fix.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
Apologies, I managed to hit the send button before finishing. My veto can be cured by upgrading Log4J to ** 2.17.0 ** . See https://logging.apache.org/log4j/2.x/security.html. On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell wrote: > -1 (binding) > > The Log4J issues are not fixed by 2.15. > > I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I know > they have plans to upgrade. It does not seem advisable to use Log4J 2 at > all actually. Another option that does not include such a dangerous > reference/rewrite mechanism might be preferable. > > On Sat, Dec 18, 2021 at 12:02 PM Josh Elser wrote: > >> +1 (binding) >> >> * Xsums/sigs good >> * Can build from source >> * Log4j 2.15 is included (more on this in the below) >> * log4j2.formatMsgNoLookups=true is set (multiple times per process, but >> properly set) >> * hbase-config.sh issue is fixed over rc1 >> >> Best as I've been able to keep up, it seems like we should already >> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings >> that 2.16 may have issues still. It's my opinion that the changes we >> have here in rc2 are a massive improvement over before. I think this is >> fine; I just wanted to acknowledge that we may still need to update >> again real soon. >> >> Thanks for your release manager work, Duo! >> >> On 12/14/21 9:06 AM, Duo Zhang wrote: >> > Please vote on this Apache hbase release candidate, >> > hbase-3.0.0-alpha-2RC1 >> > >> > The VOTE will remain open for at least 72 hours. >> > >> > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 >> > [ ] -1 Do not release this package because ... >> > >> > The tag to be voted on is 3.0.0-alpha-2RC1: >> > >> >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1 >> > >> > This tag currently points to git reference >> > >> >a3ff8e4c812eefab6ad32af45ca449a1395a6510 >> > >> > The release files, including signatures, digests, as well as CHANGES.md >> > and RELEASENOTES.md included in this RC can be found at: >> > >> >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/ >> > >> > Maven artifacts are available in a staging repository at: >> > >> > >> https://repository.apache.org/content/repositories/orgapachehbase-1473/ >> > >> > Artifacts were signed with the 9AD2AE49 key which can be found in: >> > >> >https://downloads.apache.org/hbase/KEYS >> > >> > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release >> line. >> > HBase 3.0.0 includes the following big feature/changes: >> >Synchronous Replication >> >OpenTelemetry Tracing >> >Distributed MOB Compaction >> >Backup and Restore >> >Move RSGroup balancer to core >> >Reimplement sync client on async client >> >CPEPs on shaded proto >> >Move the logging framework from log4j to log4j2 >> > >> > 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2 >> > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade >> > to 3.0.0-alpha-2 ASAP. >> > >> > Notice that this is not a production ready release. It is used to let >> our >> > users try and test the new major release, to get feedback before the >> final >> > GA release is out. >> > So please do NOT use it in production. Just try it and report back >> > everything you find unusual. >> > >> > And this time we will not include CHANGES.md and RELEASENOTE.md >> > in our source code, you can find it on the download site. For getting >> these >> > two files for old releases, please go to >> > >> >https://archive.apache.org/dist/hbase/ >> > >> > To learn more about Apache hbase, please see >> > >> >http://hbase.apache.org/ >> > >> > Thanks, >> > Your HBase Release Manager >> > >> > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands >- A23, Crosstalk > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
-1 (binding) The Log4J issues are not fixed by 2.15. I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I know they have plans to upgrade. It does not seem advisable to use Log4J 2 at all actually. Another option that does not include such a dangerous reference/rewrite mechanism might be preferable. On Sat, Dec 18, 2021 at 12:02 PM Josh Elser wrote: > +1 (binding) > > * Xsums/sigs good > * Can build from source > * Log4j 2.15 is included (more on this in the below) > * log4j2.formatMsgNoLookups=true is set (multiple times per process, but > properly set) > * hbase-config.sh issue is fixed over rc1 > > Best as I've been able to keep up, it seems like we should already > upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings > that 2.16 may have issues still. It's my opinion that the changes we > have here in rc2 are a massive improvement over before. I think this is > fine; I just wanted to acknowledge that we may still need to update > again real soon. > > Thanks for your release manager work, Duo! > > On 12/14/21 9:06 AM, Duo Zhang wrote: > > Please vote on this Apache hbase release candidate, > > hbase-3.0.0-alpha-2RC1 > > > > The VOTE will remain open for at least 72 hours. > > > > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 > > [ ] -1 Do not release this package because ... > > > > The tag to be voted on is 3.0.0-alpha-2RC1: > > > >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1 > > > > This tag currently points to git reference > > > >a3ff8e4c812eefab6ad32af45ca449a1395a6510 > > > > The release files, including signatures, digests, as well as CHANGES.md > > and RELEASENOTES.md included in this RC can be found at: > > > >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/ > > > > Maven artifacts are available in a staging repository at: > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1473/ > > > > Artifacts were signed with the 9AD2AE49 key which can be found in: > > > >https://downloads.apache.org/hbase/KEYS > > > > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release > line. > > HBase 3.0.0 includes the following big feature/changes: > >Synchronous Replication > >OpenTelemetry Tracing > >Distributed MOB Compaction > >Backup and Restore > >Move RSGroup balancer to core > >Reimplement sync client on async client > >CPEPs on shaded proto > >Move the logging framework from log4j to log4j2 > > > > 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2 > > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade > > to 3.0.0-alpha-2 ASAP. > > > > Notice that this is not a production ready release. It is used to let our > > users try and test the new major release, to get feedback before the > final > > GA release is out. > > So please do NOT use it in production. Just try it and report back > > everything you find unusual. > > > > And this time we will not include CHANGES.md and RELEASENOTE.md > > in our source code, you can find it on the download site. For getting > these > > two files for old releases, please go to > > > >https://archive.apache.org/dist/hbase/ > > > > To learn more about Apache hbase, please see > > > >http://hbase.apache.org/ > > > > Thanks, > > Your HBase Release Manager > > > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
[VOTE] First release candidate for HBase 2.4.9 (RC0) is available
Please vote on this Apache HBase release candidate, hbase-2.4.9RC0 The VOTE will remain open for at least 72 hours. [ ] +1 Release this package as Apache HBase 2.4.9 [ ] -1 Do not release this package because ... The tag to be voted on is 2.4.9RC0: https://github.com/apache/hbase/tree/2.4.9RC0 This tag currently points to git reference c49f7f63fc. The release files, including signatures, digests, as well as CHANGES.md and RELEASENOTES.md included in this RC can be found at: https://dist.apache.org/repos/dist/dev/hbase/2.4.9RC0/ The API compatibility report can be found at: https://dist.apache.org/repos/dist/dev/hbase/2.4.9RC0/api_compare_2.4.8_to_2.4.9RC0.html There are no reported compatibility issues. There are known flaky unit tests, see HBASE-26254. Maven artifacts are available in a staging repository at: https://repository.apache.org/content/repositories/orgapachehbase-1477/ Artifacts were signed with the 0xD5365CCD key which can be found in: https://dist.apache.org/repos/dist/release/hbase/KEYS To learn more about Apache HBase, please see http://hbase.apache.org/ Thanks, Your HBase Release Manager
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
+1 (binding) * Xsums/sigs good * Can build from source * Log4j 2.15 is included (more on this in the below) * log4j2.formatMsgNoLookups=true is set (multiple times per process, but properly set) * hbase-config.sh issue is fixed over rc1 Best as I've been able to keep up, it seems like we should already upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings that 2.16 may have issues still. It's my opinion that the changes we have here in rc2 are a massive improvement over before. I think this is fine; I just wanted to acknowledge that we may still need to update again real soon. Thanks for your release manager work, Duo! On 12/14/21 9:06 AM, Duo Zhang wrote: Please vote on this Apache hbase release candidate, hbase-3.0.0-alpha-2RC1 The VOTE will remain open for at least 72 hours. [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 [ ] -1 Do not release this package because ... The tag to be voted on is 3.0.0-alpha-2RC1: https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1 This tag currently points to git reference a3ff8e4c812eefab6ad32af45ca449a1395a6510 The release files, including signatures, digests, as well as CHANGES.md and RELEASENOTES.md included in this RC can be found at: https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/ Maven artifacts are available in a staging repository at: https://repository.apache.org/content/repositories/orgapachehbase-1473/ Artifacts were signed with the 9AD2AE49 key which can be found in: https://downloads.apache.org/hbase/KEYS 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line. HBase 3.0.0 includes the following big feature/changes: Synchronous Replication OpenTelemetry Tracing Distributed MOB Compaction Backup and Restore Move RSGroup balancer to core Reimplement sync client on async client CPEPs on shaded proto Move the logging framework from log4j to log4j2 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2 CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade to 3.0.0-alpha-2 ASAP. Notice that this is not a production ready release. It is used to let our users try and test the new major release, to get feedback before the final GA release is out. So please do NOT use it in production. Just try it and report back everything you find unusual. And this time we will not include CHANGES.md and RELEASENOTE.md in our source code, you can find it on the download site. For getting these two files for old releases, please go to https://archive.apache.org/dist/hbase/ To learn more about Apache hbase, please see http://hbase.apache.org/ Thanks, Your HBase Release Manager
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
+1 * Signature: ok * Checksum : ok * Rat check (1.8.0_301): ok - mvn clean apache-rat:check * Built from source (1.8.0_301): ok - mvn clean install -DskipTests * Unit tests pass (1.8.0_301): ok - mvn package -P runSmallTests -Dsurefire.rerunFailingTestsCount=3 * Nightly build results look good On Tue, Dec 14, 2021 at 7:46 PM Duo Zhang wrote: > Please vote on this Apache hbase release candidate, > hbase-3.0.0-alpha-2RC1 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 3.0.0-alpha-2RC1: > > https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1 > > This tag currently points to git reference > > a3ff8e4c812eefab6ad32af45ca449a1395a6510 > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1473/ > > Artifacts were signed with the 9AD2AE49 key which can be found in: > > https://downloads.apache.org/hbase/KEYS > > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line. > HBase 3.0.0 includes the following big feature/changes: > Synchronous Replication > OpenTelemetry Tracing > Distributed MOB Compaction > Backup and Restore > Move RSGroup balancer to core > Reimplement sync client on async client > CPEPs on shaded proto > Move the logging framework from log4j to log4j2 > > 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2 > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade > to 3.0.0-alpha-2 ASAP. > > Notice that this is not a production ready release. It is used to let our > users try and test the new major release, to get feedback before the final > GA release is out. > So please do NOT use it in production. Just try it and report back > everything you find unusual. > > And this time we will not include CHANGES.md and RELEASENOTE.md > in our source code, you can find it on the download site. For getting these > two files for old releases, please go to > > https://archive.apache.org/dist/hbase/ > > To learn more about Apache hbase, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager >
[jira] [Resolved] (HBASE-26580) The message of StoreTooBusy is confused
[
https://issues.apache.org/jira/browse/HBASE-26580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-26580.
---
Fix Version/s: 2.5.0
3.0.0-alpha-3
2.4.10
Hadoop Flags: Reviewed
Resolution: Fixed
Pushed to branch-2.4+.
Thanks [~zhengzhuobinzzb]!
> The message of StoreTooBusy is confused
> ---
>
> Key: HBASE-26580
> URL: https://issues.apache.org/jira/browse/HBASE-26580
> Project: HBase
> Issue Type: Task
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Trivial
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
>
> When check Store limit. We both check parallelPutToStoreThreadLimit and
> parallelPreparePutToStoreThreadLimit.
> {code:java}
> if (store.getCurrentParallelPutCount() > this.parallelPutToStoreThreadLimit
> || preparePutCount > this.parallelPreparePutToStoreThreadLimit) {
> tooBusyStore = (tooBusyStore == null ?
> store.getColumnFamilyName() :
> tooBusyStore + "," + store.getColumnFamilyName());
> } {code}
> But we only print Above parallelPutToStoreThreadLimit only.
>
>
> {code:java}
> if (tooBusyStore != null) {
> String msg =
> "StoreTooBusy," + this.region.getRegionInfo().getRegionNameAsString() +
> ":" + tooBusyStore
> + " Above parallelPutToStoreThreadLimit(" +
> this.parallelPutToStoreThreadLimit + ")";
> if (LOG.isTraceEnabled()) {
> LOG.trace(msg);
> }
> throw new RegionTooBusyException(msg);
> }{code}
> It confused me a lot time ..
> Just add message of parallelPreparePutToStoreThreadLimit
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26604) Replace new allocation with ThreadLocal in CellBlockBuilder to reduce GC
Yutong Xiao created HBASE-26604:
---
Summary: Replace new allocation with ThreadLocal in
CellBlockBuilder to reduce GC
Key: HBASE-26604
URL: https://issues.apache.org/jira/browse/HBASE-26604
Project: HBase
Issue Type: Task
Reporter: Yutong Xiao
Assignee: Yutong Xiao
In CellBlockBuilder decompress method, we currently allocate a new
ByteBufferOutputStream object each invoke.
{code:java}
try {
// TODO: This is ugly. The buffer will be resized on us if we guess wrong.
//TODO: reuse buffers.
bbos = new ByteBufferOutputStream(osInitialSize);
IOUtils.copy(cis, bbos);
bbos.close();
return bbos.getByteBuffer();
} finally {
CodecPool.returnDecompressor(poolDecompressor);
}
{code}
We can use a ThreadLocal variable to reuse the buffer in each Thread. As:
{code:java}
try {
// TODO: This is ugly. The buffer will be resized on us if we guess wrong.
if (this.decompressBuff.get() == null) {
this.decompressBuff.set(new ByteBufferOutputStream(osInitialSize));
}
ByteBufferOutputStream localBbos = this.decompressBuff.get();
localBbos.clear();
IOUtils.copy(cis, localBbos);
return localBbos.getByteBuffer();
} finally {
CodecPool.returnDecompressor(poolDecompressor);
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
+1 (binding) Checked sigs and sums: Matched Rat check: Passed Built from src: Succeeded Run UTs: Passed CHANGES and RELEASENOTES: Missed two issues, for generating these two files and change version in pom. This is not an actual code problem so I always do not want to sink an RC due to the problems on these two files. In the future hbase-operator-tools should consider removing these two files from git too. Log4j2: Checked the lib directory, the log4j2 version is 2.16.0, which is good. Run it against a hbase cluster: Use it with the cluster deployed by me for testing HBASE-26233, the default command failed because of can not recognize the 'hdfs' protocol. I needed to manually prepend 'INTERNAL_CLASSPATH=true' when executing the command, like this INTERNAL_CLASSPATH=true ./bin/hbase hbck -j /home/sa/disk1/hbase-operator-tools-1.2.0/hbase-hbck2/hbase-hbck2-1.2.0.jar extraRegionsInMeta default:IntegrationTestRegionReplicaReplication But this is not a problem of hbase-operator-tools, it is because we do not shade hadoop-hdfs in our shaded jar. And it is strange that why hbck uses hbase-shaded-client instead of hbase-shaded-mapreduce. The stacktrace 17:04:41.024 [main] ERROR org.apache.hbase.HBCK2 - Error on checking extra regions: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs" at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hbase.HBCKFsUtils.getRootDir(HBCKFsUtils.java:106) ~[hbase-hbck2-1.2.0.jar:1.2.0] at org.apache.hbase.FsRegionsMetaRecoverer.(FsRegionsMetaRecoverer.java:66) ~[hbase-hbck2-1.2.0.jar:1.2.0] at org.apache.hbase.HBCK2.extraRegionsInMeta(HBCK2.java:268) [hbase-hbck2-1.2.0.jar:1.2.0] at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:1020) [hbase-hbck2-1.2.0.jar:1.2.0] at org.apache.hbase.HBCK2.run(HBCK2.java:830) [hbase-hbck2-1.2.0.jar:1.2.0] at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2] at org.apache.hbase.HBCK2.main(HBCK2.java:1145) [hbase-hbck2-1.2.0.jar:1.2.0 张铎(Duo Zhang) 于2021年12月18日周六 16:36写道： > The readme of the project says we have two tools, one is HBCK2, the other > is TableReporter, but in the binary I can only see HBCK2... > > But anyway, checking the previous release, we did not include > TableReporter either, so not this release's fault... > > Josh Elser 于2021年12月17日周五 23:03写道： > >> +1 (binding) >> >> Thanks for putting this together, Guangxu! >> >> * xsums/sigs are great >> * RAT check passes on src release >> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8) >> * Can build from source >> * CHANGES and RELEASENOTES look fine at a glance >> * Public key published in KEYS >> * Verified log4j2.16 is in the binary release (both as a jar and shaded >> inside hbase-hbck) >> >> - Josh >> >> On 12/14/21 10:32 PM, Guangxu Cheng wrote: >> > Please vote on this Apache hbase operator tools release candidate, >> > hbase-operator-tools-1.2.0RC0 >> > >> > The VOTE will remain open for at least 72 hours. >> > >> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0 >> > [ ] -1 Do not release this package because ... >> > >> > The tag to be voted on is 1.2.0RC0: >> > >> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 >> > >> > This tag currently points to git reference >> > >> >76d68624cebb66ec0e509b0a4c0d96445a601685 >> > >> > The release files, including signatures, digests, as well as CHANGES.md >> > and RELEASENOTES.md included in this RC can be found at: >> > >> > >> > >> https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/ >> > >> > Maven artifacts are available in a staging repository at: >> > >> > >> https://repository.apache.org/content/repositories/orgapachehbase-1474/ >> > Artifacts were signed with the 5EF3A66D57EC647A key which can be found >> in: >> > >> >https://dist.apache.org/repos/dist/release/hbase/KEYS >> > >> > hbase-operator-tools 1.2.0 contains a critical security fix for >> addressing >> > the log4j2 >> > CVE-2021-44228. All users who use hbase-operator-tools should
Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
The readme of the project says we have two tools, one is HBCK2, the other is TableReporter, but in the binary I can only see HBCK2... But anyway, checking the previous release, we did not include TableReporter either, so not this release's fault... Josh Elser 于2021年12月17日周五 23:03写道： > +1 (binding) > > Thanks for putting this together, Guangxu! > > * xsums/sigs are great > * RAT check passes on src release > * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8) > * Can build from source > * CHANGES and RELEASENOTES look fine at a glance > * Public key published in KEYS > * Verified log4j2.16 is in the binary release (both as a jar and shaded > inside hbase-hbck) > > - Josh > > On 12/14/21 10:32 PM, Guangxu Cheng wrote: > > Please vote on this Apache hbase operator tools release candidate, > > hbase-operator-tools-1.2.0RC0 > > > > The VOTE will remain open for at least 72 hours. > > > > [ ] +1 Release this package as Apache hbase operator tools 1.2.0 > > [ ] -1 Do not release this package because ... > > > > The tag to be voted on is 1.2.0RC0: > > > >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 > > > > This tag currently points to git reference > > > >76d68624cebb66ec0e509b0a4c0d96445a601685 > > > > The release files, including signatures, digests, as well as CHANGES.md > > and RELEASENOTES.md included in this RC can be found at: > > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/ > > > > Maven artifacts are available in a staging repository at: > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1474/ > > Artifacts were signed with the 5EF3A66D57EC647A key which can be found > in: > > > >https://dist.apache.org/repos/dist/release/hbase/KEYS > > > > hbase-operator-tools 1.2.0 contains a critical security fix for > addressing > > the log4j2 > > CVE-2021-44228. All users who use hbase-operator-tools should upgrade > > to hbase-operator-tools 1.2.0 ASAP. > > > > To learn more about Apache hbase operator tools, please see > > > >http://hbase.apache.org/ > > > > Thanks, > > Your HBase Release Manager > > > > -- > > Best Regards, > > Guangxu > > >
[VOTE] Merge feature branch HBASE-26233 back to master
There is a previous thread[1] to mention that the main development work on HBASE-26233[2] is done. And now the testing work has been done too, so I send this official vote to merge for HBASE-26233 back into master. For those who are not familiar with HBASE-26233, it is for re-implementing the region replication framework, to decouple with the general replication framework. You could see the design doc[3] for more details. I've also opened a PR against master[4], PTAL if you have interest. You can see the final test result on HBASE-26487[5]. In conclusion, there is no regression on performance when running PE. Using the new tool introduced in HBASE-26540[6], the new implementation can perform much better than the old implementation. When inserting 1M rows, the max lag on secondary replica is 335ms, and the 99.9% lag is 3 only 3ms, while the lag of the old implementation highly depends on the replication.source.sleepforretries config but you should not set it to a very small value as it will waste a lot of resources. On IntegrationTestRegionReplicaReplication, both implementations can not pass but the problems are not related to the changing part, so should not be a blocker. Will file follow on issues to address these problems. The vote will open for at least 72 hours. Please vote: +1: Merge the changes from HBASE-26233 to master -1: Do not merge these changes because ... Thanks. 1. https://lists.apache.org/thread/t8w053kpy9pj1vg1x3c0kjjh4ongrxzy 2. https://issues.apache.org/jira/browse/HBASE-26233 3. https://docs.google.com/document/d/1WPSBwRqIPgTrPRM2wtzAe-3XUvA9S7auWy3Fy8xuZLc/edit 4. https://github.com/apache/hbase/pull/3891 5. https://issues.apache.org/jira/browse/HBASE-26487 6. https://issues.apache.org/jira/browse/HBASE-26540
