Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-18 Thread Tak Lon (Stephen) Wu
+1 on the log4j2 2.17.0

-Stephen

On Sat, Dec 18, 2021 at 10:14 PM 张铎(Duo Zhang) 
wrote:

> Let's also update log4j2 to 2.17.0 for hbase-oeprator-tools?
>
> Thanks.
>
> 张铎(Duo Zhang)  于2021年12月18日周六 17:07写道:
>
> > +1 (binding)
> >
> > Checked sigs and sums: Matched
> > Rat check: Passed
> > Built from src: Succeeded
> > Run UTs: Passed
> > CHANGES and RELEASENOTES: Missed two issues, for generating these two
> > files and change version in pom. This is not an actual code problem so I
> > always do not want to sink an RC due to the problems on these two files.
> In
> > the future hbase-operator-tools should consider removing these two files
> > from git too.
> > Log4j2: Checked the lib directory, the log4j2 version is 2.16.0, which is
> > good.
> > Run it against a hbase cluster: Use it with the cluster deployed by me
> for
> > testing HBASE-26233, the default command failed because of can
> > not recognize the 'hdfs' protocol. I needed to manually prepend
> > 'INTERNAL_CLASSPATH=true' when executing the command, like this
> >   INTERNAL_CLASSPATH=true ./bin/hbase hbck -j
> >
> /home/sa/disk1/hbase-operator-tools-1.2.0/hbase-hbck2/hbase-hbck2-1.2.0.jar
> > extraRegionsInMeta default:IntegrationTestRegionReplicaReplication
> >   But this is not a problem of hbase-operator-tools, it is because we do
> > not shade hadoop-hdfs in our shaded jar. And it is strange that why hbck
> > uses hbase-shaded-client instead of hbase-shaded-mapreduce.
> >   The stacktrace
> > 17:04:41.024 [main] ERROR org.apache.hbase.HBCK2 - Error on checking
> extra
> > regions:
> > org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for
> > scheme "hdfs"
> > at
> > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> > ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hbase.HBCKFsUtils.getRootDir(HBCKFsUtils.java:106)
> > ~[hbase-hbck2-1.2.0.jar:1.2.0]
> > at
> >
> org.apache.hbase.FsRegionsMetaRecoverer.(FsRegionsMetaRecoverer.java:66)
> > ~[hbase-hbck2-1.2.0.jar:1.2.0]
> > at org.apache.hbase.HBCK2.extraRegionsInMeta(HBCK2.java:268)
> > [hbase-hbck2-1.2.0.jar:1.2.0]
> > at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:1020)
> > [hbase-hbck2-1.2.0.jar:1.2.0]
> > at org.apache.hbase.HBCK2.run(HBCK2.java:830)
> [hbase-hbck2-1.2.0.jar:1.2.0]
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> > [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> > [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> > at org.apache.hbase.HBCK2.main(HBCK2.java:1145)
> > [hbase-hbck2-1.2.0.jar:1.2.0
> >
> > 张铎(Duo Zhang)  于2021年12月18日周六 16:36写道:
> >
> >> The readme of the project says we have two tools, one is HBCK2, the
> other
> >> is TableReporter, but in the binary I can only see HBCK2...
> >>
> >> But anyway, checking the previous release, we did not include
> >> TableReporter either, so not this release's fault...
> >>
> >> Josh Elser  于2021年12月17日周五 23:03写道:
> >>
> >>> +1 (binding)
> >>>
> >>> Thanks for putting this together, Guangxu!
> >>>
> >>> * xsums/sigs are great
> >>> * RAT check passes on src release
> >>> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8)
> >>> * Can build from source
> >>> * CHANGES and RELEASENOTES look fine at a glance
> >>> * Public key published in KEYS
> >>> * Verified log4j2.16 is in the binary release (both as a jar and shaded
> >>> inside hbase-hbck)
> >>>
> >>> - Josh
> >>>
> >>> On 12/14/21 10:32 PM, Guangxu Cheng wrote:
> >>> > Please vote on this Apache hbase operator tools release candidate,
> >>> > hbase-operator-tools-1.2.0RC0
> >>> >
> >>> > The VOTE will remain open for at least 72 hours.
> >>> >
> >>> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0
> >>> > [ ] -1 Do not release this package because ...
> >>> >
> >>> > The tag to be voted on is 1.2.0RC0:
> >>> >
> >>> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0
> >>> >
> >>> > This tag currently points to git reference
> >>> >
> >>> >76d68624cebb66ec0e509b0a4c0d96445a601685
> >>> >
> >>> > The release files, including signatures, digests, as well as
> CHANGES.md
> >>> > and RELEASENOTES.md included 

Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Duo Zhang
https://github.com/apache/hbase/pull/3965

Andrew Purtell  于2021年12月19日周日 13:51写道:

> Sure, we are on the same page about this RC.
>
> > On Dec 18, 2021, at 9:46 PM, 张铎  wrote:
> >
> > I think we are on the same page that we should upgrade to the newest
> log4j2
> > version since the final release has not been published yet.
> >
> > But on log4j1, in our community we have discussed this before when there
> is
> > a CVE for it. You can view this page
> >
> > https://logging.apache.org/log4j/1.2/
> >
> > And even for the recent CVE, log4j1 is also affected, as listed on the
> page
> > you provided.
> >
> > Log4j 1.x mitigation
> >
> > Log4j 1.x does not have Lookups so the risk is lower. Applications using
> > Log4j 1.x are only vulnerable to this attack when they use JNDI in their
> > configuration. A separate CVE (CVE-2021-4104) has been filed for this
> > vulnerability. To mitigate: Audit your logging configuration to ensure it
> > has no JMSAppender configured. Log4j 1.x configurations without
> JMSAppender
> > are not impacted by this vulnerability.
> >
> > It is as you said in the first paragraph, log4j1 has a special CVE for
> it,
> > and it will never be fixed. We need to say that ‘yes it is affected but
> > only if you bla bla’, not good for end users right?
> >
> > So I still stand my point that, stay on log4j1 is not a good choice, it
> is
> > not because we have already done the work, it is our duty to keep our
> users
> > safe from security problem.
> >
> > And on the Hadoop part, it is also me that trying to upgrade to log4j2.
> But
> > as you know Hadoop is actually constructed by several projects, it is
> > really not easy to do this work, like what we have done in HBase.
> >
> > Anyway, let me prepare a new RC.
> >
> > Thanks.
> > Andrew Purtell 于2021年12月19日 周日09:12写道:
> >
> >> As to your first point, I think it is a simple consideration: A user’s
> >> security department or compliance regulator will ask: “Does this version
> >> include log4j with a known CVE?” Why would we provide a release where
> they
> >> have to answer “yes” when we can provide them a release where they can
> >> answer “no”? Based on todays knowledge. (And yes I am aware that a user
> can
> >> manually upgrade the jar versions in place after unpacking the tarballs.
> >> Nonetheless.)
> >>
> >> I disagree that there was a real need to upgrade log4j because 1.x was
> EOL
> >> but I won’t argue that staying with old dependencies is automatically
> good.
> >> It’s done, anyway. The main point I would like to make here is should a
> >> good alternative emerge from this mess I am going to look at replacing
> >> log4j 2 with it. Or, if log4j decides to accept the inevitable and
> remove
> >> the dangerous substitution/rewrite feature then that would be fine too.
> >>
>  On Dec 18, 2021, at 4:42 PM, 张铎  wrote:
> >>>
> >>> After 2.15.0, all the problems require you manually put some special
> >>> markers in the pattern layout in your configuration file, so it is
> >> already
> >>> less hurt, we do not have something like %m{lookup} in the pattern
> layout
> >>> by default.
> >>>
> >>> Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to
> the
> >>> newest version.
> >>>
> >>> But stay on log4j1 should not be considered as a solution. Log4j1 is
> >>> already dead long ago and it has several CVEs where no one wants to fix
> >>> them, and our statement was just ‘do not use the feature’. That’s why
> we
> >>> want to migrate to log4j2. Every project may have CVEs, so I think
> >> whether
> >>> there are still enough people who are still maintaining the project is
> >> the
> >>> most important thing. Log4j2 is already the most active logging
> >> framework,
> >>> another option is logback, but there were no releases for nearly 4
> years
> >>> before 2021…
> >>>
> >>> Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
> >>>
> >>> Andrew Purtell 于2021年12月19日 周日05:25写道:
> >>>
>  Apologies, I managed to hit the send button before finishing. My veto
> >> can
>  be cured by upgrading Log4J to ** 2.17.0 ** . See
>  https://logging.apache.org/log4j/2.x/security.html.
> 
> > On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell 
> > wrote:
> >
> > -1 (binding)
> >
> > The Log4J issues are not fixed by 2.15.
> >
> > I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
>  know
> > they have plans to upgrade. It does not seem advisable to use Log4J 2
> >> at
> > all actually. Another option that does not include such a dangerous
> > reference/rewrite mechanism might be preferable.
> >
> >> On Sat, Dec 18, 2021 at 12:02 PM Josh Elser 
> >> wrote:
> >
> >> +1 (binding)
> >>
> >> * Xsums/sigs good
> >> * Can build from source
> >> * Log4j 2.15 is included (more on this in the below)
> >> * log4j2.formatMsgNoLookups=true is set (multiple times per process,
> >> but
> >> properly set)
> >> * 

[jira] [Created] (HBASE-26607) Put up 3.0.0-alpha-2RC2

2021-12-18 Thread Duo Zhang (Jira)
Duo Zhang created HBASE-26607:
-

 Summary: Put up 3.0.0-alpha-2RC2
 Key: HBASE-26607
 URL: https://issues.apache.org/jira/browse/HBASE-26607
 Project: HBase
  Issue Type: Sub-task
Reporter: Duo Zhang






--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Resolved] (HBASE-26569) Put up 3.0.0-alpha-2RC1

2021-12-18 Thread Duo Zhang (Jira)


 [ 
https://issues.apache.org/jira/browse/HBASE-26569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26569.
---
Resolution: Fixed

Still need to upgrade log4j2.

> Put up 3.0.0-alpha-2RC1
> ---
>
> Key: HBASE-26569
> URL: https://issues.apache.org/jira/browse/HBASE-26569
> Project: HBase
>  Issue Type: Sub-task
>  Components: community
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-18 Thread Duo Zhang
Let's also update log4j2 to 2.17.0 for hbase-oeprator-tools?

Thanks.

张铎(Duo Zhang)  于2021年12月18日周六 17:07写道:

> +1 (binding)
>
> Checked sigs and sums: Matched
> Rat check: Passed
> Built from src: Succeeded
> Run UTs: Passed
> CHANGES and RELEASENOTES: Missed two issues, for generating these two
> files and change version in pom. This is not an actual code problem so I
> always do not want to sink an RC due to the problems on these two files. In
> the future hbase-operator-tools should consider removing these two files
> from git too.
> Log4j2: Checked the lib directory, the log4j2 version is 2.16.0, which is
> good.
> Run it against a hbase cluster: Use it with the cluster deployed by me for
> testing HBASE-26233, the default command failed because of can
> not recognize the 'hdfs' protocol. I needed to manually prepend
> 'INTERNAL_CLASSPATH=true' when executing the command, like this
>   INTERNAL_CLASSPATH=true ./bin/hbase hbck -j
> /home/sa/disk1/hbase-operator-tools-1.2.0/hbase-hbck2/hbase-hbck2-1.2.0.jar
> extraRegionsInMeta default:IntegrationTestRegionReplicaReplication
>   But this is not a problem of hbase-operator-tools, it is because we do
> not shade hadoop-hdfs in our shaded jar. And it is strange that why hbck
> uses hbase-shaded-client instead of hbase-shaded-mapreduce.
>   The stacktrace
> 17:04:41.024 [main] ERROR org.apache.hbase.HBCK2 - Error on checking extra
> regions:
> org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for
> scheme "hdfs"
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> ~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hbase.HBCKFsUtils.getRootDir(HBCKFsUtils.java:106)
> ~[hbase-hbck2-1.2.0.jar:1.2.0]
> at
> org.apache.hbase.FsRegionsMetaRecoverer.(FsRegionsMetaRecoverer.java:66)
> ~[hbase-hbck2-1.2.0.jar:1.2.0]
> at org.apache.hbase.HBCK2.extraRegionsInMeta(HBCK2.java:268)
> [hbase-hbck2-1.2.0.jar:1.2.0]
> at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:1020)
> [hbase-hbck2-1.2.0.jar:1.2.0]
> at org.apache.hbase.HBCK2.run(HBCK2.java:830) [hbase-hbck2-1.2.0.jar:1.2.0]
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> [hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
> at org.apache.hbase.HBCK2.main(HBCK2.java:1145)
> [hbase-hbck2-1.2.0.jar:1.2.0
>
> 张铎(Duo Zhang)  于2021年12月18日周六 16:36写道:
>
>> The readme of the project says we have two tools, one is HBCK2, the other
>> is TableReporter, but in the binary I can only see HBCK2...
>>
>> But anyway, checking the previous release, we did not include
>> TableReporter either, so not this release's fault...
>>
>> Josh Elser  于2021年12月17日周五 23:03写道:
>>
>>> +1 (binding)
>>>
>>> Thanks for putting this together, Guangxu!
>>>
>>> * xsums/sigs are great
>>> * RAT check passes on src release
>>> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8)
>>> * Can build from source
>>> * CHANGES and RELEASENOTES look fine at a glance
>>> * Public key published in KEYS
>>> * Verified log4j2.16 is in the binary release (both as a jar and shaded
>>> inside hbase-hbck)
>>>
>>> - Josh
>>>
>>> On 12/14/21 10:32 PM, Guangxu Cheng wrote:
>>> > Please vote on this Apache hbase operator tools release candidate,
>>> > hbase-operator-tools-1.2.0RC0
>>> >
>>> > The VOTE will remain open for at least 72 hours.
>>> >
>>> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > The tag to be voted on is 1.2.0RC0:
>>> >
>>> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0
>>> >
>>> > This tag currently points to git reference
>>> >
>>> >76d68624cebb66ec0e509b0a4c0d96445a601685
>>> >
>>> > The release files, including signatures, digests, as well as CHANGES.md
>>> > and RELEASENOTES.md included in this RC can be found at:
>>> >
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/
>>> >
>>> > Maven artifacts are available in a staging repository at:
>>> >
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehbase-1474/
>>> > Artifacts were signed with the 

[jira] [Created] (HBASE-26606) Upgrade log4j2 to 2.17.0

2021-12-18 Thread Duo Zhang (Jira)
Duo Zhang created HBASE-26606:
-

 Summary: Upgrade log4j2 to 2.17.0
 Key: HBASE-26606
 URL: https://issues.apache.org/jira/browse/HBASE-26606
 Project: HBase
  Issue Type: Task
  Components: logging, security
Reporter: Duo Zhang
Assignee: Duo Zhang
 Fix For: 3.0.0-alpha-2






--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Andrew Purtell
Sure, we are on the same page about this RC. 

> On Dec 18, 2021, at 9:46 PM, 张铎  wrote:
> 
> I think we are on the same page that we should upgrade to the newest log4j2
> version since the final release has not been published yet.
> 
> But on log4j1, in our community we have discussed this before when there is
> a CVE for it. You can view this page
> 
> https://logging.apache.org/log4j/1.2/
> 
> And even for the recent CVE, log4j1 is also affected, as listed on the page
> you provided.
> 
> Log4j 1.x mitigation
> 
> Log4j 1.x does not have Lookups so the risk is lower. Applications using
> Log4j 1.x are only vulnerable to this attack when they use JNDI in their
> configuration. A separate CVE (CVE-2021-4104) has been filed for this
> vulnerability. To mitigate: Audit your logging configuration to ensure it
> has no JMSAppender configured. Log4j 1.x configurations without JMSAppender
> are not impacted by this vulnerability.
> 
> It is as you said in the first paragraph, log4j1 has a special CVE for it,
> and it will never be fixed. We need to say that ‘yes it is affected but
> only if you bla bla’, not good for end users right?
> 
> So I still stand my point that, stay on log4j1 is not a good choice, it is
> not because we have already done the work, it is our duty to keep our users
> safe from security problem.
> 
> And on the Hadoop part, it is also me that trying to upgrade to log4j2. But
> as you know Hadoop is actually constructed by several projects, it is
> really not easy to do this work, like what we have done in HBase.
> 
> Anyway, let me prepare a new RC.
> 
> Thanks.
> Andrew Purtell 于2021年12月19日 周日09:12写道:
> 
>> As to your first point, I think it is a simple consideration: A user’s
>> security department or compliance regulator will ask: “Does this version
>> include log4j with a known CVE?” Why would we provide a release where they
>> have to answer “yes” when we can provide them a release where they can
>> answer “no”? Based on todays knowledge. (And yes I am aware that a user can
>> manually upgrade the jar versions in place after unpacking the tarballs.
>> Nonetheless.)
>> 
>> I disagree that there was a real need to upgrade log4j because 1.x was EOL
>> but I won’t argue that staying with old dependencies is automatically good.
>> It’s done, anyway. The main point I would like to make here is should a
>> good alternative emerge from this mess I am going to look at replacing
>> log4j 2 with it. Or, if log4j decides to accept the inevitable and remove
>> the dangerous substitution/rewrite feature then that would be fine too.
>> 
 On Dec 18, 2021, at 4:42 PM, 张铎  wrote:
>>> 
>>> After 2.15.0, all the problems require you manually put some special
>>> markers in the pattern layout in your configuration file, so it is
>> already
>>> less hurt, we do not have something like %m{lookup} in the pattern layout
>>> by default.
>>> 
>>> Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
>>> newest version.
>>> 
>>> But stay on log4j1 should not be considered as a solution. Log4j1 is
>>> already dead long ago and it has several CVEs where no one wants to fix
>>> them, and our statement was just ‘do not use the feature’. That’s why we
>>> want to migrate to log4j2. Every project may have CVEs, so I think
>> whether
>>> there are still enough people who are still maintaining the project is
>> the
>>> most important thing. Log4j2 is already the most active logging
>> framework,
>>> another option is logback, but there were no releases for nearly 4 years
>>> before 2021…
>>> 
>>> Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
>>> 
>>> Andrew Purtell 于2021年12月19日 周日05:25写道:
>>> 
 Apologies, I managed to hit the send button before finishing. My veto
>> can
 be cured by upgrading Log4J to ** 2.17.0 ** . See
 https://logging.apache.org/log4j/2.x/security.html.
 
> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell 
> wrote:
> 
> -1 (binding)
> 
> The Log4J issues are not fixed by 2.15.
> 
> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
 know
> they have plans to upgrade. It does not seem advisable to use Log4J 2
>> at
> all actually. Another option that does not include such a dangerous
> reference/rewrite mechanism might be preferable.
> 
>> On Sat, Dec 18, 2021 at 12:02 PM Josh Elser 
>> wrote:
> 
>> +1 (binding)
>> 
>> * Xsums/sigs good
>> * Can build from source
>> * Log4j 2.15 is included (more on this in the below)
>> * log4j2.formatMsgNoLookups=true is set (multiple times per process,
>> but
>> properly set)
>> * hbase-config.sh issue is fixed over rc1
>> 
>> Best as I've been able to keep up, it seems like we should already
>> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
>> that 2.16 may have issues still. It's my opinion that the changes we
>> have here in rc2 are a massive improvement 

Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Duo Zhang
I think we are on the same page that we should upgrade to the newest log4j2
version since the final release has not been published yet.

But on log4j1, in our community we have discussed this before when there is
a CVE for it. You can view this page

https://logging.apache.org/log4j/1.2/

And even for the recent CVE, log4j1 is also affected, as listed on the page
you provided.

Log4j 1.x mitigation

Log4j 1.x does not have Lookups so the risk is lower. Applications using
Log4j 1.x are only vulnerable to this attack when they use JNDI in their
configuration. A separate CVE (CVE-2021-4104) has been filed for this
vulnerability. To mitigate: Audit your logging configuration to ensure it
has no JMSAppender configured. Log4j 1.x configurations without JMSAppender
are not impacted by this vulnerability.

It is as you said in the first paragraph, log4j1 has a special CVE for it,
and it will never be fixed. We need to say that ‘yes it is affected but
only if you bla bla’, not good for end users right?

So I still stand my point that, stay on log4j1 is not a good choice, it is
not because we have already done the work, it is our duty to keep our users
safe from security problem.

And on the Hadoop part, it is also me that trying to upgrade to log4j2. But
as you know Hadoop is actually constructed by several projects, it is
really not easy to do this work, like what we have done in HBase.

Anyway, let me prepare a new RC.

Thanks.
Andrew Purtell 于2021年12月19日 周日09:12写道:

> As to your first point, I think it is a simple consideration: A user’s
> security department or compliance regulator will ask: “Does this version
> include log4j with a known CVE?” Why would we provide a release where they
> have to answer “yes” when we can provide them a release where they can
> answer “no”? Based on todays knowledge. (And yes I am aware that a user can
> manually upgrade the jar versions in place after unpacking the tarballs.
> Nonetheless.)
>
> I disagree that there was a real need to upgrade log4j because 1.x was EOL
> but I won’t argue that staying with old dependencies is automatically good.
> It’s done, anyway. The main point I would like to make here is should a
> good alternative emerge from this mess I am going to look at replacing
> log4j 2 with it. Or, if log4j decides to accept the inevitable and remove
> the dangerous substitution/rewrite feature then that would be fine too.
>
> > On Dec 18, 2021, at 4:42 PM, 张铎  wrote:
> >
> > After 2.15.0, all the problems require you manually put some special
> > markers in the pattern layout in your configuration file, so it is
> already
> > less hurt, we do not have something like %m{lookup} in the pattern layout
> > by default.
> >
> > Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
> > newest version.
> >
> > But stay on log4j1 should not be considered as a solution. Log4j1 is
> > already dead long ago and it has several CVEs where no one wants to fix
> > them, and our statement was just ‘do not use the feature’. That’s why we
> > want to migrate to log4j2. Every project may have CVEs, so I think
> whether
> > there are still enough people who are still maintaining the project is
> the
> > most important thing. Log4j2 is already the most active logging
> framework,
> > another option is logback, but there were no releases for nearly 4 years
> > before 2021…
> >
> > Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
> >
> > Andrew Purtell 于2021年12月19日 周日05:25写道:
> >
> >> Apologies, I managed to hit the send button before finishing. My veto
> can
> >> be cured by upgrading Log4J to ** 2.17.0 ** . See
> >> https://logging.apache.org/log4j/2.x/security.html.
> >>
> >>> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell 
> >>> wrote:
> >>>
> >>> -1 (binding)
> >>>
> >>> The Log4J issues are not fixed by 2.15.
> >>>
> >>> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
> >> know
> >>> they have plans to upgrade. It does not seem advisable to use Log4J 2
> at
> >>> all actually. Another option that does not include such a dangerous
> >>> reference/rewrite mechanism might be preferable.
> >>>
>  On Sat, Dec 18, 2021 at 12:02 PM Josh Elser 
> wrote:
> >>>
>  +1 (binding)
> 
>  * Xsums/sigs good
>  * Can build from source
>  * Log4j 2.15 is included (more on this in the below)
>  * log4j2.formatMsgNoLookups=true is set (multiple times per process,
> but
>  properly set)
>  * hbase-config.sh issue is fixed over rc1
> 
>  Best as I've been able to keep up, it seems like we should already
>  upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
>  that 2.16 may have issues still. It's my opinion that the changes we
>  have here in rc2 are a massive improvement over before. I think this
> is
>  fine; I just wanted to acknowledge that we may still need to update
>  again real soon.
> 
>  Thanks for your release manager work, Duo!
> 
>  On 

Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Andrew Purtell
As to your first point, I think it is a simple consideration: A user’s security 
department or compliance regulator will ask: “Does this version include log4j 
with a known CVE?” Why would we provide a release where they have to answer 
“yes” when we can provide them a release where they can answer “no”? Based on 
todays knowledge. (And yes I am aware that a user can manually upgrade the jar 
versions in place after unpacking the tarballs. Nonetheless.)

I disagree that there was a real need to upgrade log4j because 1.x was EOL but 
I won’t argue that staying with old dependencies is automatically good. It’s 
done, anyway. The main point I would like to make here is should a good 
alternative emerge from this mess I am going to look at replacing log4j 2 with 
it. Or, if log4j decides to accept the inevitable and remove the dangerous 
substitution/rewrite feature then that would be fine too. 

> On Dec 18, 2021, at 4:42 PM, 张铎  wrote:
> 
> After 2.15.0, all the problems require you manually put some special
> markers in the pattern layout in your configuration file, so it is already
> less hurt, we do not have something like %m{lookup} in the pattern layout
> by default.
> 
> Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
> newest version.
> 
> But stay on log4j1 should not be considered as a solution. Log4j1 is
> already dead long ago and it has several CVEs where no one wants to fix
> them, and our statement was just ‘do not use the feature’. That’s why we
> want to migrate to log4j2. Every project may have CVEs, so I think whether
> there are still enough people who are still maintaining the project is the
> most important thing. Log4j2 is already the most active logging framework,
> another option is logback, but there were no releases for nearly 4 years
> before 2021…
> 
> Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.
> 
> Andrew Purtell 于2021年12月19日 周日05:25写道:
> 
>> Apologies, I managed to hit the send button before finishing. My veto can
>> be cured by upgrading Log4J to ** 2.17.0 ** . See
>> https://logging.apache.org/log4j/2.x/security.html.
>> 
>>> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell 
>>> wrote:
>>> 
>>> -1 (binding)
>>> 
>>> The Log4J issues are not fixed by 2.15.
>>> 
>>> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
>> know
>>> they have plans to upgrade. It does not seem advisable to use Log4J 2 at
>>> all actually. Another option that does not include such a dangerous
>>> reference/rewrite mechanism might be preferable.
>>> 
 On Sat, Dec 18, 2021 at 12:02 PM Josh Elser  wrote:
>>> 
 +1 (binding)
 
 * Xsums/sigs good
 * Can build from source
 * Log4j 2.15 is included (more on this in the below)
 * log4j2.formatMsgNoLookups=true is set (multiple times per process, but
 properly set)
 * hbase-config.sh issue is fixed over rc1
 
 Best as I've been able to keep up, it seems like we should already
 upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
 that 2.16 may have issues still. It's my opinion that the changes we
 have here in rc2 are a massive improvement over before. I think this is
 fine; I just wanted to acknowledge that we may still need to update
 again real soon.
 
 Thanks for your release manager work, Duo!
 
 On 12/14/21 9:06 AM, Duo Zhang wrote:
> Please vote on this Apache hbase release candidate,
> hbase-3.0.0-alpha-2RC1
> 
> The VOTE will remain open for at least 72 hours.
> 
> [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> [ ] -1 Do not release this package because ...
> 
> The tag to be voted on is 3.0.0-alpha-2RC1:
> 
>   https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
> 
> This tag currently points to git reference
> 
>   a3ff8e4c812eefab6ad32af45ca449a1395a6510
> 
> The release files, including signatures, digests, as well as
>> CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
> 
>   https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
> 
> Maven artifacts are available in a staging repository at:
> 
> 
 https://repository.apache.org/content/repositories/orgapachehbase-1473/
> 
> Artifacts were signed with the 9AD2AE49 key which can be found in:
> 
>   https://downloads.apache.org/hbase/KEYS
> 
> 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
 line.
> HBase 3.0.0 includes the following big feature/changes:
>   Synchronous Replication
>   OpenTelemetry Tracing
>   Distributed MOB Compaction
>   Backup and Restore
>   Move RSGroup balancer to core
>   Reimplement sync client on async client
>   CPEPs on shaded proto
>   Move the logging framework from log4j to log4j2
> 
> 3.0.0-alpha-2 contains a critical security fix for addressing the
>> log4j2
> 

Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Duo Zhang
After 2.15.0, all the problems require you manually put some special
markers in the pattern layout in your configuration file, so it is already
less hurt, we do not have something like %m{lookup} in the pattern layout
by default.

Anyway, since we haven’t released 3.0.0-alpha-2 yet, let’s upgrade to the
newest version.

But stay on log4j1 should not be considered as a solution. Log4j1 is
already dead long ago and it has several CVEs where no one wants to fix
them, and our statement was just ‘do not use the feature’. That’s why we
want to migrate to log4j2. Every project may have CVEs, so I think whether
there are still enough people who are still maintaining the project is the
most important thing. Log4j2 is already the most active logging framework,
another option is logback, but there were no releases for nearly 4 years
before 2021…

Thanks. Let me upgrade the log4j2 to 2.17.0 and send out RC2.

Andrew Purtell 于2021年12月19日 周日05:25写道:

> Apologies, I managed to hit the send button before finishing. My veto can
> be cured by upgrading Log4J to ** 2.17.0 ** . See
> https://logging.apache.org/log4j/2.x/security.html.
>
> On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell 
> wrote:
>
> > -1 (binding)
> >
> > The Log4J issues are not fixed by 2.15.
> >
> > I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I
> know
> > they have plans to upgrade. It does not seem advisable to use Log4J 2 at
> > all actually. Another option that does not include such a dangerous
> > reference/rewrite mechanism might be preferable.
> >
> > On Sat, Dec 18, 2021 at 12:02 PM Josh Elser  wrote:
> >
> >> +1 (binding)
> >>
> >> * Xsums/sigs good
> >> * Can build from source
> >> * Log4j 2.15 is included (more on this in the below)
> >> * log4j2.formatMsgNoLookups=true is set (multiple times per process, but
> >> properly set)
> >> * hbase-config.sh issue is fixed over rc1
> >>
> >> Best as I've been able to keep up, it seems like we should already
> >> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
> >> that 2.16 may have issues still. It's my opinion that the changes we
> >> have here in rc2 are a massive improvement over before. I think this is
> >> fine; I just wanted to acknowledge that we may still need to update
> >> again real soon.
> >>
> >> Thanks for your release manager work, Duo!
> >>
> >> On 12/14/21 9:06 AM, Duo Zhang wrote:
> >> > Please vote on this Apache hbase release candidate,
> >> > hbase-3.0.0-alpha-2RC1
> >> >
> >> > The VOTE will remain open for at least 72 hours.
> >> >
> >> > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> >> > [ ] -1 Do not release this package because ...
> >> >
> >> > The tag to be voted on is 3.0.0-alpha-2RC1:
> >> >
> >> >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
> >> >
> >> > This tag currently points to git reference
> >> >
> >> >a3ff8e4c812eefab6ad32af45ca449a1395a6510
> >> >
> >> > The release files, including signatures, digests, as well as
> CHANGES.md
> >> > and RELEASENOTES.md included in this RC can be found at:
> >> >
> >> >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
> >> >
> >> > Maven artifacts are available in a staging repository at:
> >> >
> >> >
> >> https://repository.apache.org/content/repositories/orgapachehbase-1473/
> >> >
> >> > Artifacts were signed with the 9AD2AE49 key which can be found in:
> >> >
> >> >https://downloads.apache.org/hbase/KEYS
> >> >
> >> > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
> >> line.
> >> > HBase 3.0.0 includes the following big feature/changes:
> >> >Synchronous Replication
> >> >OpenTelemetry Tracing
> >> >Distributed MOB Compaction
> >> >Backup and Restore
> >> >Move RSGroup balancer to core
> >> >Reimplement sync client on async client
> >> >CPEPs on shaded proto
> >> >Move the logging framework from log4j to log4j2
> >> >
> >> > 3.0.0-alpha-2 contains a critical security fix for addressing the
> log4j2
> >> > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
> >> > to 3.0.0-alpha-2 ASAP.
> >> >
> >> > Notice that this is not a production ready release. It is used to let
> >> our
> >> > users try and test the new major release, to get feedback before the
> >> final
> >> > GA release is out.
> >> > So please do NOT use it in production. Just try it and report back
> >> > everything you find unusual.
> >> >
> >> > And this time we will not include CHANGES.md and RELEASENOTE.md
> >> > in our source code, you can find it on the download site. For getting
> >> these
> >> > two files for old releases, please go to
> >> >
> >> >https://archive.apache.org/dist/hbase/
> >> >
> >> > To learn more about Apache hbase, please see
> >> >
> >> >http://hbase.apache.org/
> >> >
> >> > Thanks,
> >> > Your HBase Release Manager
> >> >
> >>
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > 

[jira] [Created] (HBASE-26605) TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths

2021-12-18 Thread Josh Elser (Jira)
Josh Elser created HBASE-26605:
--

 Summary: TestHStore#testRefreshStoreFiles broken due to 
unqualified and qualified paths
 Key: HBASE-26605
 URL: https://issues.apache.org/jira/browse/HBASE-26605
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 2.4.9
Reporter: Josh Elser
Assignee: Josh Elser


Was looking at a failures of this method where 
{noformat}
[ERROR] org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles  
Time elapsed: 4.2 s  <<< ERROR!
java.util.NoSuchElementException
    at 
org.apache.hbase.thirdparty.com.google.common.collect.AbstractIndexedListIterator.next(AbstractIndexedListIterator.java:75)
    at 
org.apache.hadoop.hbase.regionserver.TestHStore.closeCompactedFile(TestHStore.java:962)
    at 
org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:1000)
 {noformat}
This was on a branch where I had some HBASE-26067 changes backported, so I 
thought the problem _was_ those changes. After a bit of digging, I believe the 
test case itself is "broken" (the test passes, but for the wrong reasons).

This test methods adds some files to a Store (via memstore flush or direct 
addition of a file) and eventually tries to get the first file which is 
candidate to be removed. The test {*}never compacted any files{*}. This was the 
first sign that the test itself was wrong.

After lots of comparison with the HBASE-26067 logging to compare against, I 
found that the Store was listing a file which was created by the memstore flush 
as a file to retain AND a file to remove. Second warning. Upon closer 
inspection, I finally noticed that one of the files was qualified with the 
filesystem URI and the other was not.
{noformat}
2021-12-18 16:57:10,903 INFO  [Time-limited test] regionserver.HStore(675): 
toBeAddedFiles=[file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/dd8a430b391546d8b9bdc39bb77d447b,
 
file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73],
 
toBeRemovedFiles=[/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73]
 {noformat}
{{d4c5442b772c43fd9ebdfed1a11c0e73}} how are we both adding and removing this 
file! Turns out, this is because one of them is "/..." and the other is 
"file:/...". Either the problem is in TestHStore in how it is creating/adding 
these files behind the scenes or we should be qualifying the Path inside of 
StoreFileInfo with the filesystem that we're using.

I remember too vividly the problems when trying to separate the rootdir and 
waldir from each other and am cautious against adding something to 
StoreFileInfo to {{{}fs.qualifyPath(p){}}}. Need to look some more, but will 
get a patch up to fix.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Andrew Purtell
Apologies, I managed to hit the send button before finishing. My veto can
be cured by upgrading Log4J to ** 2.17.0 ** . See
https://logging.apache.org/log4j/2.x/security.html.

On Sat, Dec 18, 2021 at 1:22 PM Andrew Purtell  wrote:

> -1 (binding)
>
> The Log4J issues are not fixed by 2.15.
>
> I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I know
> they have plans to upgrade. It does not seem advisable to use Log4J 2 at
> all actually. Another option that does not include such a dangerous
> reference/rewrite mechanism might be preferable.
>
> On Sat, Dec 18, 2021 at 12:02 PM Josh Elser  wrote:
>
>> +1 (binding)
>>
>> * Xsums/sigs good
>> * Can build from source
>> * Log4j 2.15 is included (more on this in the below)
>> * log4j2.formatMsgNoLookups=true is set (multiple times per process, but
>> properly set)
>> * hbase-config.sh issue is fixed over rc1
>>
>> Best as I've been able to keep up, it seems like we should already
>> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
>> that 2.16 may have issues still. It's my opinion that the changes we
>> have here in rc2 are a massive improvement over before. I think this is
>> fine; I just wanted to acknowledge that we may still need to update
>> again real soon.
>>
>> Thanks for your release manager work, Duo!
>>
>> On 12/14/21 9:06 AM, Duo Zhang wrote:
>> > Please vote on this Apache hbase release candidate,
>> > hbase-3.0.0-alpha-2RC1
>> >
>> > The VOTE will remain open for at least 72 hours.
>> >
>> > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
>> > [ ] -1 Do not release this package because ...
>> >
>> > The tag to be voted on is 3.0.0-alpha-2RC1:
>> >
>> >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
>> >
>> > This tag currently points to git reference
>> >
>> >a3ff8e4c812eefab6ad32af45ca449a1395a6510
>> >
>> > The release files, including signatures, digests, as well as CHANGES.md
>> > and RELEASENOTES.md included in this RC can be found at:
>> >
>> >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
>> >
>> > Maven artifacts are available in a staging repository at:
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachehbase-1473/
>> >
>> > Artifacts were signed with the 9AD2AE49 key which can be found in:
>> >
>> >https://downloads.apache.org/hbase/KEYS
>> >
>> > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
>> line.
>> > HBase 3.0.0 includes the following big feature/changes:
>> >Synchronous Replication
>> >OpenTelemetry Tracing
>> >Distributed MOB Compaction
>> >Backup and Restore
>> >Move RSGroup balancer to core
>> >Reimplement sync client on async client
>> >CPEPs on shaded proto
>> >Move the logging framework from log4j to log4j2
>> >
>> > 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2
>> > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
>> > to 3.0.0-alpha-2 ASAP.
>> >
>> > Notice that this is not a production ready release. It is used to let
>> our
>> > users try and test the new major release, to get feedback before the
>> final
>> > GA release is out.
>> > So please do NOT use it in production. Just try it and report back
>> > everything you find unusual.
>> >
>> > And this time we will not include CHANGES.md and RELEASENOTE.md
>> > in our source code, you can find it on the download site. For getting
>> these
>> > two files for old releases, please go to
>> >
>> >https://archive.apache.org/dist/hbase/
>> >
>> > To learn more about Apache hbase, please see
>> >
>> >http://hbase.apache.org/
>> >
>> > Thanks,
>> > Your HBase Release Manager
>> >
>>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk


Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Andrew Purtell
-1 (binding)

The Log4J issues are not fixed by 2.15.

I wish we had remained on Log4J 1. Hadoop 3 is still on 1, although I know
they have plans to upgrade. It does not seem advisable to use Log4J 2 at
all actually. Another option that does not include such a dangerous
reference/rewrite mechanism might be preferable.

On Sat, Dec 18, 2021 at 12:02 PM Josh Elser  wrote:

> +1 (binding)
>
> * Xsums/sigs good
> * Can build from source
> * Log4j 2.15 is included (more on this in the below)
> * log4j2.formatMsgNoLookups=true is set (multiple times per process, but
> properly set)
> * hbase-config.sh issue is fixed over rc1
>
> Best as I've been able to keep up, it seems like we should already
> upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings
> that 2.16 may have issues still. It's my opinion that the changes we
> have here in rc2 are a massive improvement over before. I think this is
> fine; I just wanted to acknowledge that we may still need to update
> again real soon.
>
> Thanks for your release manager work, Duo!
>
> On 12/14/21 9:06 AM, Duo Zhang wrote:
> > Please vote on this Apache hbase release candidate,
> > hbase-3.0.0-alpha-2RC1
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 3.0.0-alpha-2RC1:
> >
> >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
> >
> > This tag currently points to git reference
> >
> >a3ff8e4c812eefab6ad32af45ca449a1395a6510
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
> >
> > Maven artifacts are available in a staging repository at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1473/
> >
> > Artifacts were signed with the 9AD2AE49 key which can be found in:
> >
> >https://downloads.apache.org/hbase/KEYS
> >
> > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
> line.
> > HBase 3.0.0 includes the following big feature/changes:
> >Synchronous Replication
> >OpenTelemetry Tracing
> >Distributed MOB Compaction
> >Backup and Restore
> >Move RSGroup balancer to core
> >Reimplement sync client on async client
> >CPEPs on shaded proto
> >Move the logging framework from log4j to log4j2
> >
> > 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2
> > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
> > to 3.0.0-alpha-2 ASAP.
> >
> > Notice that this is not a production ready release. It is used to let our
> > users try and test the new major release, to get feedback before the
> final
> > GA release is out.
> > So please do NOT use it in production. Just try it and report back
> > everything you find unusual.
> >
> > And this time we will not include CHANGES.md and RELEASENOTE.md
> > in our source code, you can find it on the download site. For getting
> these
> > two files for old releases, please go to
> >
> >https://archive.apache.org/dist/hbase/
> >
> > To learn more about Apache hbase, please see
> >
> >http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk


[VOTE] First release candidate for HBase 2.4.9 (RC0) is available

2021-12-18 Thread Andrew Purtell
Please vote on this Apache HBase release candidate, hbase-2.4.9RC0

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache HBase 2.4.9
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.4.9RC0:

  https://github.com/apache/hbase/tree/2.4.9RC0

This tag currently points to git reference c49f7f63fc.

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/2.4.9RC0/

The API compatibility report can be found at:


https://dist.apache.org/repos/dist/dev/hbase/2.4.9RC0/api_compare_2.4.8_to_2.4.9RC0.html

There are no reported compatibility issues.

There are known flaky unit tests, see HBASE-26254.

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1477/

Artifacts were signed with the 0xD5365CCD key which can be found in:

  https://dist.apache.org/repos/dist/release/hbase/KEYS

To learn more about Apache HBase, please see http://hbase.apache.org/

Thanks,
Your HBase Release Manager


Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Josh Elser

+1 (binding)

* Xsums/sigs good
* Can build from source
* Log4j 2.15 is included (more on this in the below)
* log4j2.formatMsgNoLookups=true is set (multiple times per process, but 
properly set)

* hbase-config.sh issue is fixed over rc1

Best as I've been able to keep up, it seems like we should already 
upgrade to log4j 2.16 due to issues in 2.15. There are alos rumblings 
that 2.16 may have issues still. It's my opinion that the changes we 
have here in rc2 are a massive improvement over before. I think this is 
fine; I just wanted to acknowledge that we may still need to update 
again real soon.


Thanks for your release manager work, Duo!

On 12/14/21 9:06 AM, Duo Zhang wrote:

Please vote on this Apache hbase release candidate,
hbase-3.0.0-alpha-2RC1

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
[ ] -1 Do not release this package because ...

The tag to be voted on is 3.0.0-alpha-2RC1:

   https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1

This tag currently points to git reference

   a3ff8e4c812eefab6ad32af45ca449a1395a6510

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

   https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/

Maven artifacts are available in a staging repository at:

   https://repository.apache.org/content/repositories/orgapachehbase-1473/

Artifacts were signed with the 9AD2AE49 key which can be found in:

   https://downloads.apache.org/hbase/KEYS

3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line.
HBase 3.0.0 includes the following big feature/changes:
   Synchronous Replication
   OpenTelemetry Tracing
   Distributed MOB Compaction
   Backup and Restore
   Move RSGroup balancer to core
   Reimplement sync client on async client
   CPEPs on shaded proto
   Move the logging framework from log4j to log4j2

3.0.0-alpha-2 contains a critical security fix for addressing the log4j2
CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
to 3.0.0-alpha-2 ASAP.

Notice that this is not a production ready release. It is used to let our
users try and test the new major release, to get feedback before the final
GA release is out.
So please do NOT use it in production. Just try it and report back
everything you find unusual.

And this time we will not include CHANGES.md and RELEASENOTE.md
in our source code, you can find it on the download site. For getting these
two files for old releases, please go to

   https://archive.apache.org/dist/hbase/

To learn more about Apache hbase, please see

   http://hbase.apache.org/

Thanks,
Your HBase Release Manager



Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-18 Thread Viraj Jasani
+1

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
  - mvn clean apache-rat:check
* Built from source (1.8.0_301): ok
  - mvn clean install -DskipTests
* Unit tests pass (1.8.0_301): ok
  - mvn package -P runSmallTests -Dsurefire.rerunFailingTestsCount=3
* Nightly build results look good


On Tue, Dec 14, 2021 at 7:46 PM Duo Zhang  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-3.0.0-alpha-2RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 3.0.0-alpha-2RC1:
>
>   https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
>
> This tag currently points to git reference
>
>   a3ff8e4c812eefab6ad32af45ca449a1395a6510
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1473/
>
> Artifacts were signed with the 9AD2AE49 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line.
> HBase 3.0.0 includes the following big feature/changes:
>   Synchronous Replication
>   OpenTelemetry Tracing
>   Distributed MOB Compaction
>   Backup and Restore
>   Move RSGroup balancer to core
>   Reimplement sync client on async client
>   CPEPs on shaded proto
>   Move the logging framework from log4j to log4j2
>
> 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2
> CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
> to 3.0.0-alpha-2 ASAP.
>
> Notice that this is not a production ready release. It is used to let our
> users try and test the new major release, to get feedback before the final
> GA release is out.
> So please do NOT use it in production. Just try it and report back
> everything you find unusual.
>
> And this time we will not include CHANGES.md and RELEASENOTE.md
> in our source code, you can find it on the download site. For getting these
> two files for old releases, please go to
>
>   https://archive.apache.org/dist/hbase/
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>


[jira] [Resolved] (HBASE-26580) The message of StoreTooBusy is confused

2021-12-18 Thread Duo Zhang (Jira)


 [ 
https://issues.apache.org/jira/browse/HBASE-26580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26580.
---
Fix Version/s: 2.5.0
   3.0.0-alpha-3
   2.4.10
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.4+.

Thanks [~zhengzhuobinzzb]!

> The message of StoreTooBusy is confused
> ---
>
> Key: HBASE-26580
> URL: https://issues.apache.org/jira/browse/HBASE-26580
> Project: HBase
>  Issue Type: Task
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Trivial
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
>  
> When check Store limit. We both check parallelPutToStoreThreadLimit and 
> parallelPreparePutToStoreThreadLimit. 
> {code:java}
> if (store.getCurrentParallelPutCount() > this.parallelPutToStoreThreadLimit
> || preparePutCount > this.parallelPreparePutToStoreThreadLimit) {
>   tooBusyStore = (tooBusyStore == null ?
>   store.getColumnFamilyName() :
>   tooBusyStore + "," + store.getColumnFamilyName());
> } {code}
> But we only print Above parallelPutToStoreThreadLimit only. 
>  
>  
> {code:java}
> if (tooBusyStore != null) {
>   String msg =
>   "StoreTooBusy," + this.region.getRegionInfo().getRegionNameAsString() + 
> ":" + tooBusyStore
>   + " Above parallelPutToStoreThreadLimit(" + 
> this.parallelPutToStoreThreadLimit + ")";
>   if (LOG.isTraceEnabled()) {
> LOG.trace(msg);
>   }
>   throw new RegionTooBusyException(msg);
> }{code}
> It confused me a lot time .. 
> Just add message of parallelPreparePutToStoreThreadLimit
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Created] (HBASE-26604) Replace new allocation with ThreadLocal in CellBlockBuilder to reduce GC

2021-12-18 Thread Yutong Xiao (Jira)
Yutong Xiao created HBASE-26604:
---

 Summary: Replace new allocation with ThreadLocal in 
CellBlockBuilder to reduce GC
 Key: HBASE-26604
 URL: https://issues.apache.org/jira/browse/HBASE-26604
 Project: HBase
  Issue Type: Task
Reporter: Yutong Xiao
Assignee: Yutong Xiao


In CellBlockBuilder decompress method, we currently allocate a new 
ByteBufferOutputStream object each invoke. 

{code:java}
try {
  // TODO: This is ugly. The buffer will be resized on us if we guess wrong.
  //TODO: reuse buffers.
  bbos = new ByteBufferOutputStream(osInitialSize);
  IOUtils.copy(cis, bbos);
  bbos.close();
  return bbos.getByteBuffer();
 } finally {
  CodecPool.returnDecompressor(poolDecompressor);
}
{code}

We can use a ThreadLocal variable to reuse the buffer in each Thread. As:

{code:java}
try {
  // TODO: This is ugly. The buffer will be resized on us if we guess wrong.
  if (this.decompressBuff.get() == null) {
this.decompressBuff.set(new ByteBufferOutputStream(osInitialSize));
  }
  ByteBufferOutputStream localBbos = this.decompressBuff.get();
  localBbos.clear();
  IOUtils.copy(cis, localBbos);
  return localBbos.getByteBuffer();
} finally {
  CodecPool.returnDecompressor(poolDecompressor);
}
{code}





--
This message was sent by Atlassian Jira
(v8.20.1#820001)


Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-18 Thread Duo Zhang
+1 (binding)

Checked sigs and sums: Matched
Rat check: Passed
Built from src: Succeeded
Run UTs: Passed
CHANGES and RELEASENOTES: Missed two issues, for generating these two files
and change version in pom. This is not an actual code problem so I always
do not want to sink an RC due to the problems on these two files. In the
future hbase-operator-tools should consider removing these two files from
git too.
Log4j2: Checked the lib directory, the log4j2 version is 2.16.0, which is
good.
Run it against a hbase cluster: Use it with the cluster deployed by me for
testing HBASE-26233, the default command failed because of can
not recognize the 'hdfs' protocol. I needed to manually prepend
'INTERNAL_CLASSPATH=true' when executing the command, like this
  INTERNAL_CLASSPATH=true ./bin/hbase hbck -j
/home/sa/disk1/hbase-operator-tools-1.2.0/hbase-hbck2/hbase-hbck2-1.2.0.jar
extraRegionsInMeta default:IntegrationTestRegionReplicaReplication
  But this is not a problem of hbase-operator-tools, it is because we do
not shade hadoop-hdfs in our shaded jar. And it is strange that why hbck
uses hbase-shaded-client instead of hbase-shaded-mapreduce.
  The stacktrace
17:04:41.024 [main] ERROR org.apache.hbase.HBCK2 - Error on checking extra
regions:
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for
scheme "hdfs"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
~[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hbase.HBCKFsUtils.getRootDir(HBCKFsUtils.java:106)
~[hbase-hbck2-1.2.0.jar:1.2.0]
at
org.apache.hbase.FsRegionsMetaRecoverer.(FsRegionsMetaRecoverer.java:66)
~[hbase-hbck2-1.2.0.jar:1.2.0]
at org.apache.hbase.HBCK2.extraRegionsInMeta(HBCK2.java:268)
[hbase-hbck2-1.2.0.jar:1.2.0]
at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:1020)
[hbase-hbck2-1.2.0.jar:1.2.0]
at org.apache.hbase.HBCK2.run(HBCK2.java:830) [hbase-hbck2-1.2.0.jar:1.2.0]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
[hbase-shaded-client-3.0.0-alpha-2.jar:3.0.0-alpha-2]
at org.apache.hbase.HBCK2.main(HBCK2.java:1145) [hbase-hbck2-1.2.0.jar:1.2.0

张铎(Duo Zhang)  于2021年12月18日周六 16:36写道:

> The readme of the project says we have two tools, one is HBCK2, the other
> is TableReporter, but in the binary I can only see HBCK2...
>
> But anyway, checking the previous release, we did not include
> TableReporter either, so not this release's fault...
>
> Josh Elser  于2021年12月17日周五 23:03写道:
>
>> +1 (binding)
>>
>> Thanks for putting this together, Guangxu!
>>
>> * xsums/sigs are great
>> * RAT check passes on src release
>> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8)
>> * Can build from source
>> * CHANGES and RELEASENOTES look fine at a glance
>> * Public key published in KEYS
>> * Verified log4j2.16 is in the binary release (both as a jar and shaded
>> inside hbase-hbck)
>>
>> - Josh
>>
>> On 12/14/21 10:32 PM, Guangxu Cheng wrote:
>> > Please vote on this Apache hbase operator tools release candidate,
>> > hbase-operator-tools-1.2.0RC0
>> >
>> > The VOTE will remain open for at least 72 hours.
>> >
>> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0
>> > [ ] -1 Do not release this package because ...
>> >
>> > The tag to be voted on is 1.2.0RC0:
>> >
>> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0
>> >
>> > This tag currently points to git reference
>> >
>> >76d68624cebb66ec0e509b0a4c0d96445a601685
>> >
>> > The release files, including signatures, digests, as well as CHANGES.md
>> > and RELEASENOTES.md included in this RC can be found at:
>> >
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/
>> >
>> > Maven artifacts are available in a staging repository at:
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachehbase-1474/
>> > Artifacts were signed with the 5EF3A66D57EC647A key which can be found
>> in:
>> >
>> >https://dist.apache.org/repos/dist/release/hbase/KEYS
>> >
>> > hbase-operator-tools 1.2.0 contains a critical security fix for
>> addressing
>> > the log4j2
>> > CVE-2021-44228. All users who use hbase-operator-tools should 

Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-18 Thread Duo Zhang
The readme of the project says we have two tools, one is HBCK2, the other
is TableReporter, but in the binary I can only see HBCK2...

But anyway, checking the previous release, we did not include TableReporter
either, so not this release's fault...

Josh Elser  于2021年12月17日周五 23:03写道:

> +1 (binding)
>
> Thanks for putting this together, Guangxu!
>
> * xsums/sigs are great
> * RAT check passes on src release
> * Can run unit tests (against 2.3.7, 2.4.4, 2.4.8)
> * Can build from source
> * CHANGES and RELEASENOTES look fine at a glance
> * Public key published in KEYS
> * Verified log4j2.16 is in the binary release (both as a jar and shaded
> inside hbase-hbck)
>
> - Josh
>
> On 12/14/21 10:32 PM, Guangxu Cheng wrote:
> > Please vote on this Apache hbase operator tools release candidate,
> > hbase-operator-tools-1.2.0RC0
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache hbase operator tools 1.2.0
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 1.2.0RC0:
> >
> >https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0
> >
> > This tag currently points to git reference
> >
> >76d68624cebb66ec0e509b0a4c0d96445a601685
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/
> >
> > Maven artifacts are available in a staging repository at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1474/
> > Artifacts were signed with the 5EF3A66D57EC647A key which can be found
> in:
> >
> >https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> > hbase-operator-tools 1.2.0 contains a critical security fix for
> addressing
> > the log4j2
> > CVE-2021-44228. All users who use hbase-operator-tools should upgrade
> > to hbase-operator-tools 1.2.0 ASAP.
> >
> > To learn more about Apache hbase operator tools, please see
> >
> >http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
> >
> > --
> > Best Regards,
> > Guangxu
> >
>


[VOTE] Merge feature branch HBASE-26233 back to master

2021-12-18 Thread Duo Zhang
There is a previous thread[1] to mention that the main development work on
HBASE-26233[2] is done. And now the testing work has been done too, so I
send this official vote to merge for HBASE-26233 back into master.

For those who are not familiar with HBASE-26233, it is for re-implementing
the region replication framework, to decouple with the general replication
framework. You could see the design doc[3] for more details. I've also
opened a PR against master[4], PTAL if you have interest.

You can see the final test result on HBASE-26487[5].
In conclusion, there is no regression on performance when running PE.
Using the new tool introduced in HBASE-26540[6], the new implementation can
perform much better than the old implementation. When inserting 1M rows,
the max lag on secondary replica is 335ms, and the 99.9% lag is 3 only 3ms,
while the lag of the old implementation highly depends on the
replication.source.sleepforretries config but you should not set it to a
very small value as it will waste a lot of resources.
On IntegrationTestRegionReplicaReplication, both implementations can not
pass but the problems are not related to the changing part, so should not
be a blocker. Will file follow on issues to address these problems.

The vote will open for at least 72 hours.

Please vote:
+1: Merge the changes from HBASE-26233 to master
-1: Do not merge these changes because ...

Thanks.

1. https://lists.apache.org/thread/t8w053kpy9pj1vg1x3c0kjjh4ongrxzy
2. https://issues.apache.org/jira/browse/HBASE-26233
3.
https://docs.google.com/document/d/1WPSBwRqIPgTrPRM2wtzAe-3XUvA9S7auWy3Fy8xuZLc/edit
4. https://github.com/apache/hbase/pull/3891
5. https://issues.apache.org/jira/browse/HBASE-26487
6. https://issues.apache.org/jira/browse/HBASE-26540