Re: [DISCUSS] Move our official slack channel to the one in the-asf.slack.com

[Wei-Chiu Chuang]

+1

On Sun, Jul 7, 2024 at 8:07 AM Duo Zhang  wrote:

> As I mentioned in another thread, now slack will hide the comments
> before 90 days in the current apache-hbase.slack.com, which is really
> not good for finding useful discussions.
>
> According to the documentation here
>
> https://infra.apache.org/slack.html
>
> We could invite people which do not have at apache dot org email
> address as a guest to the slack channel, so there is no concerns about
> only committers can join the channel.
>
> Thoughts?
>
> Thanks.
>

[jira] [Resolved] (HBASE-28656) Optimize the verifyCopyResult logic in ExportSnapshot

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-28656.
-
Resolution: Fixed

> Optimize the verifyCopyResult logic in ExportSnapshot
> -
>
> Key: HBASE-28656
> URL: https://issues.apache.org/jira/browse/HBASE-28656
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liangjun He
>Assignee: Liangjun He
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.0.0-beta-2
>
>
> In [HBASE-28625|https://issues.apache.org/jira/browse/HBASE-28625], we added 
> checksum logic comparison for ExportSnapshot files, but the checksum 
> verification scenarios were too simple. This issue aims to address the 
> existing problem. For detailed discussion, please refer to PR 
> [5950|https://github.com/apache/hbase/pull/5950].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28637) asyncwal should attempt to recover lease if close fails

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-28637.
-
Resolution: Fixed

> asyncwal should attempt to recover lease if close fails
> ---
>
> Key: HBASE-28637
> URL: https://issues.apache.org/jira/browse/HBASE-28637
> Project: HBase
>  Issue Type: Bug
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0-beta-2
>
>
> AsyncFSWAL does not verify that WAL gets closed properly.
> For example, HDDS-10609 found that Ozone close may fail because it runs out 
> of space. In such a case, the executor thread responsible for closing WAL 
> file simply exits, and the WAL remains open.
> It is okay for HDFS WAL to stay open, but Ozone does not support renaming 
> open files, and therefore WAL archiving failed.
> See HDDS-10609 for more details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28637) asyncwal should attempt to recover lease if close fails

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-28637:
---

 Summary: asyncwal should attempt to recover lease if close fails
 Key: HBASE-28637
 URL: https://issues.apache.org/jira/browse/HBASE-28637
 Project: HBase
  Issue Type: Bug
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


AsyncFSWAL does not verify that WAL gets closed properly.

For example, HDDS-10609 found that Ozone close may fail because it runs out of 
space. In such a case, the executor thread responsible for closing WAL file 
simply exits, and the WAL remains open.

It is okay for HDFS WAL to stay open, but Ozone does not support renaming 
directories that has open files, and therefore WAL archiving failed.

See HDDS-10609 for more details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28546) Make WAL rolling exception clear

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-28546.
-
Resolution: Fixed

> Make WAL rolling exception clear
> 
>
> Key: HBASE-28546
> URL: https://issues.apache.org/jira/browse/HBASE-28546
> Project: HBase
>  Issue Type: Bug
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 3.0.0-beta-2, 2.6.1, 2.5.9
>
>
> Occasionally we see errors like this that doesn't really give much clue what 
> went wrong.
> {noformat}
> 2024-04-22 08:08:02,026 ERROR org.apache.hadoop.hbase.master.HMaster: * 
> ABORTING master ccycloud-7.ozn-hb973chf3oz.xyz,22001,1713770648404: Log 
> rolling failed *
> java.lang.RuntimeException
> at 
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALMetadata(AsyncProtobufLogWriter.java:217)
> at 
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:223)
> at 
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:164)
> at 
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
> at 
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:726)
> at 
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:129)
> at 
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:886)
> at 
> org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:304)
> at 
> org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211)
> {noformat}
> In this case, it was due to a time out exception. It would be helpful to make 
> the log message more friendly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28448) CompressionTest hangs when run over a Ozone ofs path

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-28448.
-
Resolution: Fixed

> CompressionTest hangs when run over a Ozone ofs path
> 
>
> Key: HBASE-28448
> URL: https://issues.apache.org/jira/browse/HBASE-28448
> Project: HBase
>  Issue Type: Bug
>Reporter: Pratyush Bhatt
>        Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: ozone, pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2, 2.6.1
>
> Attachments: hbase_ozone_compression.jstack
>
>
> If we run the Compression test over HDFS path, it works fine:
> {code:java}
> hbase org.apache.hadoop.hbase.util.CompressionTest 
> hdfs://ns1/tmp/dir1/dir2/test_file.txt snappy
> 24/03/20 06:08:43 WARN impl.MetricsConfig: Cannot locate configuration: tried 
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 24/03/20 06:08:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot 
> period at 10 second(s).
> 24/03/20 06:08:43 INFO impl.MetricsSystemImpl: HBase metrics system started
> 24/03/20 06:08:43 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:08:44 INFO compress.CodecPool: Got brand-new decompressor 
> [.snappy]
> SUCCESS {code}
> The command exits, but when the same is tried over a ofs path, the command 
> hangs.
> {code:java}
> hbase org.apache.hadoop.hbase.util.CompressionTest 
> ofs://ozone1710862004/test-222compression-vol/compression-buck2/test_file.txt 
> snappy
> 24/03/20 06:05:19 INFO protocolPB.OmTransportFactory: Loading OM transport 
> implementation 
> org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified 
> by configuration.
> 24/03/20 06:05:20 INFO client.ClientTrustManager: Loading certificates for 
> client.
> 24/03/20 06:05:20 WARN impl.MetricsConfig: Cannot locate configuration: tried 
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 24/03/20 06:05:20 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot 
> period at 10 second(s).
> 24/03/20 06:05:20 INFO impl.MetricsSystemImpl: HBase metrics system started
> 24/03/20 06:05:20 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 24/03/20 06:05:20 INFO rpc.RpcClient: Creating Volume: 
> test-222compression-vol, with om as owner and space quota set to -1 bytes, 
> counts quota set to -1
> 24/03/20 06:05:20 INFO rpc.RpcClient: Creating Bucket: 
> test-222compression-vol/compression-buck2, with bucket layout 
> FILE_SYSTEM_OPTIMIZED, om as owner, Versioning false, Storage Type set to 
> DISK and Encryption set to false, Replication Type set to server-side default 
> replication type, Namespace Quota set to -1, Space Quota set to -1
> 24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:05:21 WARN impl.MetricsSystemImpl: HBase metrics system already 
> initialized!
> 24/03/20 06:05:21 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl
> 24/03/20 06:05:22 INFO compress.CodecPool: Got brand-new decompressor 
> [.snappy]
> SUCCESS 
> .
> .
> .{code}
> The command doesnt exit.
> Attaching the jstack of the process below:
> [^hbase_ozone_compression.jstack]
> cc: [~weichiu] 
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28546) Make WAL rolling exception clear

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-28546:
---

 Summary: Make WAL rolling exception clear
 Key: HBASE-28546
 URL: https://issues.apache.org/jira/browse/HBASE-28546
 Project: HBase
  Issue Type: Bug
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


Occasionally we see errors like this that doesn't really give much clue what 
went wrong.

{noformat}
2024-04-22 08:08:02,026 ERROR org.apache.hadoop.hbase.master.HMaster: * 
ABORTING master ccycloud-7.ozn-hb973chf3oz.xyz,22001,1713770648404: Log rolling 
failed *
java.lang.RuntimeException
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALMetadata(AsyncProtobufLogWriter.java:217)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:223)
at 
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:164)
at 
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:726)
at 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:129)
at 
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:886)
at 
org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:304)
at 
org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211)
{noformat}

In this case, it was due to a time out exception. It would be helpful to make 
the log message more friendly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28448) CompressionTest hangs when run over a Ozone ofs path

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reopened HBASE-28448:
-

> CompressionTest hangs when run over a Ozone ofs path
> 
>
> Key: HBASE-28448
> URL: https://issues.apache.org/jira/browse/HBASE-28448
> Project: HBase
>  Issue Type: Bug
>Reporter: Pratyush Bhatt
>        Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: ozone, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Attachments: hbase_ozone_compression.jstack
>
>
> If we run the Compression test over HDFS path, it works fine:
> {code:java}
> hbase org.apache.hadoop.hbase.util.CompressionTest 
> hdfs://ns1/tmp/dir1/dir2/test_file.txt snappy
> 24/03/20 06:08:43 WARN impl.MetricsConfig: Cannot locate configuration: tried 
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 24/03/20 06:08:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot 
> period at 10 second(s).
> 24/03/20 06:08:43 INFO impl.MetricsSystemImpl: HBase metrics system started
> 24/03/20 06:08:43 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:08:44 INFO compress.CodecPool: Got brand-new decompressor 
> [.snappy]
> SUCCESS {code}
> The command exits, but when the same is tried over a ofs path, the command 
> hangs.
> {code:java}
> hbase org.apache.hadoop.hbase.util.CompressionTest 
> ofs://ozone1710862004/test-222compression-vol/compression-buck2/test_file.txt 
> snappy
> 24/03/20 06:05:19 INFO protocolPB.OmTransportFactory: Loading OM transport 
> implementation 
> org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified 
> by configuration.
> 24/03/20 06:05:20 INFO client.ClientTrustManager: Loading certificates for 
> client.
> 24/03/20 06:05:20 WARN impl.MetricsConfig: Cannot locate configuration: tried 
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 24/03/20 06:05:20 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot 
> period at 10 second(s).
> 24/03/20 06:05:20 INFO impl.MetricsSystemImpl: HBase metrics system started
> 24/03/20 06:05:20 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 24/03/20 06:05:20 INFO rpc.RpcClient: Creating Volume: 
> test-222compression-vol, with om as owner and space quota set to -1 bytes, 
> counts quota set to -1
> 24/03/20 06:05:20 INFO rpc.RpcClient: Creating Bucket: 
> test-222compression-vol/compression-buck2, with bucket layout 
> FILE_SYSTEM_OPTIMIZED, om as owner, Versioning false, Storage Type set to 
> DISK and Encryption set to false, Replication Type set to server-side default 
> replication type, Namespace Quota set to -1, Space Quota set to -1
> 24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:05:21 WARN impl.MetricsSystemImpl: HBase metrics system already 
> initialized!
> 24/03/20 06:05:21 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl
> 24/03/20 06:05:22 INFO compress.CodecPool: Got brand-new decompressor 
> [.snappy]
> SUCCESS 
> .
> .
> .{code}
> The command doesnt exit.
> Attaching the jstack of the process below:
> [^hbase_ozone_compression.jstack]
> cc: [~weichiu] 
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28448) CompressionTest hangs when run over a Ozone ofs path

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-28448.
-
Fix Version/s: 4.0.0-alpha-1
   Resolution: Fixed

> CompressionTest hangs when run over a Ozone ofs path
> 
>
> Key: HBASE-28448
> URL: https://issues.apache.org/jira/browse/HBASE-28448
> Project: HBase
>  Issue Type: Bug
>Reporter: Pratyush Bhatt
>        Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: ozone, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Attachments: hbase_ozone_compression.jstack
>
>
> If we run the Compression test over HDFS path, it works fine:
> {code:java}
> hbase org.apache.hadoop.hbase.util.CompressionTest 
> hdfs://ns1/tmp/dir1/dir2/test_file.txt snappy
> 24/03/20 06:08:43 WARN impl.MetricsConfig: Cannot locate configuration: tried 
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 24/03/20 06:08:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot 
> period at 10 second(s).
> 24/03/20 06:08:43 INFO impl.MetricsSystemImpl: HBase metrics system started
> 24/03/20 06:08:43 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:08:43 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:08:44 INFO compress.CodecPool: Got brand-new decompressor 
> [.snappy]
> SUCCESS {code}
> The command exits, but when the same is tried over a ofs path, the command 
> hangs.
> {code:java}
> hbase org.apache.hadoop.hbase.util.CompressionTest 
> ofs://ozone1710862004/test-222compression-vol/compression-buck2/test_file.txt 
> snappy
> 24/03/20 06:05:19 INFO protocolPB.OmTransportFactory: Loading OM transport 
> implementation 
> org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified 
> by configuration.
> 24/03/20 06:05:20 INFO client.ClientTrustManager: Loading certificates for 
> client.
> 24/03/20 06:05:20 WARN impl.MetricsConfig: Cannot locate configuration: tried 
> hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
> 24/03/20 06:05:20 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot 
> period at 10 second(s).
> 24/03/20 06:05:20 INFO impl.MetricsSystemImpl: HBase metrics system started
> 24/03/20 06:05:20 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> 24/03/20 06:05:20 INFO rpc.RpcClient: Creating Volume: 
> test-222compression-vol, with om as owner and space quota set to -1 bytes, 
> counts quota set to -1
> 24/03/20 06:05:20 INFO rpc.RpcClient: Creating Bucket: 
> test-222compression-vol/compression-buck2, with bucket layout 
> FILE_SYSTEM_OPTIMIZED, om as owner, Versioning false, Storage Type set to 
> DISK and Encryption set to false, Replication Type set to server-side default 
> replication type, Namespace Quota set to -1, Space Quota set to -1
> 24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:05:21 INFO compress.CodecPool: Got brand-new compressor [.snappy]
> 24/03/20 06:05:21 WARN impl.MetricsSystemImpl: HBase metrics system already 
> initialized!
> 24/03/20 06:05:21 INFO metrics.MetricRegistries: Loaded MetricRegistries 
> class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl
> 24/03/20 06:05:22 INFO compress.CodecPool: Got brand-new decompressor 
> [.snappy]
> SUCCESS 
> .
> .
> .{code}
> The command doesnt exit.
> Attaching the jstack of the process below:
> [^hbase_ozone_compression.jstack]
> cc: [~weichiu] 
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Considering deprecation and removal of XZ compression (hbase-compression-xz)

[Wei-Chiu Chuang]

+1 same here. gzip/lzo in the past, Snappy or zstd now.

On Tue, Apr 2, 2024 at 7:50 PM 张铎(Duo Zhang)  wrote:

> For me I've never seen people actually use the xz compression.
>
> For size, usually people will choose gzip, and for speed, in the past
> people will choose lzo and now they choose snappy or zstd.
>
> So for me I prefer we just deprecated the xz compression immediately
> and remove it 2.6.0.
>
> Thanks.
>
> Andrew Purtell  于2024年4月2日周二 08:02写道：
> >
> > Red Hat filed CVE-2024-3094 late last week on 2024-03-29. This implicates
> > recent releases of the native liblzma library as a vector for malicious
> > code.
> >
> > This is not the pure Java version that we depend upon for HBase's support
> > for the LZMA algorithm (
> >
> https://github.com/apache/hbase/tree/master/hbase-compression/hbase-compression-xz
> ).
> > We depend on version 1.9 of xz-java, which was published in 2021, well
> > before maintenance changes in the project and the involvement of a person
> > who is now believed to be a malicious actor. Projects like HBase that
> > depend on xz-java have no reason to be concerned about the issues
> affecting
> > the native xz library.
> >
> > How the backdoor was introduced calls into question the trustworthiness
> and
> > viability of the XZ project. GitHub has disabled all repositories related
> > to XZ and liblzma, even xz-java. The webpage for XZ and xz-java is down.
> > The open source software community is responding vigorously.
> CVE-2024-3094
> > has a CVSS score 10, the highest possible score. Your security team may
> > become interested in HBase because of hbase-compression-xz's dependency
> on
> > xz-java. It is likely any discovered dependency on any LZMA
> implementation
> > will at least raise questions.
> >
> > For now xz-java remains available in Maven central. (See
> > https://central.sonatype.com/artifact/org.tukaani/xz/versions) We may
> have
> > no choice but to immediately remove hbase-compression-xz if Maven blocks
> or
> > drops xz-java too, because that will break our builds.
> >
> > There is no immediate cause for concern. Still, we believe XZ compression
> > provides little to no value over more modern alternatives, like
> ZStandard,
> > that can also achieve similar compression ratios. XZ, and alternatives
> like
> > ZStandard with the compression level set to a high value, are also
> suitable
> > only for archival use cases and unsuitable for compression of flush files
> > or for use in minor compactions. Given how niche any use of XZ
> > compression could
> > be, we are wondering if there are actually any users of it.
> >
> > If we have no users of hbase-compression-xz, then it provides little to
> no
> > value and continued maintenance of hbase-compression-xz given the issues
> > with its dependency does not make sense.
> >
> > Do you use XZ compression, or are you planning to?
> >
> > If we deprecate XZ compression immediately and then remove it in 2.6,
> would
> > this present a problem? In a private discussion we reached consensus on
> > this approach, but, of course, that is not yet a plan, and something that
> > could easily change based on feedback.
> >
> > From https://nvd.nist.gov/vuln/detail/CVE-2024-3094:
> > "Malicious code was discovered in the upstream tarballs of xz, starting
> > with version 5.6.0. Through a series of complex obfuscations, the liblzma
> > build process extracts a prebuilt object file from a disguised test file
> > existing in the source code, which is then used to modify specific
> > functions in the liblzma code. This results in a modified liblzma library
> > that can be used by any software linked against this library,
> intercepting
> > and modifying the data interaction with this library."
> >
> > --
> > Best regards,
> > Andrew
>

[jira] [Created] (HBASE-28454) Make Outputstream writeExecutor daemon threads

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-28454:
---

 Summary: Make Outputstream writeExecutor daemon threads
 Key: HBASE-28454
 URL: https://issues.apache.org/jira/browse/HBASE-28454
 Project: HBase
  Issue Type: Bug
Reporter: Wei-Chiu Chuang


Found via the HBase CompressionTest HBASE-28448.

We should consider making the threads daemon, not user thread. Otherwise 
process may hang.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28446) Remove the reflection for ByteBufferPositionedReadable

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-28446:
---

 Summary: Remove the reflection for ByteBufferPositionedReadable
 Key: HBASE-28446
 URL: https://issues.apache.org/jira/browse/HBASE-28446
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


HBASE-21946 used reflection to access the ByteBufferPositionedReadable API 
that's only available in Hadoop 3.3.

Now that HBase branch-2.6 and above updated hadoop three dependency to 3.3, we 
can get rid of the reflection.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28419) Allow Action and Policies of ServerKillingMonkey to be configurable

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-28419.
-
Fix Version/s: 4.0.0-alpha-1
   Resolution: Fixed

> Allow Action and Policies of ServerKillingMonkey to be configurable
> ---
>
> Key: HBASE-28419
> URL: https://issues.apache.org/jira/browse/HBASE-28419
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Reporter: Pratyush Bhatt
>    Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
>
> Currently for ServerKillingMonkeyFactory, actions and policies have hardcoded 
> timeouts.
> {code:java}
>     Action[] actions1 = new Action[] {
>       new RestartRandomRsExceptMetaAction(6),
>       new RestartActiveMasterAction(5000),
>       // only allow 2 servers to be dead
>       new RollingBatchRestartRsAction(5000, 1.0f, 2, true),
>       new ForceBalancerAction(),
>       new GracefulRollingRestartRsAction(gracefulRollingRestartTSSLeepTime),
>       new RollingBatchSuspendResumeRsAction(rollingBatchSuspendRSSleepTime,
>           rollingBatchSuspendtRSRatio)
>     }; {code}
> and
> {code:java}
>     return new PolicyBasedChaosMonkey(properties, util,
>       new CompositeSequentialPolicy(new DoActionsOncePolicy(60 * 1000, 
> actions1),
>         new PeriodicRandomActionPolicy(60 * 1000, actions1)),
>       new PeriodicRandomActionPolicy(60 * 1000, actions2));
>   } {code}
> We should allow these to be configurable too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-27769) Use hasPathCapability to support recoverLease, setSafeMode, isFileClosed for non-HDFS file system

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-27769.
-
Resolution: Fixed

Pushed to branch HBASE-27740. Thanks [~taklwu] and [~wchevreuil]!

> Use hasPathCapability to support recoverLease, setSafeMode, isFileClosed for 
> non-HDFS file system
> -
>
> Key: HBASE-27769
> URL: https://issues.apache.org/jira/browse/HBASE-27769
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-3, 2.4.16, 2.5.3
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: HBASE-27740
>
>
> after HADOOP-18671 , we will change the hbase-asyncfs to use use 
> hasPathCapability to support recoverLease, setSafeMode, isFileClosed for 
> non-HDFS file system instead of directly casting only HDFS in 
> RecoverLeaseFSUtils



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] About the topics and time for next meetup

[Wei-Chiu Chuang]

Not sure if folks would feel repetitive, but since the Community over Code
was not recorded/live streamed, I am wondering if we could use this as an
opportunity to repeat some of the talks in the conference.

On Mon, Oct 16, 2023 at 5:47 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> Thanks, Stephen. I'm +1 for the two regional meetups proposal.
>
> As per the topics suggested above, I think all are relevant. Some extra
> thoughts on those:
>
> 1) Roadmap
> - Do we have any roadmap currently published anywhere?
> - If not, we could use the meetup to start listing what development tasks
> folks are currently working on or what would be nice to have.
>
> 2) Attract more contributors
> - Maybe survey the user/dev lists or other hbase forums for folks willing
> to contribute?
>
> 3) Contribute/Housekeeping
> - We could agree to try cleanup stalled jira's/github cleanup;
> - Maybe redesign the website and documentation? Would be nice to have a
> "designer" contributor...
>
> For those interested in having/attending a virtual meetup, we could start
> to share the available time range/timezone. Mine would be between 10AM and
> 4PM in UTC+1.
>
> Regards,
> Wellington
>
> Em dom., 15 de out. de 2023 às 00:15, Tak Lon (Stephen) Wu <
> tak...@apache.org> escreveu:
>
> > move this discussion to dev list
> >
> > -Stephen
> >
> > On Tue, Oct 10, 2023 at 10:36 AM Tak Lon (Stephen) Wu  >
> > wrote:
> >
> > > Hi guys,
> > >
> > > I met a few folks in the community over code and was discussing the
> next
> > > hbase virtual meetup, but it's better to also put the discussion on
> here
> > > and start with the private list first .
> > >
> > >
> > > Few questions came to mind:
> > >
> > > 1. Topics, could we revisit or define a roadmap for next year or two ?
> > >- how do we formalize and publish the roadmap at least to the
> > > developer? Or better documentation ?
> > >- do we have any missing features from those existing tasks?
> > > 2. About the time, if the format is virtual meetup, would it be better
> to
> > > have regional meetup, e.g. at least two regional meetups, one for
> people
> > in
> > > the time zone between UTC+2 and UTC +10, and rest in the other zone.
> Date
> > > TBH.
> > > 3. How do we attract more developers? e.g. presenting roadmap idea for
> > > users, what problems we are facing and what options could we adopt ?
> > > 4. Community housekeeping about JIRAs and standard ?
> > >
> > > Please feel free to leave any comments.
> > >
> > > Thanks,
> > > Stephen
> > >
> > >
> > >
> >
>

[jira] [Created] (HBASE-27982) Synchronous replication should check if the file system supports truncate API

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-27982:
---

 Summary: Synchronous replication should check if the file system 
supports truncate API
 Key: HBASE-27982
 URL: https://issues.apache.org/jira/browse/HBASE-27982
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


Ok. I missed this but I was just told that the synchronous replication 
leverages the truncate() FS API.

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/SyncReplicationReplayWALManager.java#L282

Ozone does not implement truncate so calling this method on the WAL FS will 
result in an exception. It would be a better user experience to alert user that 
this is not supported from the start.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Wei-Chiu Chuang]

I am +1 to use a feature branch.

On Tue, Jun 20, 2023 at 10:20 AM Tak Lon (Stephen) Wu 
wrote:

> or maybe we create a new feature branch hadoop-33-ozone that has these
> interfaces and ozone related support, then we put all the feature changes
> into this feature branch and merge later?
>
> The problem I see is that it's hard and very confusing to maintain two
> hadoop3 profiles, and I can see sooner or later hadoop 3.2.x could be EOL.
>
> Thanks,
> Stephen
>
>
>
> On Fri, Jun 16, 2023 at 11:32 AM Viraj Jasani  wrote:
>
> > How about using a new hadoop 3.3 profile for features that are explicitly
> > present in 3.3 (like FileSystem changes)? When the time comes, we switch
> to
> > 3.3 profile by default and drop old hadoop 3 profile that supports 3.2.x
> > versions as of today?
> >
> >
> > On Fri, Jun 16, 2023 at 7:11 AM 张铎(Duo Zhang) 
> > wrote:
> >
> >> In general, in HBase, we will use the last patch release of the oldest
> >> supported hadoop release line as our default hadoop dependency.
> >>
> >> For example, since we claim that 3.x will support hadoop 3.2.x and
> >> 3.3.x, then we will declare the default hadoop version as 3.2.4.
> >>
> >> I think we can discuss whether to move up to 3.3.6 as the default
> >> version, if there are no compatibility issues when communicating with
> >> 3.2.x hadoop clusters.
> >>
> >> But if we want to use the features which are only provided in 3.3.6,
> >> then we should be careful as this means our users can not build hbase
> >> with 3.2.x any more, which means we have dropped the support for
> >> 3.2.x.
> >>
> >> Thanks.
> >>
> >> Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
> >> >
> >> > Hi HBase devs,
> >> >
> >> > Over the past few years HBase supports the default Hadoop version of
> >> 3.2.x
> >> > but it also works on Hadoop 3.3.x.
> >> >
> >> > I'm wondering if it makes sense to move the current default
> >> hadoop.version
> >> > <https://github.com/apache/hbase/blob/master/pom.xml#L800> from 3.2.4
> >> to
> >> > 3.3.x.
> >> >
> >> > Why?
> >> >
> >> > 1. From a stability and security point of view, Hadoop 3.3 is the most
> >> up
> >> > to date release line. And all HBase tests pass using 3.3.x. There
> hasn't
> >> > been a new Hadoop 3.2.x release for over a year.
> >> >
> >> > 2. We have a feature (using HBase on Ozone) that depends on an API in
> >> > Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the
> default
> >> > hadoop version to 3.3.6 will save a lot of hassle.
> >> >
> >> > Thoughts?
> >> >
> >> > Best,
> >> > Weichiu
> >>
> >
>

[DISCUSS] Move default Hadoop version to 3.3.x

[Wei-Chiu Chuang]

Hi HBase devs,

Over the past few years HBase supports the default Hadoop version of 3.2.x
but it also works on Hadoop 3.3.x.

I'm wondering if it makes sense to move the current default hadoop.version
 from 3.2.4 to
3.3.x.

Why?

1. From a stability and security point of view, Hadoop 3.3 is the most up
to date release line. And all HBase tests pass using 3.3.x. There hasn't
been a new Hadoop 3.2.x release for over a year.

2. We have a feature (using HBase on Ozone) that depends on an API in
Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
hadoop version to 3.3.6 will save a lot of hassle.

Thoughts?

Best,
Weichiu

[jira] [Created] (HBASE-27931) Update hadoop.version to 3.3.6

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-27931:
---

 Summary: Update hadoop.version to 3.3.6
 Key: HBASE-27931
 URL: https://issues.apache.org/jira/browse/HBASE-27931
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


HBase's default Hadoop3 version is 3.2.4 but HBase already supports Haddoop 
3.3.x.

Hadoop 3.2 line has not been updated for over a year. It is perhaps the time to 
update the Hadoop dependency to the 3.3.x line. (I'll start a DISCUSS thread if 
the test goes well)

3.3.6 RC is out which fixed a bunch of CVEs and I'd like to test HBase against 
it. Additionally, Hadoop 3.3.6 will permit us to use non-HDFS as WAL storage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New HBase committer Nihal Jain

[Wei-Chiu Chuang]

Congrats!

On Wed, May 3, 2023 at 5:42 AM rajeshb...@apache.org <
chrajeshbab...@gmail.com> wrote:

> Congratulations Nihal!!!
>
>
> Thanks,
> Rajeshbabu.
>
> On Wed, May 3, 2023, 5:42 PM Nick Dimiduk  wrote:
>
> > Hello!
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Nihal
> Jain
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Nihal's generous contributions thus far and look
> forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Nihal Jain!
> >
> > Thanks,
> > Nick
> >
>

[jira] [Resolved] (HBASE-27693) Support for Hadoop's LDAP Authentication mechanism (Web UI only)

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-27693.
-
Fix Version/s: 3.0.0-alpha-4
   Resolution: Fixed

> Support for Hadoop's LDAP Authentication mechanism (Web UI only)
> 
>
> Key: HBASE-27693
> URL: https://issues.apache.org/jira/browse/HBASE-27693
> Project: HBase
>  Issue Type: New Feature
>Reporter: Yash Dodeja
>Assignee: Yash Dodeja
>Priority: Major
> Fix For: 3.0.0-alpha-4
>
> Attachments: Screenshot 2023-03-27 at 10.53.26 AM.png
>
>
> Hadoop's AuthenticationFilter has changed and now has support for ldap 
> mechanism too. HBase still uses an older version tightly coupled with 
> kerberos and spnego as the only auth mechanisms. HADOOP-12082 has added 
> support for multiple auth handlers including LDAP. On trying to use Hadoop's 
> AuthenticationFilterInitializer in hbase.http.filter.initializers, there is a 
> casting exception as HBase requires it to extend 
> org.apache.hadoop.hbase.http.FilterInitializer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27746) Check if the file system supports storage policy before invoking setStoragePolicy()

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-27746:
---

 Summary: Check if the file system supports storage policy before 
invoking setStoragePolicy()
 Key: HBASE-27746
 URL: https://issues.apache.org/jira/browse/HBASE-27746
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


Found these messages on an Ozone cluster:

{noformat}
2023-03-20 12:27:09,185 WARN org.apache.hadoop.hbase.util.CommonFSUtils: Unable 
to set storagePolicy=HOT for 
path=ofs://ozone1/vol1/bucket1/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc.
 DEBUG log level might have more details.
java.lang.UnsupportedOperationException: RootedOzoneFileSystem doesn't support 
setStoragePolicy
at 
org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3227)
at 
org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:521)
at 
org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:504)
at 
org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:477)
at 
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:225)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:275)
at 
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6387)
at 
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1115)
at 
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1112)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}

Ozone does not support storage policy. If we use FileSystem.hasPathCapability() 
API to check before invoking the API, these warning messages can be avoided.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27740) Support Ozone as a WAL backing storage

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-27740:
---

 Summary: Support Ozone as a WAL backing storage
 Key: HBASE-27740
 URL: https://issues.apache.org/jira/browse/HBASE-27740
 Project: HBase
  Issue Type: New Feature
Reporter: Wei-Chiu Chuang


As discussed in the 
[thread|https://lists.apache.org/thread/tcrp8vxxs3z12y36mpzx35txhpp7tvxv], we'd 
like to make HBase workloads possible on Ozone.

This features is to be built on top of 
# HDDS-7593 (support hsync and lease recovery in Ozone), and
# HADOOP-18671 (Add recoverLease(), setSafeMode(), isFileClosed() APIs to 
FileSystem).




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27739) Remove Java reflection used in FSUtils.create()

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-27739:
---

 Summary: Remove Java reflection used in FSUtils.create() 
 Key: HBASE-27739
 URL: https://issues.apache.org/jira/browse/HBASE-27739
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


FSUtils.create() uses reflection to access a HDFS API 
DistributedFileSystem.create() that supports favored node.

This API is added in Hadoop 2.1.0-beta (HDFS-2576) so we can use it directly 
without reflection.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27738) Remove DNS reflection helper method

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-27738:
---

 Summary: Remove DNS reflection helper method
 Key: HBASE-27738
 URL: https://issues.apache.org/jira/browse/HBASE-27738
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


HBASE-14594 used a reflection helper method to use an Hadoop API added in 
2.8.0. We should remove this reflection from the master branch now.

https://github.com/apache/hbase/blob/fbe3b90e0c4eef1dc13fb2a5ed9381106ca671dd/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java#L57



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?

[Wei-Chiu Chuang]

hsync/hflush, or any input/output stream APIs for that matter, can be
probed using StreamCapabilities.hasCapabilitiy() API.

lease recovery isn't (DistributedFIleSystem.recoverLease()). safe mode
check isn't. There are a number of HDFS specific APIs that HBase uses.

I'm all for abstracting out FS implementation details. But it would be an
overkill to try and add every single FS specific APIs to the generic
FileSystem API interface.

One idea that Stephen had was to add a RecoverableFileSystem interface in
Hadoop which adds lease recovery capability, and then HDFS or Ozone can
implement this interface.

In a less ideal world, I imagine we could have one hbase module for
utilities that does FS specific tasks. That way it is future proofing.

On Wed, Mar 15, 2023 at 12:50 PM Andrew Purtell  wrote:

> Does Hadoop have a marker interface that lets an application know its
> FileSystem instances can support hsync/hflush? Ideally all we should need
> to do is test with instanceof for that marker and use reflection (in the
> worst case) to get a handle to the hsync or hflush method, and then call
> it. This approach should be taken wherever we have a requirement to use a
> special WAL specific API provided by the underlying FileSystem, so we can
> abstract it sufficiently to not require a direct dependency on Ozone or S3A
> or any non HDFS filesystem.
>
> On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu 
> wrote:
>
> > Hi team,
> >
> > Recently, Wei-Chiu and I have been discussing about if HBase can use
> > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1])
> > and HFile, for HFile it’s pluggable by configuring the file system to
> > use Ozone File System (Ozone)
> >
> > But we found that the WAL it’s a bit different, especially
> > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if
> > the file system is an instance of HDFS, and thus WAL recovery to
> > execute file lease recovery from RS crashes. Here, if we would like to
> > add Ozone, it does not matter by importing as a direct dependency to
> > perform similar lease recovery or via reflection by class name in
> > plaintext String, we still need to somehow introduce Ozone to be
> > another supported file system. (we can discuss how we can implement
> > better as well)
> >
> > We also found other places e.g. FSUtils and HFileSystem have used
> > DistributedFileSystem, but it should be able to move them into either
> > hbase-asyncfs or a new FS related component to separate the use of
> > different supported file systems.
> >
> > So, we’re wondering if anyone would have any objections to adding
> > Ozone as a dependency to hbase-asyncfs? or if you have a better idea
> > how this could be added without adding Ozone as dependency, please
> > feel free to comment on this thread.
> >
> >
> > [1] Ozone is working on support for hsync and hflush,
> > https://issues.apache.org/jira/browse/HDDS-7593,
> > https://issues.apache.org/jira/browse/HDDS-4353
> > [2] RecoverLeaseFSUtils#recoverFileLease,
> >
> >
> https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63
> >
> > Thanks,
> > Stephen
> >
>
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
>- A23, Welcome, Apocalypse
>

Re: [ANNOUNCE] New HBase committer Rushabh Shah

[Wei-Chiu Chuang]

Congrats!

On Thu, Dec 15, 2022 at 7:50 AM Geoffrey Jacoby  wrote:

> Congratulations, Rushabh!
>
> Geoffrey
>
> On Thu, Dec 15, 2022 at 10:26 AM Tak Lon (Stephen) Wu 
> wrote:
>
> > Congratulations, and welcome!
> >
> > -Stephen
> >
> > On Thu, Dec 15, 2022 at 5:39 AM Viraj Jasani  wrote:
> >
> > > Very well deserved! Congratulations and Welcome, Rushabh!!
> > >
> > >
> > > On Wed, Dec 14, 2022 at 10:57 PM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > > Rushabh Shah(shahrs87)
> > > > has accepted the PMC's invitation to become a committer on the
> > > > project. We appreciate all
> > > > of Rushabh's generous contributions thus far and look forward to his
> > > > continued involvement.
> > > >
> > > > Congratulations and welcome, Rushabh Shah!
> > > >
> > > > 我很高兴代表 Apache HBase PMC 宣布 Rushabh Shah 已接受我们的邀请，成
> > > > 为 Apache HBase 项目的 Committer。感谢 Rushabh Shah 一直以来为 HBase 项目
> > > > 做出的贡献，并期待他在未来继续承担更多的责任。
> > > >
> > > > 欢迎 Rushabh Shah！
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase Committer Liangjun He

[Wei-Chiu Chuang]

Congrats!

On Sun, Dec 4, 2022 at 5:33 AM 宾莉金（binlijin）  wrote:

> Congratulations!
>
> 张铎(Duo Zhang)  于2022年12月3日周六 22:28写道：
>
> > Congratulations!
> >
> > Yu Li  于2022年12月3日周六 21:51写道：
> > >
> > > Hi All,
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Liangjun
> > > He (heliangjun) has accepted the PMC's invitation to become a committer
> > on
> > > the project. We appreciate all of Liangjun's generous contributions
> thus
> > > far and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Liangjun!
> > >
> > > 我很高兴代表 Apache HBase PMC 宣布 Liangjun He (何良均) 已接受我们的邀请，成为 Apache HBase
> 项目的
> > > Committer。感谢何良均一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎良均！
> > >
> > > Best Regards,
> > > Yu
> > > --
> > > Best Regards,
> > > Yu
> >
>
>
> --
> *Best Regards,*
>  lijin bin
>

Re: HBase 2.4.x + Spark 3.3

[Wei-Chiu Chuang]

This looks similar to what we are seeing with Ozone running w/ Spark 3.3
HDDS-6926 

My guess is Spark 3.3 switched to the shaded Hadoop client jar, where
protobuf message classes are shaded, and somehow that breaks applications
using old Hadoop classes.

If you can share some stack traces that would be great.


On Mon, Oct 24, 2022 at 11:26 AM Lars Francke 
wrote:

> Hi Andrew,
>
> okay, we'll try that and will report back when/if we get it working.
>
> Cheers,
> Lars
>
> On Thu, Oct 20, 2022 at 2:29 AM Andrew Purtell 
> wrote:
>
> > No, that is insufficient. HBase must be recompiled against Hadoop 3 first
> >
> > cd /path/to/hbase
> > mvn clean install assembly:single -DskipTests -Dhadoop.profile=3.0
> > -Dhadoop-three.version=XXX
> >
> > Then once the results are in your local maven cache or nexus instance,
> you
> > can compile Spark as indicated.
> >
> >
> > On Tue, Oct 18, 2022 at 11:39 PM Lars Francke 
> > wrote:
> >
> > > Hi Andrew,
> > >
> > > thanks for the reply.
> > > I should have been more specific: We only tried to compile the "client"
> > > part that's used in Spark itself and we used the proper versions
> > >
> > > mvn -Dspark.version=XXX -Dscala.version=XXX -Dhadoop-three.version=XXX
> > > -Dscala.binary.version=XXX -Dhbase.version=XXX clean package
> > >
> > > I assume that should pull in the correct dependencies but I have to
> admit
> > > that I didn't check, took it straight from the readme.
> > > We wanted to try the server bit for the RegionServers afterwards but
> > didn't
> > > even get to it yet.
> > >
> > > We have this on our radar though and might try to work through those
> > issues
> > > at some point.
> > > If we get started on that I'll ping the list.
> > >
> > > Cheers,
> > > Lars
> > >
> > > On Wed, Oct 19, 2022 at 1:41 AM Andrew Purtell 
> > > wrote:
> > >
> > > > Out of the box use is going to be problematic without recompiling
> HBase
> > > for
> > > > Hadoop 3. Spark 3.3 ships with Hadoop 3.3.2. Apache HBase 2.4.x (and
> > all
> > > > 2.x) releases are compiled against Hadoop 2. Link errors
> > (ClassNotFound,
> > > > NoClassDef, etc) I think are to be expected because the class
> > hierarchies
> > > > of various Hadoop things have been incompatibly changed in 3.x
> releases
> > > > relative to 2.x. This is not unreasonable. Semantic versioning
> suggests
> > > > breaking changes can be expected in a major version increment.
> > > >
> > > > Users probably need to do a holistic (or hermetic, if you prefer)
> build
> > > of
> > > > their bill of materials before testing it or certainly before
> shipping
> > > it.
> > > > Build your HBase for the version of Hadoop you are actually shipping
> it
> > > > with, as opposed to whatever the upstream project picks as a default
> > > build
> > > > target. They are called "convenience binaries" by the project and the
> > > > Foundation for a reason. Convenience may vary according to your
> > > > circumstances. When HBase finally ships builds compiled against
> Hadoop
> > 3
> > > by
> > > > default, anyone still using 2.x in production will face the same
> > problem
> > > > (in reverse). The Phoenix project also faces this issue for what it's
> > > > worth. Their readme and build instructions walk users through
> > rebuilding
> > > > HBase using -Dhadoop.profile=3.0 as a first step as well.
> > > >
> > > >
> > > > On Mon, Oct 17, 2022 at 1:52 PM Lars Francke  >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > we've just recently tried getting the HBase Spark connector running
> > > > against
> > > > > Spark 3.3 and HBase 2.4.x and failed miserably. It was a mess of
> > Scala
> > > > and
> > > > > Java issues, classpath, NoClassDef etc.
> > > > >
> > > > > The trauma is too recent for me to dig up the details but if
> needed I
> > > can
> > > > > ;-)
> > > > >
> > > > > For now I'm just wondering if anyone has succeeded using this
> > > > combination?
> > > > >
> > > > > Cheers,
> > > > > Lars
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > > It's what we’ve earned
> > > > Welcome, apocalypse, what’s taken you so long?
> > > > Bring us the fitting end that we’ve been counting on
> > > >- A23, Welcome, Apocalypse
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> >- A23, Welcome, Apocalypse
> >
>

Re: Updated project lead on JIRA

[Wei-Chiu Chuang]

I'm just curious if there is anything that only jira project lead can do
that project administrators can't do.
Obviously if that is anything that helps us work better I am very
interested in knowing more.

On Thu, Oct 6, 2022 at 12:21 PM Andrew Purtell  wrote:

> I realized today that I could update the user that JIRA considers project
> lead, so I updated that role on our JIRA project to our current Chair, Duo
> Zhang. (The previous setting was Michael Stack.)
>
> Sure, this is pretty random. :-) But why not. Please let me know if you
> have any concerns.
>
> --
> Best regards,
> Andrew
>

Re: Become a contributor

[Wei-Chiu Chuang]

I've added your user id to the contributor group. Let me know if you have
any questions.

On Thu, Jun 2, 2022 at 2:23 PM Hernan Gelaf-Romer
 wrote:

> Hello,
>
> I’m a new developer on the HBase team at HubSpot and would like to be
> added as a contributor. My Jira username is hgromer.
>
> Thank you!
>
> Hernan
>
>

Fwd: [NOTICE] Dependabot Updates enabled for all projects

[Wei-Chiu Chuang]

-- Forwarded message -
From: Chris Lambertus 
Date: Tue, Apr 5, 2022 at 12:38 PM
Subject: [NOTICE] Dependabot Updates enabled for all projects
To: 



Hi folks,

Infra is pleased to announce that GitHub’s Dependabot service has been
approved for use by ASF Legal and Infra, and is now enabled for all repos.
Dependabot will create PRs in your repo with recommended security updates
for your project. It is entirely up to the project to accept or reject
these PRs.

Dependabot Alerts can also be configured per-project, but currently the
notifications go to Org Admins only. If your project wishes to receive
Dependabot Alerts via email, please open an Infra Jira ticket so that we
can add your committer team to the alerts.

-Chris
ASF Infra

Re: [ANNOUNCE] New HBase committer Yutong Xiao(肖禹同)

[Wei-Chiu Chuang]

Congrats! Yutong did a ton of work. I'm happy to see Yuton getting the
recognition!

On Fri, Mar 4, 2022 at 5:24 PM Nick Dimiduk  wrote:

> Congratulations Yutong, and thank you for all your contributions!
>
> On Wed, Mar 2, 2022 at 8:50 AM 张铎(Duo Zhang) 
> wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Yutong
> > Xiao(YutSean) has accepted the PMC's invitation to become a committer on
> > the project. We appreciate all of Yutong's generous contributions thus
> far
> > and look forward to his continued involvement.
> >
> > Congratulations and welcome, Yutong Xiao!
> >
> > 我很高兴代表 Apache HBase PMC 宣布肖禹同已接受我们的邀请，成为 Apache HBase 项目的
> > Committer。感谢肖禹同一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> >
> > 欢迎肖禹同！
> >
>

Re: [DISCUSS] Replace log4j with reload4j for branch-2.x

[Wei-Chiu Chuang]

https://www.slf4j.org/news.html
BTW, a new slf4j-log4j12 release is made where it switches dependency on
log4j12 to reload4j:
https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12/1.7.35

This should hopefully make the migration slightly easier.

On Tue, Jan 25, 2022 at 5:16 PM 张铎(Duo Zhang)  wrote:

> I would prefer we have LOG4J2-3341 first before releasing any 2.x releases
> with log4j2. I pinged the log4j community once but no response yet. Will
> try to ping again soon.
>
> Andrew Purtell  于2022年1月25日周二 10:09写道：
>
> > Reload4J is a great option when we really should maintain fairly strict
> > compatibility (patch releases) and less so when not (minors, majors).
> >
> > We haven’t released 2.5.0 yet. My fault, mainly... It’s been lagging.
> > Perhaps we could backport Duo’s work from 3.0.0-alpha into branch-2.5 /
> > 2.5.0? This would be a minor release, so, while some operational
> > compatibility breaks would be somewhat surprising, it could be more
> > understandable.
> >
> > > On Jan 24, 2022, at 5:00 PM, Zach York  wrote:
> > >
> > > +1 on the original discussion
> > >
> > > I think that all makes sense, we can't know whether one dependency is
> > more
> > > vulnerable/going to be more work to maintain or not. However, I think
> > > perhaps the more interesting question is whether we should be okay with
> > > using EOL dependencies in any active release line. CVEs happen... I
> think
> > > it's important to be able to react quickly. By depending on any EOL
> code
> > in
> > > our active release lines, we severely limit our ability to quickly deal
> > > with CVEs. I understand it's not always easy (the backwards
> > incompatibility
> > > of log4j2 and the properties files are good examples), but it's also
> been
> > > 6+ years since log4j 1.x has become EOL.
> > >
> > >> On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell 
> > wrote:
> > >>
> > >> I will offer a guess as answer to the question originally proposed,
> > which
> > >> is, no the Logging PMC is not going to behave in a cooperative manner
> to
> > >> Reload4J. The best we can hope for, and I personally doubt it, is they
> > will
> > >> not be actively hostile. Decisions made by Apache PMC can
> unfortunately
> > be
> > >> made on the basis of years-long running arguments and personality
> > >> conflicts, and Logging is unfortunately one of them. Just my opinion.
> > You
> > >> won't change it. Fortunately that is neither here nor there. We aren't
> > here
> > >> to judge up front if Reload4J can respond adequately to hypothetical
> > future
> > >> security issues. That is not knowable and remains to be seen. It also
> > >> remains to be seen if Log4J 2 is going to respond adequately to future
> > >> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our other
> > risky
> > >> third party dependencies (as measured by having "interesting" CVEs).
> > What
> > >> we can do is look at the current track record, which has been adequate
> > so
> > >> far.
> > >>
> > >>
> > >> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang
> > >>  wrote:
> > >>
> > >>> The reload4j is maintained by one of the Apache Logging PMC. And a
> > >> release
> > >>> was made to address the latest log4j 1.x CVEs announced only a day
> > after.
> > >>>
> > >>> There is a thread in the private@logging that mentioned the “new”
> > >> log4j1.x
> > >>> CVEs were actually not new. They were announced years before for 2.x
> > >> which
> > >>> happens to also impact 1.x. But because 1.x was considered EOL, they
> > >> didn’t
> > >>> bother to mention 1.x were affected. It is only now that 1.x’s
> getting
> > >>> interest that they did this.
> > >>>
> > >>> Sean Busbey 於 2022年1月22日 週六，上午2:47寫道：
> > >>>
> > >>>> It's relevant to what kind of mitigation this is. The effectiveness
> of
> > >>>> reload4j to deal with "the critical CVEs of log4j 1" is limited by
> how
> > >>>> likely it is that they know about them.
> > >>>>
> > >>>> Otherwise at the next CVE we're back in the same place where
> > downstream
> > >>>> users aren't meaningfully more protected. An

Re: [DISCUSS] Replace log4j with reload4j for branch-2.x

[Wei-Chiu Chuang]

What's the plan for other repos?
hbase-filesystem, hbase-connectors, hbase-operator-tools depends on log4j
1.x

Even hbase-thirdparty has transitive dependency on log4j 1.x

On Tue, Jan 25, 2022 at 5:16 PM 张铎(Duo Zhang)  wrote:

> I would prefer we have LOG4J2-3341 first before releasing any 2.x releases
> with log4j2. I pinged the log4j community once but no response yet. Will
> try to ping again soon.
>
> Andrew Purtell  于2022年1月25日周二 10:09写道：
>
> > Reload4J is a great option when we really should maintain fairly strict
> > compatibility (patch releases) and less so when not (minors, majors).
> >
> > We haven’t released 2.5.0 yet. My fault, mainly... It’s been lagging.
> > Perhaps we could backport Duo’s work from 3.0.0-alpha into branch-2.5 /
> > 2.5.0? This would be a minor release, so, while some operational
> > compatibility breaks would be somewhat surprising, it could be more
> > understandable.
> >
> > > On Jan 24, 2022, at 5:00 PM, Zach York  wrote:
> > >
> > > +1 on the original discussion
> > >
> > > I think that all makes sense, we can't know whether one dependency is
> > more
> > > vulnerable/going to be more work to maintain or not. However, I think
> > > perhaps the more interesting question is whether we should be okay with
> > > using EOL dependencies in any active release line. CVEs happen... I
> think
> > > it's important to be able to react quickly. By depending on any EOL
> code
> > in
> > > our active release lines, we severely limit our ability to quickly deal
> > > with CVEs. I understand it's not always easy (the backwards
> > incompatibility
> > > of log4j2 and the properties files are good examples), but it's also
> been
> > > 6+ years since log4j 1.x has become EOL.
> > >
> > >> On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell 
> > wrote:
> > >>
> > >> I will offer a guess as answer to the question originally proposed,
> > which
> > >> is, no the Logging PMC is not going to behave in a cooperative manner
> to
> > >> Reload4J. The best we can hope for, and I personally doubt it, is they
> > will
> > >> not be actively hostile. Decisions made by Apache PMC can
> unfortunately
> > be
> > >> made on the basis of years-long running arguments and personality
> > >> conflicts, and Logging is unfortunately one of them. Just my opinion.
> > You
> > >> won't change it. Fortunately that is neither here nor there. We aren't
> > here
> > >> to judge up front if Reload4J can respond adequately to hypothetical
> > future
> > >> security issues. That is not knowable and remains to be seen. It also
> > >> remains to be seen if Log4J 2 is going to respond adequately to future
> > >> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our other
> > risky
> > >> third party dependencies (as measured by having "interesting" CVEs).
> > What
> > >> we can do is look at the current track record, which has been adequate
> > so
> > >> far.
> > >>
> > >>
> > >> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang
> > >>  wrote:
> > >>
> > >>> The reload4j is maintained by one of the Apache Logging PMC. And a
> > >> release
> > >>> was made to address the latest log4j 1.x CVEs announced only a day
> > after.
> > >>>
> > >>> There is a thread in the private@logging that mentioned the “new”
> > >> log4j1.x
> > >>> CVEs were actually not new. They were announced years before for 2.x
> > >> which
> > >>> happens to also impact 1.x. But because 1.x was considered EOL, they
> > >> didn’t
> > >>> bother to mention 1.x were affected. It is only now that 1.x’s
> getting
> > >>> interest that they did this.
> > >>>
> > >>> Sean Busbey 於 2022年1月22日 週六，上午2:47寫道：
> > >>>
> > >>>> It's relevant to what kind of mitigation this is. The effectiveness
> of
> > >>>> reload4j to deal with "the critical CVEs of log4j 1" is limited by
> how
> > >>>> likely it is that they know about them.
> > >>>>
> > >>>> Otherwise at the next CVE we're back in the same place where
> > downstream
> > >>>> users aren't meaningfully more protected. And in that case perhaps
> we
> > >>> would
> > >>>> do bett

Re: [DISCUSS] Replace log4j with reload4j for branch-2.x

[Wei-Chiu Chuang]

The reload4j is maintained by one of the Apache Logging PMC. And a release
was made to address the latest log4j 1.x CVEs announced only a day after.

There is a thread in the private@logging that mentioned the “new” log4j1.x
CVEs were actually not new. They were announced years before for 2.x which
happens to also impact 1.x. But because 1.x was considered EOL, they didn’t
bother to mention 1.x were affected. It is only now that 1.x’s getting
interest that they did this.

Sean Busbey 於 2022年1月22日 週六，上午2:47寫道：

> It's relevant to what kind of mitigation this is. The effectiveness of
> reload4j to deal with "the critical CVEs of log4j 1" is limited by how
> likely it is that they know about them.
>
> Otherwise at the next CVE we're back in the same place where downstream
> users aren't meaningfully more protected. And in that case perhaps we would
> do better for our users by e.g. putting more emphasis upgrading across our
> releases or providing breaking changes to get the pain over with.
>
> On Fri, Jan 21, 2022 at 11:03 AM Andrew Purtell 
> wrote:
>
> > That does not seem germane to this discussion. We don’t investigate and
> > attempt to manage the security reporting arrangement of any of our other
> > third party dependencies.
> >
> > > On Jan 21, 2022, at 7:59 AM, Sean Busbey  wrote:
> > >
> > > Has anyone asked the ASF Logging PMC if they'll forward security
> reports
> > > against log4j 1 to the reload4j project?
> > >
> > >> On Fri, Jan 21, 2022 at 3:33 AM Pankaj Kumar 
> > wrote:
> > >>
> > >> +1 for reload4j.
> > >>
> > >> Regards,
> > >> Pankaj
> > >>
> > >>> On Fri, Jan 21, 2022, 2:39 PM 张铎(Duo Zhang) 
> > wrote:
> > >>>
> > >>> Already filed HBASE-26691.
> > >>>
> > >>> Wei-Chiu Chuang  于2022年1月21日周五 16:53写道：
> > >>>
> > >>>> +1 I am doing the same in Hadoop.
> > >>>>
> > >>>> On Fri, Jan 21, 2022 at 4:51 PM Viraj Jasani 
> > >> wrote:
> > >>>>
> > >>>>> +1 for Reload4J migration in active release branches.
> > >>>>>
> > >>>>>
> > >>>>> On Fri, 21 Jan 2022 at 12:52 PM, Andrew Purtell <
> > >>>> andrew.purt...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> +1 for migrating to Reload4J. It is binary and configuration
> > >>> compatible
> > >>>>>> with log4j 1 so meets our compatibility guidelines.
> > >>>>>>
> > >>>>>> If this is an agreeable plan I can make the changes in a PR and we
> > >>> can
> > >>>> do
> > >>>>>> a round of new releases.
> > >>>>>>
> > >>>>>>> On Jan 20, 2022, at 10:16 PM, Duo Zhang 
> > >>> wrote:
> > >>>>>>>
> > >>>>>>> On master we have already migrated to log4j2, but for all other
> > >>>>> release
> > >>>>>>> lines we are still on log4j1.
> > >>>>>>>
> > >>>>>>> Recently there are several new CVEs for log4j1, so I think we
> > >>> should
> > >>>>> also
> > >>>>>>> address them for release lines other than master.
> > >>>>>>>
> > >>>>>>> One possible solution is to also migrate log4j2 but use log4j12
> > >>>> bridge
> > >>>>> to
> > >>>>>>> maintain the compatibility, but we have already known that
> > >> log4j12
> > >>>>> bridge
> > >>>>>>> can not work perfectly with hadoop, as hadoop has some customized
> > >>>>> log4j1
> > >>>>>>> appender implementations, which inherit some log4j1 appenders
> > >> which
> > >>>> are
> > >>>>>> not
> > >>>>>>> part of the log4j12 bridge.
> > >>>>>>>
> > >>>>>>> Reload4j is a fork of the log4j1 and has fixed the critical CVEs,
> > >>> so
> > >>>> it
> > >>>>>> is
> > >>>>>>> less hurt to replace log4j with reload4j.
> > >>>>>>>
> > >>>>>>> Suggestions are welcomed.
> > >>>>>>>
> > >>>>>>> Thanks. Regards
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>

Re: [DISCUSS] Replace log4j with reload4j for branch-2.x

[Wei-Chiu Chuang]

+1 I am doing the same in Hadoop.

On Fri, Jan 21, 2022 at 4:51 PM Viraj Jasani  wrote:

> +1 for Reload4J migration in active release branches.
>
>
> On Fri, 21 Jan 2022 at 12:52 PM, Andrew Purtell 
> wrote:
>
> > +1 for migrating to Reload4J. It is binary and configuration compatible
> > with log4j 1 so meets our compatibility guidelines.
> >
> > If this is an agreeable plan I can make the changes in a PR and we can do
> > a round of new releases.
> >
> > > On Jan 20, 2022, at 10:16 PM, Duo Zhang  wrote:
> > >
> > > On master we have already migrated to log4j2, but for all other
> release
> > > lines we are still on log4j1.
> > >
> > > Recently there are several new CVEs for log4j1, so I think we should
> also
> > > address them for release lines other than master.
> > >
> > > One possible solution is to also migrate log4j2 but use log4j12 bridge
> to
> > > maintain the compatibility, but we have already known that log4j12
> bridge
> > > can not work perfectly with hadoop, as hadoop has some customized
> log4j1
> > > appender implementations, which inherit some log4j1 appenders which are
> > not
> > > part of the log4j12 bridge.
> > >
> > > Reload4j is a fork of the log4j1 and has fixed the critical CVEs, so it
> > is
> > > less hurt to replace log4j with reload4j.
> > >
> > > Suggestions are welcomed.
> > >
> > > Thanks. Regards
> >
>

[jira] [Resolved] (HBASE-22953) Supporting Hadoop 3.3.0

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-22953.
-
Fix Version/s: 2.3.0
   3.0.0-alpha-1
   Resolution: Fixed

> Supporting Hadoop 3.3.0
> ---
>
> Key: HBASE-22953
> URL: https://issues.apache.org/jira/browse/HBASE-22953
> Project: HBase
>  Issue Type: Umbrella
>    Reporter: Wei-Chiu Chuang
>Priority: Major
> Fix For: 2.3.0, 3.0.0-alpha-1
>
>
> The Hadoop community has started to discuss a 3.3.0 release. 
> [http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201908.mbox/%3CCAD%2B%2BeCneLtC%2BkfxRRKferufnNxhaXXGa0YPaVp%3DEBbc-R5JfqA%40mail.gmail.com%3E]
> While still early, it wouldn't hurt to start exploring what's coming in 
> Hadoop 3.3.0. In particular, there are a bunch of new features that brings in 
> all sorts of new dependencies.
>  
> I will use this Jira to list things that are related to Hadoop 3.3.0.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] JIRA Version page only available to authenticated users

[Wei-Chiu Chuang]

Duo,
Just out of curiosity, is this release note in the JIRA populated
automatically? Or someone typed it up?

On Tue, Oct 26, 2021 at 4:25 PM 张铎(Duo Zhang)  wrote:

> I think this is what we should link in the announcement?
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12350545
>
> We should link the release note, not the page where we can generate the
> release note...
>
> Just my thoughts. Thanks.
>
> Nick Dimiduk  于2021年10月26日周二 上午1:19写道：
>
> > Heya,
> >
> > The link we include in our announcement emails is only accessible for
> > people who are logged into JIRA. Seems like bad form.
> >
> > I.e., https://issues.apache.org/jira/projects/HBASE/versions/12350545
> >
> > I doubt this is by design on our part. I wonder if JIRA has always been
> > configured this way, or if it's an (un)intentional change on the part of
> > Infra.
> >
> > Should we instead link to something else?
> >
> > Thanks,
> > Nick
> >
>

Fwd: H Node renaming

[Wei-Chiu Chuang]

It looks like a number of HBase Jenkins jobs are still running on the H
nodes today so forward it to the hbase dev community just in case.

-- Forwarded message -
From: Gavin McDonald 
Date: Tue, Sep 28, 2021 at 1:13 PM
Subject: H Node renaming
To: 


Sent to builds@ . adding here also.]]
Hi All,

I have written a short proposal to rename all the H nodes, both
within Jenkins and also to add ASF DNS for Infra use,

let me know if anyone has any questions or concerns.

Thanks

https://cwiki.apache.org/confluence/display/INFRA/H+node+renaming+Proposal



-- 

*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team

[jira] [Resolved] (HBASE-26160) Configurable disallowlist for live editing of loglevels

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-26160.
-
Resolution: Fixed

Thank you, [~bbeaudreault] for contributing the patch.

> Configurable disallowlist for live editing of loglevels
> ---
>
> Key: HBASE-26160
> URL: https://issues.apache.org/jira/browse/HBASE-26160
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6
>
>
> We currently use log4j/slf4j for audit logging in AccessController. This is 
> convenient but presents a security/compliance risk because we allow 
> live-editing of logLevels via the UI. One can simply set the logger to OFF 
> and then perform actions un-audited.
> We should add a configuration for setting certain log levels to read-only



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-21946) Use ByteBuffer pread instead of byte[] pread in HFileBlock when applicable

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-21946.
-
Resolution: Fixed

> Use ByteBuffer pread instead of byte[] pread in HFileBlock when applicable
> --
>
> Key: HBASE-21946
> URL: https://issues.apache.org/jira/browse/HBASE-21946
> Project: HBase
>  Issue Type: Improvement
>  Components: Offheaping
>Reporter: Zheng Hu
>    Assignee: Wei-Chiu Chuang
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2
>
> Attachments: HBASE-21946.HBASE-21879.v01.patch, 
> HBASE-21946.HBASE-21879.v02.patch, HBASE-21946.HBASE-21879.v03.patch, 
> HBASE-21946.HBASE-21879.v04.patch
>
>
> [~stakiar] is working on HDFS-3246,  so now we have to keep the byte[] pread 
> in HFileBlock reading.  Once it get resolved, we can upgrade the hadoop 
> version and do the replacement. 
> I think it will be a great p999 latency improvement in 100% Get case, anyway 
> file a issue address this firstly. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26049) Remove DfsBuilderUtility

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-26049.
-
Resolution: Fixed

I left the commit in master, because i forgot the replicate() API isn't 
available until Hadoop 3.

> Remove DfsBuilderUtility
> 
>
> Key: HBASE-26049
> URL: https://issues.apache.org/jira/browse/HBASE-26049
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>        Reporter: Wei-Chiu Chuang
>    Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> DfsBuilderUtility was created to reflectively access 
> DistributedFileSystem$HdfsDataOutputStreamBuilder, which was added by 
> HDFS-11170 and available since Hadoop 2.9.0.
> We can remove this class and access the HDFS builder class directly in HBase 
> 3 and 2.5.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New HBase committer Baiqiang Zhao

[Wei-Chiu Chuang]

great to hear that. Congrats!

On Sun, Jul 11, 2021 at 3:18 AM Nick Dimiduk  wrote:

> Hi everyone,
>
> On behalf of the Apache HBase PMC I am pleased to announce that Baiqiang
> Zhao has accepted the PMC's invitation to become a committer on the
> project!
>
> We appreciate all of the great contributions Baiqiang has made to
> the community thus far and we look forward to his continued involvement.
>
> Allow me to be the first to congratulate Baiqiang on his new role!
>
> Thanks,
> Nick
>

Re: [DISCUSS] Contribution of a thrift2 python api

[Wei-Chiu Chuang]

Hi
thanks for your interest in contributing the python api to the HBase
project.

I quickly check and it doesn't look like there's another active python
HBase thrift client project at this point.
I don't have a demand to use a python thrift hbase client library. If there
are people who will benefit from this library, then it's a good idea to
make sure the library is well maintained, by having it become part of the
Apache HBase project and that more developers can contribute to it.

As a hobbyist Python developer I can help review/commit the patch.

My two cents:
(1) license: the code is ASL 2.0 so it's compatible. The text "Copyright
2021 Yutong Sean" would need to be removed.
(2) Apache Infra does not manage PyPi. So we (the Apache HBase project
committers/PMC) will have to do that.
I suspect we will have to replicate this PyPi project and add the
interested HBase PMCs who's willing to do the release work.
(3) compatibility matrix: we need to document what versions of HBase server
is supported.
(3) code:
(3.1) You will need a requirements.txt and preferably specify the versions
of the dependencies.
(3.2) If the community accepts it, should it be part of the HBase main
repo, or a new, separate repo?



On Mon, Jul 5, 2021 at 7:12 PM Yutong Xiao  wrote:

> Hi,
>
> I used to have a demand to deploy hbase thrift2 service for python users.
> So that I developed a python clients API supporting python 2.7 and 3.x for
> hbase thrift2, named thbase  . Besides
> that, I also added some features to current thrift2 service (HBASE-26025
>  and
> HBASE-26037
> ). I
> deployed them in the prod environment of my company and are compatible with
> thbase and I will keep maintaining this python API and add new features.
>   I am glad to contribute thbase to the community, but I am not sure if it
> is possible that such a client could be contributed to the community. So
> that I would like to get some advice about this.
>
> Thanks,
> Yutong Sean
>

[jira] [Resolved] (HBASE-26057) Remove reflections used to access Hadoop 2 API in FanOutOneBlockAsyncDFSOutputHelper

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-26057.
-
Resolution: Fixed

> Remove reflections used to access Hadoop 2 API in 
> FanOutOneBlockAsyncDFSOutputHelper
> 
>
> Key: HBASE-26057
> URL: https://issues.apache.org/jira/browse/HBASE-26057
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Wei-Chiu Chuang
>    Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> Remove the reflections used to access Hadoop 2 APIs in HBase 3.x.
> There are still a number of reflections we can't remove now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26050) Remove the reflection used in FSUtils.isInSafeMode

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-26050.
-
Fix Version/s: 3.0.0-alpha-2
   2.5.0
   Resolution: Fixed

> Remove the reflection used in FSUtils.isInSafeMode
> --
>
> Key: HBASE-26050
> URL: https://issues.apache.org/jira/browse/HBASE-26050
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> DistributedFileSystem.setSafeMode() was added by HDFS-3507 in Hadoop 
> 2.0.3-alpha. No need to access via reflection.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26057) Remove reflections used to access Hadoop 2 API in FanOutOneBlockAsyncDFSOutputHelper

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26057:
---

 Summary: Remove reflections used to access Hadoop 2 API in 
FanOutOneBlockAsyncDFSOutputHelper
 Key: HBASE-26057
 URL: https://issues.apache.org/jira/browse/HBASE-26057
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang


Remove the reflections used to access Hadoop 2 APIs in HBase 3.x.
There are still a number of reflections we can't remove now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26051) Remove reflections used to access HDFS EC APIs

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26051:
---

 Summary: Remove reflections used to access HDFS EC APIs
 Key: HBASE-26051
 URL: https://issues.apache.org/jira/browse/HBASE-26051
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 3.0.0-alpha-1
Reporter: Wei-Chiu Chuang


HDFS EC APIs exists since Hadoop 3.0.
We can access them directly without reflections in HBase 3.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26050) Remove the reflection used in FSUtils.isInSafeMode

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26050:
---

 Summary: Remove the reflection used in FSUtils.isInSafeMode
 Key: HBASE-26050
 URL: https://issues.apache.org/jira/browse/HBASE-26050
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang


DistributedFileSystem.setSafeMode() was added by HDFS-3507 in Hadoop 
2.0.3-alpha. No need to access via reflection.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26049) Remove DfsBuilderUtility

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26049:
---

 Summary: Remove DfsBuilderUtility
 Key: HBASE-26049
 URL: https://issues.apache.org/jira/browse/HBASE-26049
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 3.0.0-alpha-1, 2.5.0
Reporter: Wei-Chiu Chuang


DfsBuilderUtility was created to reflectively access 
DistributedFileSystem$HdfsDataOutputStreamBuilder, which was added by 
HDFS-11170 and available since Hadoop 2.9.0.

We can remove this class and access the HDFS builder class directly in HBase 3 
and 2.5.0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26048) [JDK17] Replace the usage of deprecated API ThreadGroup.destroy()

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26048:
---

 Summary: [JDK17] Replace the usage of deprecated API 
ThreadGroup.destroy()
 Key: HBASE-26048
 URL: https://issues.apache.org/jira/browse/HBASE-26048
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang


According to the JDK17 doc, ThreadGroup.destroy() is deprecated because
{quote}Deprecated, for removal: This API element is subject to removal in a 
future version.
{quote}
The API and mechanism for destroying a ThreadGroup is inherently flawed. The 
ability to explicitly or automatically destroy a thread group will be removed 
in a future release.

[https://download.java.net/java/early_access/jdk17/docs/api/java.base/java/lang/ThreadGroup.html#destroy(])

We don't necessarily need to remove this usage now, but the warning sounds bad 
enough.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26047) Track JDK17 unit test failures

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26047:
---

 Summary: Track JDK17 unit test failures
 Key: HBASE-26047
 URL: https://issues.apache.org/jira/browse/HBASE-26047
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang


As of now, there are still two failed unit tests after exporting JDK internal 
modules and the modifier access hack.

{noformat}

[ERROR] Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.217 s 
<<< FAILURE! - in org.apache.hadoop.hbase.io.TestHeapSize
[ERROR] org.apache.hadoop.hbase.io.TestHeapSize.testSizes  Time elapsed: 0.041 
s  <<< FAILURE!
java.lang.AssertionError: expected:<160> but was:<152>
at 
org.apache.hadoop.hbase.io.TestHeapSize.testSizes(TestHeapSize.java:335)

[ERROR] org.apache.hadoop.hbase.io.TestHeapSize.testNativeSizes  Time elapsed: 
0.01 s  <<< FAILURE!
java.lang.AssertionError: expected:<72> but was:<64>
at 
org.apache.hadoop.hbase.io.TestHeapSize.testNativeSizes(TestHeapSize.java:134)

[INFO] Running org.apache.hadoop.hbase.io.Tes


[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.697 s 
<<< FAILURE! - in org.apache.hadoop.hbase.ipc.TestBufferChain
[ERROR] org.apache.hadoop.hbase.ipc.TestBufferChain.testWithSpy  Time elapsed: 
0.537 s  <<< ERROR!
java.lang.NullPointerException: Cannot enter synchronized block because 
"this.closeLock" is null
at 
org.apache.hadoop.hbase.ipc.TestBufferChain.testWithSpy(TestBufferChain.java:119)


{noformat}

It appears that JDK17 makes the heap size estimate different than before. Not 
sure why.

TestBufferChain.testWithSpy  failure might be because of yet another unexported 
module.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26046) Add a JDK17 profile

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26046:
---

 Summary: Add a JDK17 profile
 Key: HBASE-26046
 URL: https://issues.apache.org/jira/browse/HBASE-26046
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang


While HBase builds fine with JDK17, tests fail because a number of Java SDK 
modules are no longer exposed to unnamed modules by default. We need to open 
them up.

Without which, the tests fail for errors like:
{noformat}
[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.469 s 
<<< FAILURE! - in org.apache.hadoop.hbase.rest.model.TestNamespacesModel
[ERROR] org.apache.hadoop.hbase.rest.model.TestNamespacesModel.testBuildModel  
Time elapsed: 0.273 s  <<< ERROR!
java.lang.ExceptionInInitializerError
at 
org.apache.hadoop.hbase.rest.model.TestNamespacesModel.(TestNamespacesModel.java:43)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make 
protected final java.lang.Class 
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws 
java.lang.ClassFormatError accessible: module java.base does not "opens 
java.lang" to unnamed module @56ef9176
at 
org.apache.hadoop.hbase.rest.model.TestNamespacesModel.(TestNamespacesModel.java:43)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26041) Replace PrintThreadInfoLazyHolder's reflection usage

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26041:
---

 Summary: Replace PrintThreadInfoLazyHolder's reflection usage
 Key: HBASE-26041
 URL: https://issues.apache.org/jira/browse/HBASE-26041
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang


PrintThreadInfoLazyHolder uses reflection to access Hadoop's 
ReflectionUtils.printThreadInfo(). Replace it with HBase's 
ReflectionUtils.printThreadInfo().



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26040) Replace reflections that are redundant

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26040:
---

 Summary: Replace reflections that are redundant
 Key: HBASE-26040
 URL: https://issues.apache.org/jira/browse/HBASE-26040
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


A number of reflections were used to access (back in the time) new Hadoop APIs 
that were only available in newer Hadoop versions.

Some of them are no longer needed with the default Hadoop dependency 3.1.2, so 
they can be removed to avoid the brittle code. Also, makes it possible to 
determine compile time dependency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26038) Support JDK17

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26038:
---

 Summary: Support JDK17
 Key: HBASE-26038
 URL: https://issues.apache.org/jira/browse/HBASE-26038
 Project: HBase
  Issue Type: New Feature
Reporter: Wei-Chiu Chuang


JDK 17 is the next Java LTS, coming out this September.

It brings a number of goodies. One of which is the production-ready ZGC 
(available since JDK15). (as well as the Shenandoah GC, available since JDK15)

After September 2021, there will be three Java LTS versions: Java 8, Java 11 
and Java 17. Java 8 will still be the mainstream SDK in the foreseeable future, 
so I am not looking to take advantage of the new APIs that are only available 
in JDK17. This jira aims to support HBase on all three JDK LTS.

Porting HBase to JDK17 is not a big hurdle. HBase (master branch) builds 
successfully on JDK17. A few tests fail mostly due to the new (more strict) 
Java module isolation enforcement. I have a small PoC that I will post here in 
the coming days.

What I am trying to achieve is to add an experimental support for JDK17. It 
will be interesting to benchmark HBase on ZGC and Shenandoah and determine if 
we should set our default GC to them. By then, we'll be able to claim 
production-ready support.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-23817) The message "Please make sure that backup is enabled on the cluster." is shown even when the backup feature is enabled

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-23817.
-
Fix Version/s: 3.0.0-alpha-1
   Resolution: Fixed

Thanks for the review [~brfrn169] merged the PR.

> The message "Please make sure that backup is enabled on the cluster." is 
> shown even when the backup feature is enabled
> --
>
> Key: HBASE-23817
> URL: https://issues.apache.org/jira/browse/HBASE-23817
> Project: HBase
>  Issue Type: Bug
>Reporter: Toshihiro Suzuki
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
> The following message is shown even when the backup feature is enabled, which 
> is confusing:
> {code}
> Please make sure that backup is enabled on the cluster. To enable backup, in 
> hbase-site.xml, set:
>  hbase.backup.enable=true
> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner
> hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager
> hbase.coprocessor.region.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.BackupObserver
> and restart the cluster
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26032) Make HRegion.getStores() an O(1) operation

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26032:
---

 Summary: Make HRegion.getStores() an O(1) operation
 Key: HBASE-26032
 URL: https://issues.apache.org/jira/browse/HBASE-26032
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang
 Attachments: Screen Shot 2021-06-24 at 3.56.33 PM.png

This is a relatively minor issue, but I did spot HRegion.getStores() popping up 
in my profiler.

Checking the code, I realized that HRegion.getStores() allocates a new array 
list in it, converting the Collection<> to List<>. But it also makes it an O( n 
) in space and time complexity.

This conversion appears mostly unnecessary, because we only iterate the stores 
in production code, and so the new ArrayList object is thrown away immediately. 
Only in a number of test code where we index into the stores.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26019) Remove reflections used in HBaseConfiguration.getPassword()

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-26019.
-
Resolution: Fixed

Thanks again for the review, [~wchevreuil] [~vjasani] and [~tomscut]

The commit is cherrypicked to branch-2. Let me know if it should go to lower 
branches.

> Remove reflections used in HBaseConfiguration.getPassword()
> ---
>
> Key: HBASE-26019
> URL: https://issues.apache.org/jira/browse/HBASE-26019
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> HBaseConfiguration.getPassword() uses Hadoop API Configuration.getPassword(). 
>  The API was added in Hadoop 2.6.0. Reflection was used to access the API. 
> It's time to remove the reflection and invoke the API directly. (HBase 3.0 as 
> well as 2.x too)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: HBase

[Wei-Chiu Chuang]

What's your jira user id? I can help you added to the contributor list and
then you'll be able to assign jiras.

On Tue, Jun 22, 2021 at 10:37 AM 曹明 <13760436...@163.com> wrote:

> How can I assign the issue to me?

[jira] [Created] (HBASE-26019) Remove reflections used in HBaseConfiguration.getPassword()

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-26019:
---

 Summary: Remove reflections used in 
HBaseConfiguration.getPassword()
 Key: HBASE-26019
 URL: https://issues.apache.org/jira/browse/HBASE-26019
 Project: HBase
  Issue Type: Improvement
Reporter: Wei-Chiu Chuang


HBaseConfiguration.getPassword() uses Hadoop API Configuration.getPassword().  
The API was added in Hadoop 2.6.0. Reflection was used to access the API. It's 
time to remove the reflection and invoke the API directly. (HBase 3.0 as well 
as 2.x too)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-25861) Correct the usage of Configuration#addDeprecation

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reopened HBASE-25861:
-

Sorry to reopen. I think we need to understand the behavior change better.

CC: [~snemeth] looks like we have a problem where HBase depends on the old 
behavior of Hadoop's Configuration class prior to HADOOP-15708.

> Correct the usage of Configuration#addDeprecation
> -
>
> Key: HBASE-25861
> URL: https://issues.apache.org/jira/browse/HBASE-25861
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When I was solving HBASE-25745 
> ([PR3139|https://github.com/apache/hbase/pull/3139]), I found that our use of 
> Configuration#addDeprecation API was wrong. 
>  
> At present, we will call Configuration#addDeprecation in the static block for 
> the deprecated configuration. But after testing, it is found that this does 
> not complete backward compatibility. When user upgrades HBase and does not 
> change the deprecated configuration to the new configuration, he will find 
> that the deprecated configuration does not effect, which may not be 
> consistent with expectations. The specific test results can be seen in the PR 
> above, and we can found the calling order of Configuration#addDeprecation is 
> very important.
>  
> Configuration#addDeprecation is a Hadoop API, looking through the Hadoop 
> source code, we will find that before creating the Configuration object, the 
> addDeprecatedKeys() method will be called first: 
> [https://github.com/apache/hadoop/blob/b93e448f9aa66689f1ce5059f6cdce8add130457/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java#L34]
>  .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25928) TestHBaseConfiguration#testDeprecatedConfigurations is broken with Hadoop 3.3

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25928:
---

 Summary: TestHBaseConfiguration#testDeprecatedConfigurations is 
broken with Hadoop 3.3
 Key: HBASE-25928
 URL: https://issues.apache.org/jira/browse/HBASE-25928
 Project: HBase
  Issue Type: Bug
Affects Versions: 3.0.0-alpha-1, 2.5.0
Reporter: Wei-Chiu Chuang


The test TestHBaseConfiguration#testDeprecatedConfigurations was added recently 
by HBASE-25861 to address the usage of Hadoop Configuration addDeprecations API.

However, the API's behavior was changed to fix a bug.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25919) Support Hadoop 3.3.1

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25919:
---

 Summary: Support Hadoop 3.3.1
 Key: HBASE-25919
 URL: https://issues.apache.org/jira/browse/HBASE-25919
 Project: HBase
  Issue Type: Task
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


The Hadoop 3.3.1 is a big release, quite different from 3.3.0.

File this jira to track the support for Hadoop 3.3.1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25920) Support Hadoop 3.3.1

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25920:
---

 Summary: Support Hadoop 3.3.1
 Key: HBASE-25920
 URL: https://issues.apache.org/jira/browse/HBASE-25920
 Project: HBase
  Issue Type: Task
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


The Hadoop 3.3.1 is a big release, quite different from 3.3.0.

File this jira to track the support for Hadoop 3.3.1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25908) Exclude jakarta.activation-api

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25908:
---

 Summary: Exclude jakarta.activation-api
 Key: HBASE-25908
 URL: https://issues.apache.org/jira/browse/HBASE-25908
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: Wei-Chiu Chuang


Hadoop 3.3.1 replaced its dependency of javax.activation 1.2.0 with 
jakarta.activation 1.2.1.

They are essentially the same thing (they even have the same classpath name), 
but Eclipse took over JavaEE development and therefore changed group/artifact 
id. 
(https://stackoverflow.com/questions/46493613/what-is-the-replacement-for-javax-activation-package-in-java-9)

See HADOOP-17049 for more details. Hadoop 3.3.0 updated jackson-databind to 
2.10 which shades jakarta.activation, causing classpath conflict.

The solution to this issue will be similar to HBASE-22268



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Adding Github autolink to JIRA

[Wei-Chiu Chuang]

This is now enabled. Thanks all

On Tue, Apr 13, 2021 at 3:30 PM 张铎(Duo Zhang)  wrote:

> The screen shots are broken but I'm +1 on enabling it.
>
> Wei-Chiu Chuang  于2021年4月13日周二 下午3:20写道：
>
> >
> > A number of Apache projects have enabled "GitHub autolink". What it does
> > is convert strings "HBASE-1234" in GitHub to a link to the corresponding
> > Apache JIRA.
> >
> > [image: Screen Shot 2021-04-13 at 3.15.23 PM.png]
> >
> > [image: Screen Shot 2021-04-13 at 3.18.27 PM.png]
> > I find it quite useful. Hence the ask to enable it for HBase.
> > If the community likes to proceed, I can file an INFRA jira to enable it.
> >
>

[jira] [Created] (HBASE-25771) Recommend Hadoop 3.x

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25771:
---

 Summary: Recommend Hadoop 3.x
 Key: HBASE-25771
 URL: https://issues.apache.org/jira/browse/HBASE-25771
 Project: HBase
  Issue Type: Task
Reporter: Wei-Chiu Chuang


We have this section in the hbase book:
{quote}
Hadoop 2.x is recommended.
Hadoop 2.x is faster and includes features, such as short-circuit reads (see 
Leveraging local data), which will help improve your HBase random read profile. 
Hadoop 2.x also includes important bug fixes that will improve your overall 
HBase experience. HBase does not support running with earlier versions of 
Hadoop. See the table below for requirements specific to different HBase 
versions.

Hadoop 3.x is still in early access releases and has not yet been sufficiently 
tested by the HBase community for production use cases.
{quote}

The Hadoop 2.x development is winding down. 2.10.x will be the last dev branch. 
On the contrary, we've got years of production users on Hadoop 3.x already so I 
think it's time to change our stance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New HBase PMC Huaxiang Sun

[Wei-Chiu Chuang]

Congrats!

On Tue, Apr 13, 2021 at 3:39 PM Viraj Jasani  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Huaxiang
> Sun has accepted our invitation to become a PMC member on the HBase
> project. We appreciate Huaxiang stepping up to take more responsibility for
> the project.
>
> Congratulations and welcome, Huaxiang!
>

[DISCUSS] Adding Github autolink to JIRA

[Wei-Chiu Chuang]

A number of Apache projects have enabled "GitHub autolink". What it does is
convert strings "HBASE-1234" in GitHub to a link to the corresponding
Apache JIRA.

[image: Screen Shot 2021-04-13 at 3.15.23 PM.png]

[image: Screen Shot 2021-04-13 at 3.18.27 PM.png]
I find it quite useful. Hence the ask to enable it for HBase.
If the community likes to proceed, I can file an INFRA jira to enable it.

[jira] [Resolved] (HBASE-25472) HBASE-24260 breaks JDK7 compilation in branch-1

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-25472.
-
Resolution: Duplicate

Oh yeah you're right. My local tree wasn't updated. Close this as a dup of 
HBASE-24620.

> HBASE-24260 breaks JDK7 compilation in branch-1
> ---
>
> Key: HBASE-25472
> URL: https://issues.apache.org/jira/browse/HBASE-25472
> Project: HBase
>  Issue Type: Bug
>    Reporter: Wei-Chiu Chuang
>Priority: Blocker
>
> Found in PR-2799. 
> {code}
> public static String getZKQuorum(Configuration conf) {
> String port =
>   Integer.toString(conf.getInt(HConstants.ZOOKEEPER_CLIENT_PORT, 2181));
> String[] serverHosts = conf.getStrings(HConstants.ZOOKEEPER_QUORUM, 
> "localhost");
> for (int i = 0; i < serverHosts.length; i++) {
>   serverHosts[i] = serverHosts[i] + ":" + port;
> }
> return String.join(",", serverHosts);
>   }
> {code}
> Fails to compile 
> {quote}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile 
> (default-compile) on project hbase-it: Compilation failure
> [ERROR] 
> /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2799/src/hbase-it/src/main/java/org/apache/hadoop/hbase/chaos/ChaosUtils.java:[46,18]
>  cannot find symbol
> [ERROR] symbol:   method join(java.lang.String,java.lang.String[])
> [ERROR] location: class java.lang.String
> {quote}
> because String.join() is only available since JDK8.
> CC: [~lokiore] [~ndimiduk]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25472) HBASE-24260 breaks JDK7 compilation in branch-1

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25472:
---

 Summary: HBASE-24260 breaks JDK7 compilation in branch-1
 Key: HBASE-25472
 URL: https://issues.apache.org/jira/browse/HBASE-25472
 Project: HBase
  Issue Type: Bug
Reporter: Wei-Chiu Chuang


Found in PR-2799. 

{code}
public static String getZKQuorum(Configuration conf) {
String port =
  Integer.toString(conf.getInt(HConstants.ZOOKEEPER_CLIENT_PORT, 2181));
String[] serverHosts = conf.getStrings(HConstants.ZOOKEEPER_QUORUM, 
"localhost");
for (int i = 0; i < serverHosts.length; i++) {
  serverHosts[i] = serverHosts[i] + ":" + port;
}
return String.join(",", serverHosts);
  }
{code}

Fails to compile 
{quote}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) 
on project hbase-it: Compilation failure
[ERROR] 
/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2799/src/hbase-it/src/main/java/org/apache/hadoop/hbase/chaos/ChaosUtils.java:[46,18]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
{quote}

because String.join() is only available since JDK8.

CC: [~lokiore] [~ndimiduk]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25405) Backport HBASE-25287 to branch-1

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-25405.
-
Resolution: Fixed

> Backport HBASE-25287 to branch-1
> 
>
> Key: HBASE-25405
> URL: https://issues.apache.org/jira/browse/HBASE-25405
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 1.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25405) Backport HBASE-25287 to branch-1

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25405:
---

 Summary: Backport HBASE-25287 to branch-1
 Key: HBASE-25405
 URL: https://issues.apache.org/jira/browse/HBASE-25405
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25344) Detector

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-25344.
-
Resolution: Duplicate

> Detector
> 
>
> Key: HBASE-25344
> URL: https://issues.apache.org/jira/browse/HBASE-25344
> Project: HBase
>  Issue Type: Bug
>Reporter: junwen yang
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25262) Update jetty version in hbase-thirdparty

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-25262:
---

 Summary: Update jetty version in hbase-thirdparty
 Key: HBASE-25262
 URL: https://issues.apache.org/jira/browse/HBASE-25262
 Project: HBase
  Issue Type: Improvement
  Components: hbase-thirdparty
Reporter: Wei-Chiu Chuang


The master branch of hbase-thirdparty is on Jetty 9.4.31. The latest 9.4.x is 
Jetty 9.4.34. We should update to the latest.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Please welcome Viraj Jasani to the Apache HBase PMC

[Wei-Chiu Chuang]

Congratulations!

On Tue, Oct 6, 2020 at 5:18 AM 张铎(Duo Zhang)  wrote:

> Congratulations!
>
> zheng wang <18031...@qq.com> 于2020年10月6日周二 下午8:00写道：
>
> > CongratulationsViraj!
> >
> >
> >
> >
> > ---Original---
> > From: "Andrew Purtell" > Date: Tue, Oct 6, 2020 00:58 AM
> > To: "Hbase-User" ;
> > Subject: [ANNOUNCE] Please welcome Viraj Jasani to the Apache HBase PMC
> >
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that
> > Viraj Jasani has accepted our invitation to become a PMC member on the
> > HBase project. We appreciate Viraj stepping up to take more
> > responsibility for the project.
> >
> > Please join me in welcoming Viraj to the HBase PMC!
> >
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@hbase.apache.org
> > to let us know.
> >
> > --
> > Best regards,
> > Andrew
>

Re: [ANNOUNCE] New HBase Committer Zheng Wang(王正)

[Wei-Chiu Chuang]

Congratulations!

On Thu, Sep 24, 2020 at 12:35 AM Guanghao Zhang  wrote:

> Congratulations and welcome!
>
> Viraj Jasani  于2020年9月24日周四 下午3:24写道：
>
> > Congratulations Zheng Wang. Very well deserved !
> >
> > On 2020/09/24 02:24:16, 张铎(Duo Zhang)  wrote:
> > > On behalf of the Apache HBase PMC, I am pleased to announce that Zheng
> > Wang
> > > has accepted the PMC's invitation to become a committer on the project.
> > We
> > > appreciate all of Zheng's generous contributions thus far and look
> > forward
> > > to his continued involvement.
> > >
> > > Congratulations and welcome, Zheng Wang!
> > >
> > > 我很高兴代表Apache HBase PMC宣布王正已接受我们的邀请，成为Apache
> > > HBase项目的Committer。感谢王正一直以来为HBase项目做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎王正！
> > >
> >
>

[jira] [Resolved] (HBASE-24209) [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-24209.
-
Resolution: Won't Fix

> [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality
> -
>
> Key: HBASE-24209
> URL: https://issues.apache.org/jira/browse/HBASE-24209
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Nick Dimiduk
>    Assignee: Wei-Chiu Chuang
>Priority: Major
>
> Since HBASE-23833 we're paying attention to our builds on Hadoop trunk, 
> currently 3.3.0-SNAPSHOT. Let's add this version to the version lists in 
> hadoopcheck so our CI will let us know when things break, at least 
> compile-time anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] HBASE-20904 Prometheus /metrics http endpoint for monitoring integration

[Wei-Chiu Chuang]

Might be useful to look at the Hadoop Prometheus end point and Ozone
Prometheus end point as well.

   1. HADOOP-16398 


   1. HDDS-919 


A known issue with the Hadoop Prometheus implementation is that it lacks
SPENGO support, so there is a problem integrating it with the Kerberized
environment. The solution is to whitelist the Prometheus end point so it
doesn't require SPNEGO.


On Sat, Jun 6, 2020 at 10:00 PM ckai2480  wrote:

> Hi,
>
> I am working on HBASE-20904 (Prometheus /metrics http endpoint for
> monitoring integration) and have created a patch (
> https://github.com/apache/hbase/pull/1814). @busbey and @saintstack
> suggested some good changes which I have incorporated in my local
> branch and wanted to know other's suggestions on it before I commit to
> the remote.
>
> Prometheus
> 1. Prometheus is a monitoring software which uses HTTP to pull the
> metrics from the monitored processes.
> 2. The collected data can be used to anamoly detection, altering etc.
>
> Problem HBASE-20904 solves.
> 1. Implementing a servlet to expose these metrics.
>
> Currently I have implemented this as follows
> 1. Expose two end points
>   /prometheus:
>   exposes the metrics captured in native hbase metrics
>   /prometheus-hadoop2:
>   exposes the metrics captured using hadoop2 metrics.
>
> latter is planned to be removed once all the metric sources start
> using the native metrics API.
>
> **do the endpoints names look ok?**
>
> 2. Make the /jmx, /prometheus-hadoop2, /prometheus, /metrics servlets
> optional by providing a multivalued config key with the first two
> servlets as default (values will be classnames or aliases **which one
> do you think is good idea**).
>
> Out of curiosity.. Why the hadoop2 metrics still exists in HBase, I see
> HBASE-14282 is open and task issues linked are closed. Are there any
> unlinked issues here?
>
> Thanks,
> Madhusoodan
>
>

[jira] [Resolved] (HBASE-24225) Backport HBASE-23833 to branch-2.2

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-24225.
-
Fix Version/s: 2.2.5
   Resolution: Fixed

Merged in https://github.com/apache/hbase/pull/1567

> Backport HBASE-23833 to branch-2.2
> --
>
> Key: HBASE-24225
> URL: https://issues.apache.org/jira/browse/HBASE-24225
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 2.2.5
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24237) Backport HBASE-23998 to branch-2.2

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-24237.
-
Fix Version/s: 2.2.5
   Resolution: Fixed

This was merged by https://github.com/apache/hbase/pull/1568

> Backport HBASE-23998 to branch-2.2
> --
>
> Key: HBASE-24237
> URL: https://issues.apache.org/jira/browse/HBASE-24237
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Wei-Chiu Chuang
>        Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 2.2.5
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24236) Backport HBASE-22103 to branch-2.2

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-24236.
-
Fix Version/s: 2.2.5
   Resolution: Fixed

This was merged by https://github.com/apache/hbase/pull/1566

> Backport HBASE-22103 to branch-2.2
> --
>
> Key: HBASE-24236
> URL: https://issues.apache.org/jira/browse/HBASE-24236
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3, wal
>        Reporter: Wei-Chiu Chuang
>    Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 2.2.5
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Please vote on the first hbase-thirdparty-3.3.0 release candidate

[Wei-Chiu Chuang]

I ran OWASP dependency-check and found no known CVEs.

Nit:
The NOTICE.txt states "Copyright 2018". Should it update to "Copyright
2018-2020"?


On Tue, Apr 28, 2020 at 8:03 AM 张铎(Duo Zhang)  wrote:

> Just change the releasenote.md and make a RC1? We can not republish 3.2.0
> but since the release note for 3.3.0 contains the release note of 3.2.0, we
> can fix it.
>
> Peter Somogyi  于2020年4月28日周二 下午6:04写道：
>
> > What is your suggestion to fix the release notes for 3.2.0? That release
> > was already shipped and unfortunately this issue was not noticed on its
> > vote process.
> >
> > On Tue, Apr 28, 2020 at 9:58 AM 张铎(Duo Zhang) 
> > wrote:
> >
> > > The release note of the hbase-thirdparty-3.2.0 section has a mistake
> > >
> > > * [HBASE-23718](https://issues.apache.org/jira/browse/HBASE-23718) |
> > > *Major* | **[hbase-thirdparty] Update libs; pb from 3.9 to 3.11,
> > > etc.**
> > >
> > > gson: 2.8.5 =\> 2.8.6
> > > guava: 28.1-jre =\> 28.2-jre
> > > error\_prone: 2.3.3 =\> 2.3.4
> > > netty: 4.1.42.Final =\> 4.1.44.Final
> > > protobuf: 3.9.2 =\> 3.11.1
> > > maven-assembly-plugin: 3.1.1 =\> 3.2.0
> > >
> > >
> > > Actually we reverted gson back to 2.8.5, and now the release note on
> jira
> > > is
> > >
> > > guava: 28.1-jre => 28.2-jre
> > > error_prone: 2.3.3 => 2.3.4
> > > netty: 4.1.42.Final => 4.1.44.Final
> > > protobuf: 3.9.2 => 3.11.1
> > > maven-assembly-plugin: 3.1.1 => 3.2.0
> > >
> > >
> > > Peter Somogyi  于2020年4月25日周六 下午10:58写道：
> > >
> > > > Please vote on this Apache hbase thirdparty release candidate,
> > > > hbase-thirdparty-3.3.0RC0
> > > >
> > > > The VOTE will remain open for at least 72 hours.
> > > >
> > > > [ ] +1 Release this package as Apache hbase thirdparty 3.3.0
> > > > [ ] -1 Do not release this package because ...
> > > >
> > > > The tag to be voted on is 3.3.0RC0:
> > > >
> > > > https://github.com/apache/hbase-thirdparty/tree/3.3.0RC0
> > > >
> > > > The release files, including signatures, digests, as well as
> CHANGES.md
> > > > and RELEASENOTES.md included in this RC can be found at:
> > > >
> > > >  https://dist.apache.org/repos/dist/dev/hbase/3.3.0RC0/
> > > >
> > > > Maven artifacts are available in a staging repository at:
> > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1388/
> > > >
> > > > Artifacts were signed with the psomo...@apache.org key which can be
> > > found
> > > > in:
> > > >
> > > >  https://dist.apache.org/repos/dist/release/hbase/KEYS
> > > >
> > > >  To learn more about apache hbase thirdparty, please see
> > > > http://hbase.apache.org/
> > > >
> > > > Thanks,
> > > > Your HBase Release Manager
> > > >
> > >
> >
>

[jira] [Created] (HBASE-24258) [Hadoop3.3] Update license for org.ow2.asm:*

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24258:
---

 Summary: [Hadoop3.3] Update license for org.ow2.asm:*
 Key: HBASE-24258
 URL: https://issues.apache.org/jira/browse/HBASE-24258
 Project: HBase
  Issue Type: Task
  Components: dependencies
Reporter: Wei-Chiu Chuang


Hadoop 3.3 brings a few Jetty dependencies which transitively brings in 
org.ow2.asm:asm-analysis, org.ow2.asm:asm-commons, org.ow2.asm:asm-tree.

When testing with the latest Jetty (9.4.26.v20200117) I found its org.ow2.asm:* 
updated from 7.1 to 7.2, which changed the declared license from "BSD" to 
"BSD-3-Clause License" (The actual license text did not change). The HBase's 
license checker doesn't accept it.

File the jira to update it to "BSD 3-Clause License" so that HBase can build.
{noformat}
[INFO] |  |  |  +- 
org.eclipse.jetty.websocket:javax-websocket-server-impl:jar:9.4.26.v20200117:test
[INFO] |  |  |  |  +- 
org.eclipse.jetty:jetty-annotations:jar:9.4.26.v20200117:test
[INFO] |  |  |  |  |  +- org.eclipse.jetty:jetty-plus:jar:9.4.26.v20200117:test
[INFO] |  |  |  |  |  |  \- 
org.eclipse.jetty:jetty-jndi:jar:9.4.26.v20200117:test
[INFO] |  |  |  |  |  \- org.ow2.asm:asm-commons:jar:7.2:test
[INFO] |  |  |  |  | +- org.ow2.asm:asm-tree:jar:7.2:test
[INFO] |  |  |  |  | \- org.ow2.asm:asm-analysis:jar:7.2:test
{noformat}
{noformat}
This product includes asm-analysis licensed under the BSD-3-Clause.

ERROR: Please check  this License for acceptability here:

https://www.apache.org/legal/resolved

If it is okay, then update the list named 'non_aggregate_fine' in the 
LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.

More info on the dependency:

org.ow2.asm
asm-analysis
7.2
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24240) TestDelegationTokenWithEncryption always fails

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24240:
---

 Summary: TestDelegationTokenWithEncryption always fails
 Key: HBASE-24240
 URL: https://issues.apache.org/jira/browse/HBASE-24240
 Project: HBase
  Issue Type: Bug
  Components: security, test
Affects Versions: 3.0.0
Reporter: Wei-Chiu Chuang


TestDelegationTokenWithEncryption and TestGenerateDelegationToken _always_ fail.

 

Incidentally, they don't fail in branch-2.3 and branch-2.2.

 

I suspect there's a regression with delegation token code, because if I comment 
out the following code in the test, they pass:

 
{code:java}
try (Connection conn = 
ConnectionFactory.createConnection(TEST_UTIL.getConfiguration())) {
  Token token = TokenUtil.obtainToken(conn);
  UserGroupInformation.getCurrentUser().addToken(token);
}
{code}
Effectively, use Kerberos to login instead of delegation token.

The tests fail all the time (100%) in the last 29 runs: 
 
[https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/dashboard.html]

Initially I thought this was caused by pluggable authentication (HBASE-23347), 
but the tests don't fail in branch-2.3 so looks unlikely.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24238) Remove unused $hadoop.guava.version property from pom file

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24238:
---

 Summary: Remove unused $hadoop.guava.version property from pom file
 Key: HBASE-24238
 URL: https://issues.apache.org/jira/browse/HBASE-24238
 Project: HBase
  Issue Type: Task
Reporter: Wei-Chiu Chuang


HBASE-24170 removed hadoop-2.0 profile and hadoop.guava.version is therefore 
not used afterwards.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24237) Backport HBASE-23998 to branch-2.2

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24237:
---

 Summary: Backport HBASE-23998 to branch-2.2
 Key: HBASE-24237
 URL: https://issues.apache.org/jira/browse/HBASE-24237
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24236) Backport HBASE-22103 to branch-2.2

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24236:
---

 Summary: Backport HBASE-22103 to branch-2.2
 Key: HBASE-24236
 URL: https://issues.apache.org/jira/browse/HBASE-24236
 Project: HBase
  Issue Type: Sub-task
  Components: hadoop3, wal
Reporter: Wei-Chiu Chuang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Hadoop 3.3 support

[Wei-Chiu Chuang]

Hi there!

Want to give a heads-up to the work I'm doing: supporting Hadoop 3.3.0.

There are a few commits in the master branch:
HBASE-22103
HBASE-23861
HBASE-23833
HBASE-23998

I plan to backport the 5 HBASE commits mentioned above to lower branches:
branch-2, branch-2.3. Are there any concerns backporting them to branch-2.2?

After these commits, the master branch can build against Hadoop
3.3.0-SNAPSHOT (a RC is about to be voted) with

maven clean install -Dhadoop-three.version=3.3.0-SNAPSHOT
-Djetty.version=9.4.20.v20190813

Note the jetty version must be updated too otherwise unit tests will fail
due to classpath conflict. The build will be slightly easier after
HBASE-23811  and
HBASE-23834  when HBase
uses jetty from hbase-thirdparty.

JDK8 only. I am yet to test JDK11.

I managed to run tests via mvn test, so this includes SmallTests and
MediumTests. A few UTs failed but they failed even with Hadoop 3.1.2. So
may be my local env issue.

If you are interested, here's the list of failed UTs.
(1) TestDelegationTokenWithEncryption,TestGenerateDelegationToken --> It
appears to be a bug in delegation token. reproducible with Hadoop 3.1.2 too
in master branch.
(2) TestZooKeeper.testMasterZKSessionRecoveryFailure --> not clear why but
reproducible with Hadoop 3.1.2 in master branch.
(3) TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness -->
OOM. Passed after increasing heap size.
(4) TestRegionServerHostname.testRegionServerHostname --> multi-home. It
failed on my virtualbox interface. appears to be a local env issue.
(5) TestAlwaysStandByHMaster.testAlwaysStandBy --> reproducible in
branch-2.3 but not in branch-2.2. passed if I increase timeout.

I will file jiras for these test failures.

[jira] [Created] (HBASE-24225) Backport HBASE-23833 to branch-2.2

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24225:
---

 Summary: Backport HBASE-23833 to branch-2.2
 Key: HBASE-24225
 URL: https://issues.apache.org/jira/browse/HBASE-24225
 Project: HBase
  Issue Type: Sub-task
Reporter: Wei-Chiu Chuang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24124) hbase-filesystem to use guava from hbase-thirdparty

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-24124.
-
Resolution: Fixed

Thanks the review from [~tamaas] and [~busbey]!

> hbase-filesystem to use guava from hbase-thirdparty
> ---
>
> Key: HBASE-24124
> URL: https://issues.apache.org/jira/browse/HBASE-24124
> Project: HBase
>  Issue Type: Task
>  Components: Filesystem Integration
>Affects Versions: 1.0.0-alpha1
>    Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 1.0.0-alpha2
>
>
> hbase-filesystem repo is on guava23.0:
> {noformat}
> $ grep -r "guava" .
> ./pom.xml:23.0
> ./hbase-oss/pom.xml:  com.google.guava
> ./hbase-oss/pom.xml:  guava
> ./hbase-oss/pom.xml:  ${guava.version}
> ./hbase-oss/pom.xml:  

[jira] [Created] (HBASE-24125) hbase-filesystem to use hbase-thirdparty 3.2.0

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24125:
---

 Summary: hbase-filesystem to use hbase-thirdparty 3.2.0
 Key: HBASE-24125
 URL: https://issues.apache.org/jira/browse/HBASE-24125
 Project: HBase
  Issue Type: Task
Affects Versions: 1.0.0-alpha1
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang
 Fix For: 1.0.0-alpha2


hbase-filesystem is currently on hbase-thirdparty 2.2.1. Update it to 3.2.0 so 
we can use the latest guava.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24124) hbase-filesystem to use guava from hbase-thirdparty

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24124:
---

 Summary: hbase-filesystem to use guava from hbase-thirdparty
 Key: HBASE-24124
 URL: https://issues.apache.org/jira/browse/HBASE-24124
 Project: HBase
  Issue Type: Task
  Components: Filesystem Integration
Affects Versions: 1.0.0-alpha2
Reporter: Wei-Chiu Chuang


hbase-filesystem repo is on guava23.0:

{noformat}
$ grep -r "guava" .
./pom.xml:23.0
./hbase-oss/pom.xml:  com.google.guava
./hbase-oss/pom.xml:  guava
./hbase-oss/pom.xml:  ${guava.version}
./hbase-oss/pom.xml:  

[jira] [Created] (HBASE-24087) Backport HBASE-8868 to branch-2.2

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24087:
---

 Summary: Backport HBASE-8868 to branch-2.2
 Key: HBASE-24087
 URL: https://issues.apache.org/jira/browse/HBASE-24087
 Project: HBase
  Issue Type: Task
Reporter: Wei-Chiu Chuang


There's just a trivial conflict in import.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-23829) Get `-PrunSmallTests` passing on JDK11

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reopened HBASE-23829:
-

Reopen this.

I am getting consistent test failure as reported in HBASE-23976, and I traced 
it to this commit.
I am on JDK 8.

{noformat}
2020-03-23 20:57:04,934 ERROR [Time-limited test] bucket.BucketCache(312): 
Can't restore from 
file[/Users/weichiu/sandbox/hbase/hbase-server/target/test-data/c9d48c67-87ed-70cb-e19c-4dc6c14c29c6/bucket.persistence]
 because of 
java.io.IOException: Mismatch of checksum! The persistent checksum is 
`9"�0����X!ɍ=, but the calculate checksum is 1��h&�B���D(�
at 
org.apache.hadoop.hbase.io.hfile.bucket.PersistentIOEngine.verifyFileIntegrity(PersistentIOEngine.java:55)
at 
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.parsePB(BucketCache.java:1158)
at 
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(BucketCache.java:1106)
at 
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:310)
at 
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:258)
at 
org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile(TestVerifyBucketCacheFile.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.lang.Thread.run(Thread.java:748)
{noformat}

> Get `-PrunSmallTests` passing on JDK11
> --
>
> Key: HBASE-23829
> URL: https://issues.apache.org/jira/browse/HBASE-23829
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Start with the small tests, shaking out issues identified by the harness. So 
> far it seems like {{-Dhadoop.profile=3.0}} and 
> {{-Dhadoop-three.version=3.3.0-SNAPSHOT}} maybe be required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24027) Spotbugs: Return value of putIfAbsent is ignored

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-24027:
---

 Summary: Spotbugs: Return value of putIfAbsent is ignored
 Key: HBASE-24027
 URL: https://issues.apache.org/jira/browse/HBASE-24027
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Wei-Chiu Chuang


Looks like a regression from HBASE-23561.

[https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1301/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html]
{quote}
 Return value of putIfAbsent is ignored, but node is reused in 
org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
 Bug type RV_RETURN_VALUE_OF_PUTIFABSENT_IGNORED (click for details)
 In class org.apache.hadoop.hbase.master.assignment.RegionStates
 In method 
org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
 Called method java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(Object, 
Object)
 Type org.apache.hadoop.hbase.master.assignment.RegionStateNode
 Value loaded from node
 At RegionStates.java:[line 133]
{quote}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-23861) Reconcile Hadoop version

[Wei-Chiu Chuang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HBASE-23861.
-
Resolution: Fixed

Resolve this and leave the fix in master and branch-2.3.

I'll file a Jira to update the user guide when applicable.

 

Thanks!

> Reconcile Hadoop version
> 
>
> Key: HBASE-23861
> URL: https://issues.apache.org/jira/browse/HBASE-23861
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies
>Affects Versions: 3.0.0
> Environment: Apache Maven 3.6.1
> Hadoop 3.2.0 and above.
>    Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> I followed the HBase book 
> http://hbase.apache.org/book.html#maven.build.hadoop and wanted to build 
> HBase (master) on top of Hadoop 3.2/3.3 but tests failed right away.
> Build: 
> {code}
> mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 
> -DskipTests
> {code}
> Test:
> {code}
> mvn test -Dtest=TestHelloHBase -Dhadoop.profile=3.0 
> -Dhadoop-three.version=3.2.1
> {code}
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.296 
> s <<< FAILURE! - in 
> org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase
> [ERROR] org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase  
> Time elapsed: 1.284 s  <<< ERROR!
> java.lang.NoClassDefFoundError: 
> org/apache/hadoop/hdfs/protocol/HdfsConstants$StoragePolicySatisfierMode
> at 
> org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase.beforeClass(TestHelloHBase.java:54)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hdfs.protocol.HdfsConstants$StoragePolicySatisfierMode
> at 
> org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase.beforeClass(TestHelloHBase.java:54)
> {noformat}
> Adding mvn -X parameter, I was able to tell that it was because the 
> hbase-server module includes hadoop-distcp and hadoop-dfs-client 3.1.2 (the 
> default Hadoop 3 dependency version) while it uses version 3.2.1 of other 
> hadoop jars . The classpath conflict (the storage policy satisfier is a new 
> feature in Hadoop 3.2) failed the test.
> This is reproducible on any Hadoop version 3.2 and above. It looks to me the 
> version of hadoop-distcp and hadoop-hdfs-client should be specified at the 
> top level pom (they are specified in hbase-server/pom.xml).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23998) Update license for jetty-client

[Wei-Chiu Chuang (Jira)]

Wei-Chiu Chuang created HBASE-23998:
---

 Summary: Update license for jetty-client
 Key: HBASE-23998
 URL: https://issues.apache.org/jira/browse/HBASE-23998
 Project: HBase
  Issue Type: Bug
  Components: build, dependencies
Affects Versions: 3.0.0
 Environment: HBase master branch on Apache Hadoop 3.3.0
Reporter: Wei-Chiu Chuang


After HBASE-22103, compiling on Haddop 3.3.0 has the following error:
{{mvn clean install -Dhadoop.profile=3.0 -Dhadoop.version=3.3.0-SNAPSHOT 
-DskipTests -Dmaven.javadoc.skip=true}}
{noformat}
This product includes Jetty :: Asynchronous HTTP Client licensed under the 
Apache Software License - Version 2.0.

ERROR: Please check  this License for acceptability here:

https://www.apache.org/legal/resolved

If it is okay, then update the list named 'non_aggregate_fine' in the 
LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.

More info on the dependency:

org.eclipse.jetty
jetty-client
9.4.20.v20190813
{noformat}

This is caused by YARN-8778 which added dependency on 
org.eclipse.jetty.websocket:websocket-client, and the Jetty 9.4 update in 
HADOOP-16152.

{noformat}
[INFO] +- 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.3.0-SNAPSHOT:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-client:jar:3.3.0-SNAPSHOT:compile
[INFO] |  |  +- 
org.eclipse.jetty.websocket:websocket-client:jar:9.4.20.v20190813:compile
[INFO] |  |  |  +- org.eclipse.jetty:jetty-client:jar:9.4.20.v20190813:compile
[INFO] |  |  |  \- 
org.eclipse.jetty.websocket:websocket-common:jar:9.4.20.v20190813:compile
[INFO] |  |  | \- 
org.eclipse.jetty.websocket:websocket-api:jar:9.4.20.v20190813:compile
{noformat}

Propose: update 
hbase-resource-bundle/src/main/resources/supplemental-models.xml to update the 
license text for jetty-client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)