Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
With S3, we use reflection when loading the filesystem. As long as it is on the CP, we're okay. However, I think the difference here is the APIs that Stephen wants to use is not in the hadoop filesystem API/class, but instead in DFS I believe. Others in this thread have mentioned ways to work around this both in the short term and long term. On Thu, Mar 16, 2023, 11:06 PM 张铎(Duo Zhang) wrote: > I just mean how do we manage the dependencies. We do not ship hbase with > aws-sdk, but I believe users can put the related jars under lib and then > use S3AFileSystem. I think we can support ozone in the same way? > > Viraj Jasani 于2023年3月17日周五 13:33写道： > > > WAL is still kept on hdfs only, even when hfiles are kept in s3 AFAIK. > But > > here it seems, both WAL and HFiles can be kept in Ozone IIUC. > > > > > > On Thu, Mar 16, 2023 at 8:46 PM 张铎(Duo Zhang) > > wrote: > > > > > How do we support S3 as HFile storage currently? I do not think we have > > > added aws-sdk as a direct dependency in HBase now? > > > > > > Viraj Jasani 于2023年3月17日周五 04:37写道： > > > > > > > +1, similar to what was done in the past for using > > > > HdfsDataOutputStreamBuilder that was available since hadoop 2.9 or > so I > > > > think. > > > > > > > > > > > > On Thu, Mar 16, 2023 at 1:04 PM Andrew Purtell < > > andrew.purt...@gmail.com > > > > > > > > wrote: > > > > > > > > > It should be done with reflection rather than take a direct > > dependency, > > > > > until Hadoop common interfaces are available in what we consider > the > > > > lowest > > > > > supported version. > > > > > > > > > > > On Mar 16, 2023, at 12:35 PM, Viraj Jasani > > > wrote: > > > > > > > > > > > > It would be nice using PathCapabilities to determine lease > > recovery > > > > as a > > > > > > feature flag. > > > > > > In fact, s3a and abfs have lots of feature flags being derived > from > > > > this > > > > > > API already. It would be good for dfs and ozone to recognize > lease > > > > > recovery > > > > > > as a capability. > > > > > > > > > > > > However, this alone might not be sufficient and something like > > > > > > RecoverableFileSystem interface would be helpful as long as we > can > > > > > abstract > > > > > > out lease recovery (and safe mode etc) options as hbase anyways > > need > > > to > > > > > > perform them. > > > > > > > > > > > > Hence, having both: a) path capability to identify if lease > > recovery > > > > etc > > > > > > features are available and b) a new FileSystem interface that > both > > > dfs > > > > > and > > > > > > ozone can implement, would be great IMHO. Because even if we just > > > have > > > > > path > > > > > > capability for the feature flag, we would still end up adding > ozone > > > > > > dependency (unless done with reflection as Andrew mentioned) to > > > perform > > > > > > lease recovery unless lease recovery is abstracted out somewhere > in > > > > > hadoop. > > > > > > > > > > > >> One of the original worries is if the Hadoop/HDFS community > > > > > >> would reject our proposal when we change the base > > interface/abstract > > > > > class > > > > > >> in FileSystem (if it's non-backward compatible). > > > > > > > > > > > > I believe, new IA.Public interface in hadoop that can abstract > out > > > > lease > > > > > > recovery etc would have less likelihood of getting rejected than > > > > "making > > > > > > changes in FileSystem directly". > > > > > > > > > > > > > > > > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu < > > > > tak...@apache.org > > > > > > > > > > > >> wrote: > > > > > >> > > > > > >> In addition, I'm yet confirm but based on another search in the > > > hadoop > > > > > >> code, we may be able to add recover lease as a feature flag in > > > > > >> CommonPathCapabilities [3] and can be used by the interface of > > > > > >> PathCapabilities#hasPathCapability [4]. (this is similar to > > > > > >> StreamCapabilities as mentioned by Viraj) > > > > > >> > > > > > >> 3. > > > > > >> > > > > > > > > > > > > > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java > > > > > >> 4. > > > > > >> > > > > > > > > > > > > > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java > > > > > >> > > > > > >> -Stephen > > > > > >> > > > > > >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu < > > > > > tak...@apache.org> > > > > > >>> wrote: > > > > > >>> > > > > > >>> Thanks everyone ! Sean helped to clarify that something like > DFS > > > > > specific > > > > > >>> APIs used by HBase has been in-place in many HBase modules as > the > > > > > feature > > > > > >>> implementation but yet standardized in hadoop general > FileSystem > > > API, > > > > > >> e.g. > > > > > >>> lease recovery. One of the original worries is if the > Hadoop/HDFS > > > > > >> community > > > > > >>> would reject our proposal when we
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
I just mean how do we manage the dependencies. We do not ship hbase with aws-sdk, but I believe users can put the related jars under lib and then use S3AFileSystem. I think we can support ozone in the same way? Viraj Jasani 于2023年3月17日周五 13:33写道： > WAL is still kept on hdfs only, even when hfiles are kept in s3 AFAIK. But > here it seems, both WAL and HFiles can be kept in Ozone IIUC. > > > On Thu, Mar 16, 2023 at 8:46 PM 张铎(Duo Zhang) > wrote: > > > How do we support S3 as HFile storage currently? I do not think we have > > added aws-sdk as a direct dependency in HBase now? > > > > Viraj Jasani 于2023年3月17日周五 04:37写道： > > > > > +1, similar to what was done in the past for using > > > HdfsDataOutputStreamBuilder that was available since hadoop 2.9 or so I > > > think. > > > > > > > > > On Thu, Mar 16, 2023 at 1:04 PM Andrew Purtell < > andrew.purt...@gmail.com > > > > > > wrote: > > > > > > > It should be done with reflection rather than take a direct > dependency, > > > > until Hadoop common interfaces are available in what we consider the > > > lowest > > > > supported version. > > > > > > > > > On Mar 16, 2023, at 12:35 PM, Viraj Jasani > > wrote: > > > > > > > > > > It would be nice using PathCapabilities to determine lease > recovery > > > as a > > > > > feature flag. > > > > > In fact, s3a and abfs have lots of feature flags being derived from > > > this > > > > > API already. It would be good for dfs and ozone to recognize lease > > > > recovery > > > > > as a capability. > > > > > > > > > > However, this alone might not be sufficient and something like > > > > > RecoverableFileSystem interface would be helpful as long as we can > > > > abstract > > > > > out lease recovery (and safe mode etc) options as hbase anyways > need > > to > > > > > perform them. > > > > > > > > > > Hence, having both: a) path capability to identify if lease > recovery > > > etc > > > > > features are available and b) a new FileSystem interface that both > > dfs > > > > and > > > > > ozone can implement, would be great IMHO. Because even if we just > > have > > > > path > > > > > capability for the feature flag, we would still end up adding ozone > > > > > dependency (unless done with reflection as Andrew mentioned) to > > perform > > > > > lease recovery unless lease recovery is abstracted out somewhere in > > > > hadoop. > > > > > > > > > >> One of the original worries is if the Hadoop/HDFS community > > > > >> would reject our proposal when we change the base > interface/abstract > > > > class > > > > >> in FileSystem (if it's non-backward compatible). > > > > > > > > > > I believe, new IA.Public interface in hadoop that can abstract out > > > lease > > > > > recovery etc would have less likelihood of getting rejected than > > > "making > > > > > changes in FileSystem directly". > > > > > > > > > > > > > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu < > > > tak...@apache.org > > > > > > > > > >> wrote: > > > > >> > > > > >> In addition, I'm yet confirm but based on another search in the > > hadoop > > > > >> code, we may be able to add recover lease as a feature flag in > > > > >> CommonPathCapabilities [3] and can be used by the interface of > > > > >> PathCapabilities#hasPathCapability [4]. (this is similar to > > > > >> StreamCapabilities as mentioned by Viraj) > > > > >> > > > > >> 3. > > > > >> > > > > > > > > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java > > > > >> 4. > > > > >> > > > > > > > > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java > > > > >> > > > > >> -Stephen > > > > >> > > > > >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu < > > > > tak...@apache.org> > > > > >>> wrote: > > > > >>> > > > > >>> Thanks everyone ! Sean helped to clarify that something like DFS > > > > specific > > > > >>> APIs used by HBase has been in-place in many HBase modules as the > > > > feature > > > > >>> implementation but yet standardized in hadoop general FileSystem > > API, > > > > >> e.g. > > > > >>> lease recovery. One of the original worries is if the Hadoop/HDFS > > > > >> community > > > > >>> would reject our proposal when we change the base > > interface/abstract > > > > >> class > > > > >>> in FileSystem (if it's non-backward compatible). The discussion > > here > > > > >> helps > > > > >>> to confirm the direction, and let's see how we can make it > generic > > > and > > > > >>> could help to avoid confusion in both places. > > > > >>> > > > > >>> Thanks again, > > > > >>> Stephen > > > > >>> > > > > >>> On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell < > > > > andrew.purt...@gmail.com > > > > >>> > > > > >>> wrote: > > > > >>> > > > > Then Hadoop should add one and although we would need a > reflection > > > > >> based > > > > check in the interim we can converge toward the ide
[jira] [Created] (HBASE-27728) Implement a tool to migrate replication peer data between different storage implementation
Duo Zhang created HBASE-27728: - Summary: Implement a tool to migrate replication peer data between different storage implementation Key: HBASE-27728 URL: https://issues.apache.org/jira/browse/HBASE-27728 Project: HBase Issue Type: Sub-task Components: Replication Reporter: Duo Zhang Replication peer data is usually stable unless you manually modify it, so it is OK to not migrate automatically. Instead, we could provide a tool to migrate the replication peer data. And after migration, users can use online configuration change or via a cluster a restart to load the new configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27727) Implement filesystem based Replication peer storage
Duo Zhang created HBASE-27727: - Summary: Implement filesystem based Replication peer storage Key: HBASE-27727 URL: https://issues.apache.org/jira/browse/HBASE-27727 Project: HBase Issue Type: Sub-task Components: Replication Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
WAL is still kept on hdfs only, even when hfiles are kept in s3 AFAIK. But here it seems, both WAL and HFiles can be kept in Ozone IIUC. On Thu, Mar 16, 2023 at 8:46 PM 张铎(Duo Zhang) wrote: > How do we support S3 as HFile storage currently? I do not think we have > added aws-sdk as a direct dependency in HBase now? > > Viraj Jasani 于2023年3月17日周五 04:37写道： > > > +1, similar to what was done in the past for using > > HdfsDataOutputStreamBuilder that was available since hadoop 2.9 or so I > > think. > > > > > > On Thu, Mar 16, 2023 at 1:04 PM Andrew Purtell > > > wrote: > > > > > It should be done with reflection rather than take a direct dependency, > > > until Hadoop common interfaces are available in what we consider the > > lowest > > > supported version. > > > > > > > On Mar 16, 2023, at 12:35 PM, Viraj Jasani > wrote: > > > > > > > > It would be nice using PathCapabilities to determine lease recovery > > as a > > > > feature flag. > > > > In fact, s3a and abfs have lots of feature flags being derived from > > this > > > > API already. It would be good for dfs and ozone to recognize lease > > > recovery > > > > as a capability. > > > > > > > > However, this alone might not be sufficient and something like > > > > RecoverableFileSystem interface would be helpful as long as we can > > > abstract > > > > out lease recovery (and safe mode etc) options as hbase anyways need > to > > > > perform them. > > > > > > > > Hence, having both: a) path capability to identify if lease recovery > > etc > > > > features are available and b) a new FileSystem interface that both > dfs > > > and > > > > ozone can implement, would be great IMHO. Because even if we just > have > > > path > > > > capability for the feature flag, we would still end up adding ozone > > > > dependency (unless done with reflection as Andrew mentioned) to > perform > > > > lease recovery unless lease recovery is abstracted out somewhere in > > > hadoop. > > > > > > > >> One of the original worries is if the Hadoop/HDFS community > > > >> would reject our proposal when we change the base interface/abstract > > > class > > > >> in FileSystem (if it's non-backward compatible). > > > > > > > > I believe, new IA.Public interface in hadoop that can abstract out > > lease > > > > recovery etc would have less likelihood of getting rejected than > > "making > > > > changes in FileSystem directly". > > > > > > > > > > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu < > > tak...@apache.org > > > > > > > >> wrote: > > > >> > > > >> In addition, I'm yet confirm but based on another search in the > hadoop > > > >> code, we may be able to add recover lease as a feature flag in > > > >> CommonPathCapabilities [3] and can be used by the interface of > > > >> PathCapabilities#hasPathCapability [4]. (this is similar to > > > >> StreamCapabilities as mentioned by Viraj) > > > >> > > > >> 3. > > > >> > > > > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java > > > >> 4. > > > >> > > > > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java > > > >> > > > >> -Stephen > > > >> > > > >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu < > > > tak...@apache.org> > > > >>> wrote: > > > >>> > > > >>> Thanks everyone ! Sean helped to clarify that something like DFS > > > specific > > > >>> APIs used by HBase has been in-place in many HBase modules as the > > > feature > > > >>> implementation but yet standardized in hadoop general FileSystem > API, > > > >> e.g. > > > >>> lease recovery. One of the original worries is if the Hadoop/HDFS > > > >> community > > > >>> would reject our proposal when we change the base > interface/abstract > > > >> class > > > >>> in FileSystem (if it's non-backward compatible). The discussion > here > > > >> helps > > > >>> to confirm the direction, and let's see how we can make it generic > > and > > > >>> could help to avoid confusion in both places. > > > >>> > > > >>> Thanks again, > > > >>> Stephen > > > >>> > > > >>> On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell < > > > andrew.purt...@gmail.com > > > >>> > > > >>> wrote: > > > >>> > > > Then Hadoop should add one and although we would need a reflection > > > >> based > > > check in the interim we can converge toward the ideal. > > > > > > In any case I believe we can avoid a direct dependency on Ozone > and > > > >> should > > > strongly avoid taking such unnecessary dependencies. The Hadoop > and > > > >> HBase > > > build dependency sets are already very large and we and other > users > > > are > > > being hit with significant security issue remediation work, much > of > > > >> which > > > represents compatibility problems and is not upstreamable (like > > > >> protobuf 2 > > > removal in 2.x). We struggle with the existing depende
[DISCUSS] How to align hbase-default.xml and hbase-default.adoc
Hi everyone, I have been working on removing the ‘hbase.regionserver.hlog.writer.impl’ and ‘hbase.regionserver.hlog.reader.impl’ properties from our code base. In the process, I noticed that we have two places where we mention the default configuration values: one is hbase-default.xml in the hbase-common module, and the other is hbase-default.adoc in our asciidoc. However, these two files are not consistent with each other. I'm not sure if they are aligned in the past but at least right now, they are not aligned. This inconsistency can cause confusion for our users and developers. Therefore, I suggest that we either find a way to align these two files automatically (e.g., by generating hbase-default.adoc from hbase-defauilt.xml), or remove hbase-default.adoc altogether and refer users to hbase-default.xml for the default configuration values. What do you think? Please share your opinions and suggestions. Thanks. Regards
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
How do we support S3 as HFile storage currently? I do not think we have added aws-sdk as a direct dependency in HBase now? Viraj Jasani 于2023年3月17日周五 04:37写道： > +1, similar to what was done in the past for using > HdfsDataOutputStreamBuilder that was available since hadoop 2.9 or so I > think. > > > On Thu, Mar 16, 2023 at 1:04 PM Andrew Purtell > wrote: > > > It should be done with reflection rather than take a direct dependency, > > until Hadoop common interfaces are available in what we consider the > lowest > > supported version. > > > > > On Mar 16, 2023, at 12:35 PM, Viraj Jasani wrote: > > > > > > It would be nice using PathCapabilities to determine lease recovery > as a > > > feature flag. > > > In fact, s3a and abfs have lots of feature flags being derived from > this > > > API already. It would be good for dfs and ozone to recognize lease > > recovery > > > as a capability. > > > > > > However, this alone might not be sufficient and something like > > > RecoverableFileSystem interface would be helpful as long as we can > > abstract > > > out lease recovery (and safe mode etc) options as hbase anyways need to > > > perform them. > > > > > > Hence, having both: a) path capability to identify if lease recovery > etc > > > features are available and b) a new FileSystem interface that both dfs > > and > > > ozone can implement, would be great IMHO. Because even if we just have > > path > > > capability for the feature flag, we would still end up adding ozone > > > dependency (unless done with reflection as Andrew mentioned) to perform > > > lease recovery unless lease recovery is abstracted out somewhere in > > hadoop. > > > > > >> One of the original worries is if the Hadoop/HDFS community > > >> would reject our proposal when we change the base interface/abstract > > class > > >> in FileSystem (if it's non-backward compatible). > > > > > > I believe, new IA.Public interface in hadoop that can abstract out > lease > > > recovery etc would have less likelihood of getting rejected than > "making > > > changes in FileSystem directly". > > > > > > > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu < > tak...@apache.org > > > > > >> wrote: > > >> > > >> In addition, I'm yet confirm but based on another search in the hadoop > > >> code, we may be able to add recover lease as a feature flag in > > >> CommonPathCapabilities [3] and can be used by the interface of > > >> PathCapabilities#hasPathCapability [4]. (this is similar to > > >> StreamCapabilities as mentioned by Viraj) > > >> > > >> 3. > > >> > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java > > >> 4. > > >> > > > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java > > >> > > >> -Stephen > > >> > > >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu < > > tak...@apache.org> > > >>> wrote: > > >>> > > >>> Thanks everyone ! Sean helped to clarify that something like DFS > > specific > > >>> APIs used by HBase has been in-place in many HBase modules as the > > feature > > >>> implementation but yet standardized in hadoop general FileSystem API, > > >> e.g. > > >>> lease recovery. One of the original worries is if the Hadoop/HDFS > > >> community > > >>> would reject our proposal when we change the base interface/abstract > > >> class > > >>> in FileSystem (if it's non-backward compatible). The discussion here > > >> helps > > >>> to confirm the direction, and let's see how we can make it generic > and > > >>> could help to avoid confusion in both places. > > >>> > > >>> Thanks again, > > >>> Stephen > > >>> > > >>> On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell < > > andrew.purt...@gmail.com > > >>> > > >>> wrote: > > >>> > > Then Hadoop should add one and although we would need a reflection > > >> based > > check in the interim we can converge toward the ideal. > > > > In any case I believe we can avoid a direct dependency on Ozone and > > >> should > > strongly avoid taking such unnecessary dependencies. The Hadoop and > > >> HBase > > build dependency sets are already very large and we and other users > > are > > being hit with significant security issue remediation work, much of > > >> which > > represents compatibility problems and is not upstreamable (like > > >> protobuf 2 > > removal in 2.x). We struggle with the existing dependencies enough > > >> already > > at my employer. > > > > > On Mar 15, 2023, at 1:53 PM, Sean Busbey > wrote: > > > > > > the check that Stephen is referring to is for logic around lease > > recovery > > > and not stream flush/sync. the lease recovery is specific to DFS > > >> IIRC and > > > doesn't have a FileSystem marker. > > > > > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell < > apurt...@apache.org > > >>> > >
[jira] [Created] (HBASE-27726) ruby shell not handled SyntaxError exceptions properly
chiranjeevi created HBASE-27726: --- Summary: ruby shell not handled SyntaxError exceptions properly Key: HBASE-27726 URL: https://issues.apache.org/jira/browse/HBASE-27726 Project: HBase Issue Type: Bug Components: shell Affects Versions: 2.5.2 Reporter: chiranjeevi hbase:002:0> create 't2', 'cf' 2023-03-14 04:54:50,061 INFO [main] client.HBaseAdmin: Operation: CREATE, Table Name: default:t2, procId: 2140 completed Created table t2 Took 1.1503 seconds => Hbase::Table - t2 hbase:003:0> alter 't2', NAME ⇒ 'cf', VERSIONS ⇒ 5 SyntaxError: (hbase):3: syntax error, unexpected tIDENTIFIER alter 't2', NAME ⇒ 'cf', VERSIONS ⇒ 5 ^~~ eval at org/jruby/RubyKernel.java:1091 evaluate at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/workspace.rb:85 evaluate at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:385 eval_input at uri:classloader:/irb/hirb.rb:115 signal_status at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:647 eval_input at uri:classloader:/irb/hirb.rb:112 each_top_level_statement at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:246 loop at org/jruby/RubyKernel.java:1507 each_top_level_statement at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:232 catch at org/jruby/RubyKernel.java:1237 each_top_level_statement at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:231 eval_input at uri:classloader:/irb/hirb.rb:111 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:428 catch at org/jruby/RubyKernel.java:1237 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:226 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27725) Add ConcurrentHashMap#get() call before ConcurrentHashMap#computeIfAbsent()
chenfengge created HBASE-27725: -- Summary: Add ConcurrentHashMap#get() call before ConcurrentHashMap#computeIfAbsent() Key: HBASE-27725 URL: https://issues.apache.org/jira/browse/HBASE-27725 Project: HBase Issue Type: Improvement Components: Performance Reporter: chenfengge Made hbase pe RandomRead test for 100% memory-cache in HBase 2.5.0 before. We found that there are many blocking regionserver handler threads, and our cpu utilisation percentage is about 50% and cannot increase anymore. Here is the blocking stack: RpcServer.default.FPBQ.Fifo.handler=116,port=16020" #299 daemon prio=5 os_prio=0 tid=0x825d5000 nid=0x208e51 waiting for monitor entry [0xffbe67734000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674) - waiting to lock <0xfff75f465128> (a java.util.concurrent.ConcurrentHashMap$Node) at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:77) at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:82) at org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:93) at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:283) at org.apache.hadoop.hbase.regionserver.HRegion.metricsUpdateForGet(HRegion.java:7401) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2667) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2567) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45945) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102) at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New HBase committer Tianhang Tang(唐天航)
Congratulations Tianhang! Thanks, Rajeshbabu. On Fri, Mar 17, 2023, 1:11 AM Tak Lon (Stephen) Wu wrote: > Congratulations, Tianhang! > > -Stephen > > On Thu, Mar 16, 2023 at 12:36 PM Viraj Jasani wrote: > > > Congratulations and Welcome, Tianhang Tang ! > > > > On Wed, Mar 15, 2023 at 9:02 PM 张铎(Duo Zhang) > > wrote: > > > > > On behalf of the Apache HBase PMC, I am pleased to announce that > Tianhang > > > Tang(thangTang) > > > has accepted the PMC's invitation to become a committer on the project. > > We > > > appreciate all > > > of Tianhang's generous contributions thus far and look forward to his > > > continued involvement. > > > > > > Congratulations and welcome, Tianhang Tang! > > > > > > 我很高兴代表 Apache HBase PMC 宣布 唐天航 已接受我们的邀请，成 > > > 为 Apache HBase 项目的 Committer。感谢 唐天航 一直以来为 HBase 项目 > > > 做出的贡献，并期待他在未来继续承担更多的责任。 > > > > > > 欢迎 唐天航！ > > > > > >
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
+1, similar to what was done in the past for using HdfsDataOutputStreamBuilder that was available since hadoop 2.9 or so I think. On Thu, Mar 16, 2023 at 1:04 PM Andrew Purtell wrote: > It should be done with reflection rather than take a direct dependency, > until Hadoop common interfaces are available in what we consider the lowest > supported version. > > > On Mar 16, 2023, at 12:35 PM, Viraj Jasani wrote: > > > > It would be nice using PathCapabilities to determine lease recovery as a > > feature flag. > > In fact, s3a and abfs have lots of feature flags being derived from this > > API already. It would be good for dfs and ozone to recognize lease > recovery > > as a capability. > > > > However, this alone might not be sufficient and something like > > RecoverableFileSystem interface would be helpful as long as we can > abstract > > out lease recovery (and safe mode etc) options as hbase anyways need to > > perform them. > > > > Hence, having both: a) path capability to identify if lease recovery etc > > features are available and b) a new FileSystem interface that both dfs > and > > ozone can implement, would be great IMHO. Because even if we just have > path > > capability for the feature flag, we would still end up adding ozone > > dependency (unless done with reflection as Andrew mentioned) to perform > > lease recovery unless lease recovery is abstracted out somewhere in > hadoop. > > > >> One of the original worries is if the Hadoop/HDFS community > >> would reject our proposal when we change the base interface/abstract > class > >> in FileSystem (if it's non-backward compatible). > > > > I believe, new IA.Public interface in hadoop that can abstract out lease > > recovery etc would have less likelihood of getting rejected than "making > > changes in FileSystem directly". > > > > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu > > >> wrote: > >> > >> In addition, I'm yet confirm but based on another search in the hadoop > >> code, we may be able to add recover lease as a feature flag in > >> CommonPathCapabilities [3] and can be used by the interface of > >> PathCapabilities#hasPathCapability [4]. (this is similar to > >> StreamCapabilities as mentioned by Viraj) > >> > >> 3. > >> > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java > >> 4. > >> > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java > >> > >> -Stephen > >> > >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu < > tak...@apache.org> > >>> wrote: > >>> > >>> Thanks everyone ! Sean helped to clarify that something like DFS > specific > >>> APIs used by HBase has been in-place in many HBase modules as the > feature > >>> implementation but yet standardized in hadoop general FileSystem API, > >> e.g. > >>> lease recovery. One of the original worries is if the Hadoop/HDFS > >> community > >>> would reject our proposal when we change the base interface/abstract > >> class > >>> in FileSystem (if it's non-backward compatible). The discussion here > >> helps > >>> to confirm the direction, and let's see how we can make it generic and > >>> could help to avoid confusion in both places. > >>> > >>> Thanks again, > >>> Stephen > >>> > >>> On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell < > andrew.purt...@gmail.com > >>> > >>> wrote: > >>> > Then Hadoop should add one and although we would need a reflection > >> based > check in the interim we can converge toward the ideal. > > In any case I believe we can avoid a direct dependency on Ozone and > >> should > strongly avoid taking such unnecessary dependencies. The Hadoop and > >> HBase > build dependency sets are already very large and we and other users > are > being hit with significant security issue remediation work, much of > >> which > represents compatibility problems and is not upstreamable (like > >> protobuf 2 > removal in 2.x). We struggle with the existing dependencies enough > >> already > at my employer. > > > On Mar 15, 2023, at 1:53 PM, Sean Busbey wrote: > > > > the check that Stephen is referring to is for logic around lease > recovery > > and not stream flush/sync. the lease recovery is specific to DFS > >> IIRC and > > doesn't have a FileSystem marker. > > > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell >>> > wrote: > >> > >> So we can test StreamCapabilities in code, in worst case by wrapping > some > >> probe code during startup with try-catch and examining the > >> exception. > >> > >>> On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani > wrote: > >>> > >>> As of today, both WAL impl (fshlog and asyncfs) throw > >>> StreamLacksCapabilityException if the FS Data OutputStream probe > >> fails > >> for > >>> Hflush/Hsync: > >>
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
It should be done with reflection rather than take a direct dependency, until Hadoop common interfaces are available in what we consider the lowest supported version. > On Mar 16, 2023, at 12:35 PM, Viraj Jasani wrote: > > It would be nice using PathCapabilities to determine lease recovery as a > feature flag. > In fact, s3a and abfs have lots of feature flags being derived from this > API already. It would be good for dfs and ozone to recognize lease recovery > as a capability. > > However, this alone might not be sufficient and something like > RecoverableFileSystem interface would be helpful as long as we can abstract > out lease recovery (and safe mode etc) options as hbase anyways need to > perform them. > > Hence, having both: a) path capability to identify if lease recovery etc > features are available and b) a new FileSystem interface that both dfs and > ozone can implement, would be great IMHO. Because even if we just have path > capability for the feature flag, we would still end up adding ozone > dependency (unless done with reflection as Andrew mentioned) to perform > lease recovery unless lease recovery is abstracted out somewhere in hadoop. > >> One of the original worries is if the Hadoop/HDFS community >> would reject our proposal when we change the base interface/abstract class >> in FileSystem (if it's non-backward compatible). > > I believe, new IA.Public interface in hadoop that can abstract out lease > recovery etc would have less likelihood of getting rejected than "making > changes in FileSystem directly". > > >> On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu >> wrote: >> >> In addition, I'm yet confirm but based on another search in the hadoop >> code, we may be able to add recover lease as a feature flag in >> CommonPathCapabilities [3] and can be used by the interface of >> PathCapabilities#hasPathCapability [4]. (this is similar to >> StreamCapabilities as mentioned by Viraj) >> >> 3. >> https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java >> 4. >> https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java >> >> -Stephen >> >>> On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu >>> wrote: >>> >>> Thanks everyone ! Sean helped to clarify that something like DFS specific >>> APIs used by HBase has been in-place in many HBase modules as the feature >>> implementation but yet standardized in hadoop general FileSystem API, >> e.g. >>> lease recovery. One of the original worries is if the Hadoop/HDFS >> community >>> would reject our proposal when we change the base interface/abstract >> class >>> in FileSystem (if it's non-backward compatible). The discussion here >> helps >>> to confirm the direction, and let's see how we can make it generic and >>> could help to avoid confusion in both places. >>> >>> Thanks again, >>> Stephen >>> >>> On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell >> >>> wrote: >>> Then Hadoop should add one and although we would need a reflection >> based check in the interim we can converge toward the ideal. In any case I believe we can avoid a direct dependency on Ozone and >> should strongly avoid taking such unnecessary dependencies. The Hadoop and >> HBase build dependency sets are already very large and we and other users are being hit with significant security issue remediation work, much of >> which represents compatibility problems and is not upstreamable (like >> protobuf 2 removal in 2.x). We struggle with the existing dependencies enough >> already at my employer. > On Mar 15, 2023, at 1:53 PM, Sean Busbey wrote: > > the check that Stephen is referring to is for logic around lease recovery > and not stream flush/sync. the lease recovery is specific to DFS >> IIRC and > doesn't have a FileSystem marker. > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell >> wrote: >> >> So we can test StreamCapabilities in code, in worst case by wrapping some >> probe code during startup with try-catch and examining the >> exception. >> >>> On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani wrote: >>> >>> As of today, both WAL impl (fshlog and asyncfs) throw >>> StreamLacksCapabilityException if the FS Data OutputStream probe >> fails >> for >>> Hflush/Hsync: >>> >>> StreamLacksCapabilityException(StreamCapabilities.HFLUSH) >>> and >>> StreamLacksCapabilityException(StreamCapabilities.HSYNC) >>> >>> >>> On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell < >> apurt...@apache.org> >>> wrote: >>> Does Hadoop have a marker interface that lets an application know >> its FileSystem instances can support hsync/hflush? Ideally all we >> should >> need to do is
[jira] [Created] (HBASE-27724) [HBCK2] addFsRegionsMissingInMeta command should support dumping region list into a file which can be passed as input to assigns command
Nihal Jain created HBASE-27724:
--
Summary: [HBCK2] addFsRegionsMissingInMeta command should support
dumping region list into a file which can be passed as input to assigns command
Key: HBASE-27724
URL: https://issues.apache.org/jira/browse/HBASE-27724
Project: HBase
Issue Type: Improvement
Components: hbase-operator-tools, hbck2
Reporter: Nihal Jain
Assignee: Nihal Jain
_addFsRegionsMissingInMeta_ command currently outputs a command as last line of
output which needs to be run with hbck2
{code:java}
assigns 22d30d9e332af3272302cf780da14c3c 43245731f82e5bb907a4433f688574c1
5a19939f4f219ab177dd5b376dcb882f 774514b1027846c4e3b6702e193ce03d
7f6ad3360e0a4811c4dace8c1a901f40 8cd363e4da1b95fd43166f451546ad63
90e3414947f9500ec01f6672103f29d0{code}
This is good, but the user has to copy and format the command, which can get
really big depending on how many regions need to be assigned.
_addFsRegionsMissingInMeta_ should support a flag, say -f to facilitate dumping
region list into a file, which can be passed onto as input to _assigns_ command
via -i parameter.
Sample expected use-case:
{code:java}
# Dump output of command (in a formatted manner) to file
hbase hbck -j hbase-hbck2-version.jar addFsRegionsMissingInMeta -f
regions_to_assign.txt
# Pass file as input to assigns
hbase hbck -j hbase-hbck2-version.jar assigns -i regions_to_assign.txt{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [ANNOUNCE] New HBase committer Tianhang Tang(唐天航)
Congratulations, Tianhang! -Stephen On Thu, Mar 16, 2023 at 12:36 PM Viraj Jasani wrote: > Congratulations and Welcome, Tianhang Tang ! > > On Wed, Mar 15, 2023 at 9:02 PM 张铎(Duo Zhang) > wrote: > > > On behalf of the Apache HBase PMC, I am pleased to announce that Tianhang > > Tang(thangTang) > > has accepted the PMC's invitation to become a committer on the project. > We > > appreciate all > > of Tianhang's generous contributions thus far and look forward to his > > continued involvement. > > > > Congratulations and welcome, Tianhang Tang! > > > > 我很高兴代表 Apache HBase PMC 宣布 唐天航 已接受我们的邀请，成 > > 为 Apache HBase 项目的 Committer。感谢 唐天航 一直以来为 HBase 项目 > > 做出的贡献，并期待他在未来继续承担更多的责任。 > > > > 欢迎 唐天航！ > > >
Re: [ANNOUNCE] New HBase committer Tianhang Tang(唐天航)
Congratulations and Welcome, Tianhang Tang ! On Wed, Mar 15, 2023 at 9:02 PM 张铎(Duo Zhang) wrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Tianhang > Tang(thangTang) > has accepted the PMC's invitation to become a committer on the project. We > appreciate all > of Tianhang's generous contributions thus far and look forward to his > continued involvement. > > Congratulations and welcome, Tianhang Tang! > > 我很高兴代表 Apache HBase PMC 宣布 唐天航 已接受我们的邀请，成 > 为 Apache HBase 项目的 Committer。感谢 唐天航 一直以来为 HBase 项目 > 做出的贡献，并期待他在未来继续承担更多的责任。 > > 欢迎 唐天航！ >
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
It would be nice using PathCapabilities to determine lease recovery as a feature flag. In fact, s3a and abfs have lots of feature flags being derived from this API already. It would be good for dfs and ozone to recognize lease recovery as a capability. However, this alone might not be sufficient and something like RecoverableFileSystem interface would be helpful as long as we can abstract out lease recovery (and safe mode etc) options as hbase anyways need to perform them. Hence, having both: a) path capability to identify if lease recovery etc features are available and b) a new FileSystem interface that both dfs and ozone can implement, would be great IMHO. Because even if we just have path capability for the feature flag, we would still end up adding ozone dependency (unless done with reflection as Andrew mentioned) to perform lease recovery unless lease recovery is abstracted out somewhere in hadoop. > One of the original worries is if the Hadoop/HDFS community > would reject our proposal when we change the base interface/abstract class > in FileSystem (if it's non-backward compatible). I believe, new IA.Public interface in hadoop that can abstract out lease recovery etc would have less likelihood of getting rejected than "making changes in FileSystem directly". On Thu, Mar 16, 2023 at 2:07 AM Tak Lon (Stephen) Wu wrote: > In addition, I'm yet confirm but based on another search in the hadoop > code, we may be able to add recover lease as a feature flag in > CommonPathCapabilities [3] and can be used by the interface of > PathCapabilities#hasPathCapability [4]. (this is similar to > StreamCapabilities as mentioned by Viraj) > > 3. > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java > 4. > https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java > > -Stephen > > On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu > wrote: > > > > Thanks everyone ! Sean helped to clarify that something like DFS specific > > APIs used by HBase has been in-place in many HBase modules as the feature > > implementation but yet standardized in hadoop general FileSystem API, > e.g. > > lease recovery. One of the original worries is if the Hadoop/HDFS > community > > would reject our proposal when we change the base interface/abstract > class > > in FileSystem (if it's non-backward compatible). The discussion here > helps > > to confirm the direction, and let's see how we can make it generic and > > could help to avoid confusion in both places. > > > > Thanks again, > > Stephen > > > > On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell > > > wrote: > > > > > Then Hadoop should add one and although we would need a reflection > based > > > check in the interim we can converge toward the ideal. > > > > > > In any case I believe we can avoid a direct dependency on Ozone and > should > > > strongly avoid taking such unnecessary dependencies. The Hadoop and > HBase > > > build dependency sets are already very large and we and other users are > > > being hit with significant security issue remediation work, much of > which > > > represents compatibility problems and is not upstreamable (like > protobuf 2 > > > removal in 2.x). We struggle with the existing dependencies enough > already > > > at my employer. > > > > > > > On Mar 15, 2023, at 1:53 PM, Sean Busbey wrote: > > > > > > > > the check that Stephen is referring to is for logic around lease > > > recovery > > > > and not stream flush/sync. the lease recovery is specific to DFS > IIRC and > > > > doesn't have a FileSystem marker. > > > > > > > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell > > > > wrote: > > > >> > > > >> So we can test StreamCapabilities in code, in worst case by wrapping > > > some > > > >> probe code during startup with try-catch and examining the > exception. > > > >> > > > >>> On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani > > > wrote: > > > >>> > > > >>> As of today, both WAL impl (fshlog and asyncfs) throw > > > >>> StreamLacksCapabilityException if the FS Data OutputStream probe > fails > > > >> for > > > >>> Hflush/Hsync: > > > >>> > > > >>> StreamLacksCapabilityException(StreamCapabilities.HFLUSH) > > > >>> and > > > >>> StreamLacksCapabilityException(StreamCapabilities.HSYNC) > > > >>> > > > >>> > > > >>> On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell < > apurt...@apache.org> > > > >>> wrote: > > > >>> > > > Does Hadoop have a marker interface that lets an application know > its > > > FileSystem instances can support hsync/hflush? Ideally all we > should > > > >> need > > > to do is test with instanceof for that marker and use reflection > (in > > > >> the > > > worst case) to get a handle to the hsync or hflush method, and > then > > > >> call > > > it. This approach should be taken wherever we have a requirement > to > > > >> use a
[jira] [Created] (HBASE-27723) Fix brotli4j licence issue on native-osx-aarch64
Frens Jan Rumph created HBASE-27723:
---
Summary: Fix brotli4j licence issue on native-osx-aarch64
Key: HBASE-27723
URL: https://issues.apache.org/jira/browse/HBASE-27723
Project: HBase
Issue Type: Improvement
Reporter: Frens Jan Rumph
Apparently the licence of {{brotli4j}} is malformed and fixed in
{{{}supplemental-models.xml{}}}. It didn't however cover {{native-osx-aarch64}}
yet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27686) Recovery of BucketCache and Prefetched data after RS Crash
[ https://issues.apache.org/jira/browse/HBASE-27686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-27686. -- Resolution: Fixed Thanks for the contribution [~sk...@cloudera.com]! I have now merged this into master and branch-2. > Recovery of BucketCache and Prefetched data after RS Crash > -- > > Key: HBASE-27686 > URL: https://issues.apache.org/jira/browse/HBASE-27686 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Shanmukha Haripriya Kota >Assignee: Shanmukha Haripriya Kota >Priority: Major > > HBASE-27313 introduced the ability to persist a list of hfiles for which > prefetch has already been completed, so the we can avoid prefetching those > files again in the event of a graceful restart, but it doesn't cover crash > scenarios, as if the RS is killed or abnormally stopped, the list wouldn't be > saved. > This change aims to persist the list of already prefetched from a background > thread that periodically checks cache state and persists the list if updates > have happened. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27722) Update documentation about how to enable encryption on WAL
Duo Zhang created HBASE-27722: - Summary: Update documentation about how to enable encryption on WAL Key: HBASE-27722 URL: https://issues.apache.org/jira/browse/HBASE-27722 Project: HBase Issue Type: Task Components: documentation, wal Reporter: Duo Zhang Assignee: Duo Zhang After HBASE-27632 and HBASE-27702, we removed SecureProtobufLogReader and SecureProtobufLogWriter, users do not need to specify the reader/writer class any more, they just need to enable WAL encryption. Remove the related configurations such as 'hbase.regionserver.hlog.writer.impl' and 'hbase.regionserver.hlog.reader.impl' in hbase-default.xml, and also change ref guide to tell users how to enable WAL encryption for 2.6+. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New HBase committer Tianhang Tang(唐天航)
Congratulations and welcome! On Thu, Mar 16, 2023 at 12:02 AM 张铎(Duo Zhang) wrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Tianhang > Tang(thangTang) > has accepted the PMC's invitation to become a committer on the project. We > appreciate all > of Tianhang's generous contributions thus far and look forward to his > continued involvement. > > Congratulations and welcome, Tianhang Tang! > > 我很高兴代表 Apache HBase PMC 宣布 唐天航 已接受我们的邀请，成 > 为 Apache HBase 项目的 Committer。感谢 唐天航 一直以来为 HBase 项目 > 做出的贡献，并期待他在未来继续承担更多的责任。 > > 欢迎 唐天航！ >
Re: [DISCUSS] Upgrade Zookeeper and Curator to latest version
We only use Curator in hbase-examples IIRC, so it should be OK to upgrade it. For ZooKeeper, besides client server wire compatibility, we also need to consider java compatibility. For example, different protobuf versions can communicate with each other, but if you depend on protobuf 2.5 and 3.x in the java project, you will be in trouble as the classes are different... So the question here is, if we upgrade ZooKeeper to 3.8.x, will it break downstream users who are still on ZooKeeper 3.6.x or 3.7.x? Thanks. Villő Szűcs 于2023年2月28日周二 22:33写道： > Hi, > I’d like to upgrade zookeeper in hbase (and in other components as well) to > 3.8.1 version and curator to 5.4.0 version. > It is useful since the current zookeeper version 3.5.7 is EOL and we should > release HBASE 3 with the latest zookeeper to be on an active version. > ZooKeeper clients from 3.5.x onwards are fully compatible with 3.8.x > servers. ZooKeeper 3.8.x clients are compatible with 3.5.x, 3.6.x and 3.7.x > servers as long as we are not using new APIs not present these versions. > See ZooKeeper 3.8.0 Release Notes[1] for details. > Curator 5.0 contains a few non-backward compatible/breaking changes from > previous versions: https://curator.apache.org/breaking-changes.html, but > these changes have no effect on hbase. See Curator Release Notes[2] for > details. > Do you have any suggestions? > > [1] https://zookeeper.apache.org/doc/r3.8.0/releasenotes.html > [2] https://cwiki.apache.org/confluence/display/CURATOR/Releases >
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
In addition, I'm yet confirm but based on another search in the hadoop code, we may be able to add recover lease as a feature flag in CommonPathCapabilities [3] and can be used by the interface of PathCapabilities#hasPathCapability [4]. (this is similar to StreamCapabilities as mentioned by Viraj) 3. https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java 4. https://github.com/apache/hadoop/blob/branch-3.3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java -Stephen On Thu, Mar 16, 2023 at 12:00 AM Tak Lon (Stephen) Wu wrote: > > Thanks everyone ! Sean helped to clarify that something like DFS specific > APIs used by HBase has been in-place in many HBase modules as the feature > implementation but yet standardized in hadoop general FileSystem API, e.g. > lease recovery. One of the original worries is if the Hadoop/HDFS community > would reject our proposal when we change the base interface/abstract class > in FileSystem (if it's non-backward compatible). The discussion here helps > to confirm the direction, and let's see how we can make it generic and > could help to avoid confusion in both places. > > Thanks again, > Stephen > > On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell > wrote: > > > Then Hadoop should add one and although we would need a reflection based > > check in the interim we can converge toward the ideal. > > > > In any case I believe we can avoid a direct dependency on Ozone and should > > strongly avoid taking such unnecessary dependencies. The Hadoop and HBase > > build dependency sets are already very large and we and other users are > > being hit with significant security issue remediation work, much of which > > represents compatibility problems and is not upstreamable (like protobuf 2 > > removal in 2.x). We struggle with the existing dependencies enough already > > at my employer. > > > > > On Mar 15, 2023, at 1:53 PM, Sean Busbey wrote: > > > > > > the check that Stephen is referring to is for logic around lease > > recovery > > > and not stream flush/sync. the lease recovery is specific to DFS IIRC and > > > doesn't have a FileSystem marker. > > > > > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell > > wrote: > > >> > > >> So we can test StreamCapabilities in code, in worst case by wrapping > > some > > >> probe code during startup with try-catch and examining the exception. > > >> > > >>> On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani > > wrote: > > >>> > > >>> As of today, both WAL impl (fshlog and asyncfs) throw > > >>> StreamLacksCapabilityException if the FS Data OutputStream probe fails > > >> for > > >>> Hflush/Hsync: > > >>> > > >>> StreamLacksCapabilityException(StreamCapabilities.HFLUSH) > > >>> and > > >>> StreamLacksCapabilityException(StreamCapabilities.HSYNC) > > >>> > > >>> > > >>> On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell > > >>> wrote: > > >>> > > Does Hadoop have a marker interface that lets an application know its > > FileSystem instances can support hsync/hflush? Ideally all we should > > >> need > > to do is test with instanceof for that marker and use reflection (in > > >> the > > worst case) to get a handle to the hsync or hflush method, and then > > >> call > > it. This approach should be taken wherever we have a requirement to > > >> use a > > special WAL specific API provided by the underlying FileSystem, so we > > >> can > > abstract it sufficiently to not require a direct dependency on Ozone > > or > > >>> S3A > > or any non HDFS filesystem. > > > > On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu < > > >> tak...@apache.org > > > > wrote: > > > > > Hi team, > > > > > > Recently, Wei-Chiu and I have been discussing about if HBase can use > > > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) > > > and HFile, for HFile it’s pluggable by configuring the file system to > > > use Ozone File System (Ozone) > > > > > > But we found that the WAL it’s a bit different, especially > > > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if > > > the file system is an instance of HDFS, and thus WAL recovery to > > > execute file lease recovery from RS crashes. Here, if we would like > > >> to > > > add Ozone, it does not matter by importing as a direct dependency to > > > perform similar lease recovery or via reflection by class name in > > > plaintext String, we still need to somehow introduce Ozone to be > > > another supported file system. (we can discuss how we can implement > > > better as well) > > > > > > We also found other places e.g. FSUtils and HFileSystem have used > > > DistributedFileSystem, but it should be able to move them into either > > > hbase-asyncfs or a new FS related component to separate the use of > > > diff
[DISCUSS] Flatten plugin and project metadata
Heya, At the bottom of [0], we're having a small discussion about the impact of the flatten-maven-plugin on the poms we publish. For background, since we've started using Maven's "CI Friendly Versions" feature, we have also started using the flatten plugin on install/deploy [1]. I notice that using the default configurations, we lose almost all of the project/community metadata from the poms. I'm wondering if we should go out of our way to preserve this extra info in our poms, or if it's really just extra weight in the distribution that users can find on our website. Whichever path we choose, we should apply the setting across all of our project repositories. Thanks, Nick [0]: https://github.com/apache/hbase-operator-tools/pull/112 [1]: https://maven.apache.org/maven-ci-friendly.html#install-deploy
Re: [DISCUSS] Add Ozone as dependency to hbase-asyncfs ?
Thanks everyone ! Sean helped to clarify that something like DFS specific APIs used by HBase has been in-place in many HBase modules as the feature implementation but yet standardized in hadoop general FileSystem API, e.g. lease recovery. One of the original worries is if the Hadoop/HDFS community would reject our proposal when we change the base interface/abstract class in FileSystem (if it's non-backward compatible). The discussion here helps to confirm the direction, and let's see how we can make it generic and could help to avoid confusion in both places. Thanks again, Stephen On Wed, Mar 15, 2023 at 2:54 PM Andrew Purtell wrote: > Then Hadoop should add one and although we would need a reflection based > check in the interim we can converge toward the ideal. > > In any case I believe we can avoid a direct dependency on Ozone and should > strongly avoid taking such unnecessary dependencies. The Hadoop and HBase > build dependency sets are already very large and we and other users are > being hit with significant security issue remediation work, much of which > represents compatibility problems and is not upstreamable (like protobuf 2 > removal in 2.x). We struggle with the existing dependencies enough already > at my employer. > > > On Mar 15, 2023, at 1:53 PM, Sean Busbey wrote: > > > > the check that Stephen is referring to is for logic around lease > recovery > > and not stream flush/sync. the lease recovery is specific to DFS IIRC and > > doesn't have a FileSystem marker. > > > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell > wrote: > >> > >> So we can test StreamCapabilities in code, in worst case by wrapping > some > >> probe code during startup with try-catch and examining the exception. > >> > >>> On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani > wrote: > >>> > >>> As of today, both WAL impl (fshlog and asyncfs) throw > >>> StreamLacksCapabilityException if the FS Data OutputStream probe fails > >> for > >>> Hflush/Hsync: > >>> > >>> StreamLacksCapabilityException(StreamCapabilities.HFLUSH) > >>> and > >>> StreamLacksCapabilityException(StreamCapabilities.HSYNC) > >>> > >>> > >>> On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell > >>> wrote: > >>> > Does Hadoop have a marker interface that lets an application know its > FileSystem instances can support hsync/hflush? Ideally all we should > >> need > to do is test with instanceof for that marker and use reflection (in > >> the > worst case) to get a handle to the hsync or hflush method, and then > >> call > it. This approach should be taken wherever we have a requirement to > >> use a > special WAL specific API provided by the underlying FileSystem, so we > >> can > abstract it sufficiently to not require a direct dependency on Ozone > or > >>> S3A > or any non HDFS filesystem. > > On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu < > >> tak...@apache.org > > wrote: > > > Hi team, > > > > Recently, Wei-Chiu and I have been discussing about if HBase can use > > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) > > and HFile, for HFile it’s pluggable by configuring the file system to > > use Ozone File System (Ozone) > > > > But we found that the WAL it’s a bit different, especially > > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if > > the file system is an instance of HDFS, and thus WAL recovery to > > execute file lease recovery from RS crashes. Here, if we would like > >> to > > add Ozone, it does not matter by importing as a direct dependency to > > perform similar lease recovery or via reflection by class name in > > plaintext String, we still need to somehow introduce Ozone to be > > another supported file system. (we can discuss how we can implement > > better as well) > > > > We also found other places e.g. FSUtils and HFileSystem have used > > DistributedFileSystem, but it should be able to move them into either > > hbase-asyncfs or a new FS related component to separate the use of > > different supported file systems. > > > > So, we’re wondering if anyone would have any objections to adding > > Ozone as a dependency to hbase-asyncfs? or if you have a better idea > > how this could be added without adding Ozone as dependency, please > > feel free to comment on this thread. > > > > > > [1] Ozone is working on support for hsync and hflush, > > https://issues.apache.org/jira/browse/HDDS-7593, > > https://issues.apache.org/jira/browse/HDDS-4353 > > [2] RecoverLeaseFSUtils#recoverFileLease, > > > > > > >>> > >> > https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 > > > > Thanks, > > Stephen > >>> > >> >
