Then Hadoop should add one and although we would need a reflection based check in the interim we can converge toward the ideal.
In any case I believe we can avoid a direct dependency on Ozone and should strongly avoid taking such unnecessary dependencies. The Hadoop and HBase build dependency sets are already very large and we and other users are being hit with significant security issue remediation work, much of which represents compatibility problems and is not upstreamable (like protobuf 2 removal in 2.x). We struggle with the existing dependencies enough already at my employer. > On Mar 15, 2023, at 1:53 PM, Sean Busbey <bus...@apache.org> wrote: > > the check that Stephen is referring to is for logic around lease recovery > and not stream flush/sync. the lease recovery is specific to DFS IIRC and > doesn't have a FileSystem marker. > >> On Wed, Mar 15, 2023 at 3:22 PM Andrew Purtell <apurt...@apache.org> wrote: >> >> So we can test StreamCapabilities in code, in worst case by wrapping some >> probe code during startup with try-catch and examining the exception. >> >>> On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani <vjas...@apache.org> wrote: >>> >>> As of today, both WAL impl (fshlog and asyncfs) throw >>> StreamLacksCapabilityException if the FS Data OutputStream probe fails >> for >>> Hflush/Hsync: >>> >>> StreamLacksCapabilityException(StreamCapabilities.HFLUSH) >>> and >>> StreamLacksCapabilityException(StreamCapabilities.HSYNC) >>> >>> >>> On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell <apurt...@apache.org> >>> wrote: >>> >>>> Does Hadoop have a marker interface that lets an application know its >>>> FileSystem instances can support hsync/hflush? Ideally all we should >> need >>>> to do is test with instanceof for that marker and use reflection (in >> the >>>> worst case) to get a handle to the hsync or hflush method, and then >> call >>>> it. This approach should be taken wherever we have a requirement to >> use a >>>> special WAL specific API provided by the underlying FileSystem, so we >> can >>>> abstract it sufficiently to not require a direct dependency on Ozone or >>> S3A >>>> or any non HDFS filesystem. >>>> >>>> On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu < >> tak...@apache.org >>>> >>>> wrote: >>>> >>>>> Hi team, >>>>> >>>>> Recently, Wei-Chiu and I have been discussing about if HBase can use >>>>> Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) >>>>> and HFile, for HFile it’s pluggable by configuring the file system to >>>>> use Ozone File System (Ozone) >>>>> >>>>> But we found that the WAL it’s a bit different, especially >>>>> RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if >>>>> the file system is an instance of HDFS, and thus WAL recovery to >>>>> execute file lease recovery from RS crashes. Here, if we would like >> to >>>>> add Ozone, it does not matter by importing as a direct dependency to >>>>> perform similar lease recovery or via reflection by class name in >>>>> plaintext String, we still need to somehow introduce Ozone to be >>>>> another supported file system. (we can discuss how we can implement >>>>> better as well) >>>>> >>>>> We also found other places e.g. FSUtils and HFileSystem have used >>>>> DistributedFileSystem, but it should be able to move them into either >>>>> hbase-asyncfs or a new FS related component to separate the use of >>>>> different supported file systems. >>>>> >>>>> So, we’re wondering if anyone would have any objections to adding >>>>> Ozone as a dependency to hbase-asyncfs? or if you have a better idea >>>>> how this could be added without adding Ozone as dependency, please >>>>> feel free to comment on this thread. >>>>> >>>>> >>>>> [1] Ozone is working on support for hsync and hflush, >>>>> https://issues.apache.org/jira/browse/HDDS-7593, >>>>> https://issues.apache.org/jira/browse/HDDS-4353 >>>>> [2] RecoverLeaseFSUtils#recoverFileLease, >>>>> >>>>> >>>> >>> >> https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 >>>>> >>>>> Thanks, >>>>> Stephen >>> >>
