As of today, both WAL impl (fshlog and asyncfs) throw StreamLacksCapabilityException if the FS Data OutputStream probe fails for Hflush/Hsync:
StreamLacksCapabilityException(StreamCapabilities.HFLUSH) and StreamLacksCapabilityException(StreamCapabilities.HSYNC) On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell <apurt...@apache.org> wrote: > Does Hadoop have a marker interface that lets an application know its > FileSystem instances can support hsync/hflush? Ideally all we should need > to do is test with instanceof for that marker and use reflection (in the > worst case) to get a handle to the hsync or hflush method, and then call > it. This approach should be taken wherever we have a requirement to use a > special WAL specific API provided by the underlying FileSystem, so we can > abstract it sufficiently to not require a direct dependency on Ozone or S3A > or any non HDFS filesystem. > > On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu <tak...@apache.org> > wrote: > > > Hi team, > > > > Recently, Wei-Chiu and I have been discussing about if HBase can use > > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) > > and HFile, for HFile it’s pluggable by configuring the file system to > > use Ozone File System (Ozone) > > > > But we found that the WAL it’s a bit different, especially > > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if > > the file system is an instance of HDFS, and thus WAL recovery to > > execute file lease recovery from RS crashes. Here, if we would like to > > add Ozone, it does not matter by importing as a direct dependency to > > perform similar lease recovery or via reflection by class name in > > plaintext String, we still need to somehow introduce Ozone to be > > another supported file system. (we can discuss how we can implement > > better as well) > > > > We also found other places e.g. FSUtils and HFileSystem have used > > DistributedFileSystem, but it should be able to move them into either > > hbase-asyncfs or a new FS related component to separate the use of > > different supported file systems. > > > > So, we’re wondering if anyone would have any objections to adding > > Ozone as a dependency to hbase-asyncfs? or if you have a better idea > > how this could be added without adding Ozone as dependency, please > > feel free to comment on this thread. > > > > > > [1] Ozone is working on support for hsync and hflush, > > https://issues.apache.org/jira/browse/HDDS-7593, > > https://issues.apache.org/jira/browse/HDDS-4353 > > [2] RecoverLeaseFSUtils#recoverFileLease, > > > > > https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 > > > > Thanks, > > Stephen > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
