Inline > On Mar 15, 2023, at 2:11 PM, Wei-Chiu Chuang <weic...@apache.org> wrote: > > hsync/hflush, or any input/output stream APIs for that matter, can be > probed using StreamCapabilities.hasCapabilitiy() API. > > lease recovery isn't (DistributedFIleSystem.recoverLease()). safe mode > check isn't. There are a number of HDFS specific APIs that HBase uses. > > I'm all for abstracting out FS implementation details. But it would be an > overkill to try and add every single FS specific APIs to the generic > FileSystem API interface. > > One idea that Stephen had was to add a RecoverableFileSystem interface in > Hadoop which adds lease recovery capability, and then HDFS or Ozone can > implement this interface.
Yes, please. > > In a less ideal world, I imagine we could have one hbase module for > utilities that does FS specific tasks. That way it is future proofing. Also a good idea. > >> On Wed, Mar 15, 2023 at 12:50 PM Andrew Purtell <apurt...@apache.org> wrote: >> >> Does Hadoop have a marker interface that lets an application know its >> FileSystem instances can support hsync/hflush? Ideally all we should need >> to do is test with instanceof for that marker and use reflection (in the >> worst case) to get a handle to the hsync or hflush method, and then call >> it. This approach should be taken wherever we have a requirement to use a >> special WAL specific API provided by the underlying FileSystem, so we can >> abstract it sufficiently to not require a direct dependency on Ozone or S3A >> or any non HDFS filesystem. >> >> On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu <tak...@apache.org> >> wrote: >> >>> Hi team, >>> >>> Recently, Wei-Chiu and I have been discussing about if HBase can use >>> Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) >>> and HFile, for HFile it’s pluggable by configuring the file system to >>> use Ozone File System (Ozone) >>> >>> But we found that the WAL it’s a bit different, especially >>> RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if >>> the file system is an instance of HDFS, and thus WAL recovery to >>> execute file lease recovery from RS crashes. Here, if we would like to >>> add Ozone, it does not matter by importing as a direct dependency to >>> perform similar lease recovery or via reflection by class name in >>> plaintext String, we still need to somehow introduce Ozone to be >>> another supported file system. (we can discuss how we can implement >>> better as well) >>> >>> We also found other places e.g. FSUtils and HFileSystem have used >>> DistributedFileSystem, but it should be able to move them into either >>> hbase-asyncfs or a new FS related component to separate the use of >>> different supported file systems. >>> >>> So, we’re wondering if anyone would have any objections to adding >>> Ozone as a dependency to hbase-asyncfs? or if you have a better idea >>> how this could be added without adding Ozone as dependency, please >>> feel free to comment on this thread. >>> >>> >>> [1] Ozone is working on support for hsync and hflush, >>> https://issues.apache.org/jira/browse/HDDS-7593, >>> https://issues.apache.org/jira/browse/HDDS-4353 >>> [2] RecoverLeaseFSUtils#recoverFileLease, >>> >>> >> https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 >>> >>> Thanks, >>> Stephen >>> >> >> >> -- >> Best regards, >> Andrew >> >> Unrest, ignorance distilled, nihilistic imbeciles - >> It's what we’ve earned >> Welcome, apocalypse, what’s taken you so long? >> Bring us the fitting end that we’ve been counting on >> - A23, Welcome, Apocalypse >>
