I do have a WiP library to hide that hoop-jumping behind a normal API, with a goal of 3.2+ support only. It does actually compile against hadoop 2, but it isn't tested there
https://github.com/steveloughran/fs-api-shim 1. my goal is to make it a hadoop library like hadoop-third-party, with its own release cycle etc 2. testing is complex as it needs to have contract tests (lifted from hadoop-common) applied through all versions of hadoop so far openFile(), ByteBufferPositionedReadable, PathCapabilities *and* VectorIO are in, though the latter has no tests yet. If we can get this to work then (a) HBase can delegate suffering and (b) libraries like parquet & ORC can use vector IO while still compiling against older versions On Tue, 28 Mar 2023 at 19:02, Nick Dimiduk <ndimi...@apache.org> wrote: > On Mon, Mar 27, 2023 at 20:29 Wei-Chiu Chuang <weic...@apache.org> wrote: > > > For complex applications such as > > HBase it is almost impossible to achieve true FS agnosticity without > proper > > contract tests, as now I am starting to realize. > > > > This is absolutely true. HBase jumps through all sorts of painful > reflective hoops to achieve reliable behavior across Hadoop versions. > Steve’s proposal of self-describing APIs over opaque implementations would > be drastically better than our current approach of reflection for > inspecting and interacting with the internal details of any given client > implementation. > > Thank you very much for your studious pursuit of this goal. > > Thanks, > Nick > > On Mon, Mar 27, 2023 at 4:58 AM Steve Loughran <ste...@cloudera.com.invalid > > > > wrote: > > > > > side issue, as i think about what bulk delete call would also keep > hbase > > > happy > > > https://issues.apache.org/jira/browse/HADOOP-18679 > > > > > > should we think about new API calls only raising RuntimeExceptions? > > > > > > The more work I do on futures the more the way we always raise IOEs > > > complicates life. java has outgrown checked exceptions > > > > > > On Fri, 24 Mar 2023 at 09:44, Steve Loughran <ste...@cloudera.com> > > wrote: > > > > > > > > > > > > > > > On Thu, 23 Mar 2023 at 10:07, Ayush Saxena <ayush...@gmail.com> > wrote: > > > > > > > >> > > > >> Second idea mentioned in the original mail is also similar to > > mentioned > > > in > > > >> the comment in the above ticket and is still quite acceptable, name > > can > > > be > > > >> negotiated though, Add an interface to pull the relevant methods up > in > > > >> that > > > >> without touching FileSystem class, we can have DFS implement that > and > > > >> Ozone > > > >> FS implement them as well. We should be sorted: No Hacking, No > > Bothering > > > >> FileSystem and still things can work > > > >> > > > >> > > > >> > > > > This is the way we should be thinking about it. an interface which > > > > filesystems MAY implement, but many do not. > > > > > > > > this has happened with some of the recent apis. > > > > > > > > presence of the API doesn't guarantee the api is active, only that it > > may > > > > be possible to call...callers should use PathCapabilities api to see > if > > > it > > > > is live > > > > > > > > > > > >> > > > > > >
