Silence from the HDFS team
Hadoop 3.2.2 is in an RC; it has the new FS API call. I really don't want to veto the release just because someone pulled up a method without doing the due diligence. Is anyone in the HDFS going to do that due diligence or should we include something in the release notes "msync()" must be considered unstable. Then we can do a proper msync(). If it goes into FS/FC, what does it do for a viewfs with >1 mounted HDFS? Should it take path, msync(path) so that viewFS knows where to forward it? Alternatively: go with an MSync interface which those few FS which implement it (hdfs) can do that, and the fact that it doesn't have doc or tests won't be a blocker any more? -steve On Thu, 10 Dec 2020 at 12:41, Steve Loughran <ste...@cloudera.com> wrote: > > Gosh, has it really been only since february since I last asked the HDFS > dev list to stop adding anything to FileSystem/FileContext APIs without > > * mentioning this on the hadoop-common list. > * specifying what it does in filesystem.md > * with a contract test > * a new hasPathCapabilities probe. Throwing UnsupportedOperationException > only lets people work out if it is unsupported through invocation. Being > able to probe for it is better. > * ViewFS support. > * And, for any new API, one which works well for high-latency object > stores: returning Future<Something> and Future<RemoteIterator<Something> > when > 1 result is returned > > This needs to hold even for pulling something up from HDFS. Because if > another FS wants to implement it, they need to know what it does, and have > tests to verify this. I say this as someone who has tried to document HDFS > rename() semantics and gave up. > > It's really frustrating that every time someone does an FS API change like > this in the past (most recently HDFS-13616) I am the one who has to keep > sending the reminders out, and then having to try and clean up/. > > So what now? > > Options > 1. I roll HDFS-15567 back "please be follow process" > 2. Someone does a followup patch with specification and contract test, > view FS. Add even more to the java > 3. We do as per HADOOP-16898 into an MSyncable interface and then > FileSystem & HDFS can implement. ViewFS and filterFS still need to pass > through. > > *If nobody is going to volunteer for the specification/test changes, I'm > happy for the rollback. It'll remind people about process, * > > Pre-emptive Warning: No matter what we do for this patch, I will roll back > the next change which adds a new API if it's not accompanied by > specification and tests. > > Unhappily yours, > > Steve >
