Thanks Steve very much for your discussion here. Leave some comments inline. Will focus on this thread to wait for the final conclusion to decide if we should prepare another release candidate of 3.2.2. Thanks Steve and Chao again for your warm discussions.
On Sat, Dec 12, 2020 at 7:18 PM Steve Loughran <ste...@cloudera.com.invalid> wrote: > Maybe it's not in the release; it's certainly in the 3.2 branch. Will check > further. If it's in the release I was thinking of adding a warning in the > notes "unstable API"; stable if invoked from DFSClient On Fri, 11 Dec 2020 at 18:21, Chao Sun <sunc...@apache.org> wrote: > > > I'm just curious why this is included in the 3.2.2 release? HDFS-15567 is > > tagged with 3.2.3 and the corresponding HDFS-14272 on server side is > tagged > > with 3.3.0. > Just checked that HDFS-15567 has been involved in Hadoop-3.2.2 RC4. IIRC, I have cut branch-3.2.2 in early October, at that time branch-3.2.3 has created but source code not freeze completely because several blocked issues reported and code freeze has done about mid October. Some issues which are tagged with 3.2.3 has also been involved in 3.2.2 during that period, include HDFS-15567. I will check them later, and make sure that we have mark the correct tags. > > > > > If it goes into FS/FC, what does it do for a viewfs with >1 mounted > HDFS? > > Should it take path, msync(path) so that viewFS knows where to forward > it? > > > > The API shouldn't take any path - for viewFS I think it should call this > on > > all the child file systems. It might also need to handle the case where > > some downstream clusters support this capability while others don't. > > > > That's an extra bit of work for ViewFS then. It should probe for capability > and invoke as/when supported. > > > > > > Options > > 1. I roll HDFS-15567 back "please be follow process" > > 2. Someone does a followup patch with specification and contract test, > view > > FS. Add even more to the java > > 3. We do as per HADOOP-16898 into an MSyncable interface and then > > FileSystem & HDFS can implement. ViewFS and filterFS still need to pass > > through. > > > > I'm slightly in favor of the hasPathCapabilities approach and make this a > > mixin where FS impls can optionally support. Happy to hear what others > > think. > > > > Mixins are great when FC and FS can both implement; makes it easier to code > against either. All the filtering/aggregating FS's will have to implement > it, which means that presence of the interface doesn't guarantee support. > > This is an API where it'd be ok to have a no-op if not implemented, > correct? Or is there an requirement like Syncable that specific guarantees > are met? > > > > > Chao > > > > > > On Fri, Dec 11, 2020 at 9:00 AM Steve Loughran > <ste...@cloudera.com.invalid > > > > > wrote: > > > > > Silence from the HDFS team > > > > > > > > > Hadoop 3.2.2 is in an RC; it has the new FS API call. I really don't > want > > > to veto the release just because someone pulled up a method without > doing > > > the due diligence. > Thanks Steve started this discussion here. I agree to roll back HDFS-15567 if there are still some incompatible issues not resolved completely. And release will not be the blocked things here, I would like to prepare another RC if we would reach common agreement. To be honest, I think it is better to involve Shvachko here. > > > Is anyone in the HDFS going to do that due diligence or should we > include > > > something in the release notes "msync()" must be considered unstable. > > > > > > Then we can do a proper msync(). > > > > > > If it goes into FS/FC, what does it do for a viewfs with >1 mounted > HDFS? > > > Should it take path, msync(path) so that viewFS knows where to forward > > it? > > > > > > Alternatively: go with an MSync interface which those few FS which > > > implement it (hdfs) can do that, and the fact that it doesn't have doc > or > > > tests won't be a blocker any more? > > > > > > -steve > > > > > > > > > > > > > > > On Thu, 10 Dec 2020 at 12:41, Steve Loughran <ste...@cloudera.com> > > wrote: > > > > > > > > > > > Gosh, has it really been only since february since I last asked the > > HDFS > > > > dev list to stop adding anything to FileSystem/FileContext APIs > without > > > > > > > > * mentioning this on the hadoop-common list. > > > > * specifying what it does in filesystem.md > > > > * with a contract test > > > > * a new hasPathCapabilities probe. Throwing > > UnsupportedOperationException > > > > only lets people work out if it is unsupported through invocation. > > Being > > > > able to probe for it is better. > > > > * ViewFS support. > > > > * And, for any new API, one which works well for high-latency object > > > > stores: returning Future<Something> and > > Future<RemoteIterator<Something> > > > > when > 1 result is returned > > > > > > > > This needs to hold even for pulling something up from HDFS. Because > if > > > > another FS wants to implement it, they need to know what it does, and > > > have > > > > tests to verify this. I say this as someone who has tried to document > > > HDFS > > > > rename() semantics and gave up. > > > > > > > > It's really frustrating that every time someone does an FS API change > > > like > > > > this in the past (most recently HDFS-13616) I am the one who has to > > keep > > > > sending the reminders out, and then having to try and clean up/. > > > > > > > > So what now? > > > > > > > > Options > > > > 1. I roll HDFS-15567 back "please be follow process" > > > > 2. Someone does a followup patch with specification and contract > test, > > > > view FS. Add even more to the java > > > > 3. We do as per HADOOP-16898 into an MSyncable interface and then > > > > FileSystem & HDFS can implement. ViewFS and filterFS still need to > pass > > > > through. > > > > > > > > *If nobody is going to volunteer for the specification/test changes, > > I'm > > > > happy for the rollback. It'll remind people about process, * > > > > > > > > Pre-emptive Warning: No matter what we do for this patch, I will roll > > > back > > > > the next change which adds a new API if it's not accompanied by > > > > specification and tests. > > > > > > > > Unhappily yours, > > > > > > > > Steve > > > > > > > > > >
