Somewhat related, I'd love to see the overall JNI footprint go down where possible. For example, HADOOP-17895 proposes switching to the newer JDK APIs for chmod.
Chris Nauroth On Fri, Nov 11, 2022 at 10:50 AM Iñigo Goiri <elgo...@gmail.com> wrote: > Gautham, thank you very much for the summary. > Do you have a time-line for when we can get rid of winutils? > My idea was to get this and the YARN federation hardening work into a 3.4 > release. > > > > On Fri, Nov 11, 2022, 10:15 Gautham Banasandra <gaur...@apache.org> wrote: > >> Hi folks, >> >> >> What have we done so far? >> ------------------------------------ >> Inigo and I have been working for quite some time now on this topic, >> but our efforts have mostly been oriented towards making Hadoop >> cross-platform compatible. Our focus has been on streamlining the >> process of building Hadoop on Windows so that one can easily >> build and run Hadoop, just like on Linux. We reached this milestone >> quite recently and I've documented the steps for doing so here - >> >> https://github.com/apache/hadoop/blob/5bb11cecea136acccac2563b37021b554e517012/BUILDING.txt#L493-L622 >> >> >> >> Is winutils still required? >> ------------------------------- >> As Steve mentioned, we would still require winutils for running >> Hadoop on Windows. The major change here is that winutils >> need not come from a third-party repository anymore, rather it >> gets built along with the Hadoop codebase itself henceforth. >> However, I agree that we need to deprecate winutils and >> replace it with something better so that Hadoop users can have >> a smoother experience. >> >> >> What's the best way to deprecate winutils? >> -------------------------------------------------------- >> Over all the time that I've spent making Hadoop cross-platform >> compatible, I've realized that the best way would be to have a >> JNI interface that wraps around a native layer. This native layer >> could be implemented majorly in C++. C++17 provides the >> std::filesystem namespace that can satisfy most of the native >> filesystem API requirements. Since std::filesystem is part of "The >> Standard Libray", these APIs will be present on most/all the C++ >> compilers of the various OS platforms. For those parts that can't >> be satisfied by std::filesystem, we'll have to delve into this part >> by writing C code that makes system calls. Please note that >> these C files will need to be implemented specifically for each >> platform. I took this approach when I wrote x-platform library to >> make HDFS native client cross-platform compatible - >> >> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform >> >> >> What am I focussing on currently? >> ------------------------------------------------ >> So far, I've focussed on getting the build to work seamlessly >> on Windows. I'm now trying to protect this from breaking by >> setting up CI on Jenkins that builds Hadoop on Windows >> for the precommit validation - >> https://issues.apache.org/jira/browse/INFRA-23809 >> Yes, it does involve getting >> Yetus to run on Windows. I can work on deprecating winutils >> after this. >> >> Thanks, >> --Gautham >> >> On Fri, 11 Nov 2022 at 19:51, Steve Loughran <ste...@cloudera.com.invalid> >> wrote: >> >>> It's time to reach for the axe. >>> >>> We haven't shipped eight version of Apache hadoop which builds and runs >>> on >>> windows for a long long time. I the only people trying to use the library >>> is on windows Will have been people trying to use spark on their laptops >>> with "small" dataset of only a are few tens of gigabytes at a time, the >>> kind of work where 32GB of ram and 16 cores is enough. Put differently: >>> in >>> storage and performance of Single laptop means that it is perfectly >>> suitable for doing reasonable amounts of work and the main barrier to >>> doing >>> so is getting a copy of the winutils lib. >>> >>> I know Gautham and Inigo I trying to get windows to work as a location >>> for >>> yarn again; not sure about hdfs. And there, yes, we have to say "they >>> likely to need an extra binary" >>> >>> But for someone wanting to count the number of rows in an avro file? do a >>> simple bit of filtering on some parquet data? Is these are the kind of >>> things that anyone with a linux/mac laptop can do with ease and it is not >>> fair to put suffering on to others. And well we could just say "why do >>> you >>> just install Lynnox on that laptop then?", I have someone who has had a >>> Linux laptop for many years I know the written strong arguments against >>> it >>> even beyond the "my employer demand windows with their IT software" as "a >>> latop which comes out of sleep reliably" is kind of important too. >>> >>> I how can we let the people who have to live in this world – And we have >>> someone who is clearly willing to help –Live a better life. Funnily >>> enough, >>> the fact that we have not shipped a working version of when you tails >>> for a >>> long time actually gives us an advantage: we can pick incompatible >>> changes >>> and be confident that most people aren't going to notice. >>> >>> I think a good first step would be for Shell to work well if winutils >>> isn't >>> around -get rid of that static, WINUTILS string and path/file >>> equivalents, >>> the ones deprecated in 2015. We can rip them out knowing no external code >>> is using them. >>> >>> Then we should look very closely at FileUtil to see how much of that is >>> needed and how can we isolate it better. If you look at the change log of >>> that file, we do have to consider that every time it execs a shell >>> command >>> I there's a security risk and more than once we've had to fix it. Not >>> executing any external shell commands is good everywhere. >>> >>> >>> >>> >>> On Thu, 10 Nov 2022 at 19:00, Chris Nauroth <cnaur...@apache.org> wrote: >>> >>> > Symlink support on the local file system is still used. One example I >>> can >>> > think of is YARN container launch [1]. >>> > >>> > I would welcome removal of winutils, as already described in various >>> JIRA >>> > issues. I think the biggest challenge we'll have is testing of a >>> transition >>> > from winutils to the newer Java APIs. The contract tests help, but >>> > historically there was also a tendency to break things in downstream >>> > dependent projects. >>> > >>> > I'd suggest taking this on piecemeal, transitioning small pieces of >>> > FileSystem off of winutils one at a time. >>> > >>> > [1] >>> > >>> > >>> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L1508-L1509 >>> > >>> > Chris Nauroth >>> > >>> > >>> > On Thu, Nov 10, 2022 at 10:33 AM Wei-Chiu Chuang <weic...@apache.org> >>> > wrote: >>> > >>> > > > >>> > > > >>> > > > >>> > > > * Bare Naked Local File System v0.1.0 doesn't (yet) support >>> symlinks >>> > > > or the sticky bit. >>> > > > >>> > > ok to not support symlinks. The symlinks of HDFS are not being >>> maintained >>> > > and I am not aware of anything relying on it. >>> > > So I assume people don't need it. >>> > > >>> > > Sticky bit would be useful, I guess. >>> > > >>> > > I suppose folks working at Microsoft would be more interested in this >>> > work? >>> > > Last time I heard, Gautham and Inigo were revamping Hadoop's Windows >>> > > support. >>> > > >>> > > >>> > > > * But the bigger issue is how to excise Winutils completely in >>> the >>> > > > existing Hadoop code. Winutils assumptions are hard-coded at a >>> low >>> > > > level across various classes—even code that has nothing to do >>> with >>> > > > the file system. The startup configuration for example calls >>> > > > `StringUtils.equalsIgnoreCase("true", valueString)` which >>> loads the >>> > > > `StringUtils` class, which has a static reference to `Shell`, >>> which >>> > > > has a static block that checks for `WINUTILS_EXE`. >>> > > > * For the most part there should no longer even be a need for >>> > anything >>> > > > but direct Java API access for the local file system. But >>> muddling >>> > > > things further, the existing `RawLocalFileSystem` >>> implementation >>> > has >>> > > > /four/ ways to access the local file system: Winutils, JNI >>> calls, >>> > > > shell access, and a "new" approach using "stat". The "stat" >>> > approach >>> > > > has been switched off with a hard-coded >>> `useDeprecatedFileStatus = >>> > > > true` because of HADOOP-9652 >>> > > > <https://issues.apache.org/jira/browse/HADOOP-9652>. >>> > > > * Local file access is not contained within `RawLocalFileSystem` >>> but >>> > > > is scattered across other classes; `FileUtil.readLink()` for >>> > example >>> > > > (which `RawLocalFileSystem` calls because of the deprecation >>> issue >>> > > > above) uses the shell approach without any option to change it. >>> > > > (This implementation-specific decision should have been >>> contained >>> > > > within the `FileSystem` implementation itself.) >>> > > > >>> > > > In short, it's a mess that has accumulated over years and getting >>> > worse, >>> > > > charging high interest on what at first was a small, self-contained >>> > > > technical debt. >>> > > > >>> > > > I would welcome the opportunity to clean up this mess. I'm >>> probably as >>> > > > qualified as anyone to make the changes. This is one of my areas of >>> > > > expertise: I was designing a full abstract file system interface >>> (with >>> > > > pure-Java from-scratch implementations for the local file system, >>> > > > Subversion, and WebDAV—even the WebDAV HTTP implementation was from >>> > > > scratch) around the time Apache Nutch was getting off the ground. >>> Most >>> > > > recently I've worked on the Hadoop `FileSystem` API contracting for >>> > > > LinkedIn, discovering (what I consider to be) a huge bug in >>> > > > ViewFilesystem, HADOOP-18525 >>> > > > <https://issues.apache.org/jira/browse/HADOOP-18525>. >>> > > > >>> > > > The cleanup should be done in several stages (e.g. consolidating >>> > > > WinUtils access; replacing code with pure Java API calls; >>> undeprecating >>> > > > the new Stat code and relegating it to a different class, etc.). >>> > > > Unfortunately it's not financially feasible for me to sit here for >>> > > > several months and revamp the Hadoop `FileSystem` subsystem for fun >>> > > > (even though I wish I could). Perhaps there is job opening at a >>> company >>> > > > related to Hadoop that would be interested in hiring me and >>> devoting a >>> > > > certain percentage of my time to fixing local `FileSystem` access. >>> If >>> > > > so, let me know where I should send my resume >>> > > > <https://www.garretwilson.com/about/resume>. >>> > > > >>> > > > Otherwise let me know if any ideas for a way forward. If there >>> proves >>> > to >>> > > > be interest in GlobalMentor Hadoop Bare Naked Local FileSystem >>> > > > <https://github.com/globalmentor/hadoop-bare-naked-local-fs> on >>> GitHub >>> > > > I'll try to maintain and improve it, but really what needs to be >>> > > > revamped is the Hadoop codebase itself. I'll be happy when Hadoop >>> is >>> > > > fixed so that both Steve's code and my code are no longer needed. >>> > > > >>> > > > Garret >>> > > > >>> > > >>> > >>> >>
