Re: way forward for Winutils excision from `FileSystem`

ok, so let's start: what the easy bits we do to make life easier for people trying to use the hadoop file:// client on a windows box, which essentially means "people trying to use spark standalone?" torg.apache.hadoop.util.Shell is the obvious first target. i had a go at getting it fail more

Re: way forward for Winutils excision from `FileSystem`

As Chris mentioned earlier, it would be wise to do this in pieces that can be reviewed properly. Bringing large refactorings in all at once, as Garret mentioned, is not likely to just get a +1. We do have a feature branch process and criteria and we could determine specific criteria for such a

Re: way forward for Winutils excision from `FileSystem`

On 11/29/2022 8:16 AM, Gautham Banasandra wrote: … However, I don't see anyone stopping you from working on removing winutils. I encourage you to put across a PR and I would be glad to review the same. That's not how it works. This is an intense undertaking. If I spend six months with no

Re: way forward for Winutils excision from `FileSystem`

Hi Steve, I feel we won't have the confidence with major changes until we have a CI that runs the full suite of tests. Thanks, --Gautham On Tue, 29 Nov 2022 at 16:12, Steve Loughran wrote: > he's got a point. > > what is to stop us cutting it from trunk? > > On Mon, 28 Nov 2022 at 18:12,

Re: way forward for Winutils excision from `FileSystem`

Hi Garret, My responses hitherto indicate that I'm currently focusing on setting up a CI for Windows for building Hadoop and getting the unit tests to run on it. I believe, without these, no matter the improvements, the build is bound to be broken someday. I would thus like to prioritize setting

Re: way forward for Winutils excision from `FileSystem`

he's got a point. what is to stop us cutting it from trunk? On Mon, 28 Nov 2022 at 18:12, Garret Wilson wrote: > On 11/14/2022 12:04 PM, Gautham Banasandra wrote: > > … > I would like to aim for winutils deprecation by the end of the first > quarter of 2023. > > The key word is "deprecation".

Re: way forward for Winutils excision from `FileSystem`

On 11/14/2022 12:04 PM, Gautham Banasandra wrote: … I would like to aim for winutils deprecation by the end of the first quarter of 2023. The key word is "deprecation". Note that the sentence doesn't say the references in the code to Winutils will be removed by this time. I offered to

Re: way forward for Winutils excision from `FileSystem`

Hi Steve, There are quite a few unit test failures on Windows. Some of them are major ones. I feel we need to fix those before we do a release. I'm trying to focus on setting up the CI so that we can incrementally fix the unit tests and avoid any possible regressions. Thanks, --Gautham On Tue,

Re: way forward for Winutils excision from `FileSystem`

you up for doing a winutils build on the 3.3.5 release? i'm going to do the arm64 binaries On Mon, 14 Nov 2022 at 18:08, Gautham Banasandra wrote: > Also, I plan to do a Windows release once I setup the CI for Windows and > after I get > the major unit tests to pass. It would still contain

Re: way forward for Winutils excision from `FileSystem`

Also, I plan to do a Windows release once I setup the CI for Windows and after I get the major unit tests to pass. It would still contain winutils though. However, we can do another release after deprecating winutils. Thanks, --Gautham On Mon, 14 Nov 2022 at 23:34, Gautham Banasandra wrote: >

Re: way forward for Winutils excision from `FileSystem`

Hi Iñigo, I would like to aim for winutils deprecation by the end of the first quarter of 2023. It really depends on how fast I can wrap up with setting up CI for Windows. Given that this involves getting Yetus to work properly on Windows, I feel it's a bit ambitious. But if things fall into

Re: way forward for Winutils excision from `FileSystem`

Somewhat related, I'd love to see the overall JNI footprint go down where possible. For example, HADOOP-17895 proposes switching to the newer JDK APIs for chmod. Chris Nauroth On Fri, Nov 11, 2022 at 10:50 AM Iñigo Goiri wrote: > Gautham, thank you very much for the summary. > Do you have a

Re: way forward for Winutils excision from `FileSystem`

Gautham, thank you very much for the summary. Do you have a time-line for when we can get rid of winutils? My idea was to get this and the YARN federation hardening work into a 3.4 release. On Fri, Nov 11, 2022, 10:15 Gautham Banasandra wrote: > Hi folks, > > > What have we done so far? >

Re: way forward for Winutils excision from `FileSystem`

Hi folks, What have we done so far? Inigo and I have been working for quite some time now on this topic, but our efforts have mostly been oriented towards making Hadoop cross-platform compatible. Our focus has been on streamlining the process of building

Re: way forward for Winutils excision from `FileSystem`

It's time to reach for the axe. We haven't shipped eight version of Apache hadoop which builds and runs on windows for a long long time. I the only people trying to use the library is on windows Will have been people trying to use spark on their laptops with "small" dataset of only a are few tens

Re: way forward for Winutils excision from `FileSystem`

Symlink support on the local file system is still used. One example I can think of is YARN container launch [1]. I would welcome removal of winutils, as already described in various JIRA issues. I think the biggest challenge we'll have is testing of a transition from winutils to the newer Java

Re: way forward for Winutils excision from `FileSystem`

> > > > * Bare Naked Local File System v0.1.0 doesn't (yet) support symlinks > or the sticky bit. > ok to not support symlinks. The symlinks of HDFS are not being maintained and I am not aware of anything relying on it. So I assume people don't need it. Sticky bit would be useful, I guess.

way forward for Winutils excision from `FileSystem`

Steve Loughran and I have been discussing on Stack Overflow a way forward for removing the Winutils requirement from the local `FileSystem` implementations. Hadoop's FileSystem API has a lot of *nix assumptions which originally made it not possible to