Apologies for not participating in this discussion earlier today. > If the Rust developers collectively want to move everything to a different GitHub repository and use GitHub issues, let's go ahead and do it and live with the consequences. But these principles above simply must be followed.
I do think that the time is right for us to proceed with this and I will start a new email thread to discuss this as soon as I can. It may be later this evening or more likely tomorrow morning. I want to be careful to think this through first so that we don't just discuss a decision to move the code but also discuss the implications of this and the work that will need to be done to implement new release processes etc. I do think that moving to separate GitHub repo(s), using GitHub issues, and having an independent release process will address the majority of the concerns that I have heard from Arrow / DataFusion / Ballista contributors. I strongly agree with the point that Wes makes that these changes will not change the requirement for us to communicate in the open (mailing list, GitHub issues) and make sure we are communicating any major proposals widely. We should also document the process for future projects like Ballista and Arrow2 so that we can do these kinds of long-running R&D projects within the community. Thanks, Andy. On Thu, Apr 8, 2021 at 1:07 PM Wes McKinney <wesmck...@gmail.com> wrote: > Thanks for pointing this out. I'd be supportive of the same type of > policy for Arrow (unversioned "revolution" branches -- or separate git > repositories -- where development happens in a less regimented fashion > — i.e. without pull requests, issues, etc., but with periodic written > community updates). > > Keep in mind using https://selfserve.apache.org we can create new > apache/* git repositories anytime (so we could create > apache/arrow-rust2, for example) that wouldn't have the same kind of > development process as trunk. > > On Thu, Apr 8, 2021 at 11:08 AM Julian Hyde <jh...@apache.org> wrote: > > > > It is a propos to mention "rules for revolutionaries", a way of making > > big changes in projects by forking. It was proposed by James Duncan > > Davidson, then PMC chair of Tomcat, in 2000, but remains popular > > within the ASF (as evidenced by this blog post [1] from 2020 by > > Bertrand Delacrétaz, ASF board member). > > > > Julian > > > > [1] > https://grep.codeconsult.ch/2020/04/07/rules-for-revolutionaries-2000-edition/ > > > > On Thu, Apr 8, 2021 at 7:53 AM Adam Lippai <a...@rigo.sk> wrote: > > > > > > Didn't this happen in the thread "[Rust] [Discuss] proposal to redesign > > > Arrow crate to resolve safety violations" on 7th February before any > commit > > > in arrow2 (resulting in zero discussion or any objection)? > > > > > > Best regards, > > > Adam Lippai > > > > > > On Thu, Apr 8, 2021 at 4:41 PM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > > > Hi Ben, > > > > > > > > I’m not suggesting adding any extra development process to slow > things down > > > > on an experimental project like this. My principle objection is to > the lack > > > > of discussion on the public record. > > > > > > > > A short DISCUSS email from Jorge explaining the project and > soliciting > > > > feedback from the community would have been enough. It would have > also been > > > > an opportunity to set up auxiliary git repositories or branches to > > > > accommodate the work on ASF channels (to enable collaboration inside > the > > > > community) rather than occurring as a “lone wolf” project outside the > > > > community. > > > > > > > > On Thu, Apr 8, 2021 at 9:03 AM Benjamin Blodgett < > > > > benjaminblodg...@gmail.com> > > > > wrote: > > > > > > > > > Wes, > > > > > > > > > > I think we understand your administrative prerogative. I think in > both > > > > > Julia and Rust cases, you just have engineers that want to go fast > and do > > > > > very needed, deep changes for security and performance. I think, > and > > > > this > > > > > is just my wild guess, that to go through the Apahce process > (because > > > > it's > > > > > slow and administered by part timers) for such exploratory work > would be > > > > > detrimental, because it would slow said communities down. > > > > > > > > > > I think we should let them do their 'exploratory development > branches', > > > > and > > > > > commit/merge back whatever they want to/feel they have got to a > material > > > > > contribution/end point. I think you did something similar with > Pandas2 > > > > and > > > > > feather/arrow. > > > > > > > > > > Either way, we should be praising and encouraging Jorge to go fast > and > > > > see > > > > > it as a development branch, and figure out how to remerge as we > see the > > > > > value of the new features. > > > > > > > > > > I think at this point, they are going and we should just encourage > them > > > > and > > > > > realize there is enough support in the Rust and Julia community to > deal > > > > > with the merge down the road post rewrite. I do see Jorges Arrow2 > is > > > > > substantially better in many vectors and solves a lot of problems > that > > > > have > > > > > been plaguing Arrow. Also we need a fundamental rewrite of APIs > to make > > > > a > > > > > rust version of Zero copy work, which could mean we have to have a > > > > slightly > > > > > different arrow format in how it interacts with memory. > > > > > > > > > > " I do think that I was vocal enough. At some point > > > > > the interactions here started to affect my wellbeing and I thus > decided > > > > to > > > > > scale down by efforts." > > > > > > > > > > On Thu, Apr 8, 2021 at 6:03 AM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > > > > > > > On Thu, Apr 8, 2021 at 7:49 AM Wes McKinney <wesmck...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > With both what has occurred with the Julia project and what may > > > > > > > possibly be occurring (what appears to me to be occurring) > with these > > > > > > > Rust overhaul projects, is that the communities expectations > with > > > > > > > regards to Openness aren't being followed. > > > > > > > > > > > > > > If a change is significant and will affect other developers in > the > > > > > > > community, such as a large refactor or a large PR that will > interfere > > > > > > > with development in some portion of the codebase (example, I > wrote a > > > > > > > large and disruptive C++ PR last may that affected all of our > compute > > > > > > > kernels), then it needs to be discussed in a public place as > soon as > > > > > > > you are aware of it. The Openness tenet is about an obligation > that > > > > > > > individuals have to communicate with their fellow > collaborators. > > > > > > > > > > > > For the record, the reason I'm making a fuss about this is that a > > > > > > contributor proposed a contribution moratorium in the Apache > project > > > > > > as the result of work happening outside the community > > > > > > > > > > > > "would like to raise the idea of temporarily suspending major PRs > > > > against > > > > > > Rust > > > > > > Arrow/DataFusion until the work to incorporate the two big > changes for > > > > > > Rust/DataFusion: > > > > > > > > > > > > 1. Jorge's major refactor/rewrite of the core Rust Arrow code. > > > > > > ... > > > > > > " > > > > > > > > > > > > I'm not trying to make things difficult for you all, but this > looks > > > > > > like the kind of thing we would like to avoid. > > > > > > > > > > > > > Jorge has said that "that [the community] is unwilling to > change some > > > > > > > of its processes" — if changes would involve allowing > development to > > > > > > > happen in private or outside the community (as what has > occurred with > > > > > > > Julia), it is simply not possible to accept these changes. We > have > > > > > > > said repeatedly that we would accommodate Julia's needs with > respect > > > > > > > to releasing. > > > > > > > > > > > > > > The discussions that we have had recently have centered around > > > > > > > communication questions. > > > > > > > > > > > > > > * Mailing list discussions serve to raise awareness around > matters of > > > > > > > importance to individuals in the community > > > > > > > * Issues serve to raise awareness around prospective or in > progress > > > > > > > projects, and to help document the community's labors > > > > > > > > > > > > > > Every PMC member has an obligation to ensure that > communications in > > > > > > > the project remain healthy and in concordance with the Apache > Way so > > > > > > > that we do not devolve into a small cabal of developers who > > > > understand > > > > > > > what is going on, while anyone outside the cabal can't > understand > > > > just > > > > > > > by reading our public communication. Other open source > projects do > > > > > > > that, but we don't do it here. > > > > > > > > > > > > > > If the Rust developers collectively want to move everything to > a > > > > > > > different GitHub repository and use GitHub issues, let's go > ahead and > > > > > > > do it and live with the consequences. But these principles > above > > > > > > > simply must be followed. > > > > > > > > > > > > > > On Thu, Apr 8, 2021 at 2:32 AM Antoine Pitrou < > anto...@python.org> > > > > > > wrote: > > > > > > > > > > > > > > > > On Thu, 8 Apr 2021 00:26:57 -0700 > > > > > > > > Julian Hyde <jhyde.apa...@gmail.com> wrote: > > > > > > > > > Antoine, > > > > > > > > > > > > > > > > > > I need to correct your assertion > > > > > > > > > > > > > > > > > > > we develop on the side every day when we submit PRs from > forks; > > > > > > > > > > it's just a matter of how much complexity is being > submitted at > > > > > > once > > > > > > > > > > > > > > > > > > Intuitively, there seems to be a continuum between a PR > developed > > > > > > within a project to a major feature/codebase developed outside > of it. > > > > But > > > > > > the ASF treats these contributions differently, and requires the > latter > > > > > > kind have to go through the IP clearance process. > > > > > > > > > > > > > > > > > > You might wonder where the line lies. The policy [1] > defines it > > > > as > > > > > > follows: > > > > > > > > > > > > > > > > > > > a substantial contribution that was not developed within > the > > > > > > > > > > ASF's source control system and on our public mailing > lists > > > > > > > > > > > > > > > > Well, a PR developed in a fork was not developed within the > ASF's > > > > > > > > source control system. > > > > > > > > > > > > > > > > (and most PRs to Arrow constitute substantial contributions > in my > > > > > > > > experience) > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
