Wes, I think we understand your administrative prerogative. I think in both Julia and Rust cases, you just have engineers that want to go fast and do very needed, deep changes for security and performance. I think, and this is just my wild guess, that to go through the Apahce process (because it's slow and administered by part timers) for such exploratory work would be detrimental, because it would slow said communities down.
I think we should let them do their 'exploratory development branches', and commit/merge back whatever they want to/feel they have got to a material contribution/end point. I think you did something similar with Pandas2 and feather/arrow. Either way, we should be praising and encouraging Jorge to go fast and see it as a development branch, and figure out how to remerge as we see the value of the new features. I think at this point, they are going and we should just encourage them and realize there is enough support in the Rust and Julia community to deal with the merge down the road post rewrite. I do see Jorges Arrow2 is substantially better in many vectors and solves a lot of problems that have been plaguing Arrow. Also we need a fundamental rewrite of APIs to make a rust version of Zero copy work, which could mean we have to have a slightly different arrow format in how it interacts with memory. " I do think that I was vocal enough. At some point the interactions here started to affect my wellbeing and I thus decided to scale down by efforts." On Thu, Apr 8, 2021 at 6:03 AM Wes McKinney <wesmck...@gmail.com> wrote: > On Thu, Apr 8, 2021 at 7:49 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > With both what has occurred with the Julia project and what may > > possibly be occurring (what appears to me to be occurring) with these > > Rust overhaul projects, is that the communities expectations with > > regards to Openness aren't being followed. > > > > If a change is significant and will affect other developers in the > > community, such as a large refactor or a large PR that will interfere > > with development in some portion of the codebase (example, I wrote a > > large and disruptive C++ PR last may that affected all of our compute > > kernels), then it needs to be discussed in a public place as soon as > > you are aware of it. The Openness tenet is about an obligation that > > individuals have to communicate with their fellow collaborators. > > For the record, the reason I'm making a fuss about this is that a > contributor proposed a contribution moratorium in the Apache project > as the result of work happening outside the community > > "would like to raise the idea of temporarily suspending major PRs against > Rust > Arrow/DataFusion until the work to incorporate the two big changes for > Rust/DataFusion: > > 1. Jorge's major refactor/rewrite of the core Rust Arrow code. > ... > " > > I'm not trying to make things difficult for you all, but this looks > like the kind of thing we would like to avoid. > > > Jorge has said that "that [the community] is unwilling to change some > > of its processes" — if changes would involve allowing development to > > happen in private or outside the community (as what has occurred with > > Julia), it is simply not possible to accept these changes. We have > > said repeatedly that we would accommodate Julia's needs with respect > > to releasing. > > > > The discussions that we have had recently have centered around > > communication questions. > > > > * Mailing list discussions serve to raise awareness around matters of > > importance to individuals in the community > > * Issues serve to raise awareness around prospective or in progress > > projects, and to help document the community's labors > > > > Every PMC member has an obligation to ensure that communications in > > the project remain healthy and in concordance with the Apache Way so > > that we do not devolve into a small cabal of developers who understand > > what is going on, while anyone outside the cabal can't understand just > > by reading our public communication. Other open source projects do > > that, but we don't do it here. > > > > If the Rust developers collectively want to move everything to a > > different GitHub repository and use GitHub issues, let's go ahead and > > do it and live with the consequences. But these principles above > > simply must be followed. > > > > On Thu, Apr 8, 2021 at 2:32 AM Antoine Pitrou <anto...@python.org> > wrote: > > > > > > On Thu, 8 Apr 2021 00:26:57 -0700 > > > Julian Hyde <jhyde.apa...@gmail.com> wrote: > > > > Antoine, > > > > > > > > I need to correct your assertion > > > > > > > > > we develop on the side every day when we submit PRs from forks; > > > > > it's just a matter of how much complexity is being submitted at > once > > > > > > > > Intuitively, there seems to be a continuum between a PR developed > within a project to a major feature/codebase developed outside of it. But > the ASF treats these contributions differently, and requires the latter > kind have to go through the IP clearance process. > > > > > > > > You might wonder where the line lies. The policy [1] defines it as > follows: > > > > > > > > > a substantial contribution that was not developed within the > > > > > ASF's source control system and on our public mailing lists > > > > > > Well, a PR developed in a fork was not developed within the ASF's > > > source control system. > > > > > > (and most PRs to Arrow constitute substantial contributions in my > > > experience) > > > > > > Regards > > > > > > Antoine. > > > > > > >
