Hi, I am writing this as an open source contributor and main developer of arrow2 and parquet2.
I am really happy to see so much excitement over these two projects. I am pretty excited about them too. They deliver higher performance (e.g. +10x reading parquet without parallelism) and security (e.g. parquet2 has zero usages of `unsafe`). They result from investigations on how to correct a major design issue in the current arrow implementation that exposes multiple hard-to-find security vulnerabilities. They constitute a re-write of the official implementations with security as a mandatory feature, as required by both Apache <https://www.apache.org/foundation/how-it-works.html#philosophy> and Rust <https://doc.rust-lang.org/1.2.0/complement-design-faq.html>. The development of those two repositories is happening outside of ASF, as per where they reside. My plan was to propose a donation to Apache Arrow once stabilized. However, imo Apache Arrow has recently shown, through how it engaged with Julia and Ballista's communities, that it is unwilling to change some of its processes to cater for ecosystem-specific standard release practices and familiar development experiences. Consequently, donating arrow2 and parquet2 to Apache Arrow would be a regression. I am hoping we can find a way of bridging this gap and bring this code base to Apache Arrow in the near future. Best, Jorge On Wed, Apr 7, 2021 at 1:28 PM Wes McKinney <wesmck...@gmail.com> wrote: > I'm sorry to be the PMC worry wart around here, but I'm curious what > is the plan (if any) with these repositories > > https://github.com/jorgecarleitao/arrow2 > https://github.com/jorgecarleitao/parquet2 > > I understand that large new projects like this are sometimes > necessary, but what some Apache projects have done is set up a project > branch where the work can take place without bothering the master > branch. We can of course also create new standalone git repositories > under the apache/ namespace for experimental work if you aren't sure > it's going to get merged into master. What we want to avoid is having > significant work happening outside the community. > > On Wed, Apr 7, 2021 at 6:19 AM Mike Seddon <seddo...@gmail.com> wrote: > > > > Hi, > > Sorry I am unable to attend today's (tomorrow's?) call but would like to > > raise the idea of temporarily suspending major PRs against Rust > > Arrow/DataFusion until the work to incorporate the two big changes for > > Rust/DataFusion: > > > > 1. Jorge's major refactor/rewrite of the core Rust Arrow code. > > 2. Andy's Ballista donation (I am not sure how much this will affect the > > core Arrow/DataFusion code). > > > > I am able to help do some of the grunt work involved with merging these > > pieces if I can be of assistance - but I feel the time has come to at > least > > face the major Rust Arrow work for all the reasons Jorge has previously > put > > forward. > > > > Cheers, > > Mike > > > > On Thu, Mar 25, 2021 at 10:41 AM Andy Grove <andygrov...@gmail.com> > wrote: > > > > > Hi Mike, > > > > > > I recently added the call details in the Arrow Rust README, here: > > > https://github.com/apache/arrow/tree/master/rust#arrow-rust-community > > > > > > I've added you to the guest list so it should sync with your calendar > now. > > > I'm happy to do this for anyone else that wants me to add them. > > > > > > Unfortunately, I couldn't make the call today and it looks like there > were > > > issues with letting people into the call. I've just been looking at the > > > settings and I'm not sure how I can change that. > > > > > > Regarding the timing of the call, I am open to moving it if we can > find a > > > time that accommodates everyone reasonably well. I think we have > regular > > > attendance from the US, UK, Europe, South Africa, and Australia so far. > > > > > > Thanks, > > > > > > Andy. > > > > > > > > > > > > On Wed, Mar 24, 2021 at 5:24 PM Mike Seddon <seddo...@gmail.com> > wrote: > > > > > > > Hi Jorge, > > > > Can you please confirm the starting time (and timezone) and correct > > > Google > > > > Meet link of the Rust and DataFusion sync calls? > > > > > > > > I missed today's due to daylight saving time changes - which is > going to > > > > make Sydney time even harder. > > > > > > > > Thanks > > > > Mike > > > > > > > > On Thu, Mar 25, 2021 at 3:15 AM Jorge Cardoso Leitão < > > > > jorgecarlei...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > If someone is trying to join the meeting, please try this link: > > > > > https://meet.google.com/pgx-xycf-zuf > > > > > > > > > > Best, > > > > > Jorge > > > > > > > > > > > > >
