Hi, Sorry for the delay.
If there is a path towards an official release under a <1.0.0 versioning schema aligned with the rest of the Rust ecosystem and in line with the stability of the API, then IMO we should move all development to within Apache experimental asap (I can handle this and the likely IP clearance round). If we require a release >=1.X.Y to it and/or a schedule, then I prefer to keep expectations aligned and postpone any movement. Under the move situation, I was thinking in something as follows: * gradually stop maintaining "arrow" in crates, offering a maintenance window over which we release patches (*) * work towards achieving feature parity on arrow2/parquet2 on the experimental repos. * keep releasing arrow2/parquet2 under a 0.X model during the step above (**) * migrate to arrow-rs and archive experimentals (***) * break arrow2 in smaller crates so that we can version the APIs at a different cadence * once a crate reaches some stability (this is always opinionated, but it is fine), we bump it to 1.0 and announce a maintenance plan ala tokio <https://tokio.rs/blog/2020-12-tokio-1-0>. (*) e.g. "we will continue to patch the arrow crate up to at least 6 months starting after the first release of arrow2 that supports a) nested parquet read and write b) union array (including IPC integration tests) c) map array (including IPC integration tests)" (**) officially or un-officially (I would suggest officially so that we can acknowledge everyone's work on it, but no strong feelings) (***) something like: 1. place arrow2 on top of a clear arrow repo so that the full contribution history up to that point preserved 2. make arrow-rs the home of arrow2 (i.e. we start releasing arrow2 from arrow-rs) and archive the experimental repos; create arrow-rs-parquet or something for parquet2. In summary, the core pain point for me is the current versioning of arrow, which I feel is incompatible with my goals for arrow2 and the ecosystem I envision it supporting :) Best, Jorge On Fri, Jul 30, 2021 at 8:44 PM Wes McKinney <wesmck...@gmail.com> wrote: > I think it would also be fine to push “beta” arrow2 crates out of a repo > under apache/ so long as they are not marked on crates.io as being > Apache-official releases. There’s a possible slippery slope there, but as > long as we are on a path to formalizing the releases I think it is okay. > > On Fri, Jul 30, 2021 at 1:07 PM Andrew Lamb <al...@influxdata.com> wrote: > > > Jorge -- do you feel like we have a resolution on what to do with arrow2 > in > > the near term? > > > > The current state of affairs seems to me that arrow2 is released from > > https://github.com/jorgecarleitao/arrow2 to crates.io (which is fine). > > Are > > you happy with keeping development in the jorgecarleitao repo where you > > will retain maximal control and flexibility until it is ready to start > > integrating? > > > > Or would you prefer to put it into one of the apache repos and subject > its > > development and release to the normal Arrow governance model (tarball, > > vote, etc)? > > > > Since you are the primary author/architect I think you should have a > > substantial say at this stage. > > > > Andrew > > > > > > On Tue, Jul 27, 2021 at 7:16 PM Andrew Lamb <al...@influxdata.com> > wrote: > > > > > I would be happy with this approach. Thank you for the suggestion > > > > > > This hybrid approach of both arrow and arrow2 in the same repo seems > > > better to me than separate repos. > > > > > > What I really care about is ensuring we don't have two crates/APIs > > > indefinitely -- as long as we are continually making progress towards > > > unification that is what is important to me. > > > > > > Andrew > > > > > > On Tue, Jul 27, 2021 at 1:40 PM Andy Grove <andygrov...@gmail.com> > > wrote: > > > > > >> Apologies for being late to this discussion. > > >> > > >> There is a hybrid option to consider here where we add the arrow2 code > > >> into > > >> the arrow crate as a separate module, so we release one crate > containing > > >> the "old" API (which we can mark as deprecated) as well as the new > API. > > >> Java did a similar thing a long time ago with "java.io" versus > > "java.nio" > > >> (new IO). > > >> > > >> I agree that the versioning wouldn't be ideal, but this seems like it > > >> might > > >> be a pragmatic compromise? > > >> > > >> Thanks, > > >> > > >> Andy. > > >> > > >> > > >> On Tue, Jul 20, 2021 at 5:41 AM Andrew Lamb <al...@influxdata.com> > > wrote: > > >> > > >> > What I meant is that when you decide arrow2 is suitable for release > to > > >> > existing arrow users, I stand ready to help you incorporate it into > > >> arrow. > > >> > > > >> > All the feedback I have heard so far from the rest of the community > is > > >> that > > >> > we are ready. One might even say we are anxious to do so :) > > >> > > > >> > Andrew > > >> > > > >> > > > > > >
