Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-20 Thread Micah Kornfield
> > I suggest we respect Jorge's decision and give him more time/space. > Considering he spent 6 months last year trying to donate arrow2 to ASF > and couldn't get it through, Of course, I apologize if I wasn't being respectful. However, this is the first I'm hearing, and I would guess others on

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-19 Thread QP Hou
I suggest we respect Jorge's decision and give him more time/space. Considering he spent 6 months last year trying to donate arrow2 to ASF and couldn't get it through, I can see why he doesn't want to retry the same effort in a short period of time. Especially when he is working on all of these in

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-19 Thread Micah Kornfield
Hi Jorge, Thanks for your response. My current hypothesis is that arrow2 will be donated to Apache Arrow, I > just don't feel comfortable and have the energy doing so right now. Aside from the burden of going through the donation process (maybe other PMCs members are willing to help here?),

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-19 Thread Jorge Cardoso Leitão
Hi, Thank you for raising this here and for your comments. I am very humbled by the feedback and adoption that arrow2 got so far. My current hypothesis is that arrow2 will be donated to Apache Arrow, I just don't feel comfortable and have the energy doing so right now. Thank you for your

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-17 Thread Wes McKinney
Sounds good, thanks all and look forward to hearing more about this. To second what Micah said, a reminder to please engage with civility. The ASF's code of conduct is found here [1]. We are all volunteering our time to try to do what is best for the developer and user communities' long-term

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-17 Thread Andrew Lamb
For what it is worth, I personally would likely spend much less time maintaining arrow-rs if datafusion switched to arrow2. That discussion is happening independently here [1]. [1] https://github.com/apache/arrow-datafusion/issues/1532 On Sun, Jan 16, 2022 at 11:17 PM Micah Kornfield wrote: >

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-16 Thread Micah Kornfield
I agree, Jorge's point of view (and anyone else who has contributed to arrow2) is important here. One thing that isn't exactly clear to me from the linked issue is how much interest there is in the community for maintaining arrow-rs? How much is a donation of arrow2 a factor here? Also, the

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-16 Thread QP Hou
Hi Wes, I believe what you mentioned is the plan, i.e. move arrow2 to ASF in the long run when it stabalize on its design/API and could benefit from a more rigorous release process. From what I have seen, the project is still undergoing major API changes on a monthly basis, so quick releases and

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-16 Thread Benjamin Blodgett
Why don’t we gift the whole arrow project to Jorge if he’s the one pushing the tech forward? I’d say the governance would be better Sent from my iPhone > On Jan 16, 2022, at 2:42 PM, Wes McKinney wrote: > > Is there a possibility of donating arrow2 to the Arrow project (at > some point)?

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-16 Thread Benjamin Blodgett
Jorge did such a great job making THE new and better columnar in mem format. Arrow 2 is just better and the future, but it should stay under Jorge’s clearly more capable hands. We should all be thanking him. I had a such a a good belly laugh at ‘is there a possibility of donating’. Pleaser

Re: [RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-16 Thread Wes McKinney
Is there a possibility of donating arrow2 to the Arrow project (at some point)? The main impact to development would be holding votes on releases, but this is probably a good thing long term from a governance standpoint. The answer may be "not right now" and that's fine. Having many of the same

[RUST][DataFusion][Arrow] Switching DataFusion to use arrow2 implementation and the future of arrow

2022-01-14 Thread Andrew Lamb
Hello, I wanted to draw your attention to two issues of significance to the Rust Arrow implementations Discussion for switching DataFusion to use arrow2: https://github.com/apache/arrow-datafusion/issues/1532 Discussion for what to do with arrow if DataFusion switches to use arrow2: