Thanks. I will review today and start a vote about the code donation. I know that we already voted to accept in Apache Parquet, but I want to double check that the Arrow community is also on board with sharing responsibility for this code. If the Parquet community wants to make Rust Parquet releases, this is something that can be discussed and handled later.
Do any corporations have IP rights in this code (to see if a Software Grant Agreement is necessary)? Otherwise your ICLAs are sufficient. On Thu, Nov 29, 2018 at 12:45 AM Chao Sun <sunc...@apache.org> wrote: > > I just opened a pull request <https://github.com/apache/arrow/pull/3050> for > the donation. Let me know if this is in the correct format and whether it's > OK to proceed to the next steps. > > Thanks, > Chao > > On Wed, Nov 21, 2018 at 9:18 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > Is the change to the nightly compiler the kind of thing that the PMC > > should vote on? > > > > A vote is not necessary if there is reasonable consensus. > > On Wed, Nov 21, 2018 at 9:11 PM paddy horan <paddyho...@hotmail.com> > > wrote: > > > > > > I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll > > take a closer look when I get a chance. I agree that lack of support for 32 > > bit should not hold this up. > > > > > > Is the change to the nightly compiler the kind of thing that the PMC > > should vote on? This could be done in advance of the code donantion. > > Specialization, in particular, would be really useful within the existing > > code base. > > > > > > Paddy > > > > > > Get Outlook for iOS > > > ________________________________ > > > From: Chao Sun <sunc...@apache.org> > > > Sent: Wednesday, November 21, 2018 2:42 PM > > > To: dev@arrow.apache.org > > > Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev > > > Subject: Re: [DISCUSS] Rust add adapter for parquet > > > > > > > A bigger issue is windows support for parquet-rs, Chao – I don’t > > believe > > > that parquet-rs supports windows, right? When I test it myself I get > > > errors regarding clang and libclang which parquet-rs must use. > > > > > > I think there are some issue regarding clang working with zstd on 32-bit > > > platform. However, it was able to compile with target: > > x86_64-pc-windows-msvc, > > > which seems to be the only one that arrow is using for windows CI. So I > > > think we can move forward and address the 32-bit platform issue later. > > > > > > On Wed, Nov 21, 2018 at 10:18 AM paddy horan <paddyho...@hotmail.com> > > wrote: > > > > > > > I think using nightly is fine for the reasons mentioned already. We > > > > should switch our CI to still run CI against stable (non fatal) so we > > know > > > > when we can make the move back to stable. > > > > > > > > > > > > > > > > A bigger issue is windows support for parquet-rs, Chao – I don’t > > believe > > > > that parquet-rs supports windows, right? When I test it myself I get > > > > errors regarding clang and libclang which parquet-rs must use. > > > > > > > > > > > > > > > > We have had windows support pretty much since the beginning for Rust. > > Is > > > > it possible to put parquet support behind a feature gate initially and > > only > > > > run CI for non-windows? I would be willing to help get windows support > > > > working after the fact, although I know very little about parquet > > right now. > > > > > > > > > > > > > > > > Are there other strategies for dealing with this? > > > > > > > > > > > > > > > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for > > > > Windows 10 > > > > > > > > > > > > > > > > ________________________________ > > > > From: Chao Sun <sunc...@apache.org> > > > > Sent: Wednesday, November 21, 2018 12:52:32 PM > > > > To: Wes McKinney > > > > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev > > > > Subject: Re: [DISCUSS] Rust add adapter for parquet > > > > > > > > > Can you remind us all why nightly is required? > > > > > > > > Here's a tracking issue < > > https://github.com/sunchao/parquet-rs/issues/119> > > > > for all the unstable features parquet-rs uses. I'm personally inclined > > to > > > > use nightly since some new features such as specialization makes > > > > development much easier. Like Andy mentioned, as we are still > > developing > > > > arrow + parquet I think we can stay with nightly and transition back to > > > > stable once major features are implemented and more and more people > > start > > > > to use it (hopefully by that time the unstable features are > > stabilized). > > > > Moreover, I've seen quite a few popular projects rely on nightly such > > as > > > > rocket, tikv, etc., so seems it is not uncommon in the Rust world. > > > > > > > > > The steps from here are for you all to get the codebase into a state > > > > that is ready for donation, including ASF license headers, etc. A pull > > > > request into apache/arrow would be the best thing > > > > > > > > Sure. I'll prepare a pull request in the next few days, and then we can > > > > proceed to the voting, ICLA, etc. Thanks. > > > > > > > > Chao > > > > > > > > > > > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > > > The steps from here are for you all to get the codebase into a state > > > > > that is ready for donation, including ASF license headers, etc. A > > pull > > > > > request into apache/arrow would be the best thing > > > > > > > > > > Then we have to do the following > > > > > > > > > > * Vote on the Arrow mailing list > > > > > * Receive ICLAs from contributors > > > > > * Complete IP clearance > > > > > * Merge codebase > > > > > > > > > > Let me know when you are ready to move forward. From start to finish > > > > > that can get done in approximately 6 days if the code is ready > > > > > > > > > > Thanks > > > > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <andygrov...@gmail.com> > > > > wrote: > > > > > > > > > > > > Renjie, > > > > > > > > > > > > Can you remind us all why nightly is required? > > > > > > > > > > > > My personal feeling is that stable is a nice-to-have, but Rust is > > still > > > > > moving fast and we are on the bleeding edge here so I'm OK with Arrow > > > > > relying on nightly for now. Maybe we can have a plan to transition > > back > > > > to > > > > > stable for a future release if we go with nightly now. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andy. > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu < > > liurenjie2...@gmail.com> > > > > > wrote: > > > > > >> > > > > > >> That sounds great. But parquet-rs currently relies on nightly > > rust, > > > > that > > > > > >> would be the first problem to resolve. > > > > > >> > > > > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrov...@gmail.com > > > > > > > > wrote: > > > > > >> > > > > > >> > This sounds like a great idea. > > > > > >> > > > > > > >> > With support for both CSV and Parquet in the Arrow crate, it > > would > > > > be > > > > > nice > > > > > >> > to design a standard interface for Arrow data sources. Maybe > > this is > > > > > as > > > > > >> > simple as implementing `Iterator<RecordBatch>`. > > > > > >> > > > > > > >> > Andy. > > > > > >> > > > > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <sunc...@apache.org> > > > > wrote: > > > > > >> > > > > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge > > > > this > > > > > into > > > > > >> > > Arrow because of the issues that you pointed out with parquet > > c++ > > > > > merge, > > > > > >> > > and I do see a tight relationship between the two projects, > > and > > > > > potential > > > > > >> > > sharing of common libraries. @Ivan Sadikov < > > > > ivan.sadi...@gmail.com> > > > > > what > > > > > >> > > do you think? > > > > > >> > > > > > > > >> > > Chao > > > > > >> > > > > > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney < > > > > wesmck...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > > >> > >> hi folks, > > > > > >> > >> > > > > > >> > >> Would you all be interested in moving forward the parquet-rs > > > > > project? > > > > > >> > >> I have a little more bandwidth to help with the code > > donation in > > > > > the > > > > > >> > >> next month or two. > > > > > >> > >> > > > > > >> > >> I know we voted on the Parquet mailing list about the > > donation > > > > > >> > >> already. One big question is whether you want to create an > > > > > >> > >> apache/parquet-rs repository or whether you want to > > co-develop > > > > > >> > >> parquet-rs together with Arrow in Rust, similar to what we > > are > > > > > doing > > > > > >> > >> with C++. It's possible you might run into the same kinds of > > > > issues > > > > > >> > >> that led us to consider the monorepo arrangement. > > > > > >> > >> > > > > > >> > >> Thanks > > > > > >> > >> Wes > > > > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu < > > > > > liurenjie2...@gmail.com> > > > > > >> > >> wrote: > > > > > >> > >> > > > > > > >> > >> > Hi, Chao: > > > > > >> > >> > I've opened an jira issue for that and planning to work on > > > > that. > > > > > >> > >> > > > > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu < > > > > > liurenjie2...@gmail.com> > > > > > >> > >> wrote: > > > > > >> > >> > > > > > > >> > >> > > Yes, it's a mistake, sorry for that > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun < > > > > sunc...@apache.org> > > > > > >> > wrote: > > > > > >> > >> > > > > > > > >> > >> > >> (s/flink/arrow - it is a mistake?) > > > > > >> > >> > >> > > > > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next > > step > > > > in > > > > > >> > >> parquet-rs > > > > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had > > a > > > > > >> > discussion > > > > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> > > about > > > > this > > > > > >> > >> recently. > > > > > >> > >> > >> Feel free to share your comments on the github. > > > > > >> > >> > >> > > > > > >> > >> > >> Best, > > > > > >> > >> > >> Chao > > > > > >> > >> > >> > > > > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu < > > > > > >> > liurenjie2...@gmail.com > > > > > >> > >> > > > > > > >> > >> > >> wrote: > > > > > >> > >> > >> > > > > > >> > >> > >>> cc:Sunchao and Any > > > > > >> > >> > >>> > > > > > >> > >> > >>> > > > > > >> > >> > >>> ---------- Forwarded message --------- > > > > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com> > > > > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM > > > > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet > > > > > >> > >> > >>> To: <dev@arrow.apache.org> > > > > > >> > >> > >>> > > > > > >> > >> > >>> > > > > > >> > >> > >>> Hello, > > > > > >> > >> > >>> > > > > > >> > >> > >>> you might also want to raise this with the > > > > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The > > overlap > > > > > >> > between > > > > > >> > >> the > > > > > >> > >> > >>> developers of this project and the Arrow Rust > > > > implementation > > > > > is > > > > > >> > >> quite large > > > > > >> > >> > >>> but still it may make sense to also start a discussion > > > > there. > > > > > >> > >> > >>> > > > > > >> > >> > >>> Uwe > > > > > >> > >> > >>> > > > > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote: > > > > > >> > >> > >>> > Hi, all: > > > > > >> > >> > >>> > > > > > > >> > >> > >>> > Now the rust component is approaching a stable state > > and > > > > > rust > > > > > >> > >> reader > > > > > >> > >> > >>> for > > > > > >> > >> > >>> > parquet is ready. I think it maybe a good time to > > start > > > > an > > > > > >> > >> adapter for > > > > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you > > guys > > > > > think > > > > > >> > >> about it? > > > > > >> > >> > >>> > -- > > > > > >> > >> > >>> > Liu, Renjie > > > > > >> > >> > >>> > Software Engineer, MVAD > > > > > >> > >> > >>> -- > > > > > >> > >> > >>> Liu, Renjie > > > > > >> > >> > >>> Software Engineer, MVAD > > > > > >> > >> > >>> > > > > > >> > >> > >> > > > > > >> > >> > >> -- > > > > > >> > >> > > Liu, Renjie > > > > > >> > >> > > Software Engineer, MVAD > > > > > >> > >> > > > > > > > >> > >> > -- > > > > > >> > >> > Liu, Renjie > > > > > >> > >> > Software Engineer, MVAD > > > > > >> > >> > > > > > >> > > > > > > > >> > > > > > > >> -- > > > > > >> Renjie Liu > > > > > >> Software Engineer, MVAD > > > > > > > > > > >
