Re: Arrow Flight in C

2021-04-08 Thread Sutou Kouhei
Hi, I hope that we implement Apache Arrow Flight bindings to Apache Arrow C GLib by 5.0.0. Are you interested in development them together? If you aren't interested in it, you can implement glue module in C++ by yourself. C interface in Apache Arrow C++ will help you: https://arrow.apache.org/doc

Arrow Flight in C

2021-04-08 Thread Tanveer Ahmad - EWI
Hi, I need some help in sending Arrow RecordBatches over Arrow Flight inside a C application. As there is no interface for Arrow Flight is available for Arrow CGlib. Does someone have some custom C interface/suggestions to use C++ functions for Arrow Flight inside a C application. Thanks. R

Re: [ANNOUNCE] [Rust] Ballista donation has been merged

2021-04-08 Thread Wes McKinney
Congrats Andy! I know this was a lot of work, but I think it speaks to a bright future for the Arrow ecosystem. Once we can sort out the release and code management concerns to suit the needs of the Rust ecosystem, I trust you will all be on a good path. On Thu, Apr 8, 2021 at 6:05 PM Andy Grove

Re: Rust sync meeting

2021-04-08 Thread Andy Grove
Apologies for not participating in this discussion earlier today. > If the Rust developers collectively want to move everything to a different GitHub repository and use GitHub issues, let's go ahead and do it and live with the consequences. But these principles above simply must be followed. I do

[ANNOUNCE] [Rust] Ballista donation has been merged

2021-04-08 Thread Andy Grove
Hi everyone. The Ballista IP Clearance vote passed earlier today and I have now merged the Ballista donation into the Arrow repo [1]. I have also filed some JIRA issues [2] for some of the follow-up work that is required from moving the code to a new location as well as for some of the features th

Re: Standard/Reserved Metadata keys

2021-04-08 Thread Neal Richardson
Like Pandas, in the R package we store R-specific metadata in the schema; this lets us preserve all R class and other attributes when round-tripping data to Feather or Parquet. These are not stored in a way that is readable or useful to other implementations, partly by design. If there were type in

Re: Rust sync meeting

2021-04-08 Thread Wes McKinney
Thanks for pointing this out. I'd be supportive of the same type of policy for Arrow (unversioned "revolution" branches -- or separate git repositories -- where development happens in a less regimented fashion — i.e. without pull requests, issues, etc., but with periodic written community updates).

Re: Standard/Reserved Metadata keys

2021-04-08 Thread Micah Kornfield
1. Do the standard libraries handle that metadata key automatically ? C++, Python and Java have facilities to support them automatically (extensions needs to register themselves), I'm not sure about other languages. 2. Are there other standard or best practice metadata keys that either people g

Standard/Reserved Metadata keys

2021-04-08 Thread Michael Lavina
Hey Team, I noticed that under extension type the Arrow docs specifically call out to `ARROW:extension:name` and `ARROW:extension:metadata` as recommended/reserved metadata keys to handle extension types. 2 quick questions 1. Do the standard libraries handle that metadata key automaticall

Re: Rust sync meeting

2021-04-08 Thread Julian Hyde
It is a propos to mention "rules for revolutionaries", a way of making big changes in projects by forking. It was proposed by James Duncan Davidson, then PMC chair of Tomcat, in 2000, but remains popular within the ASF (as evidenced by this blog post [1] from 2020 by Bertrand Delacrétaz, ASF board

Re: Rust sync meeting

2021-04-08 Thread Adam Lippai
Didn't this happen in the thread "[Rust] [Discuss] proposal to redesign Arrow crate to resolve safety violations" on 7th February before any commit in arrow2 (resulting in zero discussion or any objection)? Best regards, Adam Lippai On Thu, Apr 8, 2021 at 4:41 PM Wes McKinney wrote: > Hi Ben, >

Re: Rust sync meeting

2021-04-08 Thread Wes McKinney
Hi Ben, I’m not suggesting adding any extra development process to slow things down on an experimental project like this. My principle objection is to the lack of discussion on the public record. A short DISCUSS email from Jorge explaining the project and soliciting feedback from the community wo

Re: Rust sync meeting

2021-04-08 Thread Benjamin Blodgett
Wes, I think we understand your administrative prerogative. I think in both Julia and Rust cases, you just have engineers that want to go fast and do very needed, deep changes for security and performance. I think, and this is just my wild guess, that to go through the Apahce process (because it

Re: Rust sync meeting

2021-04-08 Thread Wes McKinney
On Thu, Apr 8, 2021 at 7:49 AM Wes McKinney wrote: > > With both what has occurred with the Julia project and what may > possibly be occurring (what appears to me to be occurring) with these > Rust overhaul projects, is that the communities expectations with > regards to Openness aren't being foll

Re: Rust sync meeting

2021-04-08 Thread Wes McKinney
With both what has occurred with the Julia project and what may possibly be occurring (what appears to me to be occurring) with these Rust overhaul projects, is that the communities expectations with regards to Openness aren't being followed. If a change is significant and will affect other develo

Re: [C++] (Eventually) merging asynchronous datasets feature

2021-04-08 Thread Weston Pace
Ok, so I have begun splitting ARROW-7001 into smaller tasks that eventually create an AsyncScanner. The plan... ARROW-12286 & ARROW-12287 are minor utilities that could have been split out anyways. ARROW-12288 Creates a `Scanner` interface and cleans up the existing implementation somewhat. This

[NIGHTLY] Arrow Build Report for Job nightly-2021-04-08-0

2021-04-08 Thread Crossbow
Arrow Build Report for Job nightly-2021-04-08-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-08-0 Failed Tasks: - gandiva-jar-ubuntu: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-08-0-github-gandiva-jar-ubuntu -

Re: Rust sync meeting

2021-04-08 Thread Antoine Pitrou
On Thu, 8 Apr 2021 00:26:57 -0700 Julian Hyde wrote: > Antoine, > > I need to correct your assertion > > > we develop on the side every day when we submit PRs from forks; > > it's just a matter of how much complexity is being submitted at once > > Intuitively, there seems to be a continuum be

Re: Rust sync meeting

2021-04-08 Thread Julian Hyde
Antoine, I need to correct your assertion > we develop on the side every day when we submit PRs from forks; > it's just a matter of how much complexity is being submitted at once Intuitively, there seems to be a continuum between a PR developed within a project to a major feature/codebase devel