[Format] Bounded numbers?

2021-06-21 Thread Ying Zhou
Hi, In data people use there are often bounded numbers, mostly integers with clear and fixed upper and lower bounds but also decimals and floats as well e.g. test scores, numerous codes in older databases, max temperature of a city, latitudes, longitudes, numerous IDs etc. I wonder whether we

Re: [Format][Important] Needed clarification of timezone-less timestamps

2021-06-21 Thread Weston Pace
I agree that a vote would be a good idea. Do you want to start a dedicated vote thread? I can write one up too if you'd rather. -Weston On Mon, Jun 21, 2021 at 4:54 PM Micah Kornfield wrote: > > I think comments on the doc are tailing off. Jorge's test cases I think > still need some more

Re: [Format][Important] Needed clarification of timezone-less timestamps

2021-06-21 Thread Micah Kornfield
I think comments on the doc are tailing off. Jorge's test cases I think still need some more careful analysis but Weston has provided an initial pass. The matter not resolved on the document is whether Timestamp with timezone logically represents multi-field date and time (that does not

Re: [ANNOUNCE] New Arrow PMC member: David M Li

2021-06-21 Thread Micah Kornfield
Congrats! On Mon, Jun 21, 2021 at 7:40 PM Weston Pace wrote: > Congratulations David! > > On Mon, Jun 21, 2021 at 2:24 PM Niranda Perera > wrote: > > > > Congrats David! :-) > > > > On Mon, Jun 21, 2021 at 6:32 PM Nate Bauernfeind < > nate.bauernfe...@gmail.com> > > wrote: > > > > >

Re: [ANNOUNCE] New Arrow PMC member: David M Li

2021-06-21 Thread Weston Pace
Congratulations David! On Mon, Jun 21, 2021 at 2:24 PM Niranda Perera wrote: > > Congrats David! :-) > > On Mon, Jun 21, 2021 at 6:32 PM Nate Bauernfeind > wrote: > > > Congratulations! Well earned! > > > > On Mon, Jun 21, 2021 at 4:20 PM Ian Cook wrote: > > > > > Congratulations, David! > > >

Re: [ANNOUNCE] New Arrow PMC member: David M Li

2021-06-21 Thread Niranda Perera
Congrats David! :-) On Mon, Jun 21, 2021 at 6:32 PM Nate Bauernfeind wrote: > Congratulations! Well earned! > > On Mon, Jun 21, 2021 at 4:20 PM Ian Cook wrote: > > > Congratulations, David! > > > > Ian > > > > > > On Mon, Jun 21, 2021 at 6:19 PM Wes McKinney > wrote: > > > > > > The Project

Re: Preparing for 5.0.0

2021-06-21 Thread Wes McKinney
Thanks Neal. I did a cull of 60-70 "old" issues created prior to 2020. Many of these would still be good to resolve, of course, but acknowledging that we've been kicking those cans for a long time and so they will have to wait for someone to resuscitate them. I'll continue going through some more

Re: [ANNOUNCE] New Arrow PMC member: David M Li

2021-06-21 Thread Nate Bauernfeind
Congratulations! Well earned! On Mon, Jun 21, 2021 at 4:20 PM Ian Cook wrote: > Congratulations, David! > > Ian > > > On Mon, Jun 21, 2021 at 6:19 PM Wes McKinney wrote: > > > > The Project Management Committee (PMC) for Apache Arrow has invited > > David M Li to become a PMC member and we are

Re: [ANNOUNCE] New Arrow PMC member: David M Li

2021-06-21 Thread Ian Cook
Congratulations, David! Ian On Mon, Jun 21, 2021 at 6:19 PM Wes McKinney wrote: > > The Project Management Committee (PMC) for Apache Arrow has invited > David M Li to become a PMC member and we are pleased to announce > that David has accepted. > > Congratulations and welcome!

[ANNOUNCE] New Arrow PMC member: David M Li

2021-06-21 Thread Wes McKinney
The Project Management Committee (PMC) for Apache Arrow has invited David M Li to become a PMC member and we are pleased to announce that David has accepted. Congratulations and welcome!

Re: [C++] Async Arrow Flight

2021-06-21 Thread Nate Bauernfeind
Google Cloud supports asynchronous grpc in C++ for parts of their API. One such client-facing API method is this: ``` future> AsyncPublish( google::cloud::CompletionQueue& cq, std::unique_ptr context, google::pubsub::v1::PublishRequest const& request) override { return

Re: [Rust] experimental parquet2 repo

2021-06-21 Thread Wes McKinney
In addition to the XML document / updating the IP clearance website, we would need to discuss the contribution and hold a brief vote to accept the donation(s), indeed same as with Ballista. On Mon, Jun 21, 2021 at 11:56 AM Neal Richardson wrote: > > Regarding IP clearance, see

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

2021-06-21 Thread Gosh Arzumanyan
Thanks David! I also responded/added more suggestions/questions to the doc. I think it makes sense to have two sections: one purely protocol oriented and second API oriented(examples in c++ or in any other language should make the idea easier to digest). Thanks for the reference too! Cheers,

Re: Preparing for 5.0.0

2021-06-21 Thread Neal Richardson
I've also made a 6.0.0 version in jira in case anyone wants to bump issues out of the 5.0.0 scope. Neal On Mon, Jun 21, 2021 at 11:00 AM Neal Richardson < neal.p.richard...@gmail.com> wrote: > Hi folks, > It's that time again--July is approaching, and with it our next major > release. I've

Preparing for 5.0.0

2021-06-21 Thread Neal Richardson
Hi folks, It's that time again--July is approaching, and with it our next major release. I've started https://cwiki.apache.org/confluence/display/ARROW/Arrow+5.0.0+Release (copied from the 4.0.0 page and updated the jira searches to 5.0.0, per usual), and we're currently showing 250 open issues

Re: [Rust] experimental parquet2 repo

2021-06-21 Thread Neal Richardson
Regarding IP clearance, see https://incubator.apache.org/ip-clearance/ and the form it references. IIRC you need to complete the XML version of the form and check it into the incubator's SVN repo, but just do whatever the form says and feel free to ask questions here--when I helped do the Julia

[ANNOUNCE] Apache 4.0.1 released

2021-06-21 Thread Jorge Cardoso Leitão
The Apache Arrow team is pleased to announce the 4.0.1 release. This release covers general bug fixes on the different implementations, notably C++, R, Python and JavaScript. The list is available [1], with the list of contributors [2] and changelog [3]. As usual, see the install page [4] for

Re: Complex Number support in Arrow

2021-06-21 Thread Simon Perkins
On Mon, Jun 21, 2021 at 4:58 PM Antoine Pitrou wrote: > > I certainly don't think we should have extension types with a different > type id. IMHO, it's a recipe for confusion. > Thanks, I think I got confused by the different perspectives in the thread. I'll do some more exploratory coding

Re: Complex Number support in Arrow

2021-06-21 Thread Antoine Pitrou
I certainly don't think we should have extension types with a different type id. IMHO, it's a recipe for confusion. Regards Antoine. Le 21/06/2021 à 15:54, Simon Perkins a écrit : To put it another way, an Extension Type technically has Type::EXTENSION, but now there's

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

2021-06-21 Thread David Li
Thanks! I've left some initial comments/suggestions to expand it in terms of the format definitions and not the C++ APIs. I'll also note something like this was proposed a long time ago - there's not very much discussion about it there but for reference:

Re: Complex Number support in Arrow

2021-06-21 Thread Simon Perkins
To put it another way, an Extension Type technically has Type::EXTENSION, but now there's Type::COMPLEX_FLOAT and Type::COMPLEX_DOUBLE. When checking enums, the code see's a Type::COMPLEX_FLOAT and seems to mismatch on ComplexFloatType::Type::type_id, as the latter is Type::EXTENSION? On Mon,

Re: Complex Number support in Arrow

2021-06-21 Thread Simon Perkins
I did some exploratory coding adding Complex Numbers as ExtensionTypes in this PR: https://github.com/apache/arrow/pull/10565 > My understanding is that it means having COMPLEX as an entry in the arrow/type_fwd.h Type enum. I agree this would make implementation work in the C++ library much more

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

2021-06-21 Thread Gosh Arzumanyan
Ah sorry, comments should work now. Cheers, Gosh On Mon., 21 Jun. 2021, 14:18 David Li, wrote: > Thanks! Will give it a look. > > Would you mind opening it up for comments? > > -David > > On 2021/06/21 11:56:24, Gosh Arzumanyan wrote: > > Hi folks, > > > > Started putting some thoughts

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

2021-06-21 Thread David Li
Thanks! Will give it a look. Would you mind opening it up for comments? -David On 2021/06/21 11:56:24, Gosh Arzumanyan wrote: > Hi folks, > > Started putting some thoughts together here: > https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing > Any

Re: Apache Arrow Cookbook

2021-06-21 Thread Michael Lavina
Awesome idea! I would definitely either copy paste or link to Paul’s codepens of Arrow-JS examples https://codepen.io/trxcllnt/pens/public That’s basically how I know, how to use arrow-js. -Michael P.S. Just a general thank you credit for Paul Taylor and his awesome work on arrow-js and those

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

2021-06-21 Thread Gosh Arzumanyan
Hi folks, Started putting some thoughts together here: https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing Any feedback is welcome! Cheers, Gosh

RE: Apache Arrow Cookbook

2021-06-21 Thread dana
Great idea! -Original Message- From: Alessandro Molina Sent: Monday, June 21, 2021 11:34 AM To: dev@arrow.apache.org Subject: Apache Arrow Cookbook Hi, I'd like to share with the ML an idea which me and Nic Crane have been experimenting with. It's still in the early stage, but we

Re: Arrow Dataset API on Ceph

2021-06-21 Thread Yibo Cai
Hi Jayjeet, I've successfully validated basic functions based on the links you provided, on both Arm64 and x86, with binaries built from your PR. Everything looks fine. From perf, I can see arrow code is running actively on ceph osd nodes. Currently, I deployed and tested on 4 VMs. For

Apache Arrow Cookbook

2021-06-21 Thread Alessandro Molina
Hi, I'd like to share with the ML an idea which me and Nic Crane have been experimenting with. It's still in the early stage, but we hope to turn it into a PR for Arrow documentation soon. The idea is to work on a Cookbook, a collection of ready made recipes, on how to use Arrow that both end