[GitHub] [arrow-julia] quinnj commented on a change in pull request #272: Configure repository metadata

2021-12-28 Thread GitBox
quinnj commented on a change in pull request #272: URL: https://github.com/apache/arrow-julia/pull/272#discussion_r776181901 ## File path: .asf.yaml ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [arrow-julia] quinnj commented on a change in pull request #272: Configure repository metadata

2021-12-28 Thread GitBox
quinnj commented on a change in pull request #272: URL: https://github.com/apache/arrow-julia/pull/272#discussion_r776181151 ## File path: .asf.yaml ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [arrow-julia] kou commented on a change in pull request #272: Configure repository metadata

2021-12-28 Thread GitBox
kou commented on a change in pull request #272: URL: https://github.com/apache/arrow-julia/pull/272#discussion_r776180085 ## File path: .asf.yaml ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

[GitHub] [arrow-julia] kou opened a new pull request #272: Configure repository metadata

2021-12-28 Thread GitBox
kou opened a new pull request #272: URL: https://github.com/apache/arrow-julia/pull/272 fix #271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [arrow-julia] kou opened a new issue #271: Configure repository metadata

2021-12-28 Thread GitBox
kou opened a new issue #271: URL: https://github.com/apache/arrow-julia/issues/271 We can configure this repository's metadata by `.asf.yaml`. See also: https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features We should change at least notification configurations.

[GitHub] [arrow-julia] kou opened a new issue #270: Document the Apache way development process

2021-12-28 Thread GitBox
kou opened a new issue #270: URL: https://github.com/apache/arrow-julia/issues/270 JuliaData/Arrow.jl has been transferred to apache/arrow-julia. See also: #265 and https://issues.apache.org/jira/browse/INFRA-22663 We discussed the development process for apache/arrow-julia on

Re: Arrow in HPC

2021-12-28 Thread Benson Muite
This is very nice. Look forward to trying it out. One should get performance improvements on hardware with better interconnects, so performance just with TCP is not illustrative of all cases. On 12/28/21 11:41 PM, David Li wrote: Thanks for the feedback! Collective operations:

Re: Arrow in HPC

2021-12-28 Thread David Li
Thanks for the feedback! Collective operations: unfortunately, these look very different from the model Flight provides (which is just RPC). If there's interest, we could consider implementing or exposing them in the future, or looking at making sure Arrow's IPC APIs play well with the MPI

Re: Arrow in HPC

2021-12-28 Thread Antoine Pitrou
Le 28/12/2021 à 20:09, David Li a écrit : Antoine/Micah raised the possibility of extending gRPC instead. That would be preferable, frankly, given otherwise we'd might have to re-implement a lot of what gRPC and HTTP2 provide by ourselves. However, the necessary proposal stalled and was

Re: Arrow in HPC

2021-12-28 Thread Niranda Perera
@david I think this is really interesting and a really good starting point. :-) Like Keith said, this paves way to use shared memory execution backends. In our experience at Cylon, these backends give good scalability for data engineering jobs. While send/ receive arrow payloads is essential, I

Re: Arrow in HPC

2021-12-28 Thread David Li
I ended up drafting an implementation of Flight based on UCX, and doing some of the necessary refactoring to support additional backends in the future. It can run the Flight benchmark, and performance is about comparable to gRPC, as tested on AWS EC2. The implementation is based on the UCP

Re: Arrow vs Artus

2021-12-28 Thread Benson Muite
The paper[1] is helpful. Compression may also be helpful - but it may be difficult to standardize this. [1] https://vldb.org/pvldb/vol12/p2022-chattopadhyay.pdf On 12/25/21 5:37 AM, Micah Kornfield wrote: What exactly are you looking for? To my knowledge neither Capacitor nor Artus have

Re: [VOTE][RUST] Release Apache Arrow Rust 6.5.0 RC1

2021-12-28 Thread Jorge Cardoso Leitão
+1 Thanks, Jorge On Fri, Dec 24, 2021 at 3:21 AM Wang Xudong wrote: > +1 (non-binding) > > Happy holidays > > --- > xudong > > Andy Grove 于2021年12月24日周五 09:19写道: > > > +1 (binding) > > > > Thanks, > > > > Andy. > > > > On Thu, Dec 23, 2021 at 2:26 PM Andrew Lamb > wrote: > > > > > Hi, > > >

Re: [Rust] Heads up: RUSTSEC security advisory against arrow-rs

2021-12-28 Thread Andrew Lamb
To finish up this thread, the rustsec advisory has been updated: https://github.com/rustsec/advisory-db/pull/1131 On Tue, Nov 30, 2021 at 5:10 PM Andrew Lamb wrote: > Well, I finally have a PR[1] that adds full ArrayData validation ready > for review Any thoughts or review comments would be