Hi,
We have some rough ideas of applying Flight in HPC (High Performance
Computation). Would like to hear comments.
HPC infrastructure normally leverages RDMA for fast data transfer among
storage nodes and compute nodes. Computation tasks are dispatched to
compute nodes with best fit resourc
Le 09/09/2021 à 12:34, Yibo Cai a écrit :
Hi,
We have some rough ideas of applying Flight in HPC (High Performance
Computation). Would like to hear comments.
HPC infrastructure normally leverages RDMA for fast data transfer among
storage nodes and compute nodes. Computation tasks are dispatch
There's nothing stopping us from transmitting HTTP/2 or another binary
protocol over UCX. You can think of UCX as a transport layer abstraction
library which allows transparently taking advantage of things like RDMA
over InfiniBand / RoCE, inter-process shared memory, TCP sockets, etc.
The other t
UCX allow the user to use various transportation protocols without having to be
aware of which protocol is being used under the hood. The image at the top of
this page gives a good explanation of how this happens
https://openucx.org/introduction/ . So the application can be agnostic if the
tra
Yibo Cai writes:
> HPC infrastructure normally leverages RDMA for fast data transfer among
> storage nodes and compute nodes. Computation tasks are dispatched to
> compute nodes with best fit resources.
>
> Concretely, we are investigating porting UCX as Flight transport layer.
> UCX is a comm
I would support doing the work necessary to get UCX (or really any other
transport) supported, even if it is a lot of work. (I'm hoping this clears the
path to supporting a Flight-to-browser transport as well; a few projects seem
to have rolled their own approaches but I think Flight itself shou
>
> I would support doing the work necessary to get UCX (or really any other
> transport) supported, even if it is a lot of work. (I'm hoping this clears
> the path to supporting a Flight-to-browser transport as well; a few
> projects seem to have rolled their own approaches but I think Flight itse
Hi Yibo,
Just curious, has there been more thought on this from your/the HPC side?
I also realized we never asked, what is motivating Flight in this space in the
first place? Presumably broader Arrow support in general?
-David
On Fri, Sep 10, 2021, at 12:27, Micah Kornfield wrote:
> >
> > I wo
On 10/26/21 10:02 PM, David Li wrote:
Hi Yibo,
Just curious, has there been more thought on this from your/the HPC side?
Yes. I will investigate the possible approach. Maybe build a quick (and
dirty) POC test at first.
I also realized we never asked, what is motivating Flight in this sp
Outside of just HPC, integrating UCX would potentially allow taking
advantage of its shared memory backend which would be interesting from a
performance perspective in the single-node, multi-process case in many
situations.
Not sure it's worth the UCX dependency in the long run, but would allow us
UCX is interesting, relatively new and seems like it may be easier to
integrate. MPI is the most commonly used backend for HPC. Influencing
the development of UCX is more difficult than influencing the
development of MPI, but both have a slower pace of development than
Arrow. One may want to co
Thanks for the clarification Yibo, looking forward to the results. Even if it
is a very hacky PoC it will be interesting to see how it affects performance,
though as Keith points out there are benefits in general to UCX (or similar
library), and we can work out the implementation plan from there
"David Li" writes:
> Thanks for the clarification Yibo, looking forward to the results. Even if it
> is a very hacky PoC it will be interesting to see how it affects performance,
> though as Keith points out there are benefits in general to UCX (or similar
> library), and we can work out the i
I ended up drafting an implementation of Flight based on UCX, and doing some
of the necessary refactoring to support additional backends in the future.
It can run the Flight benchmark, and performance is about comparable to
gRPC, as tested on AWS EC2.
The implementation is based on the UCP streams
@david
I think this is really interesting and a really good starting point. :-)
Like Keith said, this paves way to use shared memory execution backends. In
our experience at Cylon, these backends give good scalability for data
engineering jobs.
While send/ receive arrow payloads is essential, I fe
Le 28/12/2021 à 20:09, David Li a écrit :
Antoine/Micah raised the possibility of extending gRPC instead. That would
be preferable, frankly, given otherwise we'd might have to re-implement a
lot of what gRPC and HTTP2 provide by ourselves. However, the necessary
proposal stalled and was droppe
Thanks for the feedback!
Collective operations: unfortunately, these look very different from the model
Flight provides (which is just RPC). If there's interest, we could consider
implementing or exposing them in the future, or looking at making sure Arrow's
IPC APIs play well with the MPI APIs
This is very nice. Look forward to trying it out. One should get
performance improvements on hardware with better interconnects, so
performance just with TCP is not illustrative of all cases.
On 12/28/21 11:41 PM, David Li wrote:
Thanks for the feedback!
Collective operations: unfortunately,
] https://github.com/cyb70289/arrow/tree/flight-data-plane
From: David Li
Sent: Wednesday, December 29, 2021 3:09 AM
To: dev@arrow.apache.org
Subject: Re: Arrow in HPC
I ended up drafting an implementation of Flight based on UCX, and doing some
of the necessary re
gt;
> [1] Test environment
> nics: mellanox connectx5
> hosts: client (neoverse n1), server (xeon gold 5218)
> os: ubuntu 20.04, linux kernel 5.4
> test case: 128k batch size, DoGet
>
> [2] https://github.com/cyb70289/arrow/tree/flight-data-plane
>
> _____________
untu 20.04, linux kernel 5.4
test case: 128k batch size, DoGet
[2] https://github.com/cyb70289/arrow/tree/flight-data-plane
From: David Li
Sent: Wednesday, December 29, 2021 3:09 AM
To: dev@arrow.apache.org
Subject: Re: Arrow in HPC
I ended up drafting an imple
w tranport
> >> requires much more initial effort, but may payoff later. And looks these
> >> two approaches don't conflict with each other.
> >>
> >> [1] Test environment
> >> nics: mellanox connectx5
> >> hosts: client (neoverse n1), server (xeon gol
osts: client (neoverse n1), server (xeon gold 5218)
os: ubuntu 20.04, linux kernel 5.4
test case: 128k batch size, DoGet
[2] https://github.com/cyb70289/arrow/tree/flight-data-plane
From: David Li
Sent: Wednesday, December 29, 2021 3:09 AM
To: dev@arrow.apache.org
y github repo [2]. Besides the framework, I just implemented
> >>>> a shared memory data plane driver as PoC. Get/Put/Exchange unit tests
> >>>> passed, TestCancel hangs, some unit tests run longer than expected,
> >>>> still debugging.
> > >>>> Code is at my github repo [2]. Besides the framework, I just
> implemented a shared memory data plane driver as PoC. Get/Put/Exchange unit
> tests passed, TestCancel hangs, some unit tests run longer than expected,
> still debugging. The shared memory data plane perf
e
> > comparison is not fair. With David's patch, we can get a more realistic
> > comparison.
> > > >>>>
> > > >>>> I'm implementing a data plane approach to hope we can adopt new
> > data acceleration methods easily. My approach
at 04:37, Yibo Cai wrote:
>> > > >>>> Thanks David to initiate UCX integration, great work!
>> > > >>>> I think 5Gbps network is too limited for performance evaluation. I
>> > will try the patch on 100Gb RDMA network, hopefully we can see some
> >> design. After that I will fix unit test issues and deliver for
>>> > community
>>> > > >> review.
>>> > > >>
>>> > > >> Anyway, don't let me block your implementations. And if you think
>>> > > >> it's
>>> > > >
t;>> > > >>>
> >>> > > >>
> >>> > > >> Thanks for being interest in this approach. My current plan is
> to
> >>> > first
> >>> > > >> refactor shared memory data plane to verify it b
29 matches
Mail list logo
