Le 09/09/2021 à 12:34, Yibo Cai a écrit :
Hi,
We have some rough ideas of applying Flight in HPC (High Performance
Computation). Would like to hear comments.
HPC infrastructure normally leverages RDMA for fast data transfer among
storage nodes and compute nodes. Computation tasks are dispatched to
compute nodes with best fit resources.
Concretely, we are investigating porting UCX as Flight transport layer.
UCX is a communication framework for modern networks. [1]
Besides HPC usage, many projects (spark, dask, blazingsql, etc) also
adopt UCX to accelerate network transmission. [2][3]
I see a recent discussion about decoupling Flight from gRPC. Looks this
is also what we should do first to adapt UCX to Flight.
Is it actually necessary? Or is it possible to transmit HTTP/2 over UCX?
(I'm not fond of gRPC, but decoupling Flight *and* building support for
another transport layer will be quite a bit of work)
Regards
Antoine.
