I would support doing the work necessary to get UCX (or really any other transport) supported, even if it is a lot of work. (I'm hoping this clears the path to supporting a Flight-to-browser transport as well; a few projects seem to have rolled their own approaches but I think Flight itself should really handle this, too.)
>From what I understand, you could tunnel gRPC over UCX as Keith mentions, or >directly use UCX, which is what it sounds like you are thinking about. One >idea we had previously was to stick to gRPC for 'control plane' methods, and >support alternate protocols only for 'data plane' methods like DoGet - this >might be more manageable, depending on what you have in mind. In general - there's quite a bit of work here, so it would help to separate the work into phases, and share some more detailed design/implementation plans, to make review more manageable. (I realize of course this is just a general interest check right now.) Just splitting gRPC/Flight is going to take a decent amount of work, and (from what little I understand) using UCX means choosing from various communication methods it offers and writing a decent amount of scaffolding code, so it would be good to establish what exactly a 'UCX' transport means. (For instance, presumably there's no need to stick to the Protobuf-based wire format, but what format would we use?) It would also be good to expand the benchmarks, to validate the performance we get from UCX and have a way to compare it against gRPC. Anecdotally I've found gRPC isn't quite able to saturate a connection so it would be interesting to see what other transports can do. Jed - how would you see MPI and Flight interacting? As another transport/alternative to UCX? I admit I'm not familiar with the HPC space. About transferring commands with data: Flight already has an app_metadata field in various places to allow things like this, it may be interesting to combine with the ComputeIR proposal on this mailing list, and hopefully you & your colleagues can take a look there as well. -David On Thu, Sep 9, 2021, at 11:24, Jed Brown wrote: > Yibo Cai <yibo....@arm.com> writes: > > > HPC infrastructure normally leverages RDMA for fast data transfer among > > storage nodes and compute nodes. Computation tasks are dispatched to > > compute nodes with best fit resources. > > > > Concretely, we are investigating porting UCX as Flight transport layer. > > UCX is a communication framework for modern networks. [1] > > Besides HPC usage, many projects (spark, dask, blazingsql, etc) also > > adopt UCX to accelerate network transmission. [2][3] > > I'm interested in this topic and think it's important that even if the focus > is direct to UCX, that there be some thought into MPI interoperability and > support for scalable collectives. MPI considers UCX to be an implementation > detail, but the two main implementations (MPICH and Open MPI) support it and > vendor implementations are all derived from these two. >
