randomkang commented on code in PR #3144:
URL: https://github.com/apache/brpc/pull/3144#discussion_r4333559558
##########
src/brpc/rdma/rdma_helper.cpp:
##########
@@ -474,6 +479,20 @@ static ibv_context* OpenDevice(int num_total, int*
num_available_devices) {
}
static void GlobalRdmaInitializeOrDieImpl() {
+#if BRPC_WITH_GDR
+ // GPUDirect RDMA requires all received data to be written into the GPU
+ // memory buffer specified at post-receive time. When MLX5_SCATTER_TO_CQE
+ // is enabled, small messages are inlined directly into host memory, which
+ // breaks this requirement and causes operations to fail. libmlx5 reads
+ // MLX5_* environment variables when the device context is created (i.e.
+ // inside ibv_open_device below), so it must be forced to 0 before any
+ // device is opened, regardless of the user-provided value.
+ if (setenv("MLX5_SCATTER_TO_CQE", "0", 1) != 0) {
+ PLOG(ERROR) << "Fail to set MLX5_SCATTER_TO_CQE to 0";
+ ExitWithError();
+ }
+#endif // if BRPC_WITH_GDR
Review Comment:
If user do not want to use gdr, then should not build brpc with gdr, and
this environment variable setting will not take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
