sunce4t commented on issue #3132: URL: https://github.com/apache/brpc/issues/3132#issuecomment-3546266022
> > > > 提到[#3145 (comment)](https://github.com/apache/brpc/pull/3145#issuecomment-3544927795) 使用一个CQ即可。那在solicited_only=1时，怎么感知IBV_WC_SEND WC的生成呢？ > > > > > > > > > 所以在给一个 wr 设置 IBV_SEND_SIGNALED 标志的同时需要设置 IBV_SEND_SOLICITED 标志， solicited_only 的 wr 不需要感知 IBV_WC_SEND。 > > > > > > [@legionxiong](https://github.com/legionxiong) hi，本端的IBV_SEND_SOLICITED设置后影响的应当是对端是否在comp channel上生成一个event以触发poll_cq，这个问题看起来是本端一直没有event产生，导致没法进入poll_cq流程？ > > 由对端应用层发送立即数 ACK 来通知本端 poll cq， 每 1/4 窗口发送一个 signaled + solicited 的 wr， 对端收到 1/4 窗口个的消息后给本端发一个signaled + solicited wr 作为 ACK 通知本端 poll cq。当本端 poll 到 IBV_SEND_SIGNALED wr 产生的 CQE 后可以认为驱动层面释放掉了 1/4 窗口的 SQE。进一步细化的话， 当 `_sq_window_size` 小于 `_window_size` 时说明对端接收消息比本端产生 send CQE 快， 那么后续发消息可以考虑更高频次地设置 signaled + solicited。 现在brpc的CutFromIOBufList中逻辑跟你讲的类似，（在默认配置下，一次发送最多发送8KB，1/4 窗口就是31*8=248KB，所以不用考虑_unsolicited_bytes > 1048576这个分支） ``` bool solicited = false; if (window == 1 || current + 1 >= ndata) { // Only last message in the write queue or last message in the // current window will be flagged as solicited. solicited = true; } else { if (_unsolicited > _local_window_capacity / 4) { // Make sure the recv side can be signaled to return ack solicited = true; } else if (_accumulated_ack > _remote_window_capacity / 4) { // Make sure the recv side can be signaled to handle ack solicited = true; } else if (_unsolicited_bytes > 1048576) { // Make sure the recv side can be signaled when it receives enough data solicited = true; } else { ++_unsolicited; _unsolicited_bytes += this_len; _accumulated_ack += imm; } } if (solicited) { wr.send_flags |= IBV_SEND_SOLICITED; _unsolicited = 0; _unsolicited_bytes = 0; _accumulated_ack = 0; } ``` 另一处是： ``` int RdmaEndpoint::SendAck(int num) { if (_new_rq_wrs.fetch_add(num, butil::memory_order_relaxed) > _remote_window_capacity / 2) { return SendImm(_new_rq_wrs.exchange(0, butil::memory_order_relaxed)); } return 0; } int RdmaEndpoint::SendImm(uint32_t imm) { if (imm == 0) { return 0; } ibv_send_wr wr; memset(&wr, 0, sizeof(wr)); wr.opcode = IBV_WR_SEND_WITH_IMM; wr.imm_data = butil::HostToNet32(imm); wr.send_flags |= IBV_SEND_SOLICITED; wr.send_flags |= IBV_SEND_SIGNALED; ibv_send_wr* bad = NULL; int err = ibv_post_send(_resource->qp, &wr, &bad); if (err != 0) { // We use other way to guarantee the Send Queue is not full. // So we just consider this error as an unrecoverable error. LOG(WARNING) << "Fail to ibv_post_send: " << berror(err); return -1; } return 0; } ``` 这两处都会设置IBV_SEND_SOLICITED，我理解你的意思就是去修改这两处提高event生成频率吗 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
