stankiewicz commented on PR #37164: URL: https://github.com/apache/beam/pull/37164#issuecomment-4554667206
> Hi, I have a couple of concerns regarding the use of the SEMP API in this context. > > 1. SEMP API is primarily a management API, and in my opinion it should not be used for acknowledging messages. In case of high throughput there is a risk that the API will be a bottleneck. There are also major security limitations such as the lack of fine‑grained per‑user access control, which means a user could potentially execute all management actions on the Message VPN. > 2. The number of unacked messages is a critical metric to monitor because it creates direct backpressure on the broker. With high throughput, there is a real risk that the unacked count grows significantly. In fact, with the current implementation, the number of unacked message is already much higher than what we observe when using JmsIO. It's out of scope of this PR but it would be worth investigating whether the acknowledgment latency can be improved to reduce the number of unacked messages. hi @ngibanel , sorry for late reply, https://github.com/apache/beam/pull/38603 will make ack work as efficient as JmsIO. tracking total number of unacked messages may be tricky, as gauge metric gives "latest" value, which is not sum of all metrics from all readers. Alternatively we could create gauge per split (a connection). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
