RE: [PATCH 0/1] Introduce 'NOROOM' status for srv_state metric
Hi, Tanks for you feedbacks. > Indeed, so actually you found a pretty valid use case to the edge case I was > seeing as rare enough not to be of much interest :-) Just to provide some background, which might help in understanding our needs: In our real-life scenarios, servers do not have uniform performance. Our haproxy "servers", in our context, are typically containers. These containers can reside on very diverse hardware, depending on the age of the server. We attempt to mitigate these discrepancies with server weights, but that's not always sufficient; not all servers respond at the same speed. The performance can also be compromised by "noisy" neighbors co-hosted on the same server. Sometimes we even have instances that fail because the underlying code is poorly written. I don't know if it's common with others, but when a backend starts to become overloaded, it happens gradually, and during the initial days, the first symptoms are a few servers becoming FULL during peak hours. > Yes that's it. For this specific use case, watching the queue is much simpler > and provides more than just a boolean Your discussion about queues is very interesting, and I believe we will incorporate those into our metrics. My idea is not to present a boolean, but a percentage of FULL servers by backend. We believe it's more illustrative to see that around 7 a.m., all servers are functioning, but by 7:10 a.m., 10% of the servers begin to be "FULL" due to the increase in traffic. By 7:30 a.m., it's 50%, and by 8 a.m., it's 100%. We think this curve is simpler to grasp than just seeing, at 8 a.m., a queue beginning to grow, while all these servers still seem OK (haproxy_backend_agg_check_status=L7OK, and haproxy_backend_agg_server_status="UP"). Currently, we compute this metric using this Prometheus calculation: count by (proxy, instance) (haproxy_server_current_sessions > 180) (even though we have a maxconn set at 200). This calculation has several issues: - maxconn cannot be retrieved; it has to be hardcoded in Prometheus. - it forces us to use "/metrics?scope=server", which scrapes all server metrics, something we're trying to avoid, because it consumes a lot of resources to aggragate on prometheus. With my patch, we can achieve the same with count by (proxy, instance) (haproxy_backend_agg_server_status="FULL"), just by using /metrics?scope=backend. The main question I'm asking myself is whether my calculation method will work, because a server that starts to saturate will "oscillate" between FULL and OK. We risk missing it, not necessarily capturing the value at the right time. Cedric -Message d'origine----- De : Willy Tarreau Envoyé : mercredi 6 septembre 2023 10:36 À : Cedric Paillet Cc : haproxy@formilux.org Objet : Re: [PATCH 0/1] Introduce 'NOROOM' status for srv_state metric Hi Cedric, On Tue, Sep 05, 2023 at 01:40:14PM +, Cedric Paillet wrote: > We are using Prometheus to provide feedback to our users about the > status of backend servers. Currently, we have no means of informing > them if a server exceeds the maxconn limit, and consequently why it's > no longer receiving new requests. > > Therefore, we would like to be able to display when a server surpasses > the maxconn limit and is in the "noroom" state. I have prepared a > patch specifically for Prometheus, but it might be better to include a > boolean directly in the server structure indicating whether the server > was considered to have no room the last time server_has_room was > called. However, this change seems to have a significant impact on other > parts of the code. I think that it might be more suitable to use the term "FULL" that we already use for the listeners, and that also matches what's used at a few places in the doc in association with servers or backends being "full". Also, a more accurate metric that is generally watched is the queue (both server and backend): instead of being a boolean, it directly indicates how many additional servers are needed to improve the processing time. Persistent requests are placed into the server's queue but all other requests go into the backend queue. So if you have a total capacity of 4 servers * 100 connections = 400 outstanding requests, and you see a queue of 200, you know that you'd need two extra servers to process these without queuing, and you can even predict that the total processing time will decrease by 200*the average queue time, so this allows to even add just the required number of servers to keep response time below a certain limit. The only case where a server will be full without having any queue is when the total number of outstanding requests on a backend is exactly equal to the sum of t
Re: [PATCH 0/1] Introduce 'NOROOM' status for srv_state metric
On Wed, Sep 06, 2023 at 12:44:56PM +, Cedric Paillet wrote: > >I think that it might be more suitable to use the term "FULL" > Ok, no problem with that. (Perhaps we can also rename server_has_room to > !server_is_full ?) You're right, it might be clearer! > > Also, a more accurate metric that is generally watched is the queue (both > > server and backend): > > My first use case is to detect/display one or more server(s) with problems, > not to determine if the entire backend is undersized. > As I understand it, if just one server in the pool is very slow (and the load > balancing method is round-robin), the number of sessions on this server will > increase until it reaches maxconn. At this juncture, the server will no > longer be selected, and requests will be routed to the other servers. Then, > no queue (either backend or server) will start to fill up, correct? But the > slow server will cease receiving requests until its session count drops below > maxconn, right? Indeed, so actually you found a pretty valid use case to the edge case I was seeing as rare enough not to be of much interest :-) > The second use case, as you've outlined, is to detect if a backend is > undersized. My understanding is that if the backend is "nearly" undersized, > the first symptom will be some servers reporting "FULL". Only when ALL > servers are reporting "FULL" will the backend queue start to grow, correct? Yes that's it. For this specific use case, watching the queue is much simpler and provides more than just a boolean. But to be clear, the situation between one server full and all server full is very unstable, as queuing is an exponential function of response time by definition, so once a server is full, you can be certain that all other ones are about to be full as well, and between the two you'll have a lot of noise where servers are randomly full then ready depending on the load. Let's say you're having 40 servers processing 1 requests per second with a maxconn 50. That's a total capacity of 2000 outstanding requests (and as many concurrent connections). This means requests will last on average 200ms at saturation. At 196 ms response time you'll have 1960 outstanding requests, hence 49 per server, thus no server marked full. At 200ms all will be full. At 198ms, half of the servers will be marked full, and that changes for every single request, i.e. 250 times per second per server. Now you can see how sampling such a boolean only once every few seconds when it changes 250 times per second will be useless for scalability purposes, you're just sampling a random value. On the opposite, retrieving the queue length is more stable since it's for the whole backend and not just per server, plus it reports more than a boolean. Hoping this helps, Willy
RE: [PATCH 0/1] Introduce 'NOROOM' status for srv_state metric
Thanks for your review. >I think that it might be more suitable to use the term "FULL" Ok, no problem with that. (Perhaps we can also rename server_has_room to !server_is_full ?) > Also, a more accurate metric that is generally watched is the queue (both > server and backend): My first use case is to detect/display one or more server(s) with problems, not to determine if the entire backend is undersized. As I understand it, if just one server in the pool is very slow (and the load balancing method is round-robin), the number of sessions on this server will increase until it reaches maxconn. At this juncture, the server will no longer be selected, and requests will be routed to the other servers. Then, no queue (either backend or server) will start to fill up, correct? But the slow server will cease receiving requests until its session count drops below maxconn, right? The second use case, as you've outlined, is to detect if a backend is undersized. My understanding is that if the backend is "nearly" undersized, the first symptom will be some servers reporting "FULL". Only when ALL servers are reporting "FULL" will the backend queue start to grow, correct? Cédric -Message d'origine- De : Willy Tarreau Envoyé : mercredi 6 septembre 2023 10:36 À : Cedric Paillet Cc : haproxy@formilux.org Objet : Re: [PATCH 0/1] Introduce 'NOROOM' status for srv_state metric Hi Cedric, On Tue, Sep 05, 2023 at 01:40:14PM +, Cedric Paillet wrote: > We are using Prometheus to provide feedback to our users about the > status of backend servers. Currently, we have no means of informing > them if a server exceeds the maxconn limit, and consequently why it's > no longer receiving new requests. > > Therefore, we would like to be able to display when a server surpasses > the maxconn limit and is in the "noroom" state. I have prepared a > patch specifically for Prometheus, but it might be better to include a > boolean directly in the server structure indicating whether the server > was considered to have no room the last time server_has_room was > called. However, this change seems to have a significant impact on other > parts of the code. I think that it might be more suitable to use the term "FULL" that we already use for the listeners, and that also matches what's used at a few places in the doc in association with servers or backends being "full". Also, a more accurate metric that is generally watched is the queue (both server and backend): instead of being a boolean, it directly indicates how many additional servers are needed to improve the processing time. Persistent requests are placed into the server's queue but all other requests go into the backend queue. So if you have a total capacity of 4 servers * 100 connections = 400 outstanding requests, and you see a queue of 200, you know that you'd need two extra servers to process these without queuing, and you can even predict that the total processing time will decrease by 200*the average queue time, so this allows to even add just the required number of servers to keep response time below a certain limit. The only case where a server will be full without having any queue is when the total number of outstanding requests on a backend is exactly equal to the sum of the servers' maxconn values, so as you see, the extra metric covers very little compared to the queue itself. But there might be good use cases for this, I'm not denying it, I just wanted to make sure that you're going to monitor what's really relevant for your use case ;-) Willy
Re: [PATCH 0/1] Introduce 'NOROOM' status for srv_state metric
Hi Cedric, On Tue, Sep 05, 2023 at 01:40:14PM +, Cedric Paillet wrote: > We are using Prometheus to provide feedback to our users about the status of > backend servers. Currently, we have no means of informing them if a server > exceeds the maxconn limit, and consequently why it's no longer receiving new > requests. > > Therefore, we would like to be able to display when a server surpasses the > maxconn limit and is in the "noroom" state. I have prepared a patch > specifically for Prometheus, but it might be better to include a boolean > directly in the server structure indicating whether the server was considered > to have no room the last time server_has_room was called. However, this > change seems to have a significant impact on other parts of the code. I think that it might be more suitable to use the term "FULL" that we already use for the listeners, and that also matches what's used at a few places in the doc in association with servers or backends being "full". Also, a more accurate metric that is generally watched is the queue (both server and backend): instead of being a boolean, it directly indicates how many additional servers are needed to improve the processing time. Persistent requests are placed into the server's queue but all other requests go into the backend queue. So if you have a total capacity of 4 servers * 100 connections = 400 outstanding requests, and you see a queue of 200, you know that you'd need two extra servers to process these without queuing, and you can even predict that the total processing time will decrease by 200*the average queue time, so this allows to even add just the required number of servers to keep response time below a certain limit. The only case where a server will be full without having any queue is when the total number of outstanding requests on a backend is exactly equal to the sum of the servers' maxconn values, so as you see, the extra metric covers very little compared to the queue itself. But there might be good use cases for this, I'm not denying it, I just wanted to make sure that you're going to monitor what's really relevant for your use case ;-) Willy
