Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
Baoyuantop closed issue #11948: bug: lua_shared_dict prometheus-metrics overflow URL: https://github.com/apache/apisix/issues/11948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
Baoyuantop commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-4043851939 Here are some ways to alleviate this problem: 1. Configuration metrics expiration time ``` plugin_attr: prometheus: metrics: http_status: expire: 600# 10 mins http_latency: expire: 600 bandwidth: expire: 600 upstream_status: expire: 300# 5 mins ``` 2. Reduce the use of extra_labels, or remove unnecessary high-cardinality labels. Not configuring extra_labels in Prometheus's plugin_attr can significantly reduce the number of unique indicator keys. 3. Increase the capacity of a shared dictionary ``` nginx_config: http: lua_shared_dict: prometheus-metrics: 100m ``` 4. Upgrade to version 3.14.0+ The architecture improvements obtained from PR #12383 (decoupling computation and output) reduce repetitive operations and memory pressure between worker processes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
nightguide commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-3018687500 @jlaupa Hi! Could you please clarify how changing the number of workers in the Apisix configuration helped resolve the memory leak issue in lua_shared_dict prometheus-metrics? Is it really possible that these things are related? @DrJSAnD Hi! Could you tell me if changing those parameters helped you resolve the memory leak issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
DrJSAnD commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2753930661 @jlaupa , thanks! I will try change parameter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
jlaupa commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2732006920 @shihuncl @DrJSAnD Well, I finally found the error. I will share it with you in case it is your case. good luck. Apisix in my case was taking as many workers as cores available on the node. Which is wrong, it should take from the pod according to the limitation I assign. The solution was to set a number of workers for my traffic, I have decided based on the tests I did that 3 workers is fine. Now (1G ram, at most). If you have less memory, lower the workers and try to reuse as much as possible of the custom plugins. The prometheus plugin, even if you set it to 10m, if you exceed this number you will only get a warning and it won't really be a problem, but it is noisy. How did you check it? > nproc > top then I saw that there were as many workers as openresty running :D It is advised to have 1 core per worker hence I set 3. Translated with www.DeepL.com/Translator (free version) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
jlaupa commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2711070226 I have the same problem but it has been dragging since version 3.8, I have been increasing it from 40m to 100m now 200m. But in 3.10 I started to see a memory leak. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
DrJSAnD commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2720055844 > > I have the same problem but it has been dragging since version 3.8, I have been increasing it from 40m to 100m now 200m. But in 3.10 I started to see a memory leak. > > So what should you do if you don’t have enough memory for this indicator now? Restart apisix? Yes, I restart Apisix pods and lua_shared_dict prometheus-metrics released memory. Then it starts to gradually fill up again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
shihuncl commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2719501921 > I have the same problem but it has been dragging since version 3.8, I have been increasing it from 40m to 100m now 200m. But in 3.10 I started to see a memory leak. So what should you do if you don’t have enough memory for this indicator now? Restart apisix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
DrJSAnD commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2676748124 > Generally this has to do with how discrete the requests are on your API service, which is related to multiple metrics such as route_id, resp_code, or even consumer_id, etc. (https://apisix.apache.org/docs/apisix/next/plugins/prometheus/#labels) Multiply them together and that's the number of record entries you'll get, which in the prometheus or observability is called a high cardinality. If they're really large, you'll need to increase the metrics size. Will it be okay if we increase the lua_shared_dict - prometheus-metrics parameter value to 512m or 1024m? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
bzp2010 commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2676366094 Generally this has to do with how discrete the requests are on your API service, which is related to multiple metrics such as route_id, resp_code, or even consumer_id, etc. (https://apisix.apache.org/docs/apisix/next/plugins/prometheus/#labels) Multiply them together and that's the number of record entries you'll get, which in the prometheus or observability is called a high cardinality. If they're really large, you'll need to increase the metrics size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
DrJSAnD commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2671948910 > K8S部署的apisix怎么修改lua_shared_dict的大小，试了很多地方都是修改不成功 In ConfigMap - config.yaml ```yaml nginx_config: http: lua_shared_dict: prometheus-metrics: 40m ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
shihuncl commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2670305217 K8S部署的apisix怎么修改lua_shared_dict的大小，试了很多地方都是修改不成功 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
DrJSAnD commented on issue #11948:
URL: https://github.com/apache/apisix/issues/11948#issuecomment-2639726124
Prometheus plugin works on all pods and it returns 69000+ rows from each pod.
We display metrics from Prometheus in Grafana
(https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json).
When apisix_shared_dict_free_space_bytes{name="prometheus-metrics"} reached
value "0" then apisix_nginx_metric_errors_total start grows up and all apisix
metrics show incorrect values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: lua_shared_dict prometheus-metrics overflow [apisix]
qizhendong1 commented on issue #11948: URL: https://github.com/apache/apisix/issues/11948#issuecomment-2639004787 You can use ` curl 127.0.0.1:9091/apisix/prometheus/metrics` to test the prometheus plugin endpoint if it works properly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
