Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]
lunarlooppp closed issue #12524: help request: skywalking-logger plugin throws "no memory" errors under high concurrency URL: https://github.com/apache/apisix/issues/12524 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]
juzhiyuan commented on issue #12524: URL: https://github.com/apache/apisix/issues/12524#issuecomment-4366374755 In APISIX 3.13, the skywalking tracer pushes finished trace segments into ngx.shared.tracing_buffer. The skywalking-logger endpoint_addr only configures log reporting to /v3/logs; it does not configure where the skywalking tracer sends /v3/segments. From the screenshots, I can see tracing_buffer is already set to 500m, but I do not see the config.yaml plugin_attr.skywalking section. Please confirm whether you configured: ```yaml plugin_attr: skywalking: report_interval: 3 endpoint_addr: http://:12800 service_name: ... service_instance_name: ... ``` The route-level skywalking plugin config should mainly contain sample_ratio. The report_interval and endpoint_addr for trace segment reporting should be in plugin_attr.skywalking. If _plugin_attr.skywalking.endpoint_addr_ is not set correctly, APISIX may still try to report trace segments to the default http://127.0.0.1:12800. Under _ab -c 2000 -t 20 -k_ with _sample_ratio=1_, trace segments can be produced much faster than they are drained, and tracing_buffer can eventually return no memory. Could you also check whether error.log contains Segment report fails, Instance report fails, or Agent ping fails during the test? It would help to test these three cases separately: 1. only skywalking enabled 2. only skywalking-logger enabled 3. both enabled For skywalking-logger, batch_max_size=1 plus timeout=100/max_retry_count=5 can also create large pending batches if OAP is slow or unreachable, but that is separate from the tracer.lua:145 error path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]
Lunarloop-a commented on issue #12524: URL: https://github.com/apache/apisix/issues/12524#issuecomment-3190488042 > Hi, could you please provide some detailed minimal reproduction steps and the environment? Specifically, what is "under high concurrency"?你好，你能提供一些详细的最小复制步骤和环境吗？具体来说，什么是“高并发”？ Sorry, I didn’t explain what I meant by “high concurrency” clearly. What I meant was: when I run the ApacheBench command “ab -t 20 -c 2000 -k -H "Accept-Encoding: gzip, deflate" -s 8000 http://apisix.endpoint/guest/test/01”, I call it “high concurrency” ,this command makes 2000 connections keeping send requests to my service continuously for 20 seconds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]
SkyeYoung commented on issue #12524: URL: https://github.com/apache/apisix/issues/12524#issuecomment-3190317733 Hi, could you please provide some detailed minimal reproduction steps and the environment? Specifically, what is "under high concurrency"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]
Lunarloop-a commented on issue #12524: URL: https://github.com/apache/apisix/issues/12524#issuecomment-3188143741 Is my route:skywalking-logger:batch_max_size set too large? I tried setting it to a smaller value, and it seems to have worked. > Is my route:plugins:skywalking-logger:batch_max_size set too large? I tried setting it to a smaller value, and it seems to have worked. No, it still hasn’t worked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]
Lunarloop-a commented on issue #12524: URL: https://github.com/apache/apisix/issues/12524#issuecomment-3188117374 Is my route:skywalking-logger:batch_max_size set too large? I tried setting it to a smaller value, and it seems to have worked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
