On 2023/11/10 14:09, Tobias Fiebig wrote: > Moin, > i have been running into memleaks with nginx 1.24.0 for some time; > Nginx is self-build (as i need the http_sub module); It is configures > with: ./configure --with-http_sub_module --with-http_ssl_module --with- > http_stub_status_module --prefix=/usr/local --conf- > path=/etc/nginx/nginx.conf --user=www --group=www
So you didn't mention http_sub_module before, which precludes directly using packages from 7.4, but can you try building from the port so that the only change compared to what anyone else is running is enabling the additional module? (You can use 'FLAVOR="no_passenger no_lua no_njs" make package' to reduce the number of build dependencies). > This did not occur with nginx before 1.24.0; I went through all nginx > commits between 1.24.0 and the last non-leaking version, and the change > is indeed when Nginx made TLS1.3 default on. AFAIK a couple of people have tried to reproduce this and not been able to. I really think you are going to need to isolate _what_ triggers the issue and provide some config so that others can replicate it... > The workload is a transparent proxy/web-fronting, i.e., lots of > outbound TLS connections as well. When you look at the graphs (url > below), it seems like leakage 'growth' tracks ~ to the number of > connections nginx receives. Can you try to isolate whether it's on the inbound or outbound side? AFAIK you can disable 1.3 on the proxy side with proxy_ssl_protocols so that is probably worth a try. (I am using proxy_pass with https, though the bulk of my proxy_pass is just http backend, and the one with https backend only does TLS1.2). > As i currently assume that this is related to libressl, i did some > traces of nginx (MALLOC_OPTIONS 2,3,D and -i -tu) for 2 hours each > (usually enough for memory consumption to ~ double with avg. load. > > I do not really know enough to debug this any further at the moment; Is > there anything else i could do to further circle in on the cause of > this (and ideally figure out whether it is a libressl or nginx issue)? > > Some data (graphs, ktraces) here: > https://rincewind.home.aperture-labs.org/~tfiebig/malloc/ For continuity for the list archives, previous thread was here https://marc.info/?l=openbsd-misc&m=169571937015656&w=2 There was never an answer to my last email there, though the graphs give some idea of the memory allocation rate, though it's quite different between some of the restarts - 7-8 Nov around 5GB - 8-9 Nov around 12GB - 9-10 Nov around 10GB though it doesn't really increase until a while after the process started. Is there any more context to those graphs? Config changes? Different access patterns?
