Hi Ales, On 25/09/2023 16:09, Aleš Rygl via dnsdist wrote:
I would to kindly ask for help or and advice. I have just upgraded one of our dnsdist instances from 1.7.4 do 1.8.4 together with OS upgrade (Debian 11.7 to 12.1). Everything works fine, no issues observed apart some deprecated config references. What is a big surprise to me is CPU usage. The newer version has nearly two times higher CPU consumption in userspace. I am nearly at 80% CPU with 16 physical cores (was about 40%). We have a lot of TLS (DoT) sessions (30k) and 60kqps in total (30k via DoT) here. The latency measured by dnsdist went up also. We are collecting all the metrics dnsdist produces via graphite so I can check counters, what could be wrong.
Wow, that's awful. It's the first time I hear about such a regression, and I really would like to understand what is going on. 1/ Are you using our packages, compiling yourself, or perhaps using the Debian ones? 2/ Do you think it would be possible for you to try downgrading the instance to 1.7.4 on Debian 12.1? It might help us pinpointing whether the issue is related to a system change (I have seen people complain about the performance of OpenSSL 3.0.x compared to 1.1.1x, for example).
3/ Would you mind sharing your configuration?4/ And finally, do you think it would be possible for you to collect a perf trace on this instance? It would require installing linux-perf, if possible the debug symbols for dnsdist (dnsdist-dbgsym) then running 'perf record --call-graph dwarf -p <pid of running dnsdist process> -o </path/to/output/file>' for a few dozens of seconds to collect a trace, stopping it with Ctrl+C and finally getting a report with "perf report -i </path/to/previous/file> --stdio". It should tell us where the CPU usage is going.
Best regards, -- Remi Gacogne PowerDNS.COM BV - https://www.powerdns.com/
OpenPGP_signature.asc
Description: OpenPGP digital signature
_______________________________________________ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
