RE: logging through systemd-journald is bottleneck

[Markus.Storm]

Hi all,

removing the rsyslogd <-> systemd-journald integration in RedHat as Quanah 
suggested did the trick.
OpenLDAP now makes use of all available CPU cores.
Syslogd takes the rest for standard level request logging to files, consuming 
~180% of (1)CPU on a 24 core machine. Systemd-journald ain't noticeable any 
longer.

Thanks Quanah for your excellent support!

Regards
Markus

> -Original Message-
> From: Quanah Gibson-Mount 
> Sent: Wednesday, March 18, 2020 6:53 PM
> To: Storm, Markus ; openldap-
> techni...@openldap.org
> Subject: Re: logging through systemd-journald is bottleneck
> 
> 
> 
> --On Wednesday, March 18, 2020 6:16 PM + Markus.Storm@t-
> systems.com
> wrote:
> 
> >
> > Dear all,
> >
> > we're currently testing performance of OpenLDAP on Oracle/RedHat Linux
> > and quite unexpected actually hit systemd-journald to be a bottleneck.
> > While OpenLDAP happily makes use of all available CPUs, that one is
> > single threaded, braking everything. The only way around I have found
> > is to set olcLoglevel to 0, speeding up my test run by a factor of 6(!).
> > That now of course is not an option to use in production.
> > I'd happily directly write to a file as I did in the old days but I
> > cannot get olcLogfile to work. And even if I was able to get there,
> > how do I stop OpenLDAP from logging to syslogd (which is inevitably
> > forwarding everything to system-journald ….) ?
> > Can anyone give advice how to handle this ? Any hint appreciated
> > (short of "get a decent OS" – that is not an option).
> 
> Hi Markus,
> 
> We always advise our customers to disable the abominable syslog<->journald
> integration that RedHat has pushed onto their customers with complete
> disregard to the consequences of this short-sighted action.  Not only does it
> destroy performance, it can also cause the system to completely deadlock.
> 
> If you need instructions on how to do this, let me know.
> 
> Regards,
> Quanah
> 
> 
> --
> 
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>

Re: Antw: [EXT] Re: logging through systemd-journald is bottleneck

[Quanah Gibson-Mount]

--On Thursday, March 19, 2020 6:46 PM +0200 Janne Peltonen 
 wrote:



Hi!

Sorry about slightly offtopic question, but I'm curious: how much log do
you guys typically generate when it starts to act as a bottleneck?


In the environment where it deadlocked the server, there were some 2 
million active users, the site was MMR, and the log level was stats+sync. 
There was also a rogue mobile device that was sending thousands of failing 
authentications per second on top of the normal load.


Regards,
Quanah



--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Re: [EXT] Re: logging through systemd-journald is bottleneck

[Dieter Klünter]

Am Thu, 19 Mar 2020 08:57:11 +0100
schrieb "Ulrich Windl" :

> >>> Dieter Klünter  schrieb am 18.03.2020 um
> >>> 19:57 in  
> Nachricht
> <30206_1584557842_5e726f12_30206_1570_1_20200318195706.1d992...@pink.fritz.box>:
> 
> > Am Wed, 18 Mar 2020 17:16:53 +
> > schrieb :
> >   
> >> Dear all,
> >> 
> >> we're currently testing performance of OpenLDAP on Oracle/RedHat
> >> Linux and quite unexpected actually hit systemd-journald to be a
> >> bottleneck. While OpenLDAP happily makes use of all available CPUs,
> >> that one is single threaded, braking everything. The only way
> >> around I have found is to set olcLoglevel to 0, speeding up my
> >> test run by a factor of 6(!). That now of course is not an option
> >> to use in production. I'd happily directly write to a file as I
> >> did in the old days but I cannot get olcLogfile to work. And even
> >> if I was able to get there, how do I stop OpenLDAP from logging to
> >> syslogd (which is inevitably forwarding everything to
> >> system-journald ) ? Can anyone give advice how to handle this
> >> ? Any hint appreciated (short of "get a decent OS" - that is not
> >> an option).  
> > 
> > I support Qanah's advice!
> > Beside this, consider a logging strategy based on required
> > information and neglected information, as well as min. and max.
> > server load.
> > 
> > Based on my experience I would disable logging as default, but
> > enable logging for a short given time, just a modify operation on
> > atribute olcLogLevel.
> > With regard to journald I advice to define filters, see man
> > journalctl(1).
> > If syslog is a requirement, change to rsyslog. Don't make use of
> > logstash!  
> 
> Can't openLDAP simply log to a different port than the default syslog
> port? If so, just set up some alternate local syslog server.
> Or, in case an external syslog server is supported, just use one that
> isn't using systemd-journald.
> 
> Sorry, I neved had the need to use a different syslog mechanism...

mann slapd(8), syslog-local-user, slapd logs, as default, to local4

-Dieter

-- 
Dieter Klünter | Systemberatung
http://sys4.de
GPG Key ID: E9ED159B
53°37'09,95"N
10°08'02,42"E

Re: Antw: [EXT] Re: logging through systemd-journald is bottleneck

[Janne Peltonen]

Hi!

Sorry about slightly offtopic question, but I'm curious: how much log do
you guys typically generate when it starts to act as a bottleneck?

I have systemd-journald that receives the log and propagates it via
imjournal to rsyslog which dumps it on disk and forwards it to one
central logging server using the udp syslog protocol and to another
using the tcp-plain syslog protocol and I've experienced no logging
related performance issues (although I had to tune the rate limit
settings of both systemd-journald and rsyslogd's imjournal module so
that they wouldn't start discarding log; I actually had some trouble
with that because the ratelimitburst option in rsyslog is an unsigned
short but that isn't documented anywhare and rsyslog doesn't complain if
you set it higher than fits into the data type).  Typical loads here are
something like 500 slapd log lines per second, peaks something like 5000
slapd log lines per second at slapd's default logging settings. Block
devices are local (we also have some vmware nodes with vmware virtual
devices, the vmware system's data storage is on a SAN) The OS is RHEL7.
The highest %CPU I see on the two logging processes is around 20,
typically in the low single digits.


Best,

Janne Peltonen
IT Dept
University of Helsinki

Antw: [EXT] Re: logging through systemd-journald is bottleneck

[Ulrich Windl]

>>> Dieter Klünter  schrieb am 18.03.2020 um 19:57 in
Nachricht
<30206_1584557842_5e726f12_30206_1570_1_20200318195706.1d992...@pink.fritz.box>:

> Am Wed, 18 Mar 2020 17:16:53 +
> schrieb :
> 
>> Dear all,
>> 
>> we're currently testing performance of OpenLDAP on Oracle/RedHat
>> Linux and quite unexpected actually hit systemd-journald to be a
>> bottleneck. While OpenLDAP happily makes use of all available CPUs,
>> that one is single threaded, braking everything. The only way around
>> I have found is to set olcLoglevel to 0, speeding up my test run by a
>> factor of 6(!). That now of course is not an option to use in
>> production. I'd happily directly write to a file as I did in the old
>> days but I cannot get olcLogfile to work. And even if I was able to
>> get there, how do I stop OpenLDAP from logging to syslogd (which is
>> inevitably forwarding everything to system-journald ) ? Can
>> anyone give advice how to handle this ? Any hint appreciated (short
>> of "get a decent OS" - that is not an option).
> 
> I support Qanah's advice!
> Beside this, consider a logging strategy based on required information
> and neglected information, as well as min. and max. server load.
> 
> Based on my experience I would disable logging as default, but enable
> logging for a short given time, just a modify operation on  atribute
> olcLogLevel.
> With regard to journald I advice to define filters, see man
> journalctl(1).
> If syslog is a requirement, change to rsyslog. Don't make use of
> logstash!

Can't openLDAP simply log to a different port than the default syslog port?
If so, just set up some alternate local syslog server.
Or, in case an external syslog server is supported, just use one that isn't
using systemd-journald.

Sorry, I neved had the need to use a different syslog mechanism...

> 
> -Dieter
> 
> -- 
> Dieter Klünter | Systemberatung
> http://sys4.de 
> GPG Key ID: E9ED159B
> 53°37'09,95"N
> 10°08'02,42"E

Re: logging through systemd-journald is bottleneck

[Dieter Klünter]

Am Wed, 18 Mar 2020 17:16:53 +
schrieb :

> Dear all,
> 
> we're currently testing performance of OpenLDAP on Oracle/RedHat
> Linux and quite unexpected actually hit systemd-journald to be a
> bottleneck. While OpenLDAP happily makes use of all available CPUs,
> that one is single threaded, braking everything. The only way around
> I have found is to set olcLoglevel to 0, speeding up my test run by a
> factor of 6(!). That now of course is not an option to use in
> production. I'd happily directly write to a file as I did in the old
> days but I cannot get olcLogfile to work. And even if I was able to
> get there, how do I stop OpenLDAP from logging to syslogd (which is
> inevitably forwarding everything to system-journald ) ? Can
> anyone give advice how to handle this ? Any hint appreciated (short
> of "get a decent OS" - that is not an option).

I support Qanah's advice!
Beside this, consider a logging strategy based on required information
and neglected information, as well as min. and max. server load.

Based on my experience I would disable logging as default, but enable
logging for a short given time, just a modify operation on  atribute
olcLogLevel.
With regard to journald I advice to define filters, see man
journalctl(1).
If syslog is a requirement, change to rsyslog. Don't make use of
logstash!

-Dieter

-- 
Dieter Klünter | Systemberatung
http://sys4.de
GPG Key ID: E9ED159B
53°37'09,95"N
10°08'02,42"E

Re: logging through systemd-journald is bottleneck

[Quanah Gibson-Mount]

--On Wednesday, March 18, 2020 6:16 PM + markus.st...@t-systems.com 
wrote:




Dear all,

we're currently testing performance of OpenLDAP on Oracle/RedHat Linux
and quite unexpected actually hit systemd-journald to be a bottleneck.
While OpenLDAP happily makes use of all available CPUs, that one is
single threaded, braking everything. The only way around I have found is
to set olcLoglevel to 0, speeding up my test run by a factor of 6(!).
That now of course is not an option to use in production.
I'd happily directly write to a file as I did in the old days but I
cannot get olcLogfile to work. And even if I was able to get there, how
do I stop OpenLDAP from logging to syslogd (which is inevitably
forwarding everything to system-journald ….) ?
Can anyone give advice how to handle this ? Any hint appreciated (short
of "get a decent OS" – that is not an option).


Hi Markus,

We always advise our customers to disable the abominable syslog<->journald 
integration that RedHat has pushed onto their customers with complete 
disregard to the consequences of this short-sighted action.  Not only does 
it destroy performance, it can also cause the system to completely deadlock.


If you need instructions on how to do this, let me know.

Regards,
Quanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

logging through systemd-journald is bottleneck

[Markus.Storm]

Dear all,

we're currently testing performance of OpenLDAP on Oracle/RedHat Linux and 
quite unexpected actually hit systemd-journald to be a bottleneck. While 
OpenLDAP happily makes use of all available CPUs, that one is single threaded, 
braking everything. The only way around I have found is to set olcLoglevel to 
0, speeding up my test run by a factor of 6(!).
That now of course is not an option to use in production.
I'd happily directly write to a file as I did in the old days but I cannot get 
olcLogfile to work. And even if I was able to get there, how do I stop OpenLDAP 
from logging to syslogd (which is inevitably forwarding everything to 
system-journald ) ?
Can anyone give advice how to handle this ? Any hint appreciated (short of "get 
a decent OS" - that is not an option).

Thanks,
Markus