We ran into a production issue during our maintenance window when I switched
over LifeKeeper/PostgreSQL nodes from one to another. We noticed that the
database was 1000 times slower than it usually is, then realized that the
log_min_messages setting was set to 'debug5' for no reason.
Facts:
The postgresql.conf file has always been using the default value, which is
'warning'.An attempt to explicitly set log_min_messages=warning and reload
could not override the setting of 'debug5'.The heavy debug5 logging on our
production system drastically slowed down our database, and eventually filled
up the root file system (due to the huge /var/log/postgresql)The LifeKeeper
failover has been done in a recent past with the same unmodified postgresql.log
file without such an excessive logging behavior.Changing log_destination from
syslog to stderr appeared to have done the trick, after hours of
troubleshooting. The log_min_messages value took the explicit setting from
postgresql.log ('warning') and restored our database performance.
I could not find any relevant issue reported and want to see if anyone has a
clue about this issue.
Thank you,
-Kong
