Bug#927057: 1Gb of logs is too much
Control: blocks -1 by 928081 On 2019-04-27 12:09:19, Antoine Beaupré wrote: > On 2019-04-24 16:23:15, Willem Mulder wrote: >> On Wed, 24 Apr 2019 01:03:35 + Linda Lapinlampi >> wrote: >>> I also agree the logs should be compressed on daily rotation, but it >>> remains unclear to me how one would change this in Synapse without big >>> hacky behaviors. Preferably I'd use logrotate(8) if at all possible. >>> >>> This might be helpful: >>> https://docs.python.org/3/library/logging.handlers.html >>> >>> So maybe change logging.handlers.RotatingFileHandler in debian/log.yaml >>> to WatchFileHandler (use with logrotate(8)) if I've understood >>> correctly. TimedRotatingFileHandler is an alternative to let Python >> manage log >>> rotation by itself, but no logs will be compressed then. >> >> On my personal setup, I have the following changes to log.yaml: > > [...] > > I believe the attached debdiff fixes this problem. I have filed unblock request # for this and submitted this as a MR here as well: https://salsa.debian.org/matrix-team/matrix-synapse/merge_requests/1 I have proposed the update to the release team as well, in the bug that I marked as blocking to this bug. A. -- The ultimate test of your knowledge is your capacity to convey it to another. - Richard Feynman
Bug#927057: 1Gb of logs is too much
On 2019-04-24 16:23:15, Willem Mulder wrote:
> On Wed, 24 Apr 2019 01:03:35 + Linda Lapinlampi
> wrote:
>> I also agree the logs should be compressed on daily rotation, but it
>> remains unclear to me how one would change this in Synapse without big
>> hacky behaviors. Preferably I'd use logrotate(8) if at all possible.
>>
>> This might be helpful:
>> https://docs.python.org/3/library/logging.handlers.html
>>
>> So maybe change logging.handlers.RotatingFileHandler in debian/log.yaml
>> to WatchFileHandler (use with logrotate(8)) if I've understood
>> correctly. TimedRotatingFileHandler is an alternative to let Python
> manage log
>> rotation by itself, but no logs will be compressed then.
>
> On my personal setup, I have the following changes to log.yaml:
[...]
I believe the attached debdiff fixes this problem.
A.
--
Nothing in life is to be feared, it is only to be understood.
Now is the time to understand more, so that we may fear less.
- Marie Curie
diff -Nru matrix-synapse-0.99.2/debian/changelog matrix-synapse-0.99.2/debian/changelog
--- matrix-synapse-0.99.2/debian/changelog 2019-03-24 09:02:48.0 -0400
+++ matrix-synapse-0.99.2/debian/changelog 2019-04-27 11:48:31.0 -0400
@@ -1,3 +1,10 @@
+matrix-synapse (0.99.2-3.1) unstable; urgency=medium
+
+ * Non-maintainer upload.
+ * fix log rotation to be less verbose (Closes: #927057)
+
+ -- Antoine Beaupré Sat, 27 Apr 2019 11:48:31 -0400
+
matrix-synapse (0.99.2-3) unstable; urgency=medium
* Make the code querying the location of the key file actually work.
diff -Nru matrix-synapse-0.99.2/debian/log.yaml matrix-synapse-0.99.2/debian/log.yaml
--- matrix-synapse-0.99.2/debian/log.yaml 2019-03-24 09:02:48.0 -0400
+++ matrix-synapse-0.99.2/debian/log.yaml 2019-04-27 11:48:31.0 -0400
@@ -14,12 +14,12 @@
handlers:
file:
-class: logging.handlers.RotatingFileHandler
+class: logging.handlers.WatchedFileHandler
formatter: precise
filename: /var/log/matrix-synapse/homeserver.log
-maxBytes: 104857600
-backupCount: 10
filters: [context]
+level: DEBUG
+encoding: utf8
console:
class: logging.StreamHandler
formatter: precise
@@ -31,12 +31,18 @@
SYSLOG_IDENTIFIER: synapse
loggers:
+twisted:
+level: WARN
+
synapse:
-level: INFO
+level: WARN
synapse.storage.SQL:
level: INFO
+synapse.http.matrixfederationclient:
+level: ERROR
+
root:
level: INFO
handlers: [file, journal]
diff -Nru matrix-synapse-0.99.2/debian/matrix-synapse.logrotate matrix-synapse-0.99.2/debian/matrix-synapse.logrotate
--- matrix-synapse-0.99.2/debian/matrix-synapse.logrotate 1969-12-31 19:00:00.0 -0500
+++ matrix-synapse-0.99.2/debian/matrix-synapse.logrotate 2019-04-27 11:48:31.0 -0400
@@ -0,0 +1,8 @@
+/var/log/matrix-synapse/homeserver.log {
+rotate 7
+daily
+missingok
+notifempty
+delaycompress
+compress
+}
Bug#927057: 1Gb of logs is too much
On Wed, 24 Apr 2019 01:03:35 + Linda Lapinlampi
wrote:
> I also agree the logs should be compressed on daily rotation, but it
> remains unclear to me how one would change this in Synapse without big
> hacky behaviors. Preferably I'd use logrotate(8) if at all possible.
>
> This might be helpful:
> https://docs.python.org/3/library/logging.handlers.html
>
> So maybe change logging.handlers.RotatingFileHandler in debian/log.yaml
> to WatchFileHandler (use with logrotate(8)) if I've understood
> correctly. TimedRotatingFileHandler is an alternative to let Python
manage log
> rotation by itself, but no logs will be compressed then.
On my personal setup, I have the following changes to log.yaml:
diff --git a/log.yaml.dpkg-dist b/log.yaml
index caa3fac..187bc7c 100644
--- a/log.yaml.dpkg-dist
+++ b/log.yaml
@@ -14,12 +14,12 @@ filters:
handlers:
file:
- class: logging.handlers.RotatingFileHandler
+ class: logging.handlers.WatchedFileHandler
formatter: precise
filename: /var/log/matrix-synapse/homeserver.log
- maxBytes: 104857600
- backupCount: 10
filters: [context]
+ level: DEBUG
+ encoding: utf8
console:
class: logging.StreamHandler
formatter: precise
@@ -31,12 +31,18 @@ handlers:
SYSLOG_IDENTIFIER: synapse
loggers:
+ twisted:
+ level: WARN
+
synapse:
- level: INFO
+ level: WARN
synapse.storage.SQL:
level: INFO
+ synapse.http.matrixfederationclient:
+ level: ERROR
+
root:
level: INFO
handlers: [file, journal]
And the following in /etc/logrotate.d/matrix-synapse:
/var/log/matrix-synapse/homeserver.log {
rotate 7
daily
missingok
notifempty
delaycompress
compress
}
This gives the following result in /var/log/matrix-synapse:
total 12M
-rw-r--r-- 1 matrix-synapse nogroup 2.0M Apr 24 16:14 homeserver.log
-rw-r--r-- 1 matrix-synapse nogroup 8.4M Apr 24 06:24 homeserver.log.1
-rw-r--r-- 1 matrix-synapse nogroup 295K Apr 23 06:24 homeserver.log.2.gz
-rw-r--r-- 1 matrix-synapse nogroup 253K Apr 22 06:25 homeserver.log.3.gz
-rw-r--r-- 1 matrix-synapse nogroup 258K Apr 21 06:25 homeserver.log.4.gz
-rw-r--r-- 1 matrix-synapse nogroup 277K Apr 20 06:25 homeserver.log.5.gz
-rw-r--r-- 1 matrix-synapse nogroup 292K Apr 19 06:24 homeserver.log.6.gz
-rw-r--r-- 1 matrix-synapse nogroup 311K Apr 18 06:24 homeserver.log.7.gz
In my experience, this leaves logging useful enough to debug problems,
while drastically cutting in used disk space.
signature.asc
Description: OpenPGP digital signature
Bug#927057: 1Gb of logs is too much
One can also adjust /etc/matrix-synapse/log.yaml, but I agree some change should be made to debian/log.yaml (default configuration file) to reduce the maximum size of logs stored and/or log level to WARN (WARNING). I also agree the logs should be compressed on daily rotation, but it remains unclear to me how one would change this in Synapse without big hacky behaviors. Preferably I'd use logrotate(8) if at all possible. This might be helpful: https://docs.python.org/3/library/logging.handlers.html So maybe change logging.handlers.RotatingFileHandler in debian/log.yaml to WatchFileHandler (use with logrotate(8)) if I've understood correctly. TimedRotatingFileHandler is an alternative to let Python manage log rotation by itself, but no logs will be compressed then.
Bug#927057: 1Gb of logs is too much
I think the issue here is that python's logging isn't compressing older files. total 1.1G -rw-r--r-- 1 matrix-synapse matrix-synapse 68M Apr 22 02:12 homeserver.log -rw-r--r-- 1 matrix-synapse matrix-synapse 100M Apr 18 05:13 homeserver.log.1 -rw-r--r-- 1 matrix-synapse matrix-synapse 100M Feb 25 13:29 homeserver.log.10 -rw-r--r-- 1 matrix-synapse matrix-synapse 100M Apr 11 19:04 homeserver.log.2 A patch is needed upstream in order to make it compress logs w/ python's logger: https://stackoverflow.com/questions/8467978/python-want-logging-with-log-rotation-and-compression As a workaround, the package could probably just ship a script in cron.daily to do the following: #!/bin/sh -e for log in /var/log/matrix-synapse/*.log.*[02-9]; do test -f $log && gzip $log; done After compression, things are a bit more reasonable: total 213M -rw-r--r-- 1 matrix-synapse matrix-synapse 68M Apr 22 02:32 homeserver.log -rw-r--r-- 1 matrix-synapse matrix-synapse 100M Apr 18 05:13 homeserver.log.1 -rw-r--r-- 1 matrix-synapse matrix-synapse 5.0M Feb 25 13:29 homeserver.log.10.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.0M Apr 11 19:04 homeserver.log.2.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.1M Apr 6 04:48 homeserver.log.3.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.1M Apr 1 00:35 homeserver.log.4.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.1M Mar 26 15:22 homeserver.log.5.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.1M Mar 21 00:08 homeserver.log.6.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.0M Mar 15 06:03 homeserver.log.7.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.1M Mar 9 20:34 homeserver.log.8.gz -rw-r--r-- 1 matrix-synapse matrix-synapse 5.0M Mar 3 09:15 homeserver.log.9.gz
Bug#927057: 1Gb of logs is too much
Really, I was wrong, logs are rotated by synapse itself. But it will produce 1Gb of log files in default configuration. Moreover very rapidly, as default loglevel is INFO. -- sergio.
