Ori.livneh has uploaded a new change for review.
https://gerrit.wikimedia.org/r/154414
Change subject: HHVM:
......................................................................
HHVM:
* Set hhvm.debug.core_dump_report_directory to /var/log/hhvm, which
will make it write stack traces to this directory.
* Update the Upstart job configuration to look for a stack trace in
$HHVM_LOG_DIR rather than in /tmp.
* Configure rsyslog to write HHVM's error log to /var/log/hhvm.log,
and to rotate the file when its size exceeds 100 Mb.
(It's not /var/log/hhvm/error.log because that directory has to be
owned by HHVM's uid/guid, but rsyslog drops privileges and runs as
syslog:adm.)
* The output channel does not discard the message, and its priority
is lower than MediaWiki's rsyslog config, so MediaWiki is still
able to capture HHVM log messages for fluorine.
* Configure logrotate to rotate hhvm.log and stacktraces.
Change-Id: I11b85c87a8902de28cba9f71840885a9864ec1df
---
M modules/hhvm/files/hhvm.conf
A modules/hhvm/files/hhvm.logrotate
A modules/hhvm/files/hhvm.rsyslog.conf
M modules/hhvm/manifests/init.pp
4 files changed, 79 insertions(+), 6 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/14/154414/1
diff --git a/modules/hhvm/files/hhvm.conf b/modules/hhvm/files/hhvm.conf
index 282d251..31d1598 100644
--- a/modules/hhvm/files/hhvm.conf
+++ b/modules/hhvm/files/hhvm.conf
@@ -39,12 +39,17 @@
# Read the PID of the exited HHVM process and remove the PID file.
read PID </var/run/hhvm/hhvm.pid
rm -f /var/run/hhvm/hhvm.pid
+
# Check if HHVM wrote a stack trace before exiting.
+ TRACE="${HHVM_LOG_DIR:-/var/log/hhvm}/stacktrace.${PID}.log"
+ [ -r "$TRACE" ] || exit 0
+
# If a stack trace exists and a handler is defined, invoke it.
- TRACE="/tmp/stacktrace.${PID}.log"
- if [ -r "$TRACE" ] && [ -n "$HHVM_TRACE_HANDLER" ]; then
- $HHVM_TRACE_HANDLER "$TRACE"
- fi
+ [ -n "$HHVM_TRACE_HANDLER" ] && $HHVM_TRACE_HANDLER "$TRACE"
+
+ # Append a UNIX timestamp to the log file name so it doesn't get
+ # clobbered when the PID is recycled.
+ mv "$TRACE" "${TRACE}.$(date +%s)"
end script
# Don't limit the size of core dumps.
diff --git a/modules/hhvm/files/hhvm.logrotate
b/modules/hhvm/files/hhvm.logrotate
new file mode 100644
index 0000000..6c6ed09
--- /dev/null
+++ b/modules/hhvm/files/hhvm.logrotate
@@ -0,0 +1,26 @@
+# logrotate(8) config for HHVM
+# This file is managed by Puppet
+
+# HHVM's error log, managed by rsyslog
+/var/log/hhvm.log {
+ rotate 4
+ weekly
+ missingok
+ notifempty
+ compress
+ delaycompress
+ sharedscripts
+ postrotate
+ reload rsyslog >/dev/null 2>&1 || true
+ endscript
+}
+
+# Stack trace logs generated by HHVM when it crashes
+/var/log/hhvm/stacktrace.*.log.* {
+ daily
+ missingok
+ rotate 100
+ nocompress
+ nocreate
+ noolddir
+}
diff --git a/modules/hhvm/files/hhvm.rsyslog.conf
b/modules/hhvm/files/hhvm.rsyslog.conf
new file mode 100644
index 0000000..16ba251
--- /dev/null
+++ b/modules/hhvm/files/hhvm.rsyslog.conf
@@ -0,0 +1,4 @@
+# Route syslog messages emitted by HHVM to /var/log/hhvm.log
+# and rotate the log file when it exceeds 100 Mb.
+$outchannel hhvm,/var/log/hhvm.log,100000000,/usr/sbin/logrotate -f
/etc/logrotate.d/hhvm
+:programname, startswith, "hhvm" :omfile:$hhvm
diff --git a/modules/hhvm/manifests/init.pp b/modules/hhvm/manifests/init.pp
index 948da27..e039dab 100644
--- a/modules/hhvm/manifests/init.pp
+++ b/modules/hhvm/manifests/init.pp
@@ -30,6 +30,24 @@
# documentation is getting better, but expect to have to dig around in
# the source code.
#
+#
+# === Logging
+#
+# By default, this module configures HHVM to log to syslog, and it
+# configures rsyslog to route HHVM messages to /var/log/hhvm.log.
+# Finally, it configures HHVM to write stack traces to /var/log/hhvm.
+#
+# /var/log
+# │
+# ├── hhvm.log
+# │
+# └── hhvm
+# │
+# ├── stacktrace.XXXX.log
+# │
+# └── stacktrace.YYYY.log, ...
+#
+#
# === Parameters
#
# [*user*]
@@ -44,6 +62,7 @@
#
# [*fcgi_settings*]
# Ditto, except for FastCGI mode.
+#
#
# === Examples
#
@@ -84,7 +103,10 @@
},
mysql => {
slow_query_threshold => 10 * 1000, # milliseconds
- }
+ },
+ debug => {
+ core_dump_report_directory => '/var/log/hhvm',
+ },
},
}
@@ -200,7 +222,23 @@
}
- ## Run-time directories
+ ## Run-time data and logging
+
+ rsyslog::conf { 'hhvm':
+ source => 'puppet:///modules/hhvm/hhvm.rsyslog.conf',
+ priority => 20,
+ require => File['/etc/hhvm/logrotate.d/hhvm'],
+ before => Service['hhvm'],
+ }
+
+ file { '/etc/hhvm/logrotate.d/hhvm':
+ source => 'puppet:///modules/hhvm/hhvm.logrotate',
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ require => File['/var/log/hhvm'],
+ before => Service['hhvm'],
+ }
file { [ '/run/hhvm', '/var/log/hhvm' ]:
ensure => directory,
--
To view, visit https://gerrit.wikimedia.org/r/154414
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I11b85c87a8902de28cba9f71840885a9864ec1df
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ori.livneh <o...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
