Ori.livneh has submitted this change and it was merged.
Change subject: HHVM: improvements to logging
......................................................................
HHVM: improvements to logging
* Set hhvm.debug.core_dump_report_directory to /var/log/hhvm, which
will make it write stack traces to this directory.
* Update the Upstart job configuration to look for a stack trace in
$HHVM_LOG_DIR rather than in /tmp.
* Configure rsyslog to write HHVM's error log to /var/log/hhvm/error.log.
* Set the owner of /var/log/hhvm to 'syslog', so rsyslog can write to it.
* Set the permissions of /var/log/hhvm to 0775, so that HHVM's GID allows
it to write stack traces to that location.
* Set the permissions of /run/hhvm/cache to 0755 rather than 0750. MediaWiki's
database passwords, etc. are in world-readable files on the app server; I
don't think HHVM's PHP bytecode cache is an issue.
* Configure logrotate to retain 90 days of HHVM error logs, and 7 days of
stack traces.
* The HHVM rsyslog filter does not discard the message, and its priority
is lower than MediaWiki's rsyslog config, so MediaWiki is still
able to capture HHVM log messages for fluorine.
Change-Id: I11b85c87a8902de28cba9f71840885a9864ec1df
---
M modules/hhvm/files/hhvm.conf
A modules/hhvm/files/hhvm.logrotate
A modules/hhvm/files/hhvm.rsyslog.conf
M modules/hhvm/manifests/init.pp
4 files changed, 82 insertions(+), 17 deletions(-)
Approvals:
Ori.livneh: Looks good to me, approved
Filippo Giunchedi: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/hhvm/files/hhvm.conf b/modules/hhvm/files/hhvm.conf
index 282d251..fe7d408 100644
--- a/modules/hhvm/files/hhvm.conf
+++ b/modules/hhvm/files/hhvm.conf
@@ -10,9 +10,9 @@
. /etc/default/hhvm
mkdir -p -m0755 "${HHVM_RUN_DIR:=/run/hhvm}"
mkdir -p -m0750 "${HHVM_RUN_DIR}/cache"
- mkdir -p -m0755 "${HHVM_LOG_DIR:=/var/log/hhvm}"
+ mkdir -p -m0775 "${HHVM_LOG_DIR:=/var/log/hhvm}"
chown -R "${HHVM_USER:=www-data}:${HHVM_GROUP:=www-data}" "$HHVM_RUN_DIR"
- chown -R "${HHVM_USER}:${HHVM_GROUP}" "$HHVM_LOG_DIR"
+ chown -R "syslog:${HHVM_GROUP}" "$HHVM_LOG_DIR"
# Set the target of the symbolic link '/usr/lib/hphp/extensions/current'
# to the directory whose name matches the extension API version of HHVM:
@@ -39,12 +39,17 @@
# Read the PID of the exited HHVM process and remove the PID file.
read PID </var/run/hhvm/hhvm.pid
rm -f /var/run/hhvm/hhvm.pid
+
# Check if HHVM wrote a stack trace before exiting.
+ TRACE="${HHVM_LOG_DIR:-/var/log/hhvm}/stacktrace.${PID}.log"
+ [ -r "$TRACE" ] || exit 0
+
# If a stack trace exists and a handler is defined, invoke it.
- TRACE="/tmp/stacktrace.${PID}.log"
- if [ -r "$TRACE" ] && [ -n "$HHVM_TRACE_HANDLER" ]; then
- $HHVM_TRACE_HANDLER "$TRACE"
- fi
+ [ -n "$HHVM_TRACE_HANDLER" ] && $HHVM_TRACE_HANDLER "$TRACE"
+
+ # Append the current UTC date (in YYYYMMDD format) to the file
+ # name, so it doesn't get clobbered when the PID is recycled.
+ mv --backup "$TRACE" "${TRACE}.$(date -u +%Y%m%d)"
end script
# Don't limit the size of core dumps.
diff --git a/modules/hhvm/files/hhvm.logrotate
b/modules/hhvm/files/hhvm.logrotate
new file mode 100644
index 0000000..7233592
--- /dev/null
+++ b/modules/hhvm/files/hhvm.logrotate
@@ -0,0 +1,23 @@
+# logrotate(8) config for HHVM
+# This file is managed by Puppet
+
+# HHVM's error log, managed by rsyslog.
+/var/log/hhvm/error.log {
+ daily
+ dateext
+ dateyesterday
+ rotate 90
+ missingok
+ nocreate
+ delaycompress
+ sharedscripts
+ postrotate
+ reload rsyslog >/dev/null 2>&1 || true
+ endscript
+}
+
+# Delete local stack trace logs after one week.
+/var/log/hhvm/stacktrace.*.log.* {
+ rotate 0
+ maxage 7
+}
diff --git a/modules/hhvm/files/hhvm.rsyslog.conf
b/modules/hhvm/files/hhvm.rsyslog.conf
new file mode 100644
index 0000000..09ebda4
--- /dev/null
+++ b/modules/hhvm/files/hhvm.rsyslog.conf
@@ -0,0 +1,3 @@
+# rsyslogd(8) configuration file for HHVM.
+# This file is managed by Puppet.
+:programname, startswith, "hhvm" /var/log/hhvm/error.log
diff --git a/modules/hhvm/manifests/init.pp b/modules/hhvm/manifests/init.pp
index 948da27..aa34549 100644
--- a/modules/hhvm/manifests/init.pp
+++ b/modules/hhvm/manifests/init.pp
@@ -30,6 +30,20 @@
# documentation is getting better, but expect to have to dig around in
# the source code.
#
+#
+# === Logging
+#
+# This module configures HHVM to write to syslog, and it configures
+# rsyslogd(8) to write HHVM's log messages to /var/log/hhvm/error.log.
+# HHVM is also configured to write stack traces to the same directory.
+#
+# /var/log/hhvm
+# │
+# ├── error.log
+# │
+# └── stacktrace.NNN.log.YYYYMMDD, ...
+#
+#
# === Parameters
#
# [*user*]
@@ -44,6 +58,7 @@
#
# [*fcgi_settings*]
# Ditto, except for FastCGI mode.
+#
#
# === Examples
#
@@ -84,7 +99,10 @@
},
mysql => {
slow_query_threshold => 10 * 1000, # milliseconds
- }
+ },
+ debug => {
+ core_dump_report_directory => '/var/log/hhvm',
+ },
},
}
@@ -200,21 +218,37 @@
}
- ## Run-time directories
+ ## Run-time data and logging
- file { [ '/run/hhvm', '/var/log/hhvm' ]:
+ rsyslog::conf { 'hhvm':
+ source => 'puppet:///modules/hhvm/hhvm.rsyslog.conf',
+ priority => 20,
+ require => File['/etc/hhvm/logrotate.d/hhvm'],
+ before => Service['hhvm'],
+ }
+
+ file { '/etc/hhvm/logrotate.d/hhvm':
+ source => 'puppet:///modules/hhvm/hhvm.logrotate',
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ require => File['/var/log/hhvm'],
+ before => Service['hhvm'],
+ }
+
+ file { '/var/log/hhvm':
+ ensure => directory,
+ owner => 'syslog',
+ group => $group,
+ mode => '0775',
+ before => Service['hhvm'],
+ }
+
+ file { [ '/run/hhvm', '/run/hhvm/cache' ]:
ensure => directory,
owner => $user,
group => $group,
mode => '0755',
- before => Service['hhvm'],
- }
-
- file { '/run/hhvm/cache':
- ensure => directory,
- owner => $user,
- group => $group,
- mode => '0750',
before => Service['hhvm'],
}
}
--
To view, visit https://gerrit.wikimedia.org/r/154414
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I11b85c87a8902de28cba9f71840885a9864ec1df
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ori.livneh <[email protected]>
Gerrit-Reviewer: Aaron Schulz <[email protected]>
Gerrit-Reviewer: BryanDavis <[email protected]>
Gerrit-Reviewer: Filippo Giunchedi <[email protected]>
Gerrit-Reviewer: Ori.livneh <[email protected]>
Gerrit-Reviewer: Tim Starling <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
