On Fri, Dec 27, 2013 at 08:54:17PM PST, Jefferson Ogata spake thusly: > Log rotation tends to run around that time on Red Hat. Check your logrotate > configuration. Maybe something is rotating corosync logs and using the wrong > signal to start a new log file.
That was actually the first thing I looked at! I found /etc/logrotate.d/shorewall and removed it. But that seems to have had no effect on the problem. That file has been gone for 3 weeks, the machines rebooted (not that it should matter), and the problem has happened several times since then. I've searched all over and can't find anything. And it doesn't even happen every morning, just every week or two. Hard to nail down a real pattern other than usually (not always) 4am. > Or, if not that, could it be some other cronned task? These firewall machines are standard CentOS boxes. The stock crons (logrotate etc) and a 5 minute nagios passive check are the only things on them as far as I can tell. Although I haven't quite figured out what causes logrotate to run at 4am. I know it is in the /etc/cron.daily/logrotate but what runs this at 4am? Is 4am some special hard-coded time in crond? I just noticed that there is an /etc/logrotate.d/cman which rotates /var/log/cluster/*log Could this somehow be an issue? I'm running pacemaker and corosync but I'm not running cman: # /etc/init.d/cman status cman is not running Should I be? I don't think it is necessary for this particular kind of cluster... But since it isn't running it shouldn't matter. Oddly, I just noticed this in my tail -f of the logs (no idea what triggered it, but I did run /etc/init.d/cman status on the other node) which actually mentions cman: # Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: crm_client_new: Connecting 0x2485d00 for uid=0 gid=0 pid=11103 id=b507c867-cbde-4508-8813-1439720f9c6b Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.391.5) Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: crm_compress_string: Compressed 201733 bytes into 12221 (ratio 16:1) in 69ms Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: crm_client_destroy: Destroying 0 events Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: crm_client_new: Connecting 0x2485d00 for uid=0 gid=0 pid=11105 id=7e95fe48-ca73-4f29-b2b7-e43596fab588 Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.391.5) Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: crm_compress_string: Compressed 201732 bytes into 12220 (ratio 16:1) in 61ms Dec 27 21:55:05 [1541] new-fw2.mydomain.com cib: info: crm_client_destroy: Destroying 0 events Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log Dec 27 21:55:07 corosync [pcmk ] info: process_ais_conf: Reading configure Dec 27 21:55:07 corosync [pcmk ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon. Dec 27 21:55:07 corosync [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN Dec 27 21:55:07 corosync [pcmk ] info: config_find_init: Local handle: 7178156903111852040 for logging Dec 27 21:55:07 corosync [pcmk ] info: config_find_next: Processing additional logging options... Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Found 'off' for option: debug Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_logfile Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility Dec 27 21:55:07 corosync [pcmk ] info: config_find_init: Local handle: 5773499849093677065 for quorum Dec 27 21:55:07 corosync [pcmk ] info: config_find_next: No additional configuration supplied for: quorum Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: No default for option: provider Dec 27 21:55:07 corosync [pcmk ] info: config_find_init: Local handle: 7711695921217536010 for service Dec 27 21:55:07 corosync [pcmk ] info: config_find_next: Processing additional service options... Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Found '1' for option: ver Dec 27 21:55:07 corosync [pcmk ] info: process_ais_conf: Enabling MCP mode: Use the Pacemaker init script to complete Pacemaker startup Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd Dec 27 21:55:07 corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd Dec 27 21:55:07 corosync [pcmk ] info: pcmk_exec_dump: Local id: 1409417226, uname: new-fw2.mydomain.com, born: 4480 Dec 27 21:55:07 corosync [pcmk ] info: pcmk_exec_dump: Membership id: 4480, quorate: true, expected: 2, actual: 2 Dec 27 21:55:07 corosync [pcmk ] info: member_dump_fn: node id:1409417226, uname=new-fw2.mydomain.com state=member processes=0000000000000000 born=4480 seen=4480 addr=r(0) ip(10.0.2.84) version=1.1.10-14.el6 Dec 27 21:55:07 corosync [pcmk ] info: member_dump_fn: node id:1392640010, uname=new-fw1.mydomain.com state=member processes=0000000000000000 born=4472 seen=4480 addr=r(0) ip(10.0.2.83) version=1.1.10-14.el6 Dec 27 21:55:07 [1541] new-fw2.mydomain.com cib: info: crm_client_new: Connecting 0x2485d00 for uid=0 gid=0 pid=11283 id=09833e1c-00d3-45c4-8ea1-c019f7dc2a4b Dec 27 21:55:07 [1541] new-fw2.mydomain.com cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.391.5) Dec 27 21:55:07 [1541] new-fw2.mydomain.com cib: info: crm_compress_string: Compressed 201733 bytes into 12230 (ratio 16:1) in 62ms Dec 27 21:55:07 [1541] new-fw2.mydomain.com cib: info: crm_client_destroy: Destroying 0 events I don't see this directly relating to my problem but who knows... -- Tracy Reed _______________________________________________ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
