For the time being, that would make sense as a workaround. However, if you can, I suggest you test and confirm that a new inode gets created when the log is rotated. If Zimbra is doing this, then (IMO), it is the wrong way of rotating the log. The log file should be copied to a new file and then truncated to zero bytes, so that the inode does not change. If Zimra has a setting that allows you to create log files with the date/time stamp, it may solve your issue.
On Mon, Jul 11, 2011 at 4:23 PM, blacklight <vphu...@yahoo.com> wrote: > I take it that the corrective action going forward is to schedule a > restart of the OSSEC agent shortly after mailbox.log gets rotated. > > > On Jul 11, 1:04 pm, blacklight <vphu...@yahoo.com> wrote: > > I am eating my words just now, including a helping of crow protein :). > > I did restart the agent last Friday, but I did that only to make sure > > that nothing had died from the date I had restarted the agent back in > > June. > > > > I explicitly restarted the agent about 10 min ago and inserted the log > > entry into mailbox.log. Ta-da: > > > > 2011 Jul 11 12:57:05 Rule Id: 100111 level: 7 > > Location: (flanders.inv.anglerlabs.com) 10.80.80.3->/opt/zimbra/log/ > > mailbox.log > > Zimbra startup detected > > 2011-07-11 12:57:39,180 INFO [main] [] misc - version=7.1.1_GA_3213 > > release=20110624102500 builddate=20110624-1027 buildhost=zre- > > rhel4.eng.vmware.com <-- test by V. > > > > On Jul 11, 11:25 am, Christopher Moraes <cmoraes....@gmail.com> wrote: > > > > > > > > > > > > > > > > > I know OSSEC can handle log rotation in cases where the log file name > > > contains the date time stamp. This way, a new file is created when the > day > > > rolls over and OSSEC knows that it must read a new file. > > > > > However, in this case, I am not sure it is a log rotation issue. It > can be > > > tested easily by restarting the agent, and inserting the test log > before the > > > log rolls over. (I guess you've already tested this, right?) > > > > > If OSSEC is still not alerting on the event, then log rotation would > not > > > seem to be the issue. > > > > > On Mon, Jul 11, 2011 at 11:00 AM, blacklight <vphu...@yahoo.com> > wrote: > > > > Is there anything we can do when the log rotation results in an inode > > > > change? Aside from stopping the log from rotating, that is. > > > > > > On Jul 8, 4:35 pm, blacklight <vphu...@yahoo.com> wrote: > > > > > I restarted OSSEC agent at 14:43:01 - see the mailserver OSSEC > agent's > > > > > ossec.log that I posted in response to your request. > > > > > > > audit.log is not rotated and we don't have a problem with audit.log > > > > > mailbox.log is rotated, is gifted with a new inode as part of its > > > > > rotation. And mailbox.log is driving us crazy :). > > > > > > > On Jul 8, 4:22 pm, Christopher Moraes <cmoraes....@gmail.com> > wrote: > > > > > > > > When did you last restart the OSSEC agent? On jul 7th or Jul > 8th? > > > > > > > > Can you also run the same grep command on your alert log file for > today > > > > (Jul > > > > > > 08) > > > > > > > > The log rotation could be an issue if the Zimbra server is > created a > > > > new > > > > > > file (new inode) at the end of each day. (just thinking aloud) > > > > > > > > On Fri, Jul 8, 2011 at 4:12 PM, blacklight <vphu...@yahoo.com> > wrote: > > > > > > > [root@ossecserver tmp]# grep -A5 'mailbox.log' > ossec-alerts-07.log| > > > > > > > more > > > > > > > > > ossec: File rotated (inode changed): > '/opt/zimbra/log/mailbox.log'. > > > > > > > > > ** Alert 1310011255.433273: - syslog,postfix,spam, > > > > > > > 2011 Jul 07 00:00:55 (mailserver2.domain.com) > 10.80.80.4->/var/log/ > > > > > > > maillog > > > > > > > Rule: 3306 (level 6) -> 'IP Address black-listed by anti-spam > > > > > > > (blocked).' > > > > > > > Src IP: 123.27.188.67 > > > > > > > > > 2011 Jul 07 00:02:41 (mailserver.domain.com) > 10.8.8.30->/opt/zimbra/ > > > > > > > log/mailbox.log > > > > > > > Rule: 100301 (level 5) -> 'we are monitoring failed > authentication > > > > > > > attempts to zimbra' > > > > > > > Src IP: 209.85.214.134 > > > > > > > User: (none) > > > > > > > 2011-07-07 00:02:40,330 INFO [Pop3Server-39] > [ip=209.85.214.134;] > > > > > > > account - authentication failed for > hsander...@lefrakproductions.com > > > > > > > (no such account) > > > > > > > > > 2011 Jul 07 04:54:15 (mailserver.domain.com) > > > > 10.8.8.3->/opt/zimbra/log/ > > > > > > > mailbox.log > > > > > > > Rule: 1002 (level 2) -> 'Unknown problem somewhere in the > system.' > > > > > > > Src IP: (none) > > > > > > > User: (none) > > > > > > > Code:service.FAILURE > > > > > > > > > 33% of the messages refer to custom rule 100301; the balance of > the > > > > > > > messages refer to rule 1002 > > > > > > > We get one log rotation message per day. > > > > > > > > > Speaking about rotation, is "inode changed" a concern as far as > > > > > > > getting data from mailbox.log is concerned? My opinion is > "probably > > > > > > > not, because the OSSEC server is clearly receiving messages > culled > > > > > > > from mailbox.log from the log-collector daemon of the OSSSEC > agent. > > > > > > > It's just that these messages don't include the ones we want as > > > > > > > specified in our new rule. > > > > > > > > > On Jul 8, 3:19 pm, Christopher Moraes <cmoraes....@gmail.com> > wrote: > > > > > > > > For point #2 - can you go into your alerts.log file and paste > the > > > > entire > > > > > > > > alert message that is logged there. I'm interested in > knowing what > > > > alert > > > > > > > > has been generated. > > > > > > > > > > On Fri, Jul 8, 2011 at 3:10 PM, blacklight < > vphu...@yahoo.com> > > > > wrote: > > > > > > > > > (1) I restarted the OSSEC agent on the mailserver and the > the log > > > > > > > > > entry test on mailbox.log still failed. > > > > > > > > > > > (2) I got an inspiration and grepped alerts.log in the > OSSEC > > > > server > > > > > > > > > for mailbox.log, and I found recent activity: > > > > > > > > > > > 2011 Jul 08 10:47:10 (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/ > > > > > > > > > zimbra/log/mailbox.log > > > > > > > > > 2011 Jul 08 11:02:42 (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/ > > > > > > > > > zimbra/log/mailbox.log > > > > > > > > > 2011 Jul 08 11:04:39 (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/ > > > > > > > > > zimbra/log/mailbox.log > > > > > > > > > 2011 Jul 08 11:06:31 (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/ > > > > > > > > > zimbra/log/mailbox.log > > > > > > > > > 2011 Jul 08 11:07:10 (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/ > > > > > > > > > zimbra/log/mailbox.log > > > > > > > > > 2011 Jul 08 11:18:27 (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/ > > > > > > > > > zimbra/log/mailbox.log > > > > > > > > > > > This tells me that the OSSEC log-collector daemons are > doing > > > > their job > > > > > > > > > both on the OSSEC server host and on the OSSEC agent > mailserver. > > > > > > > > > > > I grepped for "buildhost" in alerts.log and found just one > > > > current > > > > > > > > > instance, and that instance was a test entry inserted in > > > > audit.log. I > > > > > > > > > am 100% sure that any instances that are archived from > alerts.log > > > > will > > > > > > > > > be test entries inserted in audit.log > > > > > > > > > > > On Jul 8, 12:24 pm, blacklight <vphu...@yahoo.com> wrote: > > > > > > > > > > 1. I'm assuming your audit.log file is on the same > server as > > > > the > > > > > > > > > > mailbox.log, right? - Correct. > > > > > > > > > > > > 2. Is OSSEC alerting on anything in the mailbox.log file? > Can > > > > you > > > > > > > > > > test with another known alert and insert it into > mailbox.log > > > > and > > > > > > > > > > verify that OSSEC is alerting on it? > > > > > > > > > > > > The log entry in /var/log/secure below > > > > > > > > > > > > Jul 5 17:09:29 mailserver sshd[19395]: Accepted password > for > > > > root > > > > > > > > > > from ::ffff:69.38.173.162 port 45026 ssh2 > > > > > > > > > > > > is captured through our custom rule 105715 > > > > > > > > > > > > <rule id="105715" level="7"> > > > > > > > > > > <if_sid>5715</if_sid> > > > > > > > > > > <user>root</user> > > > > > > > > > > <!-- match>^Accepted|authenticated.$</match --> > > > > > > > > > > <description>SSHD authentication > success.</description> > > > > > > > > > > <group>authentication_success,</group> > > > > > > > > > > </rule> > > > > > > > > > > > > OSSEC published the alert for this rule on 5 Jul 2011 > after > > > > 17:09:29: > > > > > > > > > > > > 011 Jul 05 17:09:31 Rule Id: 105715 level: 7 > > > > > > > > > > Location: (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/var/log/secure > > > > > > > > > > Src IP: ::ffff:69.38.173.162 > > > > > > > > > > SSHD authentication success. > > > > > > > > > > Jul 5 17:09:29 flanders sshd[19395]: Accepted password > for root > > > > > > > > > > from ::ffff:69.38.173.162 port 45026 ssh2 > > > > > > > > > > > > I adjusted this log entry for time (11:41:29) and date > (Jul > > > > 8), > > > > > > > > > > appended "<-- test by V." as usual and inserted it into > > > > audit.log. > > > > > > > > > > OSSEC published it almost immediately as expected: > > > > > > > > > > > > 011 Jul 08 11:41:18 Rule Id: 105715 level: 7 > > > > > > > > > > Location: (flanders.inv.anglerlabs.com) > > > > 10.80.80.3->/opt/zimbra/log/ > > > > > > > > > > audit.log > > > > > > > > > > Src IP: ::ffff:69.38.173.162 > > > > > > > > > > SSHD authentication success. > > > > > > > > > > Jul 8 11:41:29 flanders sshd[19395]: Accepted password > for root > > > > > > > > > > from ::ffff:69.38.173.162 port 45026 ssh2 <-- test by V. > > > > > > > > > > > > Unfortunately, OSSEC did not publish anything when I > inserted > > > > the > > > > > > > same > > > > > > > > > > exact entry into mailbox.log > > > > > > > > > > > > It appears at this point that OSSEC is not publishing any > alert > > > > > > > > > > nothing from mailbox.log is being published. Since all > OSSEC > > > > daemons > > > > > > > > > > on the OSSEC server host are 100% operational > > > > > > > > > > > > [root@ossecserver ~]# service ossec status > > > > > > > > > > ossec-monitord is running... > > > > > > > > > > ossec-logcollector is running... > > > > > > > > > > ossec-remoted is running... > > > > > > > > > > ossec-syscheckd is running... > > > > > > > > > > ossec-analysisd is running... > > > > > > > > > > ossec-maild is running... > > > > > > > > > > ossec-execd is running... > > > > > > > > > > ossec-csyslogd is running... > > > > > > > > > > > > and so are the OSSEC daemons on the OSSEC agent host > > > > > > > > > > > > [root@mailserver log]# service ossec status > > > > > > > > > > ossec-logcollector is running... > > > > > > > > > > ossec-syscheckd is running... > > > > > > > > > > ossec-agentd is running... > > > > > > > > > > ossec-execd is running... > > > > > > > > > > > > it appears as if OSSEC agent is not reading anything from > > > > > > > mailbox.log, > > > > > > > > > > despite the ossec.log entry in the mailserver host > claiming > > > > that the > > > > > > > > > > OSSEC agent is analyzing both audit.log and mailbox.log > as I > > > > had > > > > > > > > > > mentioned yesterday. > > > > > > > > > > > > Both audit.log and mailbox.log are in the same > /opt/zimbra/log > > > > ... > > > > read more ยป
