I have reduced this to a smaller problem to hopefully make some progress on
getting ossec's FIM to report in a timely fashion.
I have set up a single ossec server handling a single client, monitoring
nearly everything on the system (roughly 70,000 files). Both machines are
Ubuntu 14.04. Here are the syscheck sections of ossec.conf from the server
and client:
Server:
<syscheck>
<!-- global options -->
<auto_ignore>no</auto_ignore>
<alert_new_files>yes</alert_new_files>
<! -- global exclusions -->
<ignore>/etc/mtab</ignore>
<ignore>/etc/blkid.tab</ignore>
</syscheck>
Client:
<syscheck>
<frequency>43200</frequency>
<directories
report_changes="no"
realtime="yes"
check_md5sum="yes"
check_sha1sum="yes"
check_size="yes"
check_owner="yes"
check_group="yes"
check_perm="yes"
>/var/ossec/etc</directories>
<directories
report_changes="no"
realtime="yes"
check_md5sum="yes"
check_sha1sum="yes"
check_size="yes"
check_owner="yes"
check_group="yes"
check_perm="yes"
>/etc,/opt,/usr/local,/boot/grub,/mnt,/media,/var/spool/cron</directories>
<directories
report_changes="no"
realtime="yes"
check_md5sum="yes"
check_sha1sum="yes"
check_size="yes"
check_owner="yes"
check_group="yes"
check_perm="yes"
>/bin,/boot,/lib,/lib64,/root,/sbin,/srv,/usr</directories>
</syscheck>
For testing purposes, I have also made this change to internal_options.conf:
syscheck.sleep=0
I have also turned up debugging and have tried playing around with
syscheck.sleep_after before setting syscheck.sleep to zero.
For this test, I have cleared the syscheck database and confirmed that the
database file is empty. I have restarted ossec on the client, and then
watched the client ossec.log for these events:
2016/12/28 15:27:38 ossec-syscheckd: INFO: Real time file monitoring
started.
2016/12/28 15:27:38 ossec-syscheckd: INFO: Finished creating syscheck
database (pre-scan completed).
2016/12/28 15:27:48 ossec-syscheckd: INFO: Ending syscheck scan (forwarding
database).
At this point, I would expect the syscheck database to have a complete
baseline. However, I find that the syscheck database file for this host on
the server contains only about 27,000 entries.
I restarted ossec on the client, and watched its log again. When the
syscheck scan reported success, I again enumerated the database on the
server side, and found that about 4000 additional files had been added. I
repeated this process, and found that between 3000 and 4000 files got added
each time I restarted the scan.
Each monitored directory is being visted on each scan, but not all files in
the directory are reported. Here is a rough accounting of the numbers of
reports I found from each of the first three runs :
First:
/bin: 151
/boot: 237
/lib: 5325
/lib64: 1
/root: 19
/sbin: 232
/srv: 0
/usr: 19548
/etc: 1897
/usr/local: 9
Second:
/bin: 0
/boot: 10
/lib: 128
/lib64: 0
/root: 0
/sbin: 0
/srv: 0
/usr: 3969
/etc: 87
/usr/local: 0
Third:
/bin: 0
/boot: 18
/lib: 51
/lib64: 0
/root: 0
/sbin: 0
/srv: 0
/usr: 3656
/etc: 18
/usr/local: 0
I sorted the database at this point and confirmed that none of these were
duplicates; each run was adding new files.
Here is the actual number of files and directories in each of those
directories on disk:
/bin: 152
/boot: 290
/lib: 6548
/lib64: 2
/root: 22
/sbin: 233
/srv: 1
/usr: 60312
/etc: 2340
/usr/local: 40
At this rate, with syscheck running twice a day, it will take at least a
week to complete the baseline. As I pointed out in my previous post, I
have in fact seen this take months, suggesting that the number of updates
with each run continues to decline.
Any suggestions for identifying this bottleneck would be appreciated.
Thank you.
On Monday, October 17, 2016 at 11:46:41 AM UTC-7, Sunny Day wrote:
>
> Is there a hard limit on the rate at which syscheck will report
> new/changed files?
>
> I have roughly 120 clients reporting to one server. I see frequent
> occasions where new or changed files (sometimes with realtime enabled,
> sometimes not) seem to be reported by syscheck days, weeks, or even months
> after they were known to be added or modified.
>
> This usually coincides with updates to the OS and/or kernel. Looking
> through the logs, it looks to me like the server is limiting reports to
> roughly (but not always exactly) 10 reports at a time, a second or two
> apart. For updates involving many thousands of files on 120 hosts, this
> can take so long that syscheck is simply not useful.
>
> I have documented specific examples of files that I know were added to the
> system months before they were actually reported by syscheck. Here is one.
>
> As an example, this is from the ossec server's alert log on Sept. 25:
>
> ** Alert 1474812143.8448019: mail - ossec, syscheck,
> 2016 Sep 25 07:02:23 (sampleclient) 10.0.0.9->syscheck
> Rule: 554 (level 10) -> 'File added to the system.'
> New file '/usr/lib/klibc/bin/cpio' added to the file system.
>
> Yet this file was present at least as far back as May 18:
>
> $ dpkg -S /usr/lib/klibc/bin/cpio
> klibc-utils: /usr/lib/klibc/bin/cpio
>
> $ zgrep -h -B2 klibc-utils /var/log/apt/history.log*
> Start-Date: 2016-05-18 10:58:30
> Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o
> Dpkg::Options::=--force-confold dist-upgrade
> Upgrade: libnl-genl-3-200:amd64 (3.2.21-1, 3.2.21-1ubuntu1.1),
> libnl-3-200:amd64 (3.2.21-1, 3.2.21-1ubuntu1.1), klibc-utils:
> amd64
> (2.0.3-0ubuntu1, 2.0.3-0ubuntu1.14.04.1), lsb-base:amd64
> (4.1+Debian11ubuntu6, 4.1+Debian11ubuntu6.1), lsb-release:amd64
> (4.1+Debian11ubuntu6, 4.1+Debian11ubuntu6.1), libklibc:amd64
> (2.0.3-0ubuntu1, 2.0.3-0ubuntu1.14.04.1)
>
> $ stat /usr/lib/klibc/bin/cpio
> File: /usr/lib/klibc/bin/cpio
> Size: 5168 Blocks: 16 IO Block: 4096 regular file
> Device: 802h/2050d Inode: 2360114 Links: 1
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2016-09-30 08:33:46.193812724 -0700
> Modify: 2016-04-27 21:27:30.000000000 -0700
> Change: 2016-05-18 10:58:33.066735324 -0700
> Birth: -
>
>
> Below are the syscheck-related configurations on the server side which
> affect /usr on the client:
>
> <syscheck>
> <!-- global options -->
> <auto_ignore>no</auto_ignore>
> <alert_new_files>yes</alert_new_files>
>
> <! -- global exclusions -->
> <ignore>/etc/mtab</ignore>
> <ignore>/etc/blkid.tab</ignore>
> </syscheck>
>
> And here are the relevant client-side directives:
>
> <syscheck>
> <!-- Frequency in seconds that syscheck is executed -->
> <frequency>43200</frequency>
>
> <directories
> realtime="no"
> check_md5sum="no"
> check_sha1sum="yes"
> check_size="yes"
> check_owner="yes"
> check_group="yes"
>
> check_perm="yes">/bin,/boot,/lib,/lib64,/opt,/sbin,/srv,/usr</directories>
> </syscheck>
>
> I have verified that syscheck is taking about 3 hours to run, which is
> well within the specified excecution frequency.
>
> 2016/09/25 06:28:55 ossec-syscheckd: INFO: Starting syscheck scan
> (forwarding database).
> 2016/09/25 06:28:55 ossec-syscheckd: INFO: Starting syscheck database
> (pre-scan).
> 2016/09/25 09:38:29 ossec-syscheckd: INFO: Ending syscheck scan
> (forwarding database).
> 2016/09/25 21:39:02 ossec-syscheckd: INFO: Starting syscheck scan.
> 2016/09/26 00:48:32 ossec-syscheckd: INFO: Ending syscheck scan
>
> If there's something obvious that I screwed up or overlooked, can anyone
> hit me on the head with it?
>
>
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
