Re: [Simple-evcorr-users] how not to keep monitored files permanently open (not only) on NFS

[Risto Vaarandi]

hi Richard,

I have never used SEC for monitoring files on NFS file systems, but I can
provide few short comments on how input files are handled. After SEC has
successfully opened an input file, it will be kept open permanently. When
input file is removed or renamed, input file is still kept open (at that
point the file handle is pointing to disk blocks that no longer have a
name). When a new file appears on disk with the same name as original input
file, SEC will close the file handle pointing to nameless disk blocks and
will open the new input file, starting to process it from the beginning.
However, this operation is atomic and the input file will never show as
"Closed" in the dump file. Status "Closed" in dump file usually indicates
that SEC was not able to open the input file when it was started or
restarted with a signal (a common situation when file does not exist or
there is no permission to open the file), but it can also indicate that SEC
has closed the file due to IO error when reading from file (that should
normally not happen and usually means a serious disk issue).

In order to find out why input file is in closed state, I would recommend
to check the log of SEC (you can set it up with --log command line option).
For example, if the input file did not exist when SEC was started or
restarted with a signal, there should be the following lines in the log:

Opening input file test.log
Input file test.log does not exist!

Also, if file is closed due to IO error, there is a relevant message in the
log, for example: Input file test.log IO error (Input/output error),
closing the file.

Hope these comments are helpful,
risto


Hi Risto and friends,
>
> I am unsure about one conceptual question about how SEC keeps open
> monitored files.
>
> Using SEC as systemd service, when files stored in NFS (opened via
> addinput) being watched by SEC are moved elsewhere, and then their removal
> is tried, NFS persistently keeps .nfsNUMBER files in existence, unable to
> remove whole folder of them. This functionality of NFS is described e.g.
> here:
> https://www.ibm.com/su
> pport/pages/what-are-nfs-files-accumulate-and-why-cant-they-be-deleted-even-after-stopping-cognos-8
> 
>
>
> It seems, that SEC running as service (not re-opening and closing each
> monitored file in each "run") is keeping watched files permanently open,
> and it is not desirable in such setup.
>
> But: when I look into the dump, some files have "status: Open" and some
> "status: Closed", so maybe this my assumption about log files permanently
> opened by SEC is not correct - I am confused.
>
> How it is in reality? Has somebody experience with described behaviour
> (SEC+NFS, but it could arise also in other setups) and how to overcome it?
>
> Thank you.
>
> Richard
> ___
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] how not to keep monitored files permanently open (not only) on NFS

[Richard Ostrochovský]

Hi Risto and friends,

I am unsure about one conceptual question about how SEC keeps open
monitored files.

Using SEC as systemd service, when files stored in NFS (opened via
addinput) being watched by SEC are moved elsewhere, and then their removal
is tried, NFS persistently keeps .nfsNUMBER files in existence, unable to
remove whole folder of them. This functionality of NFS is described e.g.
here:
https://www.ibm.com/su
pport/pages/what-are-nfs-files-accumulate-and-why-cant-they-be-deleted-even-after-stopping-cognos-8



It seems, that SEC running as service (not re-opening and closing each
monitored file in each "run") is keeping watched files permanently open,
and it is not desirable in such setup.

But: when I look into the dump, some files have "status: Open" and some
"status: Closed", so maybe this my assumption about log files permanently
opened by SEC is not correct - I am confused.

How it is in reality? Has somebody experience with described behaviour
(SEC+NFS, but it could arise also in other setups) and how to overcome it?

Thank you.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SEC rules performance monitoring and tuning

[Risto Vaarandi]

hi Richard,

That's an interesting question. Dump files in JSON format have been
supported only across few recent versions (from 2.8.0 to 2.8.2) and don't
have long history behind them, but so far their format has stayed the same.
As for dump files in text format, there have been a number of changes since
early versions of SEC, but most of these changes have been related to
adding sections with new data into dump file, not changing the format of
existing information. I did a quick test with version 2.6.2 (released in
2012) and version 2.8.2 (most recent version), and the only difference
which can influence parsing is how rule usage statistics is represented.
With version 2.6.2, a single space character is used for separating words
in each printed line, and rule usage statistics looks like follows:

Statistics for the rules from test.sec
(loaded at Tue Feb  4 17:54:34 2020)

Rule 1 at line 1 (test) has matched 1 events
Rule 2 at line 7 (test) has matched 0 events
Rule 3 at line 16 (test) has matched 0 events
Rule 4 at line 23 (test) has matched 0 events

However, in version 2.8.2 the numerals in rule usage statistics are
space-padded on the left, and the width of each numeric field is determined
by the number of characters in largest numeral (that format change was
introduced in 2013 for readability reasons:
https://sourceforge.net/p/simple-evcorr/mailman/message/30185995/). For
example, if you have three rules that have matched 1, 823 and 7865 times,
the first numeral is printed as "   1", the second one as " 823", and the
third one as "7865". Here is an example of rule usage statistics data for
version 2.8.2:

Statistics for the rules from test.sec
(loaded at Tue Feb  4 17:53:30 2020)

Rule 1 line  1 matched 1 events (test)
Rule 2 line  7 matched 0 events (test)
Rule 3 line 16 matched 0 events (test)
Rule 4 line 23 matched 0 events (test)

The other differences between dump files are some extra data that is
printed for version 2.8.2 (for example, effective user and group IDs of the
SEC process). So one can say that the format of textual dump file has not
gone through major changes during the last 7-8 years, and also, there are
no plans for such changes in upcoming versions.

kind regards,
risto


Hi Risto,
>
> thank you for positive answer - dumping period in minutes in enough, not
> needed many times per second. And also for useful tips - I already noticed
> JSON option, just considering, that default is maybe more universally
> usable, because extra modules (JSON.pm) may not be installed or installable
> on any systems (not only due technical reasons, but also security policies
> - more code auditing needed). Maybe the question also could be, how
> frequent are changes of dump file formats during SEC development, which
> could break monitoring implemented for particular version of SEC.
>
> Richard
>
> ut 28. 1. 2020 o 12:40 Risto Vaarandi 
> napísal(a):
>
>> hi Richard,
>>
>> as I understand from your post, you would like to create SEC dump files
>> periodically, in order to monitor performance of SEC based on these dump
>> files. Let me first provide some comments on performance related question.
>> Essentially, creation of dump file involves a pass over all major internal
>> data structures, so that summaries about internal data structures can be
>> written into dump file. In fact, SEC traverses all internal data structures
>> for housekeeping purposes anyway once a second (you can change this
>> interval with --cleantime command line option). Therefore, if you re-create
>> the dump file after reasonable time intervals (say, after couple of
>> minutes), it wouldn't noticeably increase CPU consumption (things would
>> become different if dumps would be generated many times a second).
>>
>> For production use, I would provide couple of recommendations. Firstly,
>> you could consider generating dump files in JSON format which might make
>> their parsing easier. That feature was requested couple of years ago
>> specifically for monitoring purposes (
>> https://sourceforge.net/p/simple-evcorr/mailman/message/36334142/), and
>> you can activate JSON format for dump file with --dumpfjson command line
>> option. You could also consider using --dumpfts option which adds numeric
>> timestamp suffix to dump file names (e.g., /tmp/sec.dump.1580210466). SEC
>> does not overwrite the dump file if it already exists, so having timestamps
>> in file names avoids the need for dump file rotation (you would still need
>> to delete old dump files from time to time, though).
>>
>> Hopefully this information is helpful.
>>
>> kind regards,
>> risto
>>
>> Kontakt Richard Ostrochovský () kirjutas
>> kuupäeval E, 27. jaanuar 2020 kell 22:04:
>>
>>> Greetings, friends,
>>>
>>> there is an idea to monitor high-volume event flows (per rules or log
>>> files, e.g. in bytes per time unit) from dump files from 

Re: [Simple-evcorr-users] SEC rules performance monitoring and tuning

[Richard Ostrochovský]

Hi Risto,

thank you for positive answer - dumping period in minutes in enough, not
needed many times per second. And also for useful tips - I already noticed
JSON option, just considering, that default is maybe more universally
usable, because extra modules (JSON.pm) may not be installed or installable
on any systems (not only due technical reasons, but also security policies
- more code auditing needed). Maybe the question also could be, how
frequent are changes of dump file formats during SEC development, which
could break monitoring implemented for particular version of SEC.

Richard

ut 28. 1. 2020 o 12:40 Risto Vaarandi  napísal(a):

> hi Richard,
>
> as I understand from your post, you would like to create SEC dump files
> periodically, in order to monitor performance of SEC based on these dump
> files. Let me first provide some comments on performance related question.
> Essentially, creation of dump file involves a pass over all major internal
> data structures, so that summaries about internal data structures can be
> written into dump file. In fact, SEC traverses all internal data structures
> for housekeeping purposes anyway once a second (you can change this
> interval with --cleantime command line option). Therefore, if you re-create
> the dump file after reasonable time intervals (say, after couple of
> minutes), it wouldn't noticeably increase CPU consumption (things would
> become different if dumps would be generated many times a second).
>
> For production use, I would provide couple of recommendations. Firstly,
> you could consider generating dump files in JSON format which might make
> their parsing easier. That feature was requested couple of years ago
> specifically for monitoring purposes (
> https://sourceforge.net/p/simple-evcorr/mailman/message/36334142/), and
> you can activate JSON format for dump file with --dumpfjson command line
> option. You could also consider using --dumpfts option which adds numeric
> timestamp suffix to dump file names (e.g., /tmp/sec.dump.1580210466). SEC
> does not overwrite the dump file if it already exists, so having timestamps
> in file names avoids the need for dump file rotation (you would still need
> to delete old dump files from time to time, though).
>
> Hopefully this information is helpful.
>
> kind regards,
> risto
>
> Kontakt Richard Ostrochovský () kirjutas
> kuupäeval E, 27. jaanuar 2020 kell 22:04:
>
>> Greetings, friends,
>>
>> there is an idea to monitor high-volume event flows (per rules or log
>> files, e.g. in bytes per time unit) from dump files from *periodic
>> dumping*.
>>
>> The question is, if this is recommended for *production use* - answer
>> depends at least on if dump creation somehow slows down or stops SEC
>> processing, or it has not significant overhead (also for thousands of
>> rules, complex configurations).
>>
>> Are there some best practises for (or experiences with) performance
>> self-monitoring and tuning of SEC?
>>
>> Thank you.
>>
>> Richard
>> ___
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users