[Simple-evcorr-users] "multi-line" and multi-file logs - out of box

[Richard Ostrochovský]

Hello guys,

in fact, this question is about 2 independent things, but I see interesting
parallels to think about both topics together:

I know, that it je possible to create SEC rules configurations
(correlators), to process:

- "multi-line" logs (means having message separator other than "\n")

- multi-file logs (means monitoring all logfile paths matching wildcarded
pattern, all at the same time equivalently)

by converting them to "regular" lines / files, and then processing as usual.

My question is, if you see, how some of this things could be accomplished
in more generic way, without special configurations of correlation rules.
It would be great having SEC supporting such use cases "out of box", e.g.
by:

- having configurable line delimiter pattern (regular expression)

- accepting wildcard pattern as specification of input log file, to
"monitor them all" (also dynamically adding newly created files matching
wildcard and removing disappeared)

I don't have clue, how hard would be implementation of such things directly
in SEC (maybe question to Risto?), or if do you see also other, more
straightforward, solutions, without bringing more complexity to SEC rules,
I would be grateful for your know-how sharing.

Have a nice days.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] "multi-line" and multi-file logs - out of box

[Richard Ostrochovský]

Risto, thank you for your pre-analysis about multi-lines with regexp, and
also for suggestions about multi-files yet more sophisticated solution.

My comments are also inline:

st 27. 11. 2019 o 15:07 Risto Vaarandi 
napísal(a):

> hi Richard,
>
...

> In the current code base, identifying the end of each line is done with a
> simple search for newline character. The newline is searched not with a
> regular expression, but rather with index() function which is much faster.
> It is of course possible to change the code, so that a regular expression
> pattern is utilized instead, but that would introduce a noticeable
> performance penalty. For example, I made couple of quick tests with
> replacing the index() function with a regular expression that identifies
> the newline separator, and when testing modified sec code against log files
> of 4-5 million events, cpu time consumption increased by 25%.
>

Hmm, this is interesting. The philosophically principial question came to
my mind, if this penalty could be decreased (optimized), when doing
replacements of these regular newline characters ("\n") and matching
endings of "lines" with regexp - through rules (or by other more external
way) - before further processing by subsequent rules, instead of potential
built-in feature (used optionally on particular logfiles).

Introducing a custom delimiter also raises another important question --
> should it also be used for actions that utilize newline as a natural
> separator? For example, 'add' and 'event' actions split data into events by
> newline (if present in the data), 'spawn' action assumes that events from a
> child process are separated by newlines, etc. Should all these actions be
> changed to use the new separator?
>

Just thinking: if I understand correctly, about delimiter in event stores,
maybe there could be internally used "special character" other than regular
newline, which is most improbable to occur in logfiles (at least plain text
logfiles - I don't know, if SEC is being used sometimes also on binary
files, where any combination of bits can occur). I have not analyzed
current code base, but maybe at least here could be some performance
savings, when not differentiating "lines" in event stores with regular
expression matching, as when reading logfiles, but just with single
character. Maybe one of these?:

   - NUL (ASCII 0, null character)
   - ETX (ASCII 3, end of text)
   - RS (ASCII 30, record separator)

Given the performance penalty and other delimiter related questions, this
> idea needs careful thinking before implementing it in the code. (Before
> moving forward, it would be also interesting to know how many users would
> see this idea worth implementing.)
>

Let's see, if also somebody else will be interested in this topic.

>
>
>> - accepting wildcard pattern as specification of input log file, to
>> "monitor them all" (also dynamically adding newly created files matching
>> wildcard and removing disappeared)
>>
>
> It would be easier to implement that functionality, since input file
> patterns are re-evaluated on some signals, and in principle it is possible
> to invoke similar code after short time periods (e.g., once in 5 seconds).
> However, sec-2.8.X has 'addinput' and 'dropinput' actions which offer more
> general interface for dynamically adding and dropping inputs. For example,
> it is possible to start an external script with 'spawn' action which can
> detect input files not just with wildcard match but also more advanced
> criteria, and generate synthetic events like NEWFILE_ for input files
> that need opening. These synthetic events can be easily captured by a rule
> which invokes 'addinput' action for relevant files. I acknowledge that this
> functionality is somewhat different from providing wildcards in command
> line and requires writing your own script, but you can actually do more
> advanced things here.
>

Yes, this sounds yet more advanced.


> kind regards,
> risto
>

Thank you

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] "multi-line" and multi-file logs - out of box

[Richard Ostrochovský]

hi Risto,

thank you for sharing. You are undoubtedly right, it is common, that
"delimiter" pattern uses to be beginning part of the message (we can't say
this about "\n", which is not "part of the beginning"... delimiting the end
of message is then optional, in cases where detecting the beginning of the
next message is satisfactory), so my original "specification" wasn't so
useful, as it could be.

Considering myself a Perl developer too, I could surely implement something
more or less as you suggested (and I probably will). But I consider also
important to explain, why I am asking such questions.

At first:

   - I like, what SEC does
   - I like, how SEC does it (although, it is not for "everybody", as
   describing below)

I operate as monitoring consultant and developer. My job is to implement
monitoring solutions and hand over them in documented state to "operations
guys" for further administration. These guys often are not developers, but
SEC configurations are more about programming, than about plain
configurations as "regular" administrators use to do (as I see it). At
first sight, good for me, because when those guys need to change something
and configuring SEC is out of their area of expertise, they depend on me,
and I have the next job... but: this is not the way I think, and I am
really trying to built such solutions, which will be comprehensible and
maintainable also by "regular" administrators by keeping them as simple as
possible, and then I can focus more on higher-level solutions, with more
added value, when I save some of my time by not doing these low level
configurations repeatedly for them (a prefer teaching fishing than selling
fishes). There is a lot of human resources turnover in IT, and SEC training
for its administrator's replacement sounds as science-fiction for me.
Therefore I am suggesting to be able to do most common things out-of-box
from command line options or some basic configuration, without need to
configure rules for everything (yes, I know, it is not easy nor clear to
tell, which things are "most common", as you said, infinite possibilities
:) ). This is also related with my next post (which am I currently
writing), maybe you will like that other ideas more than these ones
(multi-line & multi-file).

In this light, your suggestion with RegExpN is also interesting, as it can
decrease number of rules (less rules => less complexity) and scripting
complexity - in cases, where it is applicable.

Thank you again - for SEC availability at the first place :).

Richard

št 28. 11. 2019 o 11:33 Risto Vaarandi 
napísal(a):

> hi Richard,
>
> just one followup thought -- have you considered sec native multi-line
> patterns such as RegexpN for handling multi-line logs? Of course, there are
> scenarios where the value of N (max number of lines in a multi-line event)
> can be very large and is difficult to predict, and for such cases an event
> converter (like the one from the previous post) is probably the best
> approach. However, if you know the value of N and events can be matched
> with regular expressions, RegexpN pattern type can be used for this task.
> For example, if events have the brace separated format described in the
> previous post, and events can contain up to 20 lines, one could utilize the
> following Regexp20 pattern for matching:
>
> type=Single
> ptype=RegExp20
> pattern=(?s)^(?:.+\n)?\{\n(.+\n)?\}$
> desc=match a multiline event between braces
> action=write - $1
>
> Also, if you want to convert such multi-line events into a single-line
> format with builtin features, sec 'rewrite' action allows for that. In the
> following example, the first rule takes the multi-line data between braces
> and replaces each newline with a space character, and resulting single-line
> string (with a prefix "Converted event: ") is used for overwriting sec
> event buffer. The second rule is now able to match such converted events:
>
> type=Single
> ptype=RegExp20
> pattern=(?s)^(?:.+\n)?\{\n(?:(.+)\n)?\}$
> continue=TakeNext
> desc=convert a multiline event between braces to single-line format
> action=lcall %ret $1 -> ( sub { my($t) = $_[0]; $t =~ s/\n/ /g; return $t;
> } ); \
>rewrite 20 Converted event: %ret
>
> type=Single
> ptype=RegExp
> pattern=Converted event: (.*)
> desc=match any event
> action=write - $1
>
> Maybe above examples are helpful for getting additional insights into
> different ways of processing multi-line events.
>
> kind regards,
> risto
>
>
> hi Richard,

>>> ...
>>>
 In the current code base, identifying the end of each line is done with
 a simple search for newline character. The newline is searched not with a
 regular expression, but rather with index() function which is much faster.
 It is of course possible to change the code, so that a regular expression
 pattern is utilized instead, but that would introduce a noticeable
 performance penalty. For example, I made couple of quick tests with
 replacing the index(

[Simple-evcorr-users] generate SEC configurations automatically

[Richard Ostrochovský]

Hello friends,

this post loosely follows this one:
https://sourceforge.net/p/simple-evcorr/mailman/message/36867007/.

Being monitoring consultant and developer, I have an idea to hide
complexity of SEC configurations, and still to allow to configure it also
for "regular" administrators without any developer or SEC background.

Imagined concept illustration:

[configuration DB] -> [generator(s)] -> [SEC configurations]

The punch line is, that user won't need to know anything about SEC, but
will need to understand logic of correlations employed, and their
parameters (configuration DB may have some kind of GUI). In the background,
higher-level correlations will be translated to respective SEC rules.

Maybe there exists something similar as described - if somebody knows about
something, I'd like if he or she will navigate me to it. If it does not
exist, maybe this is potential opportunity for implementation, and this way
also SEC could be more propagated, as still alive alternative to other
newer solutions usable for event correlations, e.g. based on ELK (I see big
advantage of SEC, that it does not need separate application infrastructure
for log collection and processing).

Any opinions about this topic?

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] generate SEC configurations automatically

[Richard Ostrochovský]

hi John and Risto,

Being a "regular" administrator, I claim admins
>> that can't program/use programming methods won't
>> be admins much longer. If the companies you work
>> for are depending on (rapidly turning over) junior
>> admins to administer something as important as
>> monitoring and correlation you have a difficult
>> job ahead of you.
>>
>
Sometimes yes. We see this trend "everywhere" - the more are computers (and
smartphones) doing instead of us, the less we need to know. We see this in
schools and also in IT business, it is getting harder and harder to hire
personnel qualified to do everything needed, and when you train somebody,
then he or she can easily go away (because there is much more
opportunities, than qualified people), and you can train somebody else, and
the loop continues. Or there is a possiblity to have smarter technologies,
compensating less skilled and experienced personnel, and this is the way I
see, (un?)fortunately...

>
>> Knowing regular expressions at the very least is
>> required to use SEC.
>
>
> I agree with John's opinion. Also, regular expressions are not something
> specific to SEC, but they are used by many other event log processing and
> network monitoring tools such as syslog-ng, rsyslog, suricata, etc. While
> learning regular expressions requires some time from a newcomer, it is
> difficult to see how a monitoring engineer/admin would manage without this
> knowledge.
>

I agree too. Getting familiar with regular expressions is not that hard, as
it resembles file path wildcards which are notoricaly familiar (but regular
expressions are of course more powerful), and it is not that hard to
explain basics of regular expression also to unexperienced newcomer, how to
match simple / basic patterns. There are also a lot of self-teaching
materials to this topic, online tutorials and converters / checkers, as
they are widely used and therefore harder to avoid. For me, knowing regular
expession basics are definitely basics.

For example we can have HPOM admins, using simplified "regular expressions"
by HP:
http://www.softpanorama.org/Admin/HP_operations_manager/Policies/Conditions/patterns.shtml.
Not that hard to comprehend. Nor basics of Perl regular expressions, and
this also enables to gradually becoming regular expressions expert, step by
step, and use this knowledge in various domains. As I see it, it is more
about pattern matching, than about true programming.

SEC, as I see it, is another story, as there are layers of more
sophistication, building (not only) on regular expressions, but joining
that with event correlations algorithmization and (Perl) programming. I see
this much more demanding, than just regular expressions. The area for SEC
use is much narrower, than area of use of regular regular expressions, and
that may mean more motivation for successful avoiding learning it, with
greater chance, than one won't need it at the end (before he or she changes
work position, and then the knowledge remains in vacuum).


> I would also argue that creating more advanced event correlation schemes
> (not only for SEC, but for any other event correlation tool) definitely
> requires at least some development skills, since one has to essentially
> write down event processing algorithms here.
>

Also agree. I don't suggest total avoiding programming, but "hiding"
(packaging) it from people, that don't need it, but still being able to use
it. When I know what it does, I don't always need also know how it does it.
To ride a car, I don't need to be a car designer or motor mechanic. When I
want a car, I just configure and order it, I don't need to design, develop,
and manufacture it. I use it without that knowledge, but when I want to
repair, upgrade, something, or buy a new car, I am contacting specialist
again.

>
> Having said that, there are certainly ways to improve future versions of
> SEC -- for example, re-evaluating input file patterns after short time
> intervals (suggestion from one of the previous posts) is worth considering
> and I've written it down as an idea for the next version.
>
> Getting performance info out
>> of SEC is better, but still
>> difficult. E.G. finding and fixing expensive/poor
>> regular expressions can result in a significant
>> improvement of performance/throughput along with
>> hierarchical structuring of the rulesets.
>>
>> >Imagined concept illustration:
>> >
>> >[configuration DB] -> [generator(s)] -> [SEC configurations]
>> >
>> >The punch line is, that user won't need to know
>> >anything about SEC, but will need to understand
>> >logic of correlations employed, and their
>> >parameters
>>
>> I assume you mean regular expressions, threholds,
>> actions and commands etc.
>>
>
I meant:

   - *correlation type* (selected from some catalogue, where it is
   described, what it does, without needing to know how it does it... for
   example SEC supports 12 atomic rule types, but there are infinite
   possiblilities, what can I compose from the

Re: [Simple-evcorr-users] generate SEC configurations automatically

[Richard Ostrochovský]

po 2. 12. 2019 o 14:19 Risto Vaarandi  napísal(a):

>
>> This SEC admin, as I see it, is still also from user perspective, tightly
>> bound with SEC technology and user needs to know, how SEC rules and their
>> compositions work internally. I am imagining something, that is
>> technology-agnostic, just describing logic of correlations, and from that
>> "formal description" generating SEC (or something other,
>> technology-agnostic means, that it may not to be dependent on SEC or other
>> specific technology, user needs just "what", but not "how"). Therefore
>> configuration DB should not operate with SEC-specific entities, but rather
>> something more general. Maybe something, which could be described with
>> something like EMML (Event Management Markup Language - never used that, I
>> don't know it, I just found that it exists).
>>
>> Levels of formal description of correlation types:
>>
>> - *user (admin) level* (higher-level, less detailed) - containing enough
>> information to configure and use described correlation type
>>
>> - *developer level* (lower-level, more detailed) - containing enough
>> information, to generate lower-level code (SEC configurations) from formal
>> description (from configuration DB, maybe with support of something like
>> EMML)
>>
>> Hope this helps to undertand what I wanted to say :).
>>
>
> Developing another event correlation language and tools for supporting it
> sounds like a separate major software development project to me.
> Before embarking on that, have you considered using configuration
> management frameworks (e.g., Salt or Ansible)? Among other things, they
> allow you to create configuration files in automated fashion after the user
> has set some generic parameters. It might not be exactly what you have
> envisioned, but it is definitely cheaper workaround than rolling out an
> entirely new solution.
>

Thank you for sharing another look, Risto. I know Ansible, Salt and other
tools of this category, and I did not thought about  this possibility yet,
maybe it would be worth of it.

About separate major software development project (related to SEC or maybe
other event correlation engines): I see it this way too. Such project would
need not only core development of "configuration framework", but also
living community, submitting their "correlation modules" in specified
format (e.g. some kind of SEC configuration "templates"). Ideologically
something, as you have here: https://github.com/simple-evcorr/rulesets/,
plus some additional functionalities around:

   - interface for submitting correlation modules (for review, because
   submitted modules should be verified before publication on framework site)

   - catalogue of submitted (and verified) correlation modules, e.g. as
   table or set of tables (structured by some categories), containing
   information:

   - description WHAT module does (user perspective)

  - (HOW is that functionality implemented is not needed here,
  reviewer(s) of the module could see it in bundled technical
documentation,
  source code and comments, but I see these implementation details as not
  needed for publishing in such catalogue... maybe there may be also more
  alternative implementations of the same correlation algorithms, having
  their specifics)

  - high-level picture of correlation algorithm (without low-level
  concepts, e.g. SEC-related)

  - list of configurable parameters with their types (constraints)

  - event source preconditions (format, etc.)

 Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] RegExp modifiers

[Richard Ostrochovský]

Hello colleagues,

I was searching for the answer here:
https://simple-evcorr.github.io/man.html
https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/
and haven't found the answer, so I'am putting new question here:

Does SEC in pattern= parameters support RegExp modifiers (
https://perldoc.perl.org/perlre.html#Modifiers) somehow?

E.g. modifiers /x or /xx allow writing more readable expressions by
ignoring unescaped whitespaces (implies possible multi-line regular
expressions). It could be practical in case of more complex expressions, to
let them being typed more legibly. Some simpler example:

pattern=/\
^\s*([A-Z]\s+)?\
(?\
   (\
  ([\[\d\-\.\:\s\]]*[\d\]]) |\
  (\
 (Mon|Tue|Wed|Thu|Fri|Sat|Sun) \s+\
 (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \s+ \d+ \s+
\d\d:\d\d:\d\d \s+ ([A-Z]+\s+)?\d\d\d\d\
  )\
   )\
) (?.*)/x

Thank you.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] RegExp modifiers

[Richard Ostrochovský]

Thank you for comprehensive answer, Risto. Maybe, hyperlink to it could be
added to that FAQ item.

(I found searching in this forum bit harder, due to traditional e-mail
from, and maybe we could categorize threads in this forum into topics on ,
I am inspired with this by
https://www.perlmonks.org/?node=Categorized%20Questions%20and%20Answers.
Maybe I would help with something like this, when I will have less busy
time.)

po 9. 12. 2019 o 16:43 Risto Vaarandi  napísal(a):

> hi Richard,
>
> Kontakt Richard Ostrochovský () kirjutas
> kuupäeval E, 9. detsember 2019 kell 01:57:
>
>> Hello colleagues,
>>
>> I was searching for the answer here:
>> https://simple-evcorr.github.io/man.html
>> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/
>> and haven't found the answer, so I'am putting new question here:
>>
>> Does SEC in pattern= parameters support RegExp modifiers (
>> https://perldoc.perl.org/perlre.html#Modifiers) somehow?
>>
>
> If you enclose a regular expression within /.../, SEC does not treat
> slashes as separators but rather as parts of regular expression, therefore
> you can't provide modifiers in the end of regular expression after /.
> However, Perl regular expressions allow for modifiers to be provided with
> (?) construct. For example, the following pattern matches the
> string "test" in case insensitive way:
> pattern=(?i)test
> In addition, you can use such modifiers anywhere in regular expression
> which makes them more flexible than modifiers after /. For example, the
> following pattern matches strings "test" and "tesT":
> pattern=tes(?i)t
>
> In SEC FAQ, there is also a short discussion on this topic:
> https://simple-evcorr.github.io/FAQ.html#13)
>
>
>> E.g. modifiers /x or /xx allow writing more readable expressions by
>> ignoring unescaped whitespaces (implies possible multi-line regular
>> expressions). It could be practical in case of more complex expressions, to
>> let them being typed more legibly. Some simpler example:
>>
>> pattern=/\
>> ^\s*([A-Z]\s+)?\
>> (?\
>>(\
>>   ([\[\d\-\.\:\s\]]*[\d\]]) |\
>>   (\
>>  (Mon|Tue|Wed|Thu|Fri|Sat|Sun) \s+\
>>  (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \s+ \d+ \s+
>> \d\d:\d\d:\d\d \s+ ([A-Z]+\s+)?\d\d\d\d\
>>   )\
>>)\
>> ) (?.*)/x
>>
>>
> It is a tricky question, since SEC configuration file format allows to
> natively provide regular expressions in multiple lines without the use of
> (?x) modifier. If any line in rule definition ends with backslash, the
> following line is appended to the current line and backslash is removed
> during configuration file parsing. For example, the following two pattern
> definitions are equivalent:
>
> pattern=test: \
> (\S+) \
> (\S+)$
>
> pattern=test: (\S+) (\S+)$
>
> However, it is important to remember that SEC converts multi-line rule
> fields into single-line format before any other processing, and that
> includes compiling regular expressions. In other words, if you consider the
> first multi-line regular expression pattern definition above, SEC actually
> sees it as "test: (\S+) (\S+)$" when it compiles this expression. This
> introduces the following caveat -- when using (?x) modifier for introducing
> a comment into multi-line regular expression, the expression is converted
> into single line format before expression is compiled and (?x) has any
> effect, and therefore the comment will unexpectedly run until the end of
> regular expression. Consider the following example:
>
> pattern=(?x)test:\
> # this is a comment \
> (\S+)$
>
> Internally, this definition is first converted to single line format:
> pattern=(?x)test:# this is a comment (\S+)$
> However, this means that without the comment the expression looks like
> (?x)test:
> which is not what we want.
>
> To address this issue, you can limit the scope of comments with (?#...)
> constructs that don't require (?x). For example:
>
> pattern=test:\
> (?# this is a comment )\
> (\S+)$
>
> During configuration file parsing this expression is converted into
> "test:(?# this is a comment )(\S+)$", and after dropping the comment it
> becomes "test:(\S+)$" as we expect.
>
> Hope this helps,
> risto
>
>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] using original message in TValue rule action

[Richard Ostrochovský]

Hello,

I was find out the answer in manual and also archive of this forum, but
without success, and the question seems very basic to me, so I assume 2 (3)
possible alternative answers:

- it is so easy, that I will bang my head
- it is not possible at all (in current version of SEC)
- (RegExp .* is equally efficient as TValue)

Assuming, that using TValue instead of RegExp or any other rule type in
cases, where I don't need filtering of or extraction from log messages, is
most computing power efficient, I am trying to find out a straightforward
way, how to use the original event text in event action of TValue rule.

$0 seems not to be working for TValue (I understand, that it is
RegExp-specific) in rule like this:

type=Single
ptype=TValue
pattern=TRUE
context=SVC_:tmp::home:user:somelog.log#MULTI-LINE &&
SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE
desc= SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE lines filter
action=add ( SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE) $0

$0 literally is added to context in this case.

("#" is meant to be part of the context name, not any kind of comment.)

Does somebody have any advice, how to use original event text in
TValue-type rule, without "compromising" the performance? (Assuming, that
the easiest solution, the replacement of TValue with RegExp nad TRUE with
.* would do the job, but won't be as fast as TValue.) Maybe new predefined
variable could be available (e.g. %e as event) independently on rule type.

Thank you in advance.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] SEC + AI (machine learning)

[Richard Ostrochovský]

Hello,

this is free continuation of
https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/. That
post was about possibilities of user-friendly configurations of event
correlations outside of SEC (without knowing SEC syntax and low-level
principles), and generation of SEC rules from that externalized
configurations. But still manual process.

Next step would be integrating AI (machine learning) with SEC somehow, so
that user won't need to configure correlations statically, but they would
configure and self-optimize automatically. (There still could be some input
needed from the user, but system would be also able to react on changing
log traffic, and self-evolve.)

Something like ELK+AI has usable in the log monitoring area.

Maybe some integration with MXNet?
http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html

Does anybody have any experience in this area, to explain some more or less
theoretical or practical setup of AI-generated SEC rules? (I am pretty
sure, that this is out of scope of SEC itself, and SEC would'nt know, that
AI is dynamically generating its rules on the background and probably
nobody has working solution, but maybe we could invent something together.)

Thanks for any ideas.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SEC + AI (machine learning)

[Richard Ostrochovský]

Thank you, Risto, such overview can serve as starting point, when designing
something yet more useful, than static configurations of correlations.

I am still not sure, what the optimal goal should be, I need to analyze and
study more about these topics.

I believe, that I will follow up on information you provided, in the
future, and further develop this thread.

št 23. 1. 2020 o 18:12 Risto Vaarandi  napísal(a):

> hi Richard,
>
> Next step would be integrating AI (machine learning) with SEC somehow, so
>> that user won't need to configure correlations statically, but they would
>> configure and self-optimize automatically. (There still could be some input
>> needed from the user, but system would be also able to react on changing
>> log traffic, and self-evolve.)
>>
>> Something like ELK+AI has usable in the log monitoring area.
>>
>> Maybe some integration with MXNet?
>>
>> http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html
>>
>> Does anybody have any experience in this area, to explain some more or
>> less theoretical or practical setup of AI-generated SEC rules? (I am pretty
>> sure, that this is out of scope of SEC itself, and SEC would'nt know, that
>> AI is dynamically generating its rules on the background and probably
>> nobody has working solution, but maybe we could invent something together.)
>>
>>
> Machine learning is a very wide area with a large number of different
> methods and algorithms around. These methods and algorithms are usually
> divided into two large classes:
> *) supervised algorithms which assume that you provide labeled data for
> learning (for example, a log file where some messages are labeled as
> "normal" and some messages as "system_fault"), so that the algorithm can
> learn from labeled examples how to distinguish normal messages from errors
> (note that in this simplified example, only two labels were used, but in
> more complex cases you could have more labels in play)
> *) unsupervised algorithms which are able to distinguish anomalous or
> abnormal messages without any previous training with labeled data
> So my first question is -- what is your actual setup and do you have the
> opportunity of using training data for supervised methods, or are
> unsupervised methods a better choice? After answering this question, you
> can start studying most promising methods more closely.
>
> Secondly, what is your actual goal? Do you want to:
> 1) detect an individual anomalous message or a time frame containing
> anomalous messages from event logs,
> 2) produce a warning if the number of messages from specific class (e.g.
> login failures) per N minutes increases suddenly to an unexpectedly large
> value,
> 3) use some tool for (semi)automated mining of new SEC rules,
> 4) something else?
>
> For achieving first goal, there is no silver bullet, but perhaps I can
> provide few pointers to some relevant research papers (note that there are
> many other papers in this area):
> https://ieeexplore.ieee.org/document/4781208
> https://ieeexplore.ieee.org/document/7367332
> https://dl.acm.org/doi/10.1145/3133956.3134015
>
> For achieving the second goal, you could consider using time series
> analysis methods. You could begin with a very simple moving average based
> method like the one described here:
>
> https://machinelearnings.co/data-science-tricks-simple-anomaly-detection-for-metrics-with-a-weekly-pattern-2e236970d77
> or you could employ more complex forecasting methods (before starting, it
> is probably a good idea to read this book on forecasting:
> https://otexts.com/fpp2/)
>
> If you want to mine new rules or knowledge for SEC (or for other tools)
> from event logs, I have actually done some previous research in this
> domain. Perhaps I can point you to a log mining utility called LogCluster (
> https://ristov.github.io/logcluster/) which allows for mining line
> patterns and outliers from textual events logs. Also, couple of years ago,
> an experimental system was created which was using LogCluster in a fully
> automated way for creating SEC Suppress rules, where these rules were
> essentially matching normal (expected) messages. Any message not matching
> these rules was considered an anomaly and was logged separately for manual
> review. Here is the paper that provides an overview of this system:
> https://ristov.github.io/publications/noms18-log-anomaly-web.pdf
>
> Hopefully these pointers will offer you some guidance what your precise
> research question could be, and what is the most promising avenue for
> continuing. My apologies if my answer was raising new questions, but
> machine learning is a very wide area with large number of methods for many
> different goals.
>
> kind regards,
> risto
>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] SEC rules performance monitoring and tuning

[Richard Ostrochovský]

Greetings, friends,

there is an idea to monitor high-volume event flows (per rules or log
files, e.g. in bytes per time unit) from dump files from *periodic dumping*.

The question is, if this is recommended for *production use* - answer
depends at least on if dump creation somehow slows down or stops SEC
processing, or it has not significant overhead (also for thousands of
rules, complex configurations).

Are there some best practises for (or experiences with) performance
self-monitoring and tuning of SEC?

Thank you.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SEC rules performance monitoring and tuning

[Richard Ostrochovský]

Hi Risto,

thank you for positive answer - dumping period in minutes in enough, not
needed many times per second. And also for useful tips - I already noticed
JSON option, just considering, that default is maybe more universally
usable, because extra modules (JSON.pm) may not be installed or installable
on any systems (not only due technical reasons, but also security policies
- more code auditing needed). Maybe the question also could be, how
frequent are changes of dump file formats during SEC development, which
could break monitoring implemented for particular version of SEC.

Richard

ut 28. 1. 2020 o 12:40 Risto Vaarandi  napísal(a):

> hi Richard,
>
> as I understand from your post, you would like to create SEC dump files
> periodically, in order to monitor performance of SEC based on these dump
> files. Let me first provide some comments on performance related question.
> Essentially, creation of dump file involves a pass over all major internal
> data structures, so that summaries about internal data structures can be
> written into dump file. In fact, SEC traverses all internal data structures
> for housekeeping purposes anyway once a second (you can change this
> interval with --cleantime command line option). Therefore, if you re-create
> the dump file after reasonable time intervals (say, after couple of
> minutes), it wouldn't noticeably increase CPU consumption (things would
> become different if dumps would be generated many times a second).
>
> For production use, I would provide couple of recommendations. Firstly,
> you could consider generating dump files in JSON format which might make
> their parsing easier. That feature was requested couple of years ago
> specifically for monitoring purposes (
> https://sourceforge.net/p/simple-evcorr/mailman/message/36334142/), and
> you can activate JSON format for dump file with --dumpfjson command line
> option. You could also consider using --dumpfts option which adds numeric
> timestamp suffix to dump file names (e.g., /tmp/sec.dump.1580210466). SEC
> does not overwrite the dump file if it already exists, so having timestamps
> in file names avoids the need for dump file rotation (you would still need
> to delete old dump files from time to time, though).
>
> Hopefully this information is helpful.
>
> kind regards,
> risto
>
> Kontakt Richard Ostrochovský () kirjutas
> kuupäeval E, 27. jaanuar 2020 kell 22:04:
>
>> Greetings, friends,
>>
>> there is an idea to monitor high-volume event flows (per rules or log
>> files, e.g. in bytes per time unit) from dump files from *periodic
>> dumping*.
>>
>> The question is, if this is recommended for *production use* - answer
>> depends at least on if dump creation somehow slows down or stops SEC
>> processing, or it has not significant overhead (also for thousands of
>> rules, complex configurations).
>>
>> Are there some best practises for (or experiences with) performance
>> self-monitoring and tuning of SEC?
>>
>> Thank you.
>>
>> Richard
>> ___
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] how not to keep monitored files permanently open (not only) on NFS

[Richard Ostrochovský]

Hi Risto and friends,

I am unsure about one conceptual question about how SEC keeps open
monitored files.

Using SEC as systemd service, when files stored in NFS (opened via
addinput) being watched by SEC are moved elsewhere, and then their removal
is tried, NFS persistently keeps .nfsNUMBER files in existence, unable to
remove whole folder of them. This functionality of NFS is described e.g.
here:
https://www.ibm.com/su
pport/pages/what-are-nfs-files-accumulate-and-why-cant-they-be-deleted-even-after-stopping-cognos-8



It seems, that SEC running as service (not re-opening and closing each
monitored file in each "run") is keeping watched files permanently open,
and it is not desirable in such setup.

But: when I look into the dump, some files have "status: Open" and some
"status: Closed", so maybe this my assumption about log files permanently
opened by SEC is not correct - I am confused.

How it is in reality? Has somebody experience with described behaviour
(SEC+NFS, but it could arise also in other setups) and how to overcome it?

Thank you.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] how not to keep monitored files permanently open (not only) on NFS

[Richard Ostrochovský]

hi Risto,

thank you for your helpful explanation about inner functionality of SEC.
Closed files in my case were not existing, that was the reason.

Richard

ut 4. 2. 2020 o 23:09 Risto Vaarandi  napísal(a):

> hi Richard,
>
> I have never used SEC for monitoring files on NFS file systems, but I can
> provide few short comments on how input files are handled. After SEC has
> successfully opened an input file, it will be kept open permanently. When
> input file is removed or renamed, input file is still kept open (at that
> point the file handle is pointing to disk blocks that no longer have a
> name). When a new file appears on disk with the same name as original input
> file, SEC will close the file handle pointing to nameless disk blocks and
> will open the new input file, starting to process it from the beginning.
> However, this operation is atomic and the input file will never show as
> "Closed" in the dump file. Status "Closed" in dump file usually indicates
> that SEC was not able to open the input file when it was started or
> restarted with a signal (a common situation when file does not exist or
> there is no permission to open the file), but it can also indicate that SEC
> has closed the file due to IO error when reading from file (that should
> normally not happen and usually means a serious disk issue).
>
> In order to find out why input file is in closed state, I would recommend
> to check the log of SEC (you can set it up with --log command line option).
> For example, if the input file did not exist when SEC was started or
> restarted with a signal, there should be the following lines in the log:
>
> Opening input file test.log
> Input file test.log does not exist!
>
> Also, if file is closed due to IO error, there is a relevant message in
> the log, for example: Input file test.log IO error (Input/output error),
> closing the file.
>
> Hope these comments are helpful,
> risto
>
>
> Hi Risto and friends,
>>
>> I am unsure about one conceptual question about how SEC keeps open
>> monitored files.
>>
>> Using SEC as systemd service, when files stored in NFS (opened via
>> addinput) being watched by SEC are moved elsewhere, and then their removal
>> is tried, NFS persistently keeps .nfsNUMBER files in existence, unable to
>> remove whole folder of them. This functionality of NFS is described e.g.
>> here:
>> https://www.ibm.com/su
>> pport/pages/what-are-nfs-files-accumulate-and-why-cant-they-be-deleted-even-after-stopping-cognos-8
>> 
>>
>>
>> It seems, that SEC running as service (not re-opening and closing each
>> monitored file in each "run") is keeping watched files permanently open,
>> and it is not desirable in such setup.
>>
>> But: when I look into the dump, some files have "status: Open" and some
>> "status: Closed", so maybe this my assumption about log files permanently
>> opened by SEC is not correct - I am confused.
>>
>> How it is in reality? Has somebody experience with described behaviour
>> (SEC+NFS, but it could arise also in other setups) and how to overcome it?
>>
>> Thank you.
>>
>> Richard
>> ___
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] how not to keep monitored files permanently open (not only) on NFS

[Richard Ostrochovský]

In this context I am also curious, what would be the effect of using
--check-timeout / --poll-timeout, if the log file will be closed or remain
open during timeout... I am trying to find a way, how to use SEC in "close
after read" mode - used to use this mode in previous log event correlation
solution, because keeping log files "always open" causes described problem
with their deletion (by external archivation script) on NFS...

>From SEC manual: "Each input file is tracked both by its name and i-node,
and input file rotations are handled seamlessly. If the input file is
recreated or truncated, SEC will reopen it and process its content from the
beginning. If the input file is removed (i.e., there is just an i-node left
without a name), SEC will keep the i-node open and wait for the input file
recreation."

Maybe it would be sufficient having an option to (immediately?) close
(re)moved file, instead of keeping original i-node open until its
recreation in its original location.

pi 7. 2. 2020 o 12:38 Richard Ostrochovský 
napísal(a):

> hi Risto,
>
> thank you for your helpful explanation about inner functionality of SEC.
> Closed files in my case were not existing, that was the reason.
>
> Richard
>
> ut 4. 2. 2020 o 23:09 Risto Vaarandi 
> napísal(a):
>
>> hi Richard,
>>
>> I have never used SEC for monitoring files on NFS file systems, but I can
>> provide few short comments on how input files are handled. After SEC has
>> successfully opened an input file, it will be kept open permanently. When
>> input file is removed or renamed, input file is still kept open (at that
>> point the file handle is pointing to disk blocks that no longer have a
>> name). When a new file appears on disk with the same name as original input
>> file, SEC will close the file handle pointing to nameless disk blocks and
>> will open the new input file, starting to process it from the beginning.
>> However, this operation is atomic and the input file will never show as
>> "Closed" in the dump file. Status "Closed" in dump file usually indicates
>> that SEC was not able to open the input file when it was started or
>> restarted with a signal (a common situation when file does not exist or
>> there is no permission to open the file), but it can also indicate that SEC
>> has closed the file due to IO error when reading from file (that should
>> normally not happen and usually means a serious disk issue).
>>
>> In order to find out why input file is in closed state, I would recommend
>> to check the log of SEC (you can set it up with --log command line option).
>> For example, if the input file did not exist when SEC was started or
>> restarted with a signal, there should be the following lines in the log:
>>
>> Opening input file test.log
>> Input file test.log does not exist!
>>
>> Also, if file is closed due to IO error, there is a relevant message in
>> the log, for example: Input file test.log IO error (Input/output error),
>> closing the file.
>>
>> Hope these comments are helpful,
>> risto
>>
>>
>> Hi Risto and friends,
>>>
>>> I am unsure about one conceptual question about how SEC keeps open
>>> monitored files.
>>>
>>> Using SEC as systemd service, when files stored in NFS (opened via
>>> addinput) being watched by SEC are moved elsewhere, and then their removal
>>> is tried, NFS persistently keeps .nfsNUMBER files in existence, unable to
>>> remove whole folder of them. This functionality of NFS is described e.g.
>>> here:
>>> https://www.ibm.com/su
>>> pport/pages/what-are-nfs-files-accumulate-and-why-cant-they-be-deleted-even-after-stopping-cognos-8
>>> <https://www.ibm.com/support/pages/what-are-nfs-files-accumulate-and-why-cant-they-be-deleted-even-after-stopping-cognos-8>
>>>
>>>
>>> It seems, that SEC running as service (not re-opening and closing each
>>> monitored file in each "run") is keeping watched files permanently open,
>>> and it is not desirable in such setup.
>>>
>>> But: when I look into the dump, some files have "status: Open" and some
>>> "status: Closed", so maybe this my assumption about log files permanently
>>> opened by SEC is not correct - I am confused.
>>>
>>> How it is in reality? Has somebody experience with described behaviour
>>> (SEC+NFS, but it could arise also in other setups) and how to overcome it?
>>>
>>> Thank you.
>>>
>>> Richard
>>> ___
>>> Simple-evcorr-users mailing list
>>> Simple-evcorr-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] action-list checking if log file is already open by SEC

[Richard Ostrochovský]

Hello Risto and friends,

having mechanism for dynamic opening (addinput) and closing (dropinput)
files, I would like to be able to check, if the file is already opened,
before trying to open it again, to avoid it. That way I would like to
eliminate this error message from SEC log (present also in debug level 3):

Dynamic input file '/path/to/file.log' already exists in the list of
inputs, can't add

This information is present in sec.dump, but maybe there exists more
instant and straightforward way how to achieve it (without parsing
intermediary files).

Thank you.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] SEC CPU utilization

[Richard Ostrochovský]

Hello friends,

I have SEC monitoring over 50 log files with various correlations, and it
is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole
system affected, as SEC is single-CPU application).

This could mean, that SEC does not prosecute processing of all rules, and I
am curious, what are possible effects, if this means increasing delays
(first in, processing, first out), or skipping some lines from input files,
or anything other (?).

And how to troubleshoot, finding bottlenecks. I can see quantities of log
messages per contexts or log files in sec.dump, this is some indicator. Are
there also other indicators? Is it possible, somehow, see also processing
times of patterns (per rules)?

Thank you in advance.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SEC CPU utilization

[Richard Ostrochovský]

 practise rules
templates", "rulesets / templates catalogue", possibly with possibility to
choose correlation doing some specific task, fill in parameters, and
generate SEC rules automatically... I see wide potential here, to enable
SEC users not re-inventing wheels, but build on verified setups, as
discussed here:
https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/. I may
suggest my employer to join such initiative with our designs. Maybe we
could make a teleconference call someday, as wanna-be participants of such
community, and brainstorm about possibilities, as publishing of rulesets on
GitHub or SourceForge is great, but maybe we could do yet more, and build
ineractive catalogue beyond file sharing.

Richard

st 25. 3. 2020 o 17:52 Risto Vaarandi  napísal(a):

> hi Richard,
>
> if CPU utilization has reached 100%, no rules or log file events would be
> skipped, but SEC would simply not be able to process events at their
> arrival rate and fall behind of events. If your events include timestamps,
> you would probably see events with past timestamps in dump file (among
> other data, SEC dump file reports the last processed event for each input
> file). As for debugging the reasons of high CPU utilization, I would
> recommend to have a look into rule match statistics, and make sure that
> rules with most matches appear in the top of their rule files (if
> possible). However, current versions of SEC do not report the CPU time
> spent on matching each pattern against events.
>
> Just out of curiosity -- how many rules are you currently having in your
> rule base, and are all these rules connected to each other? How many events
> are you currently receiving per second? Also, are all 50 input files
> containing the same event types (e.g., httpd events) that need to be
> processed by all rules? If this is not the case, and each input file
> contains different events which are processed by different rules, I would
> strongly recommend to consider a hierarchical setup for your rule files.
> The principles of hierarchical setup have been described in SEC official
> documentation, for example: http://simple-evcorr.github.io/man.html#lbBE.
> Also, there is a recent paper which provides a relevant example:
> https://ristov.github.io/publications/cogsima15-sec-web.pdf. In addition,
> you could also consider running several instances of SEC for your input
> files. For example, if some input files contain messages from a specific
> application which are processed by few specific rule files, a separate SEC
> process could be started for handling these messages with given rule files.
> In that way, it might be possible to divide the rule files and input files
> into several independent groups, and having a separate SEC process for each
> group allows to balance the load across several CPU's.
>
> hope this helps,
> risto
>
> Kontakt Richard Ostrochovský () kirjutas
> kuupäeval K, 25. märts 2020 kell 17:07:
>
>> Hello friends,
>>
>> I have SEC monitoring over 50 log files with various correlations, and it
>> is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole
>> system affected, as SEC is single-CPU application).
>>
>> This could mean, that SEC does not prosecute processing of all rules, and
>> I am curious, what are possible effects, if this means increasing delays
>> (first in, processing, first out), or skipping some lines from input files,
>> or anything other (?).
>>
>> And how to troubleshoot, finding bottlenecks. I can see quantities of log
>> messages per contexts or log files in sec.dump, this is some indicator. Are
>> there also other indicators? Is it possible, somehow, see also processing
>> times of patterns (per rules)?
>>
>> Thank you in advance.
>>
>> Richard
>> ___
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] action-list checking if log file is already open by SEC

[Richard Ostrochovský]

Hi Risto,

thank you for the solution. There is also concern about potential
performance impact in case of e.g. thousands of files being added by
addinput with creating extra context for each of them.

Another way could be e.g. maintaining (associative) array in action list,
with keys of paths to addinput files, reduced by dropinput, similarly as
you suggest with context creation.

And maybe the most efficient, having built-in way, how to access this list
of open files, without custom implementation of it.

How do you see these potential options in comparison by performance?

Richard

št 20. 2. 2020 o 21:23 Risto Vaarandi  napísal(a):

> hi Richard,
>
> I think this scenario is best addressed by creating a relevant SEC context
> when 'addinput' action is called. In fact, handling such scenarios is one
> of the purposes of contexts, and here is an example rule which illustrates
> this idea:
>
> type=single
> ptype=regexp
> pattern=start monitoring (\S+)
> context=!FILE_$1_MONITORED
> desc=add $1 to list of inputs
> action=addinput $1; create FILE_$1_MONITORED
>
> Whenever "start monitoring " event appears, the rule will match
> only if context FILE__MONITORED does not exist. If rule matches,
> it executes 'addinput' action for the given file and creates the context,
> in order to manifest the fact that 'addinput' has already been executed for
> the given file. Also, as you can see from the above rule, the presence of
> the context for a file will prevent the execution of 'addinput' again for
> this file. In order to keep contexts in sync with files that are monitored,
> the context for a file should be deleted when 'dropinput' action is
> executed for it.
>
> Note that when HUP signal is received, SEC will stop monitoring input
> files set up with 'addinput' action. However, on receiving HUP signal SEC
> will also drop all its contexts, so there is no need to take any extra
> steps in that case.
>
> Hope this helps,
> risto
>
> Kontakt Richard Ostrochovský () kirjutas
> kuupäeval N, 20. veebruar 2020 kell 21:43:
>
>> Hello Risto and friends,
>>
>> having mechanism for dynamic opening (addinput) and closing (dropinput)
>> files, I would like to be able to check, if the file is already opened,
>> before trying to open it again, to avoid it. That way I would like to
>> eliminate this error message from SEC log (present also in debug level 3):
>>
>> Dynamic input file '/path/to/file.log' already exists in the list of
>> inputs, can't add
>>
>> This information is present in sec.dump, but maybe there exists more
>> instant and straightforward way how to achieve it (without parsing
>> intermediary files).
>>
>> Thank you.
>>
>> Richard
>> ___
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] log files existence and accessibility

[Richard Ostrochovský]

Hello friends,

I am thinking about how to monitor not only events from log files, but also
those files existence and accessibility (for user running SEC) - in cases,
where this is considered to be a problem.

As I saw in the past, these were logged into SEC log file, but higher debug
level was required, so it is not suitable for production.

There are handful of other options how to monitor it "externally", e.g. via
some script, or other monitoring agent, but this is not ultimate solution,
as e.g. SELinux may be configured, allowing external script or agent
(running under the same user as SEC) to see nad open file for reading, but
not SEC (theoretical caveat of such solution).

So, has somebody some "best practise" for reliable and production-ready
way, how to "self-monitor" log files being accessed by SEC, if:

   - they exist
   - they are accessible by SEC

?

Thanks in advance.

Richard
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] log files existence and accessibility

[Richard Ostrochovský]

Thank you, Risto, we could try to integrate lcall with calendar into our
setup. SELinux was just theoretical example, we have no problem with that
now, I was just seeking for bullet-proof solution.

Richard

st 8. 4. 2020 o 12:05 Risto Vaarandi  napísal(a):

> hi Richard,
>
> if you want to find input files which can not be opened because of
> permission issues, and want to conduct all checks specifically from SEC
> process without forking anything, I would recommend to set up an 'lcall'
> action that runs all checks from a Perl function. Since this function is
> executed within SEC process, it will have the same permissions for
> accessing the file as SEC itself. If you return one or more values from the
> function, 'lcall' will set the output variable by joining return values
> into single string, so that newline is acting as a separator between return
> values. Also, if no values are returned from the function, output variable
> will be set to perl 'undef'. After collecting the output from the function
> in this way, you can provide the output variable to 'event' action that
> creates synthetic event from its content. Note that if multi-line string is
> submitted to 'event' action, several synthetic events are generated (one
> for each line). And now that you have synthetic events about files that are
> not accessible, you can process them further in any way you like (e.g., an
> e-mail could be sent to admin about input file with wrong permissions).
>
> Here is an example rule that executes the input file check once a minute
> from Calendar rule:
>
> type=Calendar
> time=* * * * *
> desc=check files
> action=lcall %events -> ( sub { my(@files, @list, $file); \
>@files = glob("*.log"); foreach $file (@files) { \
>  if (!open(FILE, $file)) { push @list, "$file is not accessible";
> } else { close(FILE); } \
>} return @list; } ); \
>if %events ( event %events )
>
> The action list of the Calendar rule consists of two actions -- the
> 'lcall' action and 'if' action, with 'if' action executing 'event' action
> conditionally. The 'lcall' action tries to open given input files and
> verify that each open succeeds. For all files that it could not open,
> 'lcall' returns a list of messages " is not accessible", and the list
> is assigned to %events variable as a multi-line string (each line in the
> multi-line string represents one message for some file). If the list is
> empty, %events variable is set to perl 'undef' value. After 'lcall' action,
> the 'if' action is executed which verifies that %events is true in perl
> boolean context (if %events has been previously set to 'undef', that check
> will fail). If the check succeeds (in other words, we have some messages to
> report), the 'event' action will generate synthetic events from messages.
>
> That is one way how to check input files without forking anything from
> SEC. However, if the root cause of the problem is related to SELinux, it is
> probably much better to adjust SELinux configuration (perhaps by changing
> file contexts), so that the problem would not appear.
>
> kind regards,
> risto
>
>
> Kontakt Richard Ostrochovský () kirjutas
> kuupäeval E, 6. aprill 2020 kell 17:21:
>
>> Hello friends,
>>
>> I am thinking about how to monitor not only events from log files, but
>> also those files existence and accessibility (for user running SEC) - in
>> cases, where this is considered to be a problem.
>>
>> As I saw in the past, these were logged into SEC log file, but higher
>> debug level was required, so it is not suitable for production.
>>
>> There are handful of other options how to monitor it "externally", e.g.
>> via some script, or other monitoring agent, but this is not ultimate
>> solution, as e.g. SELinux may be configured, allowing external script or
>> agent (running under the same user as SEC) to see nad open file for
>> reading, but not SEC (theoretical caveat of such solution).
>>
>> So, has somebody some "best practise" for reliable and production-ready
>> way, how to "self-monitor" log files being accessed by SEC, if:
>>
>>- they exist
>>- they are accessible by SEC
>>
>> ?
>>
>> Thanks in advance.
>>
>> Richard
>> ___
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
___
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users