Hello Risto Though this query is not related. Just got curious about using the variable in a pattern directly instead of matching later.
Log: "ASA-6-302016: Teardown UDP connection 806353 for outside: 187.189.195.208 <http://187.189.195.208:8443/>/24057 to identity: 172.18.124.136/161 duration 0:02:01 bytes 313" As per the rule, IPv4 is extracted to a variable $1 and match against %ioc hash table. Instead, is it possible to match the IOC IP's with any part of the log(either in outside:([\d.]+) or in identity:([\d.]+)). Regards, san On Tue, Sep 3, 2019 at 12:47 PM Santhosh Kumar <santhoshkmrre...@gmail.com> wrote: > Hello risto > > I ran the tests with real logs. Suggested method works exactly as expected. > > This resolves many of my other queries. Thank you for prompt response. > > Regards, > Santhosh > > > On Fri, Aug 30, 2019, 20:59 Risto Vaarandi <risto.vaara...@gmail.com> > wrote: > >> hi Santhosh, >> >> since your task involves not only matching IP addresses against a >> blacklist but also includes reporting IoC information for a bad IP address, >> I would recommend loading IoC data from file into a Perl hash which allows >> for quick lookups. The example ruleset below uses a global hash %ioc which >> is a global variable and can thus be accessed from all rules: >> >> type=Single >> ptype=RegExp >> pattern=^(SEC_STARTUP|SEC_RESTART|SEC_SOFTRESTART)$ >> context=SEC_INTERNAL_EVENT >> desc=load IoC information on SEC startup and HUP or ABRT signals >> action=lcall %count -> ( sub { %ioc = (); \ >> if (!open(FILE, "/home/risto/IOC_data_proposal.txt")) { return -1; >> } \ >> while (<FILE>) { if (/^\s*(([\d.]+):\d+\s*-.*)/) { \ >> $ioc{$2} = $1; } \ >> } close(FILE); return scalar keys %ioc; } ); \ >> logonly Loaded %count entries from IoC data file >> >> type=Single >> ptype=PerlFunc >> pattern=sub { if ($_[0] !~ /ASA-\S+: Teardown \S+ connection \d+ for >> outside: ([\d.]+)/) { return 0; } \ >> if (!exists($ioc{$1})) { return 0; } return ($1, $ioc{$1}); } >> desc=Connection to IP address $1 with IoC information $2 >> action=pipe 'Log matched IoC:%{.nl}IoC: $2%{.nl}Log: $0' /bin/mail >> some...@example.com >> >> The first rule loads IoC information from file into %ioc hash table >> whenever SEC is started or HUP or ABRT signal is received by SEC. IP >> addresses serve as keys of the hash table, while each value is an entire >> line from the IoC file. For example, if the file contains the following two >> lines >> >> 187.163.222.244:465 - emotet >> 187.189.195.208:8443 - emotet >> >> the %ioc hash table will contain the following mappings (keys and values >> are separated by ->): >> >> 187.163.222.244 -> 187.163.222.244:465 - emotet >> 187.189.195.208 -> 187.189.195.208:8443 - emotet >> >> Currently, the first rule assumes that IoC file is in the same format as >> you described in your e-mail, and the rule uses regular expression >> ^\s*(([\d.]+):\d+\s*-.*) for parsing the file and extracting relevant >> information. Should the format of the file change, this regular expression >> needs to be adjusted accordingly. Also, the rule finds the number of >> entries loaded from IoC file, stores it in %count action list variable and >> logs a debug message with this value into SEC log file. If the rule was >> unable to open the file, the value -1 is logged which is useful for >> troubleshooting purposes. >> >> The second rule uses PerlFunc pattern for matching incoming ASA firewall >> events and first verifies that incoming event matches the regular expression >> ASA-\S+: Teardown \S+ connection \d+ for outside: ([\d.]+) >> If there is a match, IP address of remote host is extracted and assigned >> to $1 variable, and %ioc hash table is looked up for IoC information for >> that IP address. If lookup is successful, PerlFunc pattern returns a list >> with two elements (IP address, IoC info) which are mapped to match >> variables $1 and $2 by SEC ($0 variable will hold the entire matching event >> log line). The match variables are then used by 'pipe' action for sending >> an e-mail to relevant mailbox. >> >> Hope this helps, >> risto >> >> Kontakt Santhosh Kumar (<santhoshkmrre...@gmail.com>) kirjutas kuupäeval >> R, 30. august 2019 kell 06:37: >> >>> Hello >>> >>> Thanks for the brief..!! >>> >>> In my case, udgram creates a event loop however tcpsock functionality >>> achived the goal. >>> >>> Regarding second scenario, the goal is to match the IP address and >>> notify. >>> >>> "ASA-6-302016: Teardown UDP connection 806353 for outside: 187.189.195.208 >>> <http://187.189.195.208:8443/>/24057 to >>> identity: 172.18.124.136/161 duration 0:02:01 bytes 313" >>> >>> List of IOC's stored in a file(IOC_data_proposal.txt) as below. the >>> rule suppose to match the IP address from the file( >>> IOC_data_proposal.txt) though the file contains other informations like >>> port number and name of the malware, etc. which is not part of the log. >>> >>> The reason for not removing extra data(port, ioc name) from the file( >>> IOC_data_proposal.txt) is that its required for a notification. >>> >>> ************************ >>> >>> IOC_data_proposal.txt >>> >>> 187.163.222.244:465 - emotet >>> >>> 187.189.195.208:8443 - emotet >>> >>> 188.166.253.46:8080 - emotet >>> >>> 189.209.217.49:80 <http://189.209.217.49/> - heartbleed >>> >>> ************************ >>> >>> >>> Expected mail Output: >>> >>> >>> Log Matched IOC, >>> >>> IOC: 187.189.195.208:8443 - emotet >>> >>> Log: "ASA-6-302016: Teardown UDP connection 806353 for outside: >>> 187.189.195.208 <http://187.189.195.208:8443/>/24057 to >>> >>> identity: 172.18.124.136/161 duration 0:02:01 bytes 313" >>> >>> >>> regards, >>> Santhosh >>> >>> On Wed, Aug 28, 2019 at 4:18 AM Risto Vaarandi <risto.vaara...@gmail.com> >>> wrote: >>> >>>> hi Santhosh, >>>> >>>> Kontakt Santhosh Kumar (<santhoshkmrre...@gmail.com>) kirjutas >>>> kuupäeval T, 27. august 2019 kell 04:55: >>>> >>>>> Hello Risto >>>>> >>>>> >>>>> I’ve been running tests on SEC for a while and stuck with below >>>>> points. I’m not familiar with Perl though I tried to find a solution from >>>>> sec mail bucket but no luck, please suggest if this can be achieved with >>>>> high performance, >>>>> >>>>> >>>>> >>>>> 1. I could see a log drops when I tested with the event rate of >>>>> 15000 logs/sec. A simple SEC rule to receive and forward all the logs >>>>> to a >>>>> destination. The output shows relatively less number of logs. This also >>>>> increases the cpu usage from 0.3% to 45% >>>>> >>>>> ************************ >>>>> >>>>> Type=single >>>>> >>>>> Ptype=regexp >>>>> >>>>> Pattern=([.\d]+) >>>>> >>>>> Desc=$1 >>>>> >>>>> Action=pipe $0 nc syslog101 514 >>>>> >>>>> ************************ >>>>> >>>> >>>> The above rule is very inefficient, since 'pipe' action pipes the value >>>> of $0 to "nc syslog101 514" command and then closes the pipe, forcing the >>>> nc command to terminate. In other words, each time the above rule matches >>>> an event, new command is forked, and if you have 15000 events per second, >>>> the rule attempts to fork 15000 processes each second. This imposes >>>> considerable load on the system and if you send in events at a high rate, >>>> the rule might easily exhaust your system resources. >>>> >>>> Instead of forking nc on each matching event, I would recommend to >>>> utilize 'tcpsock' action which transmits events over a single TCP socket >>>> (since you haven't used -u flag with nc tool, I assume that your syslog >>>> server listens on port 514/tcp). For example, consider the following rule >>>> (the rule terminates each transmitted line with the newline): >>>> >>>> type=single >>>> ptype=regexp >>>> pattern=([.\d]+) >>>> desc=test event $1 >>>> action=tcpsock syslog101:514 $0%.nl >>>> >>>> If your syslog server speaks BSD syslog protocol ( >>>> https://tools.ietf.org/html/rfc3164), and incoming events are not in >>>> that format, you could use sec builtin action list variables for formatting >>>> the event and providing fields that syslog server expects (such as >>>> timestamp). For example, the following rule transmits each event line over >>>> TCP in BSD syslog format with priority 14 (facility of "user" and severity >>>> of "info"), with hostname "myhost", with program name "myprog", and using >>>> newline as a separator between messages: >>>> >>>> type=single >>>> ptype=regexp >>>> pattern=([.\d]+) >>>> desc=test event $1 >>>> action=tcpsock syslog101:514 <14>%.monstr %.mdaystr %.hmsstr myhost >>>> myprog: $0%.nl >>>> >>>> Finally, as David suggested, you can also pass messages to local syslog >>>> server via /dev/log socket, and let the local syslog server handle the >>>> messages (note that unlike for 'tcpsock' in previous example, there is no >>>> need for hostname and terminating newline for 'udgram' action): >>>> >>>> type=single >>>> ptype=regexp >>>> pattern=([.\d]+) >>>> desc=test event $1 >>>> action=udgram /dev/log <14>%.monstr %.mdaystr %.hmsstr myprog: $0 >>>> >>>> If your local syslog server is rsyslog, you could have the following >>>> rsyslog rule for forwarding messages: >>>> >>>> if $programname == "myprog" then @@syslog101:514 >>>> >>>> As you can see, there are several ways for achieving your goal, and >>>> hopefully above examples are helpful for selecting the most convenient >>>> solution. >>>> >>>> >>>>> >>>>> 1. On a different scenario, I was interested to match the logs >>>>> with list of IOC’s. Here i was trying to mail the detected log along >>>>> with >>>>> IOC name. I could achieve it to certain level as mentioned in example >>>>> but >>>>> no luck with this cases, "Split IP's from the IOC file and use it >>>>> on the “pattern” to match IP from logs" >>>>> >>>>> ************************ >>>>> >>>>> IOC_data_proposal.txt >>>>> >>>>> 187.163.222.244:465 - emotet >>>>> >>>>> 187.189.195.208:8443 - emotet >>>>> >>>>> 188.166.253.46:8080 - emotet >>>>> >>>>> 189.209.217.49:80 - heartbleed >>>>> >>>>> ************************ >>>>> >>>>> Please check and share some insights. >>>>> >>>> >>>> I am not sure I fully understood what exactly you want to achieve here. >>>> Can you provide some examples of input events and what output you would >>>> like to generate on each match? >>>> >>>> kind regards, >>>> risto >>>> >>>> >>>> >>>>> >>>>> >>>>> >>>>> Eg: I currently tested below case and its working fine as this is a >>>>> straight forward IOC matches. >>>>> >>>>> ************************ >>>>> >>>>> #Current Rule for matching IOC: >>>>> >>>>> type=Single >>>>> >>>>> ptype=RegExp >>>>> >>>>> pattern=(?:SEC_STARTUP|SEC_RESTART|SEC_SOFTRESTART) >>>>> >>>>> desc=load IOC data >>>>> >>>>> action=logonly; delete IP; create IP; \ >>>>> >>>>> lcall %iocevents -> (sub{scalar `cat /usr/local/bin/sec-rules/ >>>>> ioc_data.txt`}); \ >>>>> >>>>> cevent IOC_IP 0 %iocevents; >>>>> >>>>> >>>>> >>>>> type=Single >>>>> >>>>> ptype=RegExp >>>>> >>>>> pattern=. >>>>> >>>>> context=IOC_IP >>>>> >>>>> desc=create an entry >>>>> >>>>> action=logonly; alias IOC IOC_$0 >>>>> >>>>> >>>>> >>>>> type=Single >>>>> >>>>> ptype=regexp >>>>> >>>>> context=IOC_$2 >>>>> >>>>> pattern= syslog.*hostname=([\w\-\d]+).*IP=([\d\.]+) >>>>> >>>>> desc=Matched host & ip: $2 && $3 >>>>> >>>>> action=pipe '$0' mail -s ‘%s’ ‘test123.gmail.com’ >>>>> >>>>> >>>>> >>>>> IOC_data.txt >>>>> >>>>> 187.163.222.244 >>>>> >>>>> 187.189.195.208 >>>>> >>>>> 188.166.253.46 >>>>> >>>>> 189.209.217.49 >>>>> >>>>> 187.163.222.244 >>>>> >>>>> 187.189.195.208 >>>>> >>>>> 188.166.253.46 >>>>> >>>>> 189.209.217.49 >>>>> >>>>> ************************ >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> san >>>>> >>>> >>> >>> -- >>> Regards, >>> SanthoshKumar S >>> >> _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > -- Regards, SanthoshKumar S
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
