Hi Jose,
I got some help to sort out the different timestamps (format) and all log types now use "Jan 27 09:41:01". You asked about the firewall, this particular one is a Checkpoint currently running version R77.20. The remaining question, that might be of interest to others on the path to OSSEC mastery ;) ;) is how to handle messages with different "format" coming from the same host. I have collected a bunch of messages that I would like to be able to decode, but I'm not sure about the most efficient way to build the parent/child decoder tree for this. With the help received previously in this thread, I currently have the following in my local_decoder and I'm experimenting with different addition - none of which is working so far ;) <decoder name="Checkpoint"> <prematch>^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d</prematch> <type>firewall</type> </decoder> <decoder name="Checkpoint-alert"> <parent>Checkpoint</parent> <regex offset="after_parent">(\w+) \p\w+ \w+ src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)</regex> <order>action,srcip,dstip</order> </decoder> <decoder name="Checkpoint-alert"> <parent>Checkpoint</parent> <regex offset="after_regex">\.*resource: (\.*);\.*product: (\.*);</regex> <order>url,extra_data</order> </decoder> Below is a collection of syslog messages recieved from the firewall where the first section is currently decoded using the local_decoder above: Mar 29 10:09:40 127.0.0.1 Mar 29 9:57:49 st4600fw01n1 block <eth6 mail src: 192.168.7.206; dst: 54.72.9.51; proto: tcp; bytes: 4962; sent_bytes: 530; received_bytes: 4432; app_id: 3404393449; browse_time: ******; Suppressed logs: 1; Referrer_self_uid: ******; product: URL Filtering; service: http; s_port: 54693; product_family: Network; Mar 29 20:57:00, st4600fw01n1, allow <eth1 mail src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; appi_name: ******; app_desc: ******; app_id: 10063753; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Microsoft IE; web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: http://aliveproxy.com/; proxy_src_ip: 192.168.5.133; product: Application Control; service: http; s_port: 63867; product_family: Network; Mar 30 09:04:14 127.0.0.1 Mar 30 8:52:22 st4600fw01n1 allow <eth6 mail src: 192.168.5.133; dst: 23.67.132.180; proto: tcp; appi_name: ******; app_desc: ******; app_id: 10003219; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; app_sig_id: 10003219:4; proxy_src_ip: 192.168.5.133; product: Application Control; service: https; s_port: 64166; product_family: Network; Mar 30 09:03:56 127.0.0.1 Mar 30 8:52:05 st4600fw01n1 allow <eth6 mail src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; bytes: 2396; sent_bytes: 1180; received_bytes: 1216; app_id: 10063753; browse_time: ******; Suppressed logs: 19; Referrer_self_uid: ******; product: Application Control; service: http; s_port: 64136; product_family: Network; Mar 30 09:03:35 127.0.0.1 Mar 30 8:51:43 st4600fw01n1 allow <eth6 mail src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; appi_name: ******; app_desc: ******; app_id: 10063753; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Chrome; web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: http://www.aliveproxy.com/; proxy_src_ip: 192.168.5.133; product: Application Control; service: http; s_port: 64136; product_family: Network; Mar 29 09:04:21 127.0.0.1 Mar 29 8:52:29 st4600fw01n1 block <eth6 mail src: 192.168.5.136; dst: 37.157.4.15; proto: tcp; appi_name: ******; app_desc: ******; app_id: 1875144601; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Other: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko; web_server_type: Other: nginx; resource: http://adx.adform.net/adx/?rp=3&pv=1&bWlkPTE1NzU3NiZybmQ9ODU1M2JjNGMtMmY2MC00YzVjLWFhMzAtYmY5NzZlNDllZDNk&callback=_adform_cb_1459234242747_7491414591872548; proxy_src_ip: 192.168.5.136; product: URL Filtering; service: http; s_port: 54051; product_family: Network; Mar 29 09:06:12 127.0.0.1 Mar 29 8:54:21 st4600fw01n1 block <eth6 mail src: 192.168.6.157; dst: 37.157.4.15; proto: tcp; appi_name: ******; app_desc: ******; app_id: 1875144601; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Other: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko; web_server_type: Other: nginx; resource: http://adx.adform.net/adx/?rp=3&pv=1&bWlkPTk1NzI4&callback=_adform_cb_1459234355177_008705563130792681; proxy_src_ip: 192.168.6.157; product: URL Filtering; service: http; s_port: 51746; product_family: Network; Mar 29 08:37:54 127.0.0.1 Mar 29 8:26:03 st4600fw01n1 block <eth6 mail src: 192.168.6.157; dst: 152.115.75.210; proto: tcp; appi_name: ******; app_desc: ******; app_id: 1875144601; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Other: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko; web_server_type: Other: nginx; resource: http://adx.adform.net/adx/?rp=3&pv=1&bWlkPTk3ODMyJmN0dXJsPWh0dHA6Ly9mdXNpb24uYm9ubmllcnRpZHNrcmlmdGVyLnNlL2V2ZW50L3VlZXVqL2J0LmVrb25vbWkucGFmLnJvcy8xNDc3OTkyNzcvY2xpY2s_dXJsPQ&bWlkPTk3ODMxJmN0dXJsPWh0dHA6Ly9mdXNpb24uYm9ubmllcnRpZHNrcmlmdGVyLnNlL2V2ZW50L3lydmd2L2J0LmVrb25vbWkucGFmLnJvcy8xNDc3OTkyNzAvY2xpY2s_dXJsPQ&callback=_adform_cb_1459232656248_7681010355476278; proxy_src_ip: 192.168.6.157; product: URL Filtering; service: http; s_port: 51104; product_family: Network; Mar 29 08:35:17 127.0.0.1 Mar 29 8:23:24 st4600fw01n1 block <eth6 mail src: 192.168.6.157; dst: 152.115.75.199; proto: tcp; appi_name: ******; app_desc: ******; app_id: 1875144601; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Other: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko; web_server_type: Other: nginx; resource: http://adx.adform.net/adx/?rp=3&pv=1&bWlkPTk1NzMw&callback=_adform_cb_1459232497393_6046677836175733; proxy_src_ip: 192.168.6.157; product: URL Filtering; service: http; s_port: 50904; product_family: Network; Mar 30 10:04:39 127.0.0.1 redirect <eth1 alert web_client_type: Chrome; resource: http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; src: 192.168.5.133; dst: 172.226.217.148; proto: tcp; session_id: {0x56fb8896,0x10009,0xc50d2e0a,0xc0000001}; Protection name: Check Point - Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence Level: 5; severity: 2; malware_action: Communication with C&C site; rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL reputation; malware_rule_id: {000000CE-00A4-0046-9658-621EA5468654}; protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 192.168.5.133; product: Anti Malware; service: http; s_port: 49244; Mar 29 20:13:14 127.0.0.1 prevent <eth1 alert web_client_type: Microsoft IE; resource: http://www.bing.com/fd/ls/GLinkPing.aspx?IG=9A0044152B65437D93F87086B9E730D9&&ID=SERP,5118.1&url=http://sc1.checkpoint.com/z...; src: 192.168.5.133; dst: 204.79.197.200; proto: tcp; session_id: {0x56fac5ba,0x10004,0xc50d2e0a,0xc0000000}; Protection name: Check Point - Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence Level: 5; severity: 2; malware_action: Communication with C&C site; rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL reputation; malware_rule_id: {000000CE-00A4-0046-9658-621EA5468654}; protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 192.168.5.133; product: Anti Malware; service: http; s_port: 63119; Mar 30 9:04:59, st4600fw01n1, allow <eth6 mail src: 192.168.5.133; dst: 207.244.85.73; proto: tcp; appi_name: ******; app_desc: ******; app_id : 10064017; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Chrome; web_server_type: Apache; app_sig_id: 10064017:2; resource: http://www.bypassthat.com/; proxy_src_ip: 192.168.5.133; product: Application Control; service: http; s_port: 64499; product_family: Network; Mar 30 08:55:41 127.0.0.1 Mar 30 8:49:25 < sto-fwm03 mail System Alert message: A Firewall Policy has been successfully installed on st4600fw01n2; Object: st4600fw01n2; Event: Change; Parameter: policy_time; Condition: changes Tue Mar 22 11:07:17 2016; Current value: Wed Mar 30 08:39:57 2016; product: System Monitor; product_family: Network; Mar 30 08:56:02 127.0.0.1 Mar 30 8:49:47 < sto-fwm03 mail System Alert message: A Firewall Policy has been successfully installed on st4600fw01n1; Object: st4600fw01n1; Event: Change; Parameter: policy_time; Condition: changes Tue Mar 22 11:09:21 2016; Current value: Wed Mar 30 08:43:12 2016; product: System Monitor; product_family: Network; On Tuesday, March 29, 2016 at 12:53:19 PM UTC+2, Jesus Linares wrote: > > Hi, > > first, I would use the same format for both messages. Two options: > > - Change log format in each device. > - Choose one: > - 1Mar2016 15:17:09 redirect st4600fw01n1 > - Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 > - This part could be your parent decoder (using regular expressions) > - Change the log received with rsyslog, for example, add a string: > - *MyFirewall *1Mar2016 15:17:09 redirect st4600fw01n1 > - So, the parent decoder will be <prematch*>^**MyFirewall > </prematch>* > > The prematch of each sub-decoder (child decoder) could be the type of log, > maybe "web_client_type" or "mail". > > What firewall are you using? Version?. > > Paste here more logs. > > Regards, > Jesus Linares > > On Thursday, March 24, 2016 at 9:47:28 PM UTC+1, Fredrik wrote: >> >> Hi Jesus, >> >> >> Got sidetracked with other projects, and finally getting back to my >> questions about handling different messages from the same device >> (firewall). Also, Jesus your suggestion about placing a prematch in the >> suggested decoder in this thread - what would be a good prematch here? >> >> Should I add an OR to the parent decoder to do the first match and then >> use different subdecoders to extract the useful information from the other >> type of message? How do you deal with these type of scenarios? >> >> Just so I got that part right. Giving two sections the same >> <decoder-name>Checkpoint-alert</decoder> in essence means that it is one >> decoder, but defined in two sections? >> >> >> Please find the two message-types below for reference. >> >> MESSAGE1: >> 1Mar2016 15:17:09 redirect st4600fw01n1 <eth1 alert web_client_type: >> Chrome; resource: http:// >> sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; >> src: 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: >> {0x56d5a2de,0x4,0xc50d2e0a,0xc0000001}; Protection name: Check Point - >> Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence >> Level: 5; severity: 2; malware_action: Communication with C&C site; >> rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL >> reputation; malware_rule_id: {000000CE-00A4-0046-9658-621EA5468654}; >> protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: >> 10.46.5.133; product: Anti Malware; service: http; s_port: 61834; >> >> MESSAGE2: >> Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 allow <eth1 mail >> src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: ******; >> app_desc: ******; app_id: 10063753; app_category: ******; >> matched_category: ******; app_properties: ******; app_risk: ******; >> app_rule_id: ******; app_rule_name: ******; web_client_type: Chrome; >> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: http:// >> www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: Application >> Control; service: http; s_port: 58579; product_family: Network; >> >> On Monday, March 7, 2016 at 12:11:21 PM UTC+1, Jesus Linares wrote: >> >>> Hi Fredrik, >>> >>> The expression "\.+" matches for anything. Usually, it is not a good >>> idea because is slow and maybe you capture something that you don't want. >>> So, *when it is possible*, it is better to use something specific. >>> >>> When you have different decoders (different name) with the same parent, >>> you should use a prematch. If you don't use prematch, it is fired the first >>> rule. In the previous example: >>> >>> Log: >>> Mar 3 12:15:24 LinMV TestDecoder[1963]: TypeB field1: hi; value2: bye; >>> value3: seeyou >>> >>> Without prematch: >>> **Phase 2: Completed decoding. >>> decoder: 'TestDecoder' >>> extra_data: 'seeyou' >>> >>> With prematch: >>> **Phase 2: Completed decoding. >>> decoder: 'TestDecoder' >>> id: 'bye;' >>> >>> >>> Without prematch, the decoder is TestDecoder-1, but it should be >>> TestDecoder2 (because it has the string "field1". In my view, it is a good >>> practice use prematch, but sometimes it is no necessary. >>> >>> Regarding your last question, could you use the same log format in your >>> firewall and in the blade?. Paste here two logs of each one (firewall and >>> blade) and your decoders, and we will take a look ;) >>> >>> Regards. >>> Jesus Linares >>> >>> On Friday, March 4, 2016 at 9:08:34 PM UTC+1, Fredrik wrote: >>>> >>>> Hi All, >>>> >>>> >>>> In this context and with your great response. What would you PROs >>>> suggest I do when decoding another type of message from the same firewall >>>> - >>>> but a different blade (i.e. module). Turns out that the messages look >>>> somewhat different. This is a sample from the other module and it won't >>>> match with the current decoder. Should I add an OR to the parent decoder >>>> to >>>> do the first match and then use different subdecoders to extract the >>>> useful >>>> information from the other type of message? How do you deal with these >>>> type >>>> of scenarios? >>>> >>>> MESSAGE: >>>> 1Mar2016 15:17:09 redirect st4600fw01n1 <eth1 alert web_client_type: >>>> Chrome; resource: http:// >>>> sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; >>>> src: 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: >>>> {0x56d5a2de,0x4,0xc50d2e0a,0xc0000001}; Protection name: Check Point - >>>> Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence >>>> Level: 5; severity: 2; malware_action: Communication with C&C site; >>>> rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL >>>> reputation; malware_rule_id: {000000CE-00A4-0046-9658-621EA5468654}; >>>> protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: >>>> 10.46.5.133; product: Anti Malware; service: http; s_port: 61834; >>>> >>>> Best regards, >>>> Fredrik >>>> >>>> >>>> On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote: >>>>> >>>>> Hi All, >>>>> >>>>> >>>>> Came across this where I think I would be helped by extracting fields >>>>> both in forward (from beginning) and in reverse (from end) order of >>>>> messages!? Is this possible, if so, is it stupid given that there are >>>>> other >>>>> (better) ways to accomplish the same thing :/ ? >>>>> >>>>> In addition to the fields my current decoder extracts, I was hoping to >>>>> extract the resource (http://www.aliveproxy.com/) and also the >>>>> product (Application Control;). My idea was to add a secondary >>>>> statement, before the <order> statement, something in the lines of: >>>>> <regex>$/p\w+\s [...] and work my way backward so that I can >>>>> extract Application Control and resource . How would you suggest I do >>>>> this?! >>>>> >>>>> Thanks again for all the great help - hope my threads (and questions) >>>>> can be useful for other newstarters outhere trying to get there feet off >>>>> the ground ;) >>>>> >>>>> Best regards, >>>>> Fredrik >>>>> >>>>> LOG-MESSAGE >>>>> >>>>> *Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 *allow <eth1 >>>>> mail src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: >>>>> ******; app_desc: ******; app_id: 10063753; app_category: ******; >>>>> matched_category: ******; app_properties: ******; app_risk: ******; >>>>> app_rule_id: ******; app_rule_name: ******; web_client_type: Chrome; >>>>> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: http >>>>> ://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: >>>>> Application Control; service: http; s_port: 58579; product_family: >>>>> Network; >>>>> >>>>> MY CURRENT DECODER >>>>> >>>>> <decoder name="Checkpoint"> >>>>> <prematch>^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d</prematch> >>>>> <type>firewall</type> >>>>> </decoder> >>>>> >>>>> <decoder name="Checkpoint-alert"> >>>>> <parent>Checkpoint</parent> >>>>> <regex offset="after_parent">(\w+) \p\w+ \w+ >>>>> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)</regex> >>>>> <order>action,srcip,dstip</order> >>>>> </decoder> >>>>> >>>>> LOGTEST OUTPUT >>>>> >>>>> >>>>> **Phase 1: Completed pre-decoding. >>>>> full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 >>>>> st4600fw01n1 allow <eth1 mail src: 192.168.1.15 dst: 89.208.212.2; proto: >>>>> tcp; appi_name: ******; app_desc: ******; app_id: 10063753; app_category: >>>>> ******; matched_category: ******; app_properties: ******; app_risk: >>>>> ******; >>>>> app_rule_id: ******; app_rule_name: ******; web_client_type: Chrome; >>>>> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: >>>>> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15; product: >>>>> Application Control; service: http; s_port: 58579; product_family: >>>>> Network;' >>>>> hostname: '127.0.0.1' >>>>> program_name: '(null)' >>>>> log: 'Jan 27 9:32:28 st4600fw01n1 allow <eth1 mail src: >>>>> 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: ******; app_desc: >>>>> ******; app_id: 10063753; app_category: ******; matched_category: ******; >>>>> app_properties: ******; app_risk: ******; app_rule_id: ******; >>>>> app_rule_name: ******; web_client_type: Chrome; web_server_type: >>>>> Microsoft-IIS; app_sig_id: 10063753:5; resource: >>>>> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15; product: >>>>> Application Control; service: http; s_port: 58579; product_family: >>>>> Network;' >>>>> >>>>> **Phase 2: Completed decoding. >>>>> decoder: 'Checkpoint' >>>>> action: 'allow' >>>>> srcip: '192.168.1.15' >>>>> dstip: '89.208.212.2' >>>>> >>>>> **Phase 3: Completed filtering (rules). >>>>> Rule id: '4100' >>>>> Level: '0' >>>>> Description: 'Firewall rules grouped.' >>>>> >>>>> -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
