Metron version – 0.4.0 Single node install, bare metal install No significant changes to base install besides maintenance mode on elasticsearch mpack and manual configuration.
I have a Windows 2012 server running AD, AD LDS, DNS, and DHCP. I installed Winlogbeat<https://www.elastic.co/downloads/beats/winlogbeat> 5.3.2 64 bit onto the server. It was configured to push logs to the Elasticsearch on my Metron install, and it works great. No issues. I modified the Winlogbeat configuration to push logs directly to Kafka as I want to enrich the logs. I followed this guide<https://www.elastic.co/guide/en/beats/winlogbeat/master/kafka-output.html>. I can see logs coming into the Kafka topic, so I built a Grok parser to slice and dice. It seems to work fine on Grok Constructor<http://grokconstructor.appspot.com/do/match> and Grok Debugger<https://grokdebug.herokuapp.com/>, but when I load it into Metron as a parser, it kills the Storm topology. It seems to be sticking on the timestamp, which is ISO_8601<https://en.wikipedia.org/wiki/ISO_8601> format (2017-05-03T21:04:33Z). My question to the group, before troubleshooting my install, is to see if anyone else has had success ingesting and parsing Windows event logs? Does anyone pull Windows log into Kafka, Nifi, or other with the intent to enrich the elements of the log? And if yes, what have you found to be most useful? FYI here is my Grok parser for reference: timestamp"\:"%{TIMESTAMP_ISO8601:timestamp}","beat"\:\{"hostname"\:%{QUOTEDSTRING:hostname},"name"\:%{QUOTEDSTRING:name},"version"\:%{QUOTEDSTRING:beat_version}\},"computer_name"\:%{QUOTEDSTRING:computer_name},"event_data"\:\{("AuthenticationPackageName"\:%{QUOTEDSTRING:AuthenticationPackageName},?)?("ImpersonationLevel"\:%{QUOTEDSTRING:ImpersonationLevel},?)?("FailureReason"\:%{QUOTEDSTRING:FailureReason},?)?("IpAddress"\:"%{IP:ip_src_addr}",?)?("IpPort"\:%{QUOTEDSTRING:IpPort},?)?("KeyLength"\:%{QUOTEDSTRING:KeyLength},?)?("LmPackageName"\:%{QUOTEDSTRING:LmPackageName},?)?("LogonGuid"\:%{QUOTEDSTRING:LogonGuid},?)?("LogonProcessName"\:%{QUOTEDSTRING:LogonProcessName},?)?("LogonType"\:%{QUOTEDSTRING:LogonType},?)?("PrivilegeList"\:%{QUOTEDSTRING:PrivilegeList},?)?("ProcessId"\:%{QUOTEDSTRING:ProcessId},?)?("ProcessName"\:%{QUOTEDSTRING:ProcessName},?)?("PackageName"\:%{QUOTEDSTRING:PackageName},?)?("Status"\:%{QUOTEDSTRING:Status},?)?("SubStatus"\:%{QUOTEDSTRING:SubStatus},?)?("SubjectDomainName"\:%{QUOTEDSTRING:SubjectDomainName},?)?("SubjectLogonId"\:%{QUOTEDSTRING:SubjectLogonId},?)?("SubjectUserName"\:%{QUOTEDSTRING:SubjectUserName},?)?("SubjectUserSid"\:%{QUOTEDSTRING:SubjectUserSid},?)?("TargetDomainName"\:%{QUOTEDSTRING:TargetDomainName},?)?("TargetLogonId"\:%{QUOTEDSTRING:TargetLogonId},?)?("TargetUserName"\:%{QUOTEDSTRING:TargetUserName},?)?("TargetUserSid"\:%{QUOTEDSTRING:TargetUserSid},?)?("TransmittedServices"\:%{QUOTEDSTRING:TransmittedServices},?)?("Workstation"\:%{QUOTEDSTRING:Workstation},?)?("WorkstationName"\:%{QUOTEDSTRING:WorkstationName},?)?\},"event_id"\:%{NUMBER:event_id},"keywords"\:\[%{QUOTEDSTRING:keywords}\],"level"\:%{QUOTEDSTRING:level},"log_name"\:%{QUOTEDSTRING:log_name},"message"\:%{QUOTEDSTRING:message},"opcode"\:%{QUOTEDSTRING:opcode},"process_id"\:%{NUMBER:process_id},"provider_guid"\:%{QUOTEDSTRING:provider_guid},"record_number"\:%{QUOTEDSTRING:record_number},"source_name"\:%{QUOTEDSTRING:source_name},"task"\:%{QUOTEDSTRING:task},"thread_id"\:%{NUMBER:thread_id},"type"\:%{QUOTEDSTRING:type},?("version"\:%{NUMBER:version},?)?\}
