Hi Simon, thanks for the quick reply.
I would love to be a part of this project, just let me know what i can do.
Also, i forgot to add that when i change things, i always
1. kill the storm topology
2. change a local parser/enrichment file
3. copy to the parser dir
*
cp winlogbeat_parser.json
/usr/metron/$METRON_VERSION/config/zookeeper/parsers/winlogbeat.json
4. push configs to metron
*
/usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -m PUSH -i
/usr/metron/$METRON_VERSION/config/zookeeper/ -z `hostname -f`:2181
5. deploy the storm topology
*
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --topic winlogbeat
--partitions 1 --replication-factor 1 -z `hostname -f`:2181
As for the issue, the Strom worker pops up, tries to live, then dies. I have
modified the parser JSON to see if i could do a STELLAR transform, but it still
fails. The parser is this:
{
"parserClassName":"org.apache.metron.parsers.GrokParser",
"sensorTopic":"winlogbeat",
"parserConfig":{
"grokPath":"/patterns/winlogbeat",
"patternLabel":"WINLOGBEAT",
"timestampField":"timestamp"
},
"fieldTransformations":[
{
"transformation":"STELLAR"
,"output":[ "utc_timestamp" ]
,"config":{
"utc_timestamp":"TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-ddTHH:mm:ssZ',
MAP_GET(dc, dc2tz, 'UTC') )"
}
}
]
,"parserConfig":{
"dc2tz":{
"nyc":"EST"
,"la":"PST"
,"london":"UTC"
}
}
}
and the error message is this:
2017-05-03 19:03:37.211 o.a.m.p.GrokParser [ERROR] Can not create a Path from a
null string
java.lang.IllegalArgumentException: Can not create a Path from a null string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
~[stormjar.jar:?]
at org.apache.hadoop.fs.Path.<init>(Path.java:134) ~[stormjar.jar:?]
at
org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:82)
~[stormjar.jar:?]
at org.apache.metron.parsers.GrokParser.init(GrokParser.java:109)
[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:98)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
[storm-core-1.0.1.2.5.5.0-157.jar:1.0.1.2.5.5.0-157]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
[storm-core-1.0.1.2.5.5.0-157.jar:1.0.1.2.5.5.0-157]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2017-05-03 19:03:37.219 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Grok parser Error: Can not create a Path from a
null string
at org.apache.metron.parsers.GrokParser.init(GrokParser.java:129)
~[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:98)
~[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
~[storm-core-1.0.1.2.5.5.0-157.jar:1.0.1.2.5.5.0-157]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
[storm-core-1.0.1.2.5.5.0-157.jar:1.0.1.2.5.5.0-157]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.IllegalArgumentException: Can not create a Path from a
null string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
~[stormjar.jar:?]
at org.apache.hadoop.fs.Path.<init>(Path.java:134) ~[stormjar.jar:?]
at
org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:82)
~[stormjar.jar:?]
at org.apache.metron.parsers.GrokParser.init(GrokParser.java:109)
~[stormjar.jar:?]
... 5 more
________________________________
From: Simon Elliston Ball <si...@simonellistonball.com>
Sent: Wednesday, May 3, 2017 5:59 PM
To: user@metron.apache.org
Subject: Re: Question on Windows event log ingest and parse
Hi Ed,
Sounds like a really nice piece of work to get pushed into the core… how would
you feel about taking that grok parser and formalising it into the core of
Metron (happy to help there by the way).
On the actual issue, is sounds like it’s likely to be something to do with
conversion of the timestamp format to the unixtime used in Metron. We can look
at that. Did you see any log messages in the storm logs from the topology that
died?
Simon
On 3 May 2017, at 22:34, ed d
<ragdel...@hotmail.com<mailto:ragdel...@hotmail.com>> wrote:
Metron version – 0.4.0
Single node install, bare metal install
No significant changes to base install besides maintenance mode on
elasticsearch mpack and manual configuration.
I have a Windows 2012 server running AD, AD LDS, DNS, and DHCP. I installed
Winlogbeat<https://www.elastic.co/downloads/beats/winlogbeat>5.3.2 64 bit onto
the server. It was configured to push logs to the Elasticsearch on my Metron
install, and it works great. No issues.
I modified the Winlogbeat configuration to push logs directly to Kafka as I
want to enrich the logs. I followed this
guide<https://www.elastic.co/guide/en/beats/winlogbeat/master/kafka-output.html>.
I can see logs coming into the Kafka topic, so I built a Grok parser to slice
and dice. It seems to work fine on Grok
Constructor<http://grokconstructor.appspot.com/do/match> and Grok
Debugger<https://grokdebug.herokuapp.com/>, but when I load it into Metron as a
parser, it kills the Storm topology. It seems to be sticking on the timestamp,
which is ISO_8601<https://en.wikipedia.org/wiki/ISO_8601> format
(2017-05-03T21:04:33Z).
My question to the group, before troubleshooting my install, is to see if
anyone else has had success ingesting and parsing Windows event logs?
Does anyone pull Windows log into Kafka, Nifi, or other with the intent to
enrich the elements of the log? And if yes, what have you found to be most
useful?
FYI here is my Grok parser for reference:
timestamp"\:"%{TIMESTAMP_ISO8601:timestamp}","beat"\:\{"hostname"\:%{QUOTEDSTRING:hostname},"name"\:%{QUOTEDSTRING:name},"version"\:%{QUOTEDSTRING:beat_version}\},"computer_name"\:%{QUOTEDSTRING:computer_name},"event_data"\:\{("AuthenticationPackageName"\:%{QUOTEDSTRING:AuthenticationPackageName},?)?("ImpersonationLevel"\:%{QUOTEDSTRING:ImpersonationLevel},?)?("FailureReason"\:%{QUOTEDSTRING:FailureReason},?)?("IpAddress"\:"%{IP:ip_src_addr}",?)?("IpPort"\:%{QUOTEDSTRING:IpPort},?)?("KeyLength"\:%{QUOTEDSTRING:KeyLength},?)?("LmPackageName"\:%{QUOTEDSTRING:LmPackageName},?)?("LogonGuid"\:%{QUOTEDSTRING:LogonGuid},?)?("LogonProcessName"\:%{QUOTEDSTRING:LogonProcessName},?)?("LogonType"\:%{QUOTEDSTRING:LogonType},?)?("PrivilegeList"\:%{QUOTEDSTRING:PrivilegeList},?)?("ProcessId"\:%{QUOTEDSTRING:ProcessId},?)?("ProcessName"\:%{QUOTEDSTRING:ProcessName},?)?("PackageName"\:%{QUOTEDSTRING:PackageName},?)?("Status"\:%{QUOTEDSTRING:Status},?)?("SubStatus"\:%{QUOTEDSTRING:SubStatus},?)?("SubjectDomainName"\:%{QUOTEDSTRING:SubjectDomainName},?)?("SubjectLogonId"\:%{QUOTEDSTRING:SubjectLogonId},?)?("SubjectUserName"\:%{QUOTEDSTRING:SubjectUserName},?)?("SubjectUserSid"\:%{QUOTEDSTRING:SubjectUserSid},?)?("TargetDomainName"\:%{QUOTEDSTRING:TargetDomainName},?)?("TargetLogonId"\:%{QUOTEDSTRING:TargetLogonId},?)?("TargetUserName"\:%{QUOTEDSTRING:TargetUserName},?)?("TargetUserSid"\:%{QUOTEDSTRING:TargetUserSid},?)?("TransmittedServices"\:%{QUOTEDSTRING:TransmittedServices},?)?("Workstation"\:%{QUOTEDSTRING:Workstation},?)?("WorkstationName"\:%{QUOTEDSTRING:WorkstationName},?)?\},"event_id"\:%{NUMBER:event_id},"keywords"\:\[%{QUOTEDSTRING:keywords}\],"level"\:%{QUOTEDSTRING:level},"log_name"\:%{QUOTEDSTRING:log_name},"message"\:%{QUOTEDSTRING:message},"opcode"\:%{QUOTEDSTRING:opcode},"process_id"\:%{NUMBER:process_id},"provider_guid"\:%{QUOTEDSTRING:provider_guid},"record_number"\:%{QUOTEDSTRING:record_number},"source_name"\:%{QUOTEDSTRING:source_name},"task"\:%{QUOTEDSTRING:task},"thread_id"\:%{NUMBER:thread_id},"type"\:%{QUOTEDSTRING:type},?("version"\:%{NUMBER:version},?)?\}
