I have opened https://issues.apache.org/jira/browse/METRON-933 for supporting regex capture groups. Not exactly what you need I think, but well, you gave me the idea :)
On May 4, 2017 at 23:19:19, Otto Fowler ([email protected]) wrote: There are a couple of different ways this could be done: https://github.com/tools4j/unix4j for example as a back end, with STELLAR integration and commands ( if the MIT lic is OK ). Also, there is I believe a Jira for running scripts from stellar that may suit this. There is an interesting possibility around using java regex as well. STELLAR does not from what I can see support unix pipes however. Do you think you can create a jira for this? or would you like me to create one for you? On May 4, 2017 at 16:48:26, ed d ([email protected]) wrote: metron - 0.4.0 parsing windows event logs being pushed from Windows 2012 via winlogbeat 5.3.2 into kafka as JSON i am parsing the JSON windows event log with the JSONMapParser but there is vital information in the non-parsed message data portion. i need to extract the workstation, username, ip address and ip port from the original string as the JSON parser does not do this. i think STELLAR can do this but i do not see anything in the STELLAR documentation page <https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common> <https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common> <https://cwiki.apache.org/confluence/display/METRON/Stellar+Query+Language> to replicate the functionality of "grep", "sed" or "cut", at least i dont think i do. if it does not exist, i am ready and willing to hack something together as i understand we can write our own STELLAR functions. before i apply my non-programming skills to the function, my question to the group is if anyone has used STELLAR to perform the function of "grep", "sed" or "cut" on a parser? below is some background information for this question. this is what i want the parser to do: { "parserClassName":"org.apache.metron.parsers.json.JSONMapParser", "sensorTopic":"winlogbeat", "parserConfig": {}, "fieldTransformations" : [ { "transformation" : "STELLAR" ,"output" : [ "TargetUserName","WorkstationName","IpAddress","IpPort" ] ,"config" : { *"TargetUserName" : "sed 's/,/\r\n/g' original_string |grep "TargetUserName"|cut -f 2 -d :|sed 's/}//'",* *"WorkstationName" : "sed 's/,/\r\n/g' original_string |grep "WorkstationName"|cut -f 2 -d :|sed 's/}//'",* *"IpAddress" : "sed 's/,/\r\n/g' original_string |grep "IpAddress"|cut -f 2 -d :|sed 's/}//'",* *"IpPort" : "sed 's/,/\r\n/g' original_string |grep "IpPort"|cut -f 2 -d :|sed 's/}//'"* } ] } this is the data blob: {"@timestamp":"2017-05-04T11:28:16.171Z","beat":{"hostname":"demometrondc01","name":"demometrondc01","version":"5.3.2"},"computer_name":"demometrondc01.demometron.local","event_data":{"AuthenticationPackageName":"NTLM","ImpersonationLevel":"%%1833","IpAddress":"192.168.56.1","IpPort":"53479","KeyLength":"128","LmPackageName":"NTLM V2","LogonGuid":"{00000000-0000-0000-0000-000000000000}","LogonProcessName":"NtLmSsp ","LogonType":"3","ProcessId":"0x0","ProcessName":"-","SubjectDomainName":"-","SubjectLogonId":"0x0","SubjectUserName":"-","SubjectUserSid":"S-1-0-0","TargetDomainName":"DEMOMETRON","TargetLogonId":"0x4fed9e","TargetUserName":"ragdelaed","TargetUserSid":"S-1-5-21-2066284954-3896131091-2738420592-1106","TransmittedServices":"-","WorkstationName":"ragdelaed_pc"},"event_id":4624,"keywords":["Audit Success"],"level":"Information","log_name":"Security","message":"An account was successfully logged on.\n\nSubject:\n\tSecurity ID:\t\tS-1-0-0\n\tAccount Name:\t\t-\n\tAccount Domain:\t\t-\n\tLogon ID:\t\t0x0\n\nLogon Type:\t\t\t3\n\nImpersonation Level:\t\tImpersonation\n\nNew Logon:\n\tSecurity ID:\t\tS-1-5-21-2066284954-3896131091-2738420592-1106\n\tAccount Name:\t\tragdelaed\n\tAccount Domain:\t\tDEMOMETRON\n\tLogon ID:\t\t0x4FED9E\n\tLogonGUID:\t\t{00000000-0000-0000-0000-000000000000}\n\nProcess Information:\n\tProcess ID:\t\t0x0\n\tProcess Name:\t\t-\n\nNetwork Information:\n\tWorkstation Name:\tUS1206123W3\n\tSource Network Address:\t192.168.56.1\n\tSource Port:\t\t53479\n\nDetailed Authentication Information:\n\tLogon Process:\t\tNtLmSsp \n\tAuthentication Package:\tNTLM\n\tTransited Services:\t-\n\tPackage Name (NTLM only):\tNTLM V2\n\tKey Length:\t\t128\n\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\n\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\n\nThe logon type field indicates the kindof logon that occurred. The most common types are 2 (interactive) and 3 (network).\n\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\n\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\n\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\n\nThe authentication information fields provide detailed information about this specific logon request.\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\n\t- Transited services indicate which intermediate services have participated in this logon request.\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\n\t- Key length indicates the length of the generated session key. This will be 0 if no session keywas requested.","opcode":"Info","process_id":448,"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","record_number":"37123","source_name":"Microsoft-Windows-Security-Auditing","task":"Logon","thread_id":2768,"type":"wineventlog","version":1} this is how it renders in Metron: message An account was successfully logged on. Subject: Security ID:S-1-0-0 Account Name:- Account Domain:- Logon ID:0x0 Logon Type: 3 Impersonation Level: Impersonation New Logon: Security ID:S-1-5-21-2066284954-3896131091-2738420592-1106 Account Name:ragdelaed Account Domain:DEMOMETRON Logon ID:0x6C7D7C Logon GUID:{00000000-0000-0000-0000-000000000000} Process Information: Process ID:0x0 Process Name:- Network Information: Workstation Name:ragdelaed_pc Source Network Address:192.168.56.1 Source Port:55418 Detailed Authentication Information: Logon Process:NtLmSsp Authentication Package:NTLM Transited Services:- Package Name (NTLM only):NTLM V2 Key Length:128
