[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285318927
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
 
 Review comment:
   I've added another, but there are no doubt more delinquent cases we will 
find in the wild.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285318853
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285318743
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
 
 Review comment:
   Fair enough, this was mainly kept the same to ensure similarity with the 
CEFParser. I've moved to the new API in this case, but there should be a follow 
on for converting other parsers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on issue #1407: METRON-2113: Update version to 0.7.2

[GitBox]

mmiklavc commented on issue #1407: METRON-2113: Update version to 0.7.2
URL: https://github.com/apache/metron/pull/1407#issuecomment-493607746
 
 
   Looks good, thanks @justinleet! +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on issue #1409: METRON-2112 Normalize parser original_string handling

[GitBox]

mmiklavc commented on issue #1409: METRON-2112 Normalize parser original_string 
handling
URL: https://github.com/apache/metron/pull/1409#issuecomment-493602054
 
 
   @simonellistonball and @ottobackwards - you guys probably want to have a 
chance to review the proposed functionality change before it goes in.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on issue #1409: METRON-2112 Normalize parser original_string handling

[GitBox]

mmiklavc commented on issue #1409: METRON-2112 Normalize parser original_string 
handling
URL: https://github.com/apache/metron/pull/1409#issuecomment-493599907
 
 
   Here is the proposed follow on Jira to normalize all existing Metron-managed 
parsers
   
   https://issues.apache.org/jira/browse/METRON-2120


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (METRON-2120) Normalize Metron parser handling of original_string

[Michael Miklavcic (JIRA)]

Michael Miklavcic created METRON-2120:
-

 Summary: Normalize Metron parser handling of original_string
 Key: METRON-2120
 URL: https://issues.apache.org/jira/browse/METRON-2120
 Project: Metron
  Issue Type: Task
Reporter: Michael Miklavcic


See linked Jira METRON-2112. This work would normalize our handling of 
original_string for all existing Metron-managed parsers. For all intents and 
purposes, this would mean removing any parser specific handling of 
original_string, updating any failing tests, and a release note about breaking 
changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285286533
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == null || 

[GitHub] [metron] mmiklavc commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493596337
 
 
   > Lots of good catches there Mike, I'll do some clean up. Many of the issues 
are inherited from the fact that I heavily 'borrowed' from the existing CEF 
parser. Do you think it would be worth fixing that up at the same time on this 
PR, since I'm refactoring bits of it anyway?
   
   Sure, that sounds reasonable. What did you have in mind?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493596017
 
 
   > > Agree with @ottobackwards. I just had to mess about in the integration 
tests for parsers, so here are some links to help that process:
   > > 
   > > * 
https://github.com/apache/metron/tree/master/metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/integration
   > > * 
https://github.com/apache/metron/tree/master/metron-platform/metron-integration-test/src/main/sample/data
   > 
   > Is it really a good idea to keep integration tests for the metron-parsers 
package in the metron-parsers-common package? Seems like we're mixing concerns 
there.
   
   I tend to agree, though a handful of them are considered "common."
   
   - "jsonMap",
   - "jsonMapQuery",
   - "jsonMapWrappedQuery",
   - "syslog3164",
   - "syslog5424"
   
   This should probably be split up a bit, but I don't think it needs to happen 
in this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] merrimanr commented on issue #1389: METRON-2087: Remove Storm dependency from metron-indexing

[GitBox]

merrimanr commented on issue #1389: METRON-2087: Remove Storm dependency from 
metron-indexing
URL: https://github.com/apache/metron/pull/1389#issuecomment-493584143
 
 
   After another round of testing I realized that we also need to separate 
`metron-elasticsearch` and `metron-solr` into separate `common` and `storm` 
modules.  This is because the indexing topology depends on uber jars built from 
the these modules.  The `storm` modules for each now contain flux property 
files, start scripts, and indexing integration tests.  The benefit of this is 
that the awkwardness of modules in `metron-indexing` that only contain a single 
test goes away.  The `metron-indexing-storm-elasticsearch` module in 
`metron-indexing` is now `metron-elasticsearch-storm` and likewise for 
`metron-indexing-storm-solr`.
   
   The changes were fairly straightforward except for the `pom.xml` files.  I 
ran into similar class version issues that we commonly run into when 
refactoring.  The `metron-elasticsearch-storm` module was particularly tricky.  
I believe this is caused by our use of a dedicated `elasticsearch-shaded` 
module that makes the dependency tree more difficult to follow.  The solution 
was to just include this module dependency at the top of the 
`metron-elasticsearch-storm` pom.  
   
   Other changes included updating paths in various tests since the module 
structure changed and modules containing these tests are now a level deeper.  
Other than that the only changes involved were just moving files.
   
   I ran through another round of tests (based on instructions in 
https://github.com/apache/metron/pull/1368) and everything seems to be working 
well.  I also tested the various meta alert functions in the Alerts UI.  I 
found a bug (https://issues.apache.org/jira/browse/METRON-2119) but I believe 
it is preexisting and unrelated to this work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (METRON-2119) Cannot remove an alert from a metaalert when Solr is enabled

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-2119:
-

 Summary: Cannot remove an alert from a metaalert when Solr is 
enabled
 Key: METRON-2119
 URL: https://issues.apache.org/jira/browse/METRON-2119
 Project: Metron
  Issue Type: Bug
Reporter: Ryan Merriman


After creating a metaalert in the Alerts UI, an alert cannot be removed.  The 
following error is returned:

 
{code:java}
Could not find collection:  {code}
I believe this happens because the request is incorrectly formatted:

 
{code:java}
{
  "metaAlertGuid":"6eaa087e-d357-4c5c-82be-ed911a8bb53e",
  "alerts":[
{
  "guid":"278d97e6-7de8-4453-8366-aadf2a057b59",
  "sensorType":"bro",
  "index":""
}
  ]
}{code}
The index property that is set to a blank string should instead be left out.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285271505
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
 
 Review comment:
   Sure, can you guess where I started from with this class ;)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285270629
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/utils/DateUtils.java
 ##
 @@ -36,6 +36,14 @@
  */
 public class DateUtils {
 
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285270083
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285264740
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
+   @Test
+   public void testInvalid() {
+   List obj = parse("test test test nonsense\n");
+   Assert.assertEquals(0, obj.size());
+   }
+
+   @Test
+   public void testTimestampPriority() throws java.text.ParseException {
+   long correctTime = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+   .getTime();
+
+   SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz");
+
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-06-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-05-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertNotNull(obj.get("timestamp"));
+   }
+
+   }
+
+   private void runMissingYear(Calendar expected, Calendar input) {
+   SimpleDateFormat sdf = new SimpleDateFormat("MMM dd 
HH:mm:ss.SSS");
+   for (JSONObject obj : 
parse("LEEF:2.0|Lancope|StealthWatch|1.0|41|\t|src=10.0.0.1\tdevTime="
+   + sdf.format(input.getTime()) +
+   "\tdevTimeFormat=MMM dd HH:mm:ss.SSS" +
+   "\tdst=2.1.2.2\tspt=1232")) {
+   

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285264348
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
+   @Test
+   public void testInvalid() {
+   List obj = parse("test test test nonsense\n");
+   Assert.assertEquals(0, obj.size());
+   }
+
+   @Test
+   public void testTimestampPriority() throws java.text.ParseException {
+   long correctTime = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+   .getTime();
+
+   SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz");
+
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-06-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-05-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertNotNull(obj.get("timestamp"));
+   }
+
+   }
+
+   private void runMissingYear(Calendar expected, Calendar input) {
+   SimpleDateFormat sdf = new SimpleDateFormat("MMM dd 
HH:mm:ss.SSS");
+   for (JSONObject obj : 
parse("LEEF:2.0|Lancope|StealthWatch|1.0|41|\t|src=10.0.0.1\tdevTime="
+   + sdf.format(input.getTime()) +
+   "\tdevTimeFormat=MMM dd HH:mm:ss.SSS" +
+   "\tdst=2.1.2.2\tspt=1232")) {
+   

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285264301
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
+   @Test
+   public void testInvalid() {
+   List obj = parse("test test test nonsense\n");
+   Assert.assertEquals(0, obj.size());
+   }
+
+   @Test
+   public void testTimestampPriority() throws java.text.ParseException {
+   long correctTime = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+   .getTime();
+
+   SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz");
+
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-06-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-05-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertNotNull(obj.get("timestamp"));
+   }
+
+   }
+
+   private void runMissingYear(Calendar expected, Calendar input) {
+   SimpleDateFormat sdf = new SimpleDateFormat("MMM dd 
HH:mm:ss.SSS");
+   for (JSONObject obj : 
parse("LEEF:2.0|Lancope|StealthWatch|1.0|41|\t|src=10.0.0.1\tdevTime="
+   + sdf.format(input.getTime()) +
+   "\tdevTimeFormat=MMM dd HH:mm:ss.SSS" +
+   "\tdst=2.1.2.2\tspt=1232")) {
+   

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285263938
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285263654
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == 

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285262583
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == 

[GitHub] [metron] simonellistonball commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on a change in pull request #1408: METRON-2118: 
Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285262315
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == 

[GitHub] [metron] simonellistonball commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493569746
 
 
   > Agree with @ottobackwards. I just had to mess about in the integration 
tests for parsers, so here are some links to help that process:
   > 
   > * 
https://github.com/apache/metron/tree/master/metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/integration
   > * 
https://github.com/apache/metron/tree/master/metron-platform/metron-integration-test/src/main/sample/data
   
   Is it really a good idea to keep integration tests for the metron-parsers 
package in the metron-parsers-common package? Seems like we're mixing concerns 
there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285255904
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
 
 Review comment:
   I don't think new parsers should implement the old interface method, this 
should instead implement the new interface method 
   `Optional> parseOptionalResult(byte[] parseMessage)`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285254280
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/cef/CEFParser.java
 ##
 @@ -45,7 +45,7 @@
private static final Charset UTF_8 = Charset.forName("UTF-8");
 
private Pattern p;
 
 Review comment:
   +1 to the static pattern


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285254850
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
 
 Review comment:
   A lot of these string can be static final defines


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285254192
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/utils/DateUtils.java
 ##
 @@ -36,6 +36,14 @@
  */
 public class DateUtils {
 
 
 Review comment:
   We mention specs here, we should link to them


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285254581
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285257033
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
 
 Review comment:
   Is this the only invalid case?  Are there almost could match cases that will 
fail or should fail?  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ottobackwards commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on a change in pull request #1408: METRON-2118: Added a 
LEEF parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285256723
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?

[GitHub] [metron] simonellistonball commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493567697
 
 
   @ottobackwards sure, integration tests will come shortly. I've not had much 
luck finding good sample data to make those exercise much more than the unit 
tests do already (except of course for the pure integration with the parser 
runner). I would love to see if anyone else has better samples they could 
contribute to beef up the tests and help us find any edge cases too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] simonellistonball commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

simonellistonball commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493566883
 
 
   Lots of good catches there Mike, I'll do some clean up. Many of the issues 
are inherited from the fact that I heavily 'borrowed' from the existing CEF 
parser. Do you think it would be worth fixing that up at the same time on this 
PR, since I'm refactoring bits of it anyway?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285249498
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/resources/org/apache/metron/parsers/leef/sample.schema
 ##
 @@ -0,0 +1,27 @@
+{
 
 Review comment:
   Cool!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285245164
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == null || 

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285244269
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
 
 Review comment:
   Just making a note as this relates to another discussion on default charsets 
- https://github.com/apache/metron/pull/1341#issuecomment-493459303. Regardless 
of which comes first, we should land on a cohesive strategy for all parsers and 
be sure this parser has the change once both PRs are complete.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285243093
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285245664
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == null || 

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285247929
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
+   @Test
+   public void testInvalid() {
+   List obj = parse("test test test nonsense\n");
+   Assert.assertEquals(0, obj.size());
+   }
+
+   @Test
+   public void testTimestampPriority() throws java.text.ParseException {
+   long correctTime = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+   .getTime();
+
+   SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz");
+
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-06-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-05-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertNotNull(obj.get("timestamp"));
+   }
+
+   }
+
+   private void runMissingYear(Calendar expected, Calendar input) {
+   SimpleDateFormat sdf = new SimpleDateFormat("MMM dd 
HH:mm:ss.SSS");
+   for (JSONObject obj : 
parse("LEEF:2.0|Lancope|StealthWatch|1.0|41|\t|src=10.0.0.1\tdevTime="
+   + sdf.format(input.getTime()) +
+   "\tdevTimeFormat=MMM dd HH:mm:ss.SSS" +
+   "\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(expected.getTime(), new 

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285247697
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
+   @Test
+   public void testInvalid() {
+   List obj = parse("test test test nonsense\n");
+   Assert.assertEquals(0, obj.size());
+   }
+
+   @Test
+   public void testTimestampPriority() throws java.text.ParseException {
+   long correctTime = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+   .getTime();
+
+   SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz");
+
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-06-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-05-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertNotNull(obj.get("timestamp"));
+   }
+
+   }
+
+   private void runMissingYear(Calendar expected, Calendar input) {
+   SimpleDateFormat sdf = new SimpleDateFormat("MMM dd 
HH:mm:ss.SSS");
+   for (JSONObject obj : 
parse("LEEF:2.0|Lancope|StealthWatch|1.0|41|\t|src=10.0.0.1\tdevTime="
+   + sdf.format(input.getTime()) +
+   "\tdevTimeFormat=MMM dd HH:mm:ss.SSS" +
+   "\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(expected.getTime(), new 

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285246944
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/test/java/org/apache/metron/parsers/leef/LEEFParserTest.java
 ##
 @@ -0,0 +1,252 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.Charset;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import com.google.common.io.Resources;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+public class LEEFParserTest {
+   private static final Charset UTF_8 = Charset.forName("utf-8");
+   private LEEFParser parser;
+
+   @Before
+   public void setUp() {
+   parser = new LEEFParser();
+   parser.init();
+   }
+
+   @Test
+   public void testInvalid() {
+   List obj = parse("test test test nonsense\n");
+   Assert.assertEquals(0, obj.size());
+   }
+
+   @Test
+   public void testTimestampPriority() throws java.text.ParseException {
+   long correctTime = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
+   .getTime();
+
+   SimpleDateFormat sdf = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss.SSSz");
+
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-06-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   "2016-05-01T09:29:11.356-04:00 host 
LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(new Date(correctTime), new 
Date((long) obj.get("timestamp")));
+   Assert.assertEquals(correctTime, obj.get("timestamp"));
+   }
+   for (JSONObject obj : parse(
+   
"LEEF:2.0|Lancope|StealthWatch|1.0|41|src=10.0.0.1\tdevTime=May 1 2016 
09:29:11.356 -0400\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertNotNull(obj.get("timestamp"));
+   }
+
+   }
+
+   private void runMissingYear(Calendar expected, Calendar input) {
+   SimpleDateFormat sdf = new SimpleDateFormat("MMM dd 
HH:mm:ss.SSS");
+   for (JSONObject obj : 
parse("LEEF:2.0|Lancope|StealthWatch|1.0|41|\t|src=10.0.0.1\tdevTime="
+   + sdf.format(input.getTime()) +
+   "\tdevTimeFormat=MMM dd HH:mm:ss.SSS" +
+   "\tdst=2.1.2.2\tspt=1232")) {
+   Assert.assertEquals(expected.getTime(), new 

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285244790
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == null || 

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285242615
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
 
 Review comment:
   Leave the existing files as they are, but for new files can you give these a 
quick reformat with 2 space indent and tabs as spaces? We're working towards 
normalizing the file formats, which makes diffs and code reviews easier and 
faster.
   
   You can manually change it in the IDE:
   
![image](https://user-images.githubusercontent.com/658443/57948789-d7f80680-789f-11e9-8124-c8824f0e9631.png)
   
   or there's a simple set of instructions for importing the Google stylesheet 
we use - see section 2.2.2 here - 
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on a change in pull request #1408: METRON-2118: Added a LEEF 
parser
URL: https://github.com/apache/metron/pull/1408#discussion_r285245851
 
 

 ##
 File path: 
metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
 ##
 @@ -0,0 +1,236 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.leef;
+
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.Charset;
+import java.text.SimpleDateFormat;
+import java.time.Clock;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.ParseException;
+import org.apache.metron.parsers.cef.CEFParser;
+import org.apache.metron.parsers.utils.DateUtils;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LEEFParser extends BasicParser {
+   private static final long serialVersionUID = 1L;
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+   private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
+   private static final Charset UTF_8 = Charset.forName("UTF-8");
+   private static final String EXTENSION_CAPTURE_PATTERN = "(?";
+   String syslogHost = "[a-z0-9\\.-_]+";
+
+   StringBuilder sb = new StringBuilder("");
+   sb.append("(?");
+   sb.append(syslogPriority);
+   sb.append(")?");
+   sb.append("(?");
+   sb.append(syslogTime);
+   sb.append("|");
+   sb.append(syslogTime5424);
+   sb.append(")?");
+
+   sb.append("(?");
+   sb.append(syslogHost);
+   sb.append(")?");
+
+   sb.append(".*");
+
+   sb.append("LEEF:(?1.0|2.0|0)?\\|");
+
+   headerBlock("DeviceVendor", sb);
+   sb.append("\\|");
+   headerBlock("DeviceProduct", sb);
+   sb.append("\\|");
+   headerBlock("DeviceVersion", sb);
+   sb.append("\\|");
+   headerBlock("DeviceEvent", sb);
+   sb.append("\\|");
+   
+   // add optional delimiter header (only applicable for LEEF 2.0)
+   sb.append("(");
+   headerBlock("Delimiter", sb);
+   sb.append("\\|");
+   sb.append(")?");
+   
+   // extension capture:
+   sb.append(" ?(?.*)");
+   String pattern = sb.toString();
+
+   p = Pattern.compile(pattern);
+
+   pext = Pattern.compile(EXTENSION_CAPTURE_PATTERN);
+   }
+
+   @SuppressWarnings("unchecked")
+   public List parse(byte[] rawMessage) {
+   List messages = new ArrayList<>();
+
+   String cefString = new String(rawMessage, UTF_8);
+
+   Matcher matcher = p.matcher(cefString);
+
+   while (matcher.find()) {
+   JSONObject obj = new JSONObject();
+   if (!matcher.matches()) {
+   break;
+   }
+   LOG.debug("Found %d groups", matcher.groupCount());
+   obj.put("DeviceVendor", matcher.group("DeviceVendor"));
+   obj.put("DeviceProduct", 
matcher.group("DeviceProduct"));
+   obj.put("DeviceVersion", 
matcher.group("DeviceVersion"));
+   obj.put("DeviceEvent", matcher.group("DeviceEvent"));
+   
+   String ext = matcher.group("extensions");
+
+   // In LEEF 2.0 the delimiter can be specified
+   String version = matcher.group("Version");
+   if (version.equals("2.0")) {
+   String delimiter = matcher.group("Delimiter");
+   if (delimiter == null || 

[GitHub] [metron] mmiklavc commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

mmiklavc commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493552590
 
 
   Agree with @ottobackwards. I just had to mess about in the integration tests 
for parsers, so here are some links to help that process:
   
   - 
https://github.com/apache/metron/tree/master/metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/integration
   - 
https://github.com/apache/metron/tree/master/metron-platform/metron-integration-test/src/main/sample/data


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc opened a new pull request #1409: METRON-2112 Normalize parser original_string handling

[GitBox]

mmiklavc opened a new pull request #1409: METRON-2112 Normalize parser 
original_string handling
URL: https://github.com/apache/metron/pull/1409
 
 
   ## Contributor Comments
   
   https://issues.apache.org/jira/browse/METRON-2112
   
   This PR introduces work to address the issue with `original_string` brought 
up in the DISCUSS thread [[DISCUSS] JsonMapParser original string 
functionality](https://lists.apache.org/thread.html/2dbf068cd2144ea3bda8d652b4e866c74b3ef9e96510c63ff27335b4@%3Cdev.metron.apache.org%3E)
   
   The discussion concluded with a solution that would:
   1. Address the regression
   2. Make the handling of original_string more generalized and universally 
applied across all parsers
   3. Allow individual parsers to override the original_string, if desired.
   4. Not screw up parser chaining
   
   This is discussed in more detail in the README, but here's a rundown of the 
settings:
   
   - Global config option added `parser.original.string.global`, defaults to 
`true`. By default, this will now enable the parser runner to append an 
`original_string` using the true raw source message. Note, for backwards 
compatibility, the implementation uses a putIfAbsent approach. This approach is 
to keep from completely breaking parser chaining due to the way enveloped 
message parsing works (hint: it's our only real special system-level case). 
Setting the property to false will mean the runner will not attempt to add 
`original_string` at all.
   - Modification to JsonMapParser to accept a new configuration option 
`overrideOriginalString`. The default of `false` addresses the regression 
introduced by jsonpquery and will not attempt to add an `original_string`. 
Setting this value to `true` will effectively override the global setting and 
apply an `original_string` per message generated that reflects the existing 
functionality.
   
   I also addressed a few random doc and test issues I noticed while modifying 
the code for this PR. e.g. global config sub-section links, missing/omitted 
test assertions, etc.
   
   More exhaustive test instructions to follow. In the meantime, this PR is 
ready for code and doc review. Most importantly, the functionality I've 
outlined here and in the README's should be reviewed. I have not modified any 
of the parsers besides JsonMapParser. If we think they should have their 
default functionality changed, I propose we open a separate DISCUSS thread for 
this and provide a migration path for existing users that may or may not be 
impacted by any change. I suspect that only JSON parsers should be affected.
   
   Considering the origin and impact of this change, I presume this will 
warrant a mention in Upgrading.md. I have not done this yet, but would like to 
hear any specific concerns or feedback on wording.
   
   ## Pull Request Checklist
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   
   ### For code changes:
   - [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
 ```
 mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
 ```
   
   - [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - n/a If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered by building and verifying the site-book? If not then run the 
following commands and the verify changes via 
`site-book/target/site/index.html`:
   
 ```
 cd site-book
 mvn site
 ```
   
   - n/a Have you ensured that any documentation diagrams have been updated, 
along with their source files, using [draw.io](https://www.draw.io/)? See 
[Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 for instructions.
   


This is an automated 

[GitHub] [metron] ottobackwards commented on issue #1408: METRON-2118: Added a LEEF parser

[GitBox]

ottobackwards commented on issue #1408: METRON-2118: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408#issuecomment-493550026
 
 
   Simon this looks very nice, I'm going to review.
   
   One thing I would ask if you can do an integration test, with integration 
test data and a configuration for this as we have done with the other parsers?
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] simonellistonball opened a new pull request #1408: Added a LEEF parser

[GitBox]

simonellistonball opened a new pull request #1408: Added a LEEF parser
URL: https://github.com/apache/metron/pull/1408
 
 
   ## Contributor Comments
   LEEF is a popular format in IBM shops as it is the default supported by 
Qradar. In a number of ways it is similar to CEF. This PR supports LEEF 1.0 and 
2.0 per the IBM guide at 
https://www.ibm.com/support/knowledgecenter/SS42VS_DSM/c_LEEF_Format_Guide_intro.html
 and also some found in the wild examples which are technically not up to the 
IBM spec, and are much closer to the CEF spec.
   
   The CEF parser has been slightly refactored in this effort to expose CEF 
extension parsing for reuse in the LEEF parser for in the wild examples of CEF 
style 'delimiters'.
   
   This has been tested against a variety of samples from public sources, and 
from synthetic data generated according to the spec in new unit tests.
   
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron.  
   Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
   Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  
   
   
   In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [X] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   
   ### For code changes:
   - [X] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [X] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
 ```
 mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
 ```
   
   - [X] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered by building and verifying the site-book? If not then run the 
following commands and the verify changes via 
`site-book/target/site/index.html`:
   
 ```
 cd site-book
 mvn site
 ```
   
   - [ ] Have you ensured that any documentation diagrams have been updated, 
along with their source files, using [draw.io](https://www.draw.io/)? See 
[Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 for instructions.
   
    Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   It is also recommended that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (METRON-2118) Parser for LEEF data

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-2118:
---

 Summary: Parser for LEEF data
 Key: METRON-2118
 URL: https://issues.apache.org/jira/browse/METRON-2118
 Project: Metron
  Issue Type: Bug
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball


LEEF is a popular format mainly in environments using Qradar as a SIEM. It 
would be useful for Metron to be able to consume data from Qradar, and related 
collectors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [metron] mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset

[GitBox]

mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default 
Charset
URL: https://github.com/apache/metron/pull/1341#issuecomment-493500295
 
 
   I think setting a default to `UTF-8` in the parsers and documenting it would 
be the way to go. Provide a per-sensor config option, e.g. `inputDataCharset` 
that lets users configure it for the edge case. Emphasis on per-sensor because 
99/100 sensors will probably be `UTF-8`, and then one will be something wild 
like `EBCDIC` because hey, why not.
   
   In general, I agree that it would be odd for any network sensors to be set 
to anything other than `UTF-8`. We're probably looking at other sources of 
mischief, though. A couple examples could be streaming and bulk loaded 
enrichments. I would not be surprised to find someone at some point loading 
`ISO-8859-1` or `Windows-1252`. In multiple big data projects prior to Metron I 
had to deal with encodings like this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset

[GitBox]

justinleet commented on issue #1341: METRON-614: Eliminate use of the default 
Charset
URL: https://github.com/apache/metron/pull/1341#issuecomment-493480413
 
 
   Ahh good call, it's been long enough that I'd forgotten about that 
discussion. There should definitely at least be a README addition that I'll add.
   
   Re: non-UTF-8 inbound data sets, that's potentially a fair problem, although 
I don't personally know what the circumstances would be where non-UTF-8 string 
data is coming from (maybe Latin-1?).  Seems like the only real way to deal 
with this is to make it configurable at the parser level, or otherwise mixing 
incoming charset encodings is a problem (Which I think it would be right now, 
if everything is just using platform default, right? Double check my thinking 
on that). Then the parser itself just reads with whatever character encoding.
   
   At that point, stuff like say GrokParser would need do something like `new 
InputStreamReader(commonInputStream, getEncoding());` or similar.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default Charset

[GitBox]

mmiklavc commented on issue #1341: METRON-614: Eliminate use of the default 
Charset
URL: https://github.com/apache/metron/pull/1341#issuecomment-493459303
 
 
   Probably worth referencing this - 
https://lists.apache.org/thread.html/55e57410cb8cd467a51545e4ae0f9f67d32312cc3f9e1afa144552f4@%3Cdev.metron.apache.org%3E
   
   Just to follow up - are there any concerns with, for example, non-UTF-8 
inbound data charsets? I think it makes sense for us to normalize, however we 
may run into trouble in our parsers, e.g 
https://github.com/apache/metron/pull/1341/files#diff-e5a5c182fb529e5e2d93fc9f06ce1012.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (METRON-2117) [UI] Aligning models to grouping feature

[Tibor Meller (JIRA)]

Tibor Meller created METRON-2117:


 Summary: [UI] Aligning models to grouping feature
 Key: METRON-2117
 URL: https://issues.apache.org/jira/browse/METRON-2117
 Project: Metron
  Issue Type: Sub-task
Reporter: Tibor Meller
Assignee: Tibor Meller


Introduce a new model for parser groups and align existing ones to the new 
functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [metron] justinleet commented on issue #1341: METRON-614: Eliminate use of the default Charset

[GitBox]

justinleet commented on issue #1341: METRON-614: Eliminate use of the default 
Charset
URL: https://github.com/apache/metron/pull/1341#issuecomment-493453296
 
 
   @mmiklavc @nickwallen @ottobackwards I merged master post-release and ran up 
in full dev again. Is everyone still good with the changes?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (METRON-2116) [UI] Removing redundant AppConfigService

[Tibor Meller (JIRA)]

Tibor Meller created METRON-2116:


 Summary: [UI] Removing redundant AppConfigService
 Key: METRON-2116
 URL: https://issues.apache.org/jira/browse/METRON-2116
 Project: Metron
  Issue Type: Sub-task
Reporter: Tibor Meller


Management UI contains two separated solutions for getting application configs.
In this changeset, I remove the older and less capable version and wiring in 
the newer AppConfigService.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-2116) [UI] Removing redundant AppConfigService

[Tibor Meller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tibor Meller reassigned METRON-2116:


Assignee: Tibor Meller

> [UI] Removing redundant AppConfigService
> 
>
> Key: METRON-2116
> URL: https://issues.apache.org/jira/browse/METRON-2116
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Tibor Meller
>Assignee: Tibor Meller
>Priority: Major
>
> Management UI contains two separated solutions for getting application 
> configs.
> In this changeset, I remove the older and less capable version and wiring in 
> the newer AppConfigService.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-2115) [UI] Aligning UI to the parser aggregation API

[Tibor Meller (JIRA)]

Tibor Meller created METRON-2115:


 Summary: [UI] Aligning UI to the parser aggregation API
 Key: METRON-2115
 URL: https://issues.apache.org/jira/browse/METRON-2115
 Project: Metron
  Issue Type: Sub-task
Reporter: Tibor Meller
Assignee: Tibor Meller


Parser Aggregation feature introduced a set of new endpoints on REST API.
As part of this changeset, I prepare the UI to start using these endpoints.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-2114) [UI] Moving components to sensor parser module

[Tibor Meller (JIRA)]

Tibor Meller created METRON-2114:


 Summary: [UI] Moving components to sensor parser module
 Key: METRON-2114
 URL: https://issues.apache.org/jira/browse/METRON-2114
 Project: Metron
  Issue Type: Sub-task
Reporter: Tibor Meller
Assignee: Tibor Meller


As part of this changeset, I'm moving the components which belong to parser 
configuration feature to a separated module file. This makes the code cleaner 
and gives us the opportunity to initialize the parser config state inside a 
separated module. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [metron] sardell commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default

[GitBox]

sardell commented on issue #1393: METRON-2092: Config UI does not require you 
to set a grok timestamp field by default
URL: https://github.com/apache/metron/pull/1393#issuecomment-493421928
 
 
   @ruffle1986 You were right, it must've been something weird with my cache 
(or maybe it was just user error :flushed:). It works as described for me now 
when testing locally on full dev. While I can verify that the UI works, I would 
like another committer who is more familiar with the timestampField to verify 
that an unformatted string value is correct. It just seems weird to me that a 
timestamp field wouldn't enforce timestamp formatting, and would default to a 
string value of "timestamp." Again, this probably stems from my own ignorance 
of that field and what it is used for on the backend.
   
   +1 pending another committer verifying that this is indeed how the field 
should be.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ruffle1986 commented on issue #1380: METRON-1253: Manual pasting of timestamps into the timestamp picker

[GitBox]

ruffle1986 commented on issue #1380: METRON-1253: Manual pasting of timestamps 
into the timestamp picker
URL: https://github.com/apache/metron/pull/1380#issuecomment-493369951
 
 
   @sardell Alright, I put the trigger on the calendar icon, but in that case I 
had to remove the toggle listener from the component. The reason is because it 
has conflicted with Pikaday and Pikaday doesn't have this "toggling" capability 
by by default and there's no option to turn it on so we lost that behaviour. 
But I think it's not a big deal so I ended going without the toggle.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] asfgit closed pull request #1388: METRON-2085: [UI] Alerts UI Details Pane: naming meta alerts is broken

[GitBox]

asfgit closed pull request #1388: METRON-2085: [UI] Alerts UI Details Pane: 
naming meta alerts is broken
URL: https://github.com/apache/metron/pull/1388
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ruffle1986 commented on issue #1380: METRON-1253: Manual pasting of timestamps into the timestamp picker

[GitBox]

ruffle1986 commented on issue #1380: METRON-1253: Manual pasting of timestamps 
into the timestamp picker
URL: https://github.com/apache/metron/pull/1380#issuecomment-493359168
 
 
   @sardell Good point and doesn't sound like a big effort. Let me check this 
out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ruffle1986 edited a comment on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default

[GitBox]

ruffle1986 edited a comment on issue #1393: METRON-2092: Config UI does not 
require you to set a grok timestamp field by default
URL: https://github.com/apache/metron/pull/1393#issuecomment-493358360
 
 
   @sardell weird. for me, it's there. looks like a caching issue. 
   ![Screen Shot 2019-05-17 at 9 49 
57](https://user-images.githubusercontent.com/2196208/57911876-3f568c00-7889-11e9-8622-b0c60df597a2.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ruffle1986 edited a comment on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default

[GitBox]

ruffle1986 edited a comment on issue #1393: METRON-2092: Config UI does not 
require you to set a grok timestamp field by default
URL: https://github.com/apache/metron/pull/1393#issuecomment-493358360
 
 
   @sardell weird. for me, it's there. looks like a caching issue. There's no 
additional step that I know about.
   
   ![Screen Shot 2019-05-17 at 9 49 
57](https://user-images.githubusercontent.com/2196208/57911876-3f568c00-7889-11e9-8622-b0c60df597a2.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [metron] ruffle1986 commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default

[GitBox]

ruffle1986 commented on issue #1393: METRON-2092: Config UI does not require 
you to set a grok timestamp field by default
URL: https://github.com/apache/metron/pull/1393#issuecomment-493358360
 
 
   @sardell weird. for me it's there. looks like a caching issue. 
   ![Screen Shot 2019-05-17 at 9 49 
57](https://user-images.githubusercontent.com/2196208/57911876-3f568c00-7889-11e9-8622-b0c60df597a2.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services