[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634167#comment-16634167
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1175


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633118#comment-16633118
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1175
  
LGTM +1, thanks Otto this is awesome!


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632979#comment-16632979
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r221426956
  
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec 
---
@@ -590,6 +591,8 @@ chkconfig --del metron-alerts-ui
 %changelog
 * Thu Aug 30 2018 Apache Metron  - 0.6.1
 - Update compiled css file name for Alerts UI
+* Fri Aug 24 2018 Apache Metron  - 0.5.1
--- End diff --

fixed


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632956#comment-16632956
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r221425083
  
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec 
---
@@ -590,6 +591,8 @@ chkconfig --del metron-alerts-ui
 %changelog
 * Thu Aug 30 2018 Apache Metron  - 0.6.1
 - Update compiled css file name for Alerts UI
+* Fri Aug 24 2018 Apache Metron  - 0.5.1
+- Add syslog5424 parser
 * Tue Aug 21 2018 Apache Metron  - 0.6.1
 - Add Profiler for REPL
 * Tue Aug 14 2018 Apache Metron  - 0.5.1
--- End diff --

It [looks 
like](https://github.com/apache/metron/blame/Metron_0.6.0/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec)
 all of these are newer than 0.6.0, but we haven't updated to 0.6.1, 0.7.0, 
etc. across the codebase.  @nickwallen does it sound right to change the Spark 
profiler line item to be 0.6.1?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632955#comment-16632955
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r221425063
  
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec 
---
@@ -590,6 +591,8 @@ chkconfig --del metron-alerts-ui
 %changelog
 * Thu Aug 30 2018 Apache Metron  - 0.6.1
 - Update compiled css file name for Alerts UI
+* Fri Aug 24 2018 Apache Metron  - 0.5.1
--- End diff --

Shouldn't this be 0.6.1?  


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632540#comment-16632540
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1175
  
@JonZeolla let me know if you are all set



> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631878#comment-16631878
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1175
  
I agree, +1 by inspection, pending Travis. This is definitely something I'm 
glad to see go in.


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631873#comment-16631873
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1175
  
This looks good to me; I'm +1 on it by inspection.  Good job, otto ;)


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631827#comment-16631827
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1175
  
Hopefully it is all set now



> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619877#comment-16619877
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1175
  
Can you deconflict please?  package-lock.json is not happy.


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602913#comment-16602913
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1175
  
New upstream integrated now.


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601770#comment-16601770
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1175
  
Fixed in upstream 0.0.8
I will update when it posts / tomorrow


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601738#comment-16601738
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1175
  
@kylerichardson Let's talk over on the upstream issue


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601641#comment-16601641
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1175
  
Filed an issue upstream.

palindromicity/simple-syslog-5424#15


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601257#comment-16601257
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user kylerichardson commented on the issue:

https://github.com/apache/metron/pull/1175
  
I can contribute a PR to upstream for this if that would be helpful too.

On Sun, Sep 2, 2018 at 1:08 PM Otto Fowler  wrote:

> Can you log an issue in upstream with your excellent description please?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>
-- 
-Kyle



> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601254#comment-16601254
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1175
  
Can you log an issue in upstream with your excellent description please?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599252#comment-16599252
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1175
  
It looks like the upstream palindromicity/simple-syslog-5424 assumes that 
the PRI will be included in a log.  While this is in the spec/RFC and sent on 
the network, it is standard practice to not write this to disk, but instead it 
is used by syslog software to choose which file to write it to, and strip it 
before writing to disk so the first component of the log is the date/timestamp. 
 Situations where syslog is pulled from disk and sent into Metron will all fail 
with a syntax error.  I would suggest that you work with the upstream lib 
(yourself) to make the PRI field optional =)

Some evidence of my claims:
 * rsyslog documentation explaining that PRI fields are sent but not 
recorded 
[here](https://www.rsyslog.com/doc/v8-stable/tutorials/recording_pri.html).
 * The rsyslog built-in templates for writing to disk exclude PRI 
([details](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-templates#brid-template-examples)).
 * Even legacy file formats only include PRI when forwarding 
([details](https://rsyslog-doc.readthedocs.io/en/latest/configuration/templates.html#legacy-string-based-template-samples)).
 * [Back in 
2010](https://serverfault.com/questions/110678/syslog-ng-how-to-log-severity-facility)
 the SUSE syslog-ng format defaults to writing without PRI.


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596422#comment-16596422
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213706134
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

Great point, latest commit has the change and the test


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596310#comment-16596310
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213670531
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

Based on looking at the docs for the syslog library, it looks like it's 
possible to not have a timestamp (or to not validly parse a timestamp).  If we 
have a nil for timestamp here, we probably want to default like we do 
elsewhere, which is to current time.  What do you think?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596304#comment-16596304
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213669448
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

If we aren't able to parse the timestamp here, I presume there will be an 
exception in the parser, right?  I just want to make sure there's no way for 
the parser to fail to return a timestamp.


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593994#comment-16593994
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213051917
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

yeah


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593974#comment-16593974
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213048634
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

Is it worth moving the SyslogParserBuilder to the configure() method and 
just storing it off, rather than recreating every time?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593970#comment-16593970
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213048275
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

Created https://issues.apache.org/jira/browse/METRON-1753


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593921#comment-16593921
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213039514
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

I don't think I want to touch all the parsers for this PR , there might be 
more than one parser follow on depending on how many have configurations.  Can 
you create  a jira or shall I?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593874#comment-16593874
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213027440
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

I'm okay with a follow-on (particularly if it's a lot of work, or if it's 
risky).  I'd just make sure the ticket includes updating any parsers to use the 
functionality.

It's probably not too hard to add (and I may be being incredibly blasé 
about something here).  I'd expect it to be:

* Add update method to `MessageParser`
* Implement `reloadCallback` to update the parser configs and make sure 
anything else in the bolt gets updated as necessary.
* Tests and docs


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593826#comment-16593826
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213016887
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

You ok with that as part of this pr?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593700#comment-16593700
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r212985523
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  /**
+   * The NilPolicy specifies how the parser handles missing fields in the 
return
+   * It can:
+   *  Omit the fields
+   *  Have a value of '-' ( as spec )
+   *  Have null values for the fields
+   * The default is to omit the fields from the return set.
+   */
+  private NilPolicy nilPolicy = NilPolicy.OMIT;
+
+  @Override
+  public void configure(Map config) {
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+
+  SyslogParser parser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
--- End diff --

This is half comment, half question.

It seems odd to recreate the SyslogParserBuilder every `parser`. I dug in a 
bit, and I expected (personally expected, not "Metron itself expects x 
condition") this process to be somewhat similar to the enrichment where the 
bolt implements the `reloadCallback` method (e.g. 
[UnifiedEnrichmentBolt](https://github.com/apache/metron/blob/1d95b8316a18097747be116a0276c56b894fb79c/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L229))
 which would then delegate updating the config of the underlying pieces.

E.g. here I would expect the `SyslogParser` to be created a priori and then 
when the parser config gets updated `reloadCallback` would be called and this 
would be updated (and in this case recreated).

Looking into it a bit further, it looks like ParserBolt has the appropriate 
method passed down to it, but chooses not to implement it.  I suspect it's 
because nothing the underlying parsers don't update configs, although I didn't 
check all of them.

Would it be reasonable to get that setup in the ParserBolt and then handle 
the `SyslogParser` object that way, rather than recreating it every time?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)