[jira] [Commented] (METRON-1060) Add performance timing logging to enrichment topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105854#comment-16105854
 ] 

ASF GitHub Bot commented on METRON-1060:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/665
  
+1 -> Love this.  Really great work.  Makes me think that we want to have 
some kind of topical area of the site-book or developers documentation ( like 
this would go with testing ) but that is another thing.

Thanks!


> Add performance timing logging to enrichment topology
> -
>
> Key: METRON-1060
> URL: https://issues.apache.org/jira/browse/METRON-1060
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105745#comment-16105745
 ] 

ASF GitHub Bot commented on METRON-777:
---

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r129968508
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/VFSClassloaderUtil.java
 ---
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator;
+import org.apache.commons.vfs2.*;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Optional;
+
+public class VFSClassloaderUtil {
+  private static final Logger LOG = 
LoggerFactory.getLogger(VFSClassloaderUtil.class);
+
+  /**
+   * Create a FileSystem manager suitable for our purposes.
+   * This manager supports files of the following types:
+   * * res - resource files
+   * * jar
+   * * tar
+   * * bz2
+   * * tgz
+   * * zip
+   * * HDFS
+   * * FTP
+   * * HTTP/S
+   * * file
+   * @return
+   * @throws FileSystemException
+   */
+  public static FileSystemManager generateVfs() throws FileSystemException 
{
+return generateVfs(null);
+  }
+  public static FileSystemManager generateVfs(String 
jarExtensionToRegister) throws FileSystemException {
+DefaultFileSystemManager vfs = new DefaultFileSystemManager();
+
+if(jarExtensionToRegister == null || 
StringUtils.isBlank(jarExtensionToRegister) ) {
+  vfs.addExtensionMap(jarExtensionToRegister, "jar");
+  vfs.addProvider(jarExtensionToRegister, new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+}
--- End diff --

This doesn't seem right.  Did you perhaps intend the negative of the 
conditional, ie
`if (jarExtensionToRegister != null && 
!StringUtils.isBlank(jarExtensionToRegister) ) {...`


> Create a plugin system for Metron based on 'NAR'
> 
>
> Key: METRON-777
> URL: https://issues.apache.org/jira/browse/METRON-777
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The success of the Metron project will be greatly dependent on community 
> participation, and with that the ability to adapt and extend Metron without 
> having to maintain a fork of the project.
> As organizations and individuals look to extend the Metron system with custom 
> parsers, enrichments, and stellar functions that may be proprietary in 
> nature, the ability to develop and deploy these extensions outside the Metron 
> code base is critically important.
> To that end, and after community discussion and proposal we create or 
> formalize the 'plugin' development story in Metron.  
> The proposal is to adapt the Apache Nifi NAR system for use in Metron.  This 
> will provide the system with:
> * archetype(s) for developer projects and independent development
> * defined packaging and metadata for 'plugin' products
> * loading and instantiation with classloader isolation capabilities
> * removing the necessity for shading plugin jars
> These capabilities will also enable other features, such as plugin lifecycle, 
> plugin configuration+redeployment, and other things.
> The plugin archetypes and their installation will be a followon



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105746#comment-16105746
 ] 

ASF GitHub Bot commented on METRON-777:
---

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r129992712
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/FileUtils.java ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSelector;
+import org.apache.commons.vfs2.FileSystemException;
+import org.slf4j.Logger;
+
+import java.io.InputStream;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class FileUtils  {
+
+  private static FileUtilities utilities;
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  public static void init(FileUtilities utils){
+utilities = utils;
+inited.set(true);
+  }
--- End diff --

Let's talk off-line about making fuller use of Commons-VFS to fix these 
abstractions.


> Create a plugin system for Metron based on 'NAR'
> 
>
> Key: METRON-777
> URL: https://issues.apache.org/jira/browse/METRON-777
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The success of the Metron project will be greatly dependent on community 
> participation, and with that the ability to adapt and extend Metron without 
> having to maintain a fork of the project.
> As organizations and individuals look to extend the Metron system with custom 
> parsers, enrichments, and stellar functions that may be proprietary in 
> nature, the ability to develop and deploy these extensions outside the Metron 
> code base is critically important.
> To that end, and after community discussion and proposal we create or 
> formalize the 'plugin' development story in Metron.  
> The proposal is to adapt the Apache Nifi NAR system for use in Metron.  This 
> will provide the system with:
> * archetype(s) for developer projects and independent development
> * defined packaging and metadata for 'plugin' products
> * loading and instantiation with classloader isolation capabilities
> * removing the necessity for shading plugin jars
> These capabilities will also enable other features, such as plugin lifecycle, 
> plugin configuration+redeployment, and other things.
> The plugin archetypes and their installation will be a followon



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1060) Add performance timing logging to enrichment topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105673#comment-16105673
 ] 

ASF GitHub Bot commented on METRON-1060:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/665
  
@ottobackwards Just submitted. I've updated the testing notes above as 
well. Note the added documentation in the metron-common README.


> Add performance timing logging to enrichment topology
> -
>
> Key: METRON-1060
> URL: https://issues.apache.org/jira/browse/METRON-1060
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-271) Add graph DB to the platform

[Nick Allen (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105533#comment-16105533
 ] 

Nick Allen commented on METRON-271:
---

I think that would be a viable option to explore.  Feel free to dig into this.

> Add graph DB to the platform
> 
>
> Key: METRON-271
> URL: https://issues.apache.org/jira/browse/METRON-271
> Project: Metron
>  Issue Type: New Feature
>Reporter: James Sirota
>Assignee: Nick Allen
>  Labels: 0.2.2BETA, METRON_ML
>
> I propose adding a graph database (Titan or others) so we can use graph 
> mining as feature inputs to some of our models and anomaly detectors



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1071) Create CONTRIBUTING.md

[Justin Leet (JIRA)]

Justin Leet created METRON-1071:
---

 Summary: Create CONTRIBUTING.md
 Key: METRON-1071
 URL: https://issues.apache.org/jira/browse/METRON-1071
 Project: Metron
  Issue Type: Improvement
Reporter: Justin Leet
Assignee: Justin Leet


The idea is to have a document on contributing to Metron that lives alongside 
our code (and we can then move away from the wiki).  This document should have 
a couple things in it:

* What we look for from code contributions
* How people can actually contribute code
* How people can contribute, even without code (e.g. reviews)
* Helpful things like setting up Travis on personal repos to avoid full testing 
time locally.

It should also integrate nicely with the site-book, so just make sure 
everything plays nicely.

See: https://github.com/blog/1184-contributing-guidelines



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1060) Add performance timing logging to enrichment topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105490#comment-16105490
 ] 

ASF GitHub Bot commented on METRON-1060:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/665
  
Nice, tag me when it is in


> Add performance timing logging to enrichment topology
> -
>
> Key: METRON-1060
> URL: https://issues.apache.org/jira/browse/METRON-1060
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1069) Ambari MPack documentation around development strategies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105432#comment-16105432
 ] 

ASF GitHub Bot commented on METRON-1069:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/673#discussion_r130153573
  
--- Diff: metron-deployment/packaging/ambari/README.md ---
@@ -0,0 +1,315 @@
+# Ambari Management Pack Development
+Typically, Ambari Management Pack development will be done in the Vagrant 
environments. These instructions are specific to Vagrant, but can be adapted 
for other environemnts (e.g. make sure to be on the correct nodes for server vs 
agent files)
+
+There is an `mpack.json` file which describes what services the mpack will 
contains, versions, etc.
+
+Alongside this are two directories, `addon-services` and `common-services`.
+
+The layout of `/common-services/METRON.CURRENT` is
+* `/configuration`
+  * This contains a set of `*-env.xml` files, relevent to particular 
components or the service as a whole.  These are where properties are defined.
+* `/package`
+  * `/files`
+* Contains files that get used as provided, in particular 
Elasticsearch templates.
+  * `/scripts`
+* A set of Python scripts that interface with Ambari to manage setup 
and install
+* `/params`
+  * A set of Python scripts for managing parameters from the 
`*-env.xml` files
+* `/templates`
+  * A set of Jinja template files which can be populated with 
properties
+* `/quicklinks`
+  * Contains `quicklinks.json` to define the Ambari quicklinks that should 
be used in the UI
+* `themes`
+  * Manages the Ambari UI themes blocks for organizing the configuration 
by relevant area.
+* `kerberos.json`
+  * Defines the keytabs and other Kerberos configuration to be used when 
Kerberizing a cluster
+* `metainfo.xml`
+  * Defines the METRON service, along with required packages, services, 
etc.
+* `service_advisor.py`
+  * Handles component layout and validation, along with handling some 
configurations for other services or that needs configs from other services.
+
+The layout of `/addon-services/METRON.CURRENT` is
+* `/repos`
+  * Contains `repoinfo.xml` that defines repositories to install packages 
from
+* `metainfo.xml`
+  * Limited info version of `/common-services/METRON.CURRENT/metainfo.xml`
+* `role_command_order.json`
+  * Defines the order of service startup and other actions relative to 
each other.
+
+## Adding a new property
+1. Add the property to the appropriate `*-env.xml` file found in 
`METRON.CURRENT/configuration`.
+  ```
+
+new_property
+New Property description
+Default Value
+New Property Pretty Name
+
+  ```
+
+2.  Reference the property in 
`METRON.CURRENT/package/scriptes/params/params_linux.py`, unless it will be 
used in Ambari's status command.  It will be stored in a variable. The name 
doesn't have to match, but it's preferred that it does.
+Make sure to use replace `metron-env` the correct `*-env` file, as noted 
above.
+  ```
+  new_property = config['configurations']['metron-env']['new_property']
+  ```
+If this property will be used in the status command, instead make this 
change in `METRON.CURRENT/package/scriptes/params/status_params.py`.
+Afterwards, in `params_linux.py`, reference the new property:
+  ```
+  new_property = status_params.new_property
+  ```
+This behavior is because Ambari doesn't send all parameters to the status, 
so it needs to be explicitly provided.
+
+3. Ambari master services can then import the params:
+
+  ```
+  from params import params
+  env.set_params(params)
+  ```
+
+4. The `*_commands.py` files receive the params as an input from the 
master services.  Once this is done, they can be accessed via the variable we 
set above:
+  ```
+  self.__params.new_property
+  ```
+
+
+### Env file property walkthrough
+To illustrate how property files are carried through to the scripts for 
our services, we'll run through an existing property, `metron_apps_hdfs_dir`.
+
+ Defining the property
+First the property appears in the appropriate `*-env.xml`, in this case 
`METRON.CURRENT/configuration/metron-env.xml`.
+```
+
+metron_apps_hdfs_dir
+/apps/metron
+Metron apps HDFS dir
+Metron apps HDFS dir
+
+```
+
+This defines several things
+1. The name of the property we'll be referencing it by in our code.
+1. The default value of the property.
+1. The description of the property for the Ambari UI.
+1. The pretty name that will be shown in Ambari for 

[jira] [Commented] (METRON-1069) Ambari MPack documentation around development strategies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105413#comment-16105413
 ] 

ASF GitHub Bot commented on METRON-1069:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/673#discussion_r130151964
  
--- Diff: metron-deployment/packaging/ambari/README.md ---
@@ -0,0 +1,315 @@
+# Ambari Management Pack Development
+Typically, Ambari Management Pack development will be done in the Vagrant 
environments. These instructions are specific to Vagrant, but can be adapted 
for other environemnts (e.g. make sure to be on the correct nodes for server vs 
agent files)
+
+There is an `mpack.json` file which describes what services the mpack will 
contains, versions, etc.
+
+Alongside this are two directories, `addon-services` and `common-services`.
+
+The layout of `/common-services/METRON.CURRENT` is
+* `/configuration`
+  * This contains a set of `*-env.xml` files, relevent to particular 
components or the service as a whole.  These are where properties are defined.
+* `/package`
+  * `/files`
+* Contains files that get used as provided, in particular 
Elasticsearch templates.
+  * `/scripts`
+* A set of Python scripts that interface with Ambari to manage setup 
and install
+* `/params`
+  * A set of Python scripts for managing parameters from the 
`*-env.xml` files
+* `/templates`
+  * A set of Jinja template files which can be populated with 
properties
+* `/quicklinks`
+  * Contains `quicklinks.json` to define the Ambari quicklinks that should 
be used in the UI
+* `themes`
+  * Manages the Ambari UI themes blocks for organizing the configuration 
by relevant area.
+* `kerberos.json`
+  * Defines the keytabs and other Kerberos configuration to be used when 
Kerberizing a cluster
+* `metainfo.xml`
+  * Defines the METRON service, along with required packages, services, 
etc.
+* `service_advisor.py`
+  * Handles component layout and validation, along with handling some 
configurations for other services or that needs configs from other services.
+
+The layout of `/addon-services/METRON.CURRENT` is
+* `/repos`
+  * Contains `repoinfo.xml` that defines repositories to install packages 
from
+* `metainfo.xml`
+  * Limited info version of `/common-services/METRON.CURRENT/metainfo.xml`
+* `role_command_order.json`
+  * Defines the order of service startup and other actions relative to 
each other.
+
+## Adding a new property
+1. Add the property to the appropriate `*-env.xml` file found in 
`METRON.CURRENT/configuration`.
+  ```
+
+new_property
+New Property description
+Default Value
+New Property Pretty Name
+
+  ```
+
+2.  Reference the property in 
`METRON.CURRENT/package/scriptes/params/params_linux.py`, unless it will be 
used in Ambari's status command.  It will be stored in a variable. The name 
doesn't have to match, but it's preferred that it does.
+Make sure to use replace `metron-env` the correct `*-env` file, as noted 
above.
+  ```
+  new_property = config['configurations']['metron-env']['new_property']
+  ```
+If this property will be used in the status command, instead make this 
change in `METRON.CURRENT/package/scriptes/params/status_params.py`.
+Afterwards, in `params_linux.py`, reference the new property:
+  ```
+  new_property = status_params.new_property
+  ```
+This behavior is because Ambari doesn't send all parameters to the status, 
so it needs to be explicitly provided.
+
+3. Ambari master services can then import the params:
+
+  ```
+  from params import params
+  env.set_params(params)
+  ```
+
+4. The `*_commands.py` files receive the params as an input from the 
master services.  Once this is done, they can be accessed via the variable we 
set above:
+  ```
+  self.__params.new_property
+  ```
+
+
+### Env file property walkthrough
+To illustrate how property files are carried through to the scripts for 
our services, we'll run through an existing property, `metron_apps_hdfs_dir`.
+
+ Defining the property
+First the property appears in the appropriate `*-env.xml`, in this case 
`METRON.CURRENT/configuration/metron-env.xml`.
+```
+
+metron_apps_hdfs_dir
+/apps/metron
+Metron apps HDFS dir
+Metron apps HDFS dir
+
+```
+
+This defines several things
+1. The name of the property we'll be referencing it by in our code.
+1. The default value of the property.
+1. The description of the property for the Ambari UI.
+1. The pretty name that will be shown in Ambari 

[jira] [Commented] (METRON-1069) Ambari MPack documentation around development strategies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105409#comment-16105409
 ] 

ASF GitHub Bot commented on METRON-1069:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/673#discussion_r130151618
  
--- Diff: metron-deployment/packaging/ambari/README.md ---
@@ -0,0 +1,315 @@
+# Ambari Management Pack Development
+Typically, Ambari Management Pack development will be done in the Vagrant 
environments. These instructions are specific to Vagrant, but can be adapted 
for other environemnts (e.g. make sure to be on the correct nodes for server vs 
agent files)
+
+There is an `mpack.json` file which describes what services the mpack will 
contains, versions, etc.
+
+Alongside this are two directories, `addon-services` and `common-services`.
+
+The layout of `/common-services/METRON.CURRENT` is
+* `/configuration`
+  * This contains a set of `*-env.xml` files, relevent to particular 
components or the service as a whole.  These are where properties are defined.
+* `/package`
+  * `/files`
+* Contains files that get used as provided, in particular 
Elasticsearch templates.
+  * `/scripts`
+* A set of Python scripts that interface with Ambari to manage setup 
and install
+* `/params`
+  * A set of Python scripts for managing parameters from the 
`*-env.xml` files
+* `/templates`
+  * A set of Jinja template files which can be populated with 
properties
+* `/quicklinks`
+  * Contains `quicklinks.json` to define the Ambari quicklinks that should 
be used in the UI
+* `themes`
+  * Manages the Ambari UI themes blocks for organizing the configuration 
by relevant area.
+* `kerberos.json`
+  * Defines the keytabs and other Kerberos configuration to be used when 
Kerberizing a cluster
+* `metainfo.xml`
+  * Defines the METRON service, along with required packages, services, 
etc.
+* `service_advisor.py`
+  * Handles component layout and validation, along with handling some 
configurations for other services or that needs configs from other services.
+
+The layout of `/addon-services/METRON.CURRENT` is
+* `/repos`
+  * Contains `repoinfo.xml` that defines repositories to install packages 
from
+* `metainfo.xml`
+  * Limited info version of `/common-services/METRON.CURRENT/metainfo.xml`
+* `role_command_order.json`
+  * Defines the order of service startup and other actions relative to 
each other.
+
+## Adding a new property
+1. Add the property to the appropriate `*-env.xml` file found in 
`METRON.CURRENT/configuration`.
+  ```
+
+new_property
+New Property description
+Default Value
+New Property Pretty Name
+
+  ```
+
+2.  Reference the property in 
`METRON.CURRENT/package/scriptes/params/params_linux.py`, unless it will be 
used in Ambari's status command.  It will be stored in a variable. The name 
doesn't have to match, but it's preferred that it does.
+Make sure to use replace `metron-env` the correct `*-env` file, as noted 
above.
+  ```
+  new_property = config['configurations']['metron-env']['new_property']
+  ```
+If this property will be used in the status command, instead make this 
change in `METRON.CURRENT/package/scriptes/params/status_params.py`.
+Afterwards, in `params_linux.py`, reference the new property:
+  ```
+  new_property = status_params.new_property
+  ```
+This behavior is because Ambari doesn't send all parameters to the status, 
so it needs to be explicitly provided.
+
+3. Ambari master services can then import the params:
+
+  ```
+  from params import params
+  env.set_params(params)
+  ```
+
+4. The `*_commands.py` files receive the params as an input from the 
master services.  Once this is done, they can be accessed via the variable we 
set above:
+  ```
+  self.__params.new_property
+  ```
+
+
+### Env file property walkthrough
+To illustrate how property files are carried through to the scripts for 
our services, we'll run through an existing property, `metron_apps_hdfs_dir`.
+
+ Defining the property
+First the property appears in the appropriate `*-env.xml`, in this case 
`METRON.CURRENT/configuration/metron-env.xml`.
+```
+
+metron_apps_hdfs_dir
+/apps/metron
+Metron apps HDFS dir
+Metron apps HDFS dir
+
+```
+
+This defines several things
+1. The name of the property we'll be referencing it by in our code.
+1. The default value of the property.
+1. The description of the property for the Ambari UI.
+1. The pretty name that will be shown in Ambari 

[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105406#comment-16105406
 ] 

ASF GitHub Bot commented on METRON-777:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
METRON-1070



> Create a plugin system for Metron based on 'NAR'
> 
>
> Key: METRON-777
> URL: https://issues.apache.org/jira/browse/METRON-777
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The success of the Metron project will be greatly dependent on community 
> participation, and with that the ability to adapt and extend Metron without 
> having to maintain a fork of the project.
> As organizations and individuals look to extend the Metron system with custom 
> parsers, enrichments, and stellar functions that may be proprietary in 
> nature, the ability to develop and deploy these extensions outside the Metron 
> code base is critically important.
> To that end, and after community discussion and proposal we create or 
> formalize the 'plugin' development story in Metron.  
> The proposal is to adapt the Apache Nifi NAR system for use in Metron.  This 
> will provide the system with:
> * archetype(s) for developer projects and independent development
> * defined packaging and metadata for 'plugin' products
> * loading and instantiation with classloader isolation capabilities
> * removing the necessity for shading plugin jars
> These capabilities will also enable other features, such as plugin lifecycle, 
> plugin configuration+redeployment, and other things.
> The plugin archetypes and their installation will be a followon



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1070) Track Nifi Nar and bundle improvements

[Otto Fowler (JIRA)]

Otto Fowler created METRON-1070:
---

 Summary: Track Nifi Nar and bundle improvements
 Key: METRON-1070
 URL: https://issues.apache.org/jira/browse/METRON-1070
 Project: Metron
  Issue Type: Task
Reporter: Otto Fowler
Assignee: Otto Fowler






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1060) Add performance timing logging to enrichment topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105391#comment-16105391
 ] 

ASF GitHub Bot commented on METRON-1060:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/665
  
Hi @ottobackwards I actually took this advice and have a commit coming your 
way. I extracted functionality into a PerformanceLogger class that allows you 
to specify a percentage probability for logging the timing in the global 
config. A value of 1 logs roughly 1% of statements, 100 will log all 
statements. I could use some feedback on the approach I took for instantiating 
the logger - I used a local member variable instead of a static variable. The 
reason being that the logger maintains a Map of marked times. 
Since multiple executors may exist within a single worker JVM, it seemed likely 
that they might trample one another.


> Add performance timing logging to enrichment topology
> -
>
> Key: METRON-1060
> URL: https://issues.apache.org/jira/browse/METRON-1060
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1069) Ambari MPack documentation around development strategies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105378#comment-16105378
 ] 

ASF GitHub Bot commented on METRON-1069:


Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/673#discussion_r130149172
  
--- Diff: metron-deployment/packaging/ambari/README.md ---
@@ -0,0 +1,315 @@
+# Ambari Management Pack Development
+Typically, Ambari Management Pack development will be done in the Vagrant 
environments. These instructions are specific to Vagrant, but can be adapted 
for other environemnts (e.g. make sure to be on the correct nodes for server vs 
agent files)
+
+There is an `mpack.json` file which describes what services the mpack will 
contains, versions, etc.
+
+Alongside this are two directories, `addon-services` and `common-services`.
+
+The layout of `/common-services/METRON.CURRENT` is
+* `/configuration`
+  * This contains a set of `*-env.xml` files, relevent to particular 
components or the service as a whole.  These are where properties are defined.
+* `/package`
+  * `/files`
+* Contains files that get used as provided, in particular 
Elasticsearch templates.
+  * `/scripts`
+* A set of Python scripts that interface with Ambari to manage setup 
and install
+* `/params`
+  * A set of Python scripts for managing parameters from the 
`*-env.xml` files
+* `/templates`
+  * A set of Jinja template files which can be populated with 
properties
+* `/quicklinks`
+  * Contains `quicklinks.json` to define the Ambari quicklinks that should 
be used in the UI
+* `themes`
+  * Manages the Ambari UI themes blocks for organizing the configuration 
by relevant area.
+* `kerberos.json`
+  * Defines the keytabs and other Kerberos configuration to be used when 
Kerberizing a cluster
+* `metainfo.xml`
+  * Defines the METRON service, along with required packages, services, 
etc.
+* `service_advisor.py`
+  * Handles component layout and validation, along with handling some 
configurations for other services or that needs configs from other services.
+
+The layout of `/addon-services/METRON.CURRENT` is
+* `/repos`
+  * Contains `repoinfo.xml` that defines repositories to install packages 
from
+* `metainfo.xml`
+  * Limited info version of `/common-services/METRON.CURRENT/metainfo.xml`
+* `role_command_order.json`
+  * Defines the order of service startup and other actions relative to 
each other.
+
+## Adding a new property
+1. Add the property to the appropriate `*-env.xml` file found in 
`METRON.CURRENT/configuration`.
+  ```
+
+new_property
+New Property description
+Default Value
+New Property Pretty Name
+
+  ```
+
+2.  Reference the property in 
`METRON.CURRENT/package/scriptes/params/params_linux.py`, unless it will be 
used in Ambari's status command.  It will be stored in a variable. The name 
doesn't have to match, but it's preferred that it does.
+Make sure to use replace `metron-env` the correct `*-env` file, as noted 
above.
+  ```
+  new_property = config['configurations']['metron-env']['new_property']
+  ```
+If this property will be used in the status command, instead make this 
change in `METRON.CURRENT/package/scriptes/params/status_params.py`.
+Afterwards, in `params_linux.py`, reference the new property:
+  ```
+  new_property = status_params.new_property
+  ```
+This behavior is because Ambari doesn't send all parameters to the status, 
so it needs to be explicitly provided.
+
+3. Ambari master services can then import the params:
+
+  ```
+  from params import params
+  env.set_params(params)
+  ```
+
+4. The `*_commands.py` files receive the params as an input from the 
master services.  Once this is done, they can be accessed via the variable we 
set above:
+  ```
+  self.__params.new_property
+  ```
+
+
+### Env file property walkthrough
+To illustrate how property files are carried through to the scripts for 
our services, we'll run through an existing property, `metron_apps_hdfs_dir`.
+
+ Defining the property
+First the property appears in the appropriate `*-env.xml`, in this case 
`METRON.CURRENT/configuration/metron-env.xml`.
+```
+
+metron_apps_hdfs_dir
+/apps/metron
+Metron apps HDFS dir
+Metron apps HDFS dir
+
+```
+
+This defines several things
+1. The name of the property we'll be referencing it by in our code.
+1. The default value of the property.
+1. The description of the property for the Ambari UI.
+1. The pretty name that will be shown in Ambari 

[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105331#comment-16105331
 ] 

ASF GitHub Bot commented on METRON-777:
---

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/530
  
@ottobackwards , I agree it isn't critical to update now, and would slow 
down this PR even more.  Do you mind opening a tracking jira to consider an 
update in the future?  (assuming they continue developing Nar further.)


> Create a plugin system for Metron based on 'NAR'
> 
>
> Key: METRON-777
> URL: https://issues.apache.org/jira/browse/METRON-777
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The success of the Metron project will be greatly dependent on community 
> participation, and with that the ability to adapt and extend Metron without 
> having to maintain a fork of the project.
> As organizations and individuals look to extend the Metron system with custom 
> parsers, enrichments, and stellar functions that may be proprietary in 
> nature, the ability to develop and deploy these extensions outside the Metron 
> code base is critically important.
> To that end, and after community discussion and proposal we create or 
> formalize the 'plugin' development story in Metron.  
> The proposal is to adapt the Apache Nifi NAR system for use in Metron.  This 
> will provide the system with:
> * archetype(s) for developer projects and independent development
> * defined packaging and metadata for 'plugin' products
> * loading and instantiation with classloader isolation capabilities
> * removing the necessity for shading plugin jars
> These capabilities will also enable other features, such as plugin lifecycle, 
> plugin configuration+redeployment, and other things.
> The plugin archetypes and their installation will be a followon



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1069) Ambari MPack documentation around development strategies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105220#comment-16105220
 ] 

ASF GitHub Bot commented on METRON-1069:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/673#discussion_r130129238
  
--- Diff: metron-deployment/packaging/ambari/README.md ---
@@ -0,0 +1,315 @@
+# Ambari Management Pack Development
+Typically, Ambari Management Pack development will be done in the Vagrant 
environments. These instructions are specific to Vagrant, but can be adapted 
for other environemnts (e.g. make sure to be on the correct nodes for server vs 
agent files)
+
+There is an `mpack.json` file which describes what services the mpack will 
contains, versions, etc.
+
+Alongside this are two directories, `addon-services` and `common-services`.
+
+The layout of `/common-services/METRON.CURRENT` is
+* `/configuration`
+  * This contains a set of `*-env.xml` files, relevent to particular 
components or the service as a whole.  These are where properties are defined.
+* `/package`
+  * `/files`
+* Contains files that get used as provided, in particular 
Elasticsearch templates.
+  * `/scripts`
+* A set of Python scripts that interface with Ambari to manage setup 
and install
+* `/params`
+  * A set of Python scripts for managing parameters from the 
`*-env.xml` files
+* `/templates`
+  * A set of Jinja template files which can be populated with 
properties
+* `/quicklinks`
+  * Contains `quicklinks.json` to define the Ambari quicklinks that should 
be used in the UI
+* `themes`
+  * Manages the Ambari UI themes blocks for organizing the configuration 
by relevant area.
+* `kerberos.json`
+  * Defines the keytabs and other Kerberos configuration to be used when 
Kerberizing a cluster
+* `metainfo.xml`
+  * Defines the METRON service, along with required packages, services, 
etc.
+* `service_advisor.py`
+  * Handles component layout and validation, along with handling some 
configurations for other services or that needs configs from other services.
+
+The layout of `/addon-services/METRON.CURRENT` is
+* `/repos`
+  * Contains `repoinfo.xml` that defines repositories to install packages 
from
+* `metainfo.xml`
+  * Limited info version of `/common-services/METRON.CURRENT/metainfo.xml`
+* `role_command_order.json`
+  * Defines the order of service startup and other actions relative to 
each other.
+
+## Adding a new property
+1. Add the property to the appropriate `*-env.xml` file found in 
`METRON.CURRENT/configuration`.
+  ```
+
+new_property
+New Property description
+Default Value
+New Property Pretty Name
+
+  ```
+
+2.  Reference the property in 
`METRON.CURRENT/package/scriptes/params/params_linux.py`, unless it will be 
used in Ambari's status command.  It will be stored in a variable. The name 
doesn't have to match, but it's preferred that it does.
+Make sure to use replace `metron-env` the correct `*-env` file, as noted 
above.
+  ```
+  new_property = config['configurations']['metron-env']['new_property']
+  ```
+If this property will be used in the status command, instead make this 
change in `METRON.CURRENT/package/scriptes/params/status_params.py`.
+Afterwards, in `params_linux.py`, reference the new property:
+  ```
+  new_property = status_params.new_property
+  ```
+This behavior is because Ambari doesn't send all parameters to the status, 
so it needs to be explicitly provided.
+
+3. Ambari master services can then import the params:
+
+  ```
+  from params import params
+  env.set_params(params)
+  ```
+
+4. The `*_commands.py` files receive the params as an input from the 
master services.  Once this is done, they can be accessed via the variable we 
set above:
+  ```
+  self.__params.new_property
+  ```
+
+
+### Env file property walkthrough
+To illustrate how property files are carried through to the scripts for 
our services, we'll run through an existing property, `metron_apps_hdfs_dir`.
+
+ Defining the property
+First the property appears in the appropriate `*-env.xml`, in this case 
`METRON.CURRENT/configuration/metron-env.xml`.
+```
+
+metron_apps_hdfs_dir
+/apps/metron
+Metron apps HDFS dir
+Metron apps HDFS dir
+
+```
+
+This defines several things
+1. The name of the property we'll be referencing it by in our code.
+1. The default value of the property.
+1. The description of the property for the Ambari UI.
+1. The pretty name that will be shown in Ambari for 

[jira] [Commented] (METRON-1069) Ambari MPack documentation around development strategies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105216#comment-16105216
 ] 

ASF GitHub Bot commented on METRON-1069:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/673#discussion_r130128846
  
--- Diff: metron-deployment/packaging/ambari/README.md ---
@@ -0,0 +1,315 @@
+# Ambari Management Pack Development
+Typically, Ambari Management Pack development will be done in the Vagrant 
environments. These instructions are specific to Vagrant, but can be adapted 
for other environemnts (e.g. make sure to be on the correct nodes for server vs 
agent files)
+
+There is an `mpack.json` file which describes what services the mpack will 
contains, versions, etc.
+
+Alongside this are two directories, `addon-services` and `common-services`.
+
+The layout of `/common-services/METRON.CURRENT` is
+* `/configuration`
+  * This contains a set of `*-env.xml` files, relevent to particular 
components or the service as a whole.  These are where properties are defined.
+* `/package`
+  * `/files`
+* Contains files that get used as provided, in particular 
Elasticsearch templates.
+  * `/scripts`
+* A set of Python scripts that interface with Ambari to manage setup 
and install
+* `/params`
+  * A set of Python scripts for managing parameters from the 
`*-env.xml` files
+* `/templates`
+  * A set of Jinja template files which can be populated with 
properties
+* `/quicklinks`
+  * Contains `quicklinks.json` to define the Ambari quicklinks that should 
be used in the UI
+* `themes`
+  * Manages the Ambari UI themes blocks for organizing the configuration 
by relevant area.
+* `kerberos.json`
+  * Defines the keytabs and other Kerberos configuration to be used when 
Kerberizing a cluster
+* `metainfo.xml`
+  * Defines the METRON service, along with required packages, services, 
etc.
+* `service_advisor.py`
+  * Handles component layout and validation, along with handling some 
configurations for other services or that needs configs from other services.
+
+The layout of `/addon-services/METRON.CURRENT` is
+* `/repos`
+  * Contains `repoinfo.xml` that defines repositories to install packages 
from
+* `metainfo.xml`
+  * Limited info version of `/common-services/METRON.CURRENT/metainfo.xml`
+* `role_command_order.json`
+  * Defines the order of service startup and other actions relative to 
each other.
+
+## Adding a new property
+1. Add the property to the appropriate `*-env.xml` file found in 
`METRON.CURRENT/configuration`.
+  ```
+
+new_property
+New Property description
+Default Value
+New Property Pretty Name
+
+  ```
+
+2.  Reference the property in 
`METRON.CURRENT/package/scriptes/params/params_linux.py`, unless it will be 
used in Ambari's status command.  It will be stored in a variable. The name 
doesn't have to match, but it's preferred that it does.
+Make sure to use replace `metron-env` the correct `*-env` file, as noted 
above.
+  ```
+  new_property = config['configurations']['metron-env']['new_property']
+  ```
+If this property will be used in the status command, instead make this 
change in `METRON.CURRENT/package/scriptes/params/status_params.py`.
+Afterwards, in `params_linux.py`, reference the new property:
+  ```
+  new_property = status_params.new_property
+  ```
+This behavior is because Ambari doesn't send all parameters to the status, 
so it needs to be explicitly provided.
+
+3. Ambari master services can then import the params:
+
+  ```
+  from params import params
+  env.set_params(params)
+  ```
+
+4. The `*_commands.py` files receive the params as an input from the 
master services.  Once this is done, they can be accessed via the variable we 
set above:
+  ```
+  self.__params.new_property
+  ```
+
+
+### Env file property walkthrough
+To illustrate how property files are carried through to the scripts for 
our services, we'll run through an existing property, `metron_apps_hdfs_dir`.
+
+ Defining the property
+First the property appears in the appropriate `*-env.xml`, in this case 
`METRON.CURRENT/configuration/metron-env.xml`.
+```
+
+metron_apps_hdfs_dir
+/apps/metron
+Metron apps HDFS dir
+Metron apps HDFS dir
+
+```
+
+This defines several things
+1. The name of the property we'll be referencing it by in our code.
+1. The default value of the property.
+1. The description of the property for the Ambari UI.
+1. The pretty name that will be shown in Ambari for 

[jira] [Commented] (METRON-982) REST STORM Support for supervisor summary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105156#comment-16105156
 ] 

ASF GitHub Bot commented on METRON-982:
---

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/607


> REST STORM Support for supervisor summary
> -
>
> Key: METRON-982
> URL: https://issues.apache.org/jira/browse/METRON-982
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The supervisor summary can be used for information on the cluster including 
> the number of slots etc
> StormStatusServiceImpl already calls the storm ui rest endpoints and should 
> be extended to call:
> {code}
> /api/v1/supervisor/summary (GET)
> returns all supervisors summary
> Response Fields:
> Field Value   Description
> idString  Supervisor's id
> host  String  Supervisor's host name
> uptimeString  Shows how long the supervisor is running
> slotsTotalInteger Total number of available worker slots for this 
> supervisor
> slotsUsed Integer Number of worker slots used on this supervisor
> Sample Response:
> json { "supervisors": [ { "id": "0b879808-2a26-442b-8f7d-23101e0c3696", 
> "host": "10.11.1.7", "uptime": "5m 58s", "slotsTotal": 4, "slotsUsed": 3 } ] }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1064) Make init script OS-agnostic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105041#comment-16105041
 ] 

ASF GitHub Bot commented on METRON-1064:


Github user james-sirota commented on the issue:

https://github.com/apache/metron/pull/669
  
+1 by inspection. thanks, ryan


> Make init script OS-agnostic
> 
>
> Key: METRON-1064
> URL: https://issues.apache.org/jira/browse/METRON-1064
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>
> Currently the init scripts for metron-rest and metron-config are specific to 
> centos.  They should also work on ubuntu.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-982) REST STORM Support for supervisor summary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104955#comment-16104955
 ] 

ASF GitHub Bot commented on METRON-982:
---

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/607
  
Just tested this on full dev and it worked great.  Thanks @ottobackwards. +1


> REST STORM Support for supervisor summary
> -
>
> Key: METRON-982
> URL: https://issues.apache.org/jira/browse/METRON-982
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The supervisor summary can be used for information on the cluster including 
> the number of slots etc
> StormStatusServiceImpl already calls the storm ui rest endpoints and should 
> be extended to call:
> {code}
> /api/v1/supervisor/summary (GET)
> returns all supervisors summary
> Response Fields:
> Field Value   Description
> idString  Supervisor's id
> host  String  Supervisor's host name
> uptimeString  Shows how long the supervisor is running
> slotsTotalInteger Total number of available worker slots for this 
> supervisor
> slotsUsed Integer Number of worker slots used on this supervisor
> Sample Response:
> json { "supervisors": [ { "id": "0b879808-2a26-442b-8f7d-23101e0c3696", 
> "host": "10.11.1.7", "uptime": "5m 58s", "slotsTotal": 4, "slotsUsed": 3 } ] }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-681) CSVConverter should trim values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104565#comment-16104565
 ] 

ASF GitHub Bot commented on METRON-681:
---

GitHub user basvdl opened a pull request:

https://github.com/apache/metron/pull/674

METRON-681 CSVConverter should trim key and values

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/basvdl/metron 
METRON-681_CSVConverter_should_trim_values

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/674.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #674


commit e93082219a23bb8188321d812ab505967f5df1bb
Author: Bas van de Lustgraaf 
Date:   2017-07-28T07:09:24Z

METRON-681 trim to keys and values




> CSVConverter should trim values
> ---
>
> Key: METRON-681
> URL: https://issues.apache.org/jira/browse/METRON-681
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>  Labels: newbie
>
> Using the CSVParser, there is no option to trim the values.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)