date:20180129


[ 
https://issues.apache.org/jira/browse/METRON-993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344569#comment-16344569
 ] 

Anand Subramanian commented on METRON-993:
--

Quick dev deployment is now obsolete.

> Quick dev broken with latest master - services unable to start
> --
>
> Key: METRON-993
> URL: https://issues.apache.org/jira/browse/METRON-993
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.0
>Reporter: Anand Subramanian
>Priority: Major
>
> I am trying to bring up a quick dev environment using the latest bits from 
> master and I noticed that metron services fails to come up. The below error 
> is seen for Metron Enrichment in Ambari:
> {code}
> stderr: 
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/enrichment_master.py",
>  line 84, in 
> Enrichment().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/enrichment_master.py",
>  line 51, in start
> metron_service.load_global_config(params)
>   File 
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/metron_service.py",
>  line 76, in load_global_config
> init_config()
>   File 
> "/var/lib/ambari-agent/cache/common-services/METRON/0.3.1/package/scripts/metron_service.py",
>  line 31, in init_config
> path=ambari_format("{java_home}/bin")
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 70, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 
> '/usr/metron/0.3.1/bin/zk_load_configs.sh --mode PUSH -i 
> /usr/metron/0.3.1/config/zookeeper -z node1:2181' returned 127. /bin/bash: 
> /usr/metron/0.3.1/bin/zk_load_configs.sh: No such file or directory
>  stdout:
> 2017-06-09 09:32:22,007 - The hadoop conf dir 
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for 
> version 2.5.3.0-37
> 2017-06-09 09:32:22,007 - Checking if need to create versioned conf dir 
> /etc/hadoop/2.5.3.0-37/0
> 2017-06-09 09:32:22,009 - call[('ambari-python-wrap', '/usr/bin/conf-select', 
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.3.0-37', 
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 
> 'stderr': -1}
> 2017-06-09 09:32:22,097 - call returned (1, '/etc/hadoop/2.5.3.0-37/0 exist 
> already', '')
> 2017-06-09 09:32:22,098 - checked_call[('ambari-python-wrap', 
> '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', 
> '--stack-version', '2.5.3.0-37', '--conf-version', '0')] {'logoutput': False, 
> 'sudo': True, 'quiet': False}
> 2017-06-09 09:32:22,304 - checked_call returned (0, '')
> 2017-06-09 09:32:22,306 - Ensuring that hadoop has the correct symlink 
> structure
> 2017-06-09 09:32:22,306 - Using hadoop conf dir: 
> /usr/hdp/current/hadoop-client/conf
> 2017-06-09 09:32:23,537 - The hadoop conf dir 
> /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for 
> version 2.5.3.0-37
> 2017-06-09 09:32:23,561 - Checking if need to create versioned conf dir 
> /etc/hadoop/2.5.3.0-37/0
> 2017-06-09 09:32:23,562 - call[('ambari-python-wrap', '/usr/bin/conf-select', 
> 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.3.0-37', 
> '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 
> 'stderr': -1}
> 2017-06-09 09:32:23,915 - call returned (1, '/etc/hadoop/2.5.3.0-37/0 exist 
> already', '')
> 2017-06-09 09:32:23,918 - checked_call[('ambari-python-wrap', 
> '/usr/bin/conf-select',

[jira] [Commented] (METRON-1399) ES 5.x requires additional configuration for OS' which use systemd


[ 
https://issues.apache.org/jira/browse/METRON-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344567#comment-16344567
 ] 

Anand Subramanian commented on METRON-1399:
---

[~nickwallen] has addressed this issue as a part of METRON-1370 
(https://github.com/apache/metron/pull/903).

> ES 5.x requires additional configuration for OS' which use systemd
> --
>
> Key: METRON-1399
> URL: https://issues.apache.org/jira/browse/METRON-1399
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.3
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>Priority: Major
>
> This is with latest master, which has elasticsearch 5.6.2.
> On a Centos 7, it is seen that the Elasticsearch service fails to start with 
> error as follows:
> {code}
> [2017-12-07T10:06:36,451][ERROR][o.e.b.Bootstrap  ] 
> [metron-1.openstacklocal] node validation exception
> [1] bootstrap checks failed
> [1]: memory locking requested for elasticsearch process but memory is not 
> locked
> [2017-12-07T10:06:36,459][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] stopping ...
> [2017-12-07T10:06:36,496][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] stopped
> [2017-12-07T10:06:36,497][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] closing ...
> [2017-12-07T10:06:36,522][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] closed
> {code}
> There was no change in behavior after adding the entry 
> "MAX_LOCKED_MEMORY=unlimited" in /etc/sysconfig/elasticsearch.
> I found that ES 5.x requires additional configuration for environments that 
> use systemd (like CentOS 7). The following link explains the changes required:
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#systemd
> This hence needs to be done at install/configuration time for ES to come up 
> properly on OS' that use 'systemd'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1399) ES 5.x requires additional configuration for OS' which use systemd


 [ 
https://issues.apache.org/jira/browse/METRON-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anand Subramanian reassigned METRON-1399:
-

Assignee: Nick Allen

> ES 5.x requires additional configuration for OS' which use systemd
> --
>
> Key: METRON-1399
> URL: https://issues.apache.org/jira/browse/METRON-1399
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.3
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>Priority: Major
>
> This is with latest master, which has elasticsearch 5.6.2.
> On a Centos 7, it is seen that the Elasticsearch service fails to start with 
> error as follows:
> {code}
> [2017-12-07T10:06:36,451][ERROR][o.e.b.Bootstrap  ] 
> [metron-1.openstacklocal] node validation exception
> [1] bootstrap checks failed
> [1]: memory locking requested for elasticsearch process but memory is not 
> locked
> [2017-12-07T10:06:36,459][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] stopping ...
> [2017-12-07T10:06:36,496][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] stopped
> [2017-12-07T10:06:36,497][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] closing ...
> [2017-12-07T10:06:36,522][INFO ][o.e.n.Node   ] 
> [metron-1.openstacklocal] closed
> {code}
> There was no change in behavior after adding the entry 
> "MAX_LOCKED_MEMORY=unlimited" in /etc/sysconfig/elasticsearch.
> I found that ES 5.x requires additional configuration for environments that 
> use systemd (like CentOS 7). The following link explains the changes required:
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#systemd
> This hence needs to be done at install/configuration time for ES to come up 
> properly on OS' that use 'systemd'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1369) squid events are not seen in Alerts UI


[ 
https://issues.apache.org/jira/browse/METRON-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344563#comment-16344563
 ] 

Anand Subramanian commented on METRON-1369:
---

In order for the squid events to show up in the Alerts UI, there needs to be a 
template added into Elasticsearch so that it gets applied dynamically as the 
data flows in.

The following needs to be done before ingesting into squid sensor:
{code:java}
curl -XPUT 'http://node1:9200/_template/squid_index' -d '
{
  "template": "squid_index*",
  "mappings": {
"squid_doc": {
  "dynamic_templates": [
  {
"geo_location_point": {
  "match": "enrichments:geo:*:location_point",
  "match_mapping_type": "*",
  "mapping": {
"type": "geo_point"
  }
}
  },
  {
"geo_country": {
  "match": "enrichments:geo:*:country",
  "match_mapping_type": "*",
  "mapping": {
"type": "keyword"
  }
}
  },
  {
"geo_city": {
  "match": "enrichments:geo:*:city",
  "match_mapping_type": "*",
  "mapping": {
"type": "keyword"
  }
}
  },
  {
"geo_location_id": {
  "match": "enrichments:geo:*:locID",
  "match_mapping_type": "*",
  "mapping": {
"type": "keyword"
  }
}
  },
  {
"geo_dma_code": {
  "match": "enrichments:geo:*:dmaCode",
  "match_mapping_type": "*",
  "mapping": {
"type": "keyword"
  }
}
  },
  {
"geo_postal_code": {
  "match": "enrichments:geo:*:postalCode",
  "match_mapping_type": "*",
  "mapping": {
"type": "keyword"
  }
}
  },
  {
"geo_latitude": {
  "match": "enrichments:geo:*:latitude",
  "match_mapping_type": "*",
  "mapping": {
"type": "float"
  }
}
  },
  {
"geo_longitude": {
  "match": "enrichments:geo:*:longitude",
  "match_mapping_type": "*",
  "mapping": {
"type": "float"
  }
}
  },
  {
"timestamps": {
  "match": "*:ts",
  "match_mapping_type": "*",
  "mapping": {
"type": "date",
"format": "epoch_millis"
  }
}
  },
  {
"threat_triage_score": {
  "mapping": {
"type": "float"
  },
  "match": "threat:triage:*score",
  "match_mapping_type": "*"
}
  },
  {
"threat_triage_reason": {
  "mapping": {
"type": "text",
"fielddata": "true"
  },
  "match": "threat:triage:rules:*:reason",
  "match_mapping_type": "*"
}
  },
  {
"threat_triage_name": {
  "mapping": {
"type": "text",
"fielddata": "true"
  },
  "match": "threat:triage:rules:*:name",
  "match_mapping_type": "*"
}
  }
  ],
  "properties": {
"timestamp": {
  "type": "date",
  "format": "epoch_millis"
},
"source:type": {
  "type": "keyword"
},
"ip_dst_addr": {
  "type": "ip"
},
"ip_dst_port": {
  "type": "integer"
},
"ip_src_addr": {
  "type": "ip"
},
"ip_src_port": {
  "type": "integer"
},
"alert": {
  "type": "nested"
},
"guid": {
  "type": "keyword"
}
  }
}
  }
}
'{code}
Validate that the template loaded as expected  by running:
{code:java}
curl -XGET 'http://node1:9200/_template/squid_index?pretty'{code}
 

[~mmiklavcic] has updated the Wiki instructions here to reflect the same. Thank 
you Mike!

[https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry]

> squid events are not seen in Alerts UI
> --
>
> Key: METRON-1369
> URL: https://issues.apache.org/jira/browse/METRON-1369
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Nick Allen
>Priority: Major
>
> On a metron cluster, I have added the squid telemetry by following the [Wiki 
> instructions|https://cwiki.apache.org/confluence/display/METRON/2016/05/02/Metron+Tutorial+-+Fundamentals+Part+4%3A+Pluggable+Threat+Intelligence].
> I am able to see the squid indices present in elasticsearch, but not able to 
> see them in the alerts UI. On further checking, I found out that this is due 
> to the missing nested "alert" field in the index mappings--which is required 
> for the Alerts UI to pick up the

[jira] [Commented] (METRON-1427) Add support for storm 1.1 and hdp 2.6


[ 
https://issues.apache.org/jira/browse/METRON-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344217#comment-16344217
 ] 

ASF GitHub Bot commented on METRON-1427:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
FYI @cestella I submitted a fix against your PR branch that should address 
the issue with the embedded handlebars in the Ambari response. 


> Add support for storm 1.1 and hdp 2.6
> -
>
> Key: METRON-1427
> URL: https://issues.apache.org/jira/browse/METRON-1427
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1 becuase 
> of some classpath issues and not supporting the stack.  We should migrate 
> fulldev to run 2.6 (while still working with 2.5) and validate that we work 
> with storm 1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1433) Only emit debugging timing fields in enrichment when debugging is turned on


[ 
https://issues.apache.org/jira/browse/METRON-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344120#comment-16344120
 ] 

ASF GitHub Bot commented on METRON-1433:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/915
  

https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java#L89


> Only emit debugging timing fields in enrichment when debugging is turned on
> ---
>
> Key: METRON-1433
> URL: https://issues.apache.org/jira/browse/METRON-1433
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Right now we always emit performance debugging fields in the split/join 
> bolts.  We should only do that when debug logging is turned on for 
> org.apache.metron.enrichment.bolt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1431) Add REGEXP_REPLACE function to Stellar


[ 
https://issues.apache.org/jira/browse/METRON-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344106#comment-16344106
 ] 

ASF GitHub Bot commented on METRON-1431:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/912


> Add REGEXP_REPLACE function to Stellar
> --
>
> Key: METRON-1431
> URL: https://issues.apache.org/jira/browse/METRON-1431
> Project: Metron
>  Issue Type: Improvement
>Reporter: 1havran
>Priority: Minor
>
> Add REGEXP_REPLACE(input, pattern, value) function to Stellar that will 
> replace all occurrences of regex pattern within the input by provided value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1431) Add REGEXP_REPLACE function to Stellar


[ 
https://issues.apache.org/jira/browse/METRON-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344107#comment-16344107
 ] 

ASF GitHub Bot commented on METRON-1431:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/912
  
please remember to take care of your jira


> Add REGEXP_REPLACE function to Stellar
> --
>
> Key: METRON-1431
> URL: https://issues.apache.org/jira/browse/METRON-1431
> Project: Metron
>  Issue Type: Improvement
>Reporter: 1havran
>Priority: Minor
>
> Add REGEXP_REPLACE(input, pattern, value) function to Stellar that will 
> replace all occurrences of regex pattern within the input by provided value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1431) Add REGEXP_REPLACE function to Stellar


[ 
https://issues.apache.org/jira/browse/METRON-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344054#comment-16344054
 ] 

ASF GitHub Bot commented on METRON-1431:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/912
  
+1 by inspection.  Thank you for the contribution



> Add REGEXP_REPLACE function to Stellar
> --
>
> Key: METRON-1431
> URL: https://issues.apache.org/jira/browse/METRON-1431
> Project: Metron
>  Issue Type: Improvement
>Reporter: 1havran
>Priority: Minor
>
> Add REGEXP_REPLACE(input, pattern, value) function to Stellar that will 
> replace all occurrences of regex pattern within the input by provided value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1431) Add REGEXP_REPLACE function to Stellar


[ 
https://issues.apache.org/jira/browse/METRON-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344051#comment-16344051
 ] 

ASF GitHub Bot commented on METRON-1431:


Github user 1havran commented on a diff in the pull request:

https://github.com/apache/metron/pull/912#discussion_r164570355
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RegExFunctionsTest.java
 ---
@@ -68,4 +68,19 @@ public void testRegExGroupVal() throws Exception {
   Assert.assertTrue("Did not fail on wrong number of 
parameters",false);
 }
   }
+
+  @Test
+  public void testRegExReplace() throws Exception {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
--- End diff --

Indeed, more tests included.


> Add REGEXP_REPLACE function to Stellar
> --
>
> Key: METRON-1431
> URL: https://issues.apache.org/jira/browse/METRON-1431
> Project: Metron
>  Issue Type: Improvement
>Reporter: 1havran
>Priority: Minor
>
> Add REGEXP_REPLACE(input, pattern, value) function to Stellar that will 
> replace all occurrences of regex pattern within the input by provided value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1431) Add REGEXP_REPLACE function to Stellar


[ 
https://issues.apache.org/jira/browse/METRON-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344050#comment-16344050
 ] 

ASF GitHub Bot commented on METRON-1431:


Github user 1havran commented on a diff in the pull request:

https://github.com/apache/metron/pull/912#discussion_r164570247
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RegExFunctions.java
 ---
@@ -100,4 +100,39 @@ public Object apply(List list) {
   return matcher.group(groupNumber);
 }
   }
+
+  @Stellar(name = "REGEXP_REPLACE",
+  description = "Replace all occurences of the regex pattern within 
the string by value",
+  params = {
+  "string - The input string",
+  "pattern - The regex pattern to be replaced. Special characters 
must be escaped (e.g. d)",
+  "value - The value to replace the regex pattern"
+  },
+  returns = "The modified input string with replaced values")
+  public static class RegexpReplace extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() != 3) {
+throw new IllegalStateException(
+"REGEXP_REPLACE expects three args: [string, pattern, value]"
++ " where pattern is a regexp pattern");
+  }
+  String str = (String) list.get(0);
+  String stringPattern = (String) list.get(1);
+  String value = (String) list.get(2);
+
--- End diff --

Thanks for a hint, it is fixed now.


> Add REGEXP_REPLACE function to Stellar
> --
>
> Key: METRON-1431
> URL: https://issues.apache.org/jira/browse/METRON-1431
> Project: Metron
>  Issue Type: Improvement
>Reporter: 1havran
>Priority: Minor
>
> Add REGEXP_REPLACE(input, pattern, value) function to Stellar that will 
> replace all occurrences of regex pattern within the input by provided value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1433) Only emit debugging timing fields in enrichment when debugging is turned on


[ 
https://issues.apache.org/jira/browse/METRON-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344016#comment-16344016
 ] 

ASF GitHub Bot commented on METRON-1433:


GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/915

METRON-1433: Only emit debugging timing fields in enrichment when debugging 
is turned on

## Contributor Comments
Right now we always emit performance debugging fields in the split/join 
bolts.  We should only do that when debug logging is turned on for 
`org.apache.metron.enrichment.bolt`

Test:
Spin up full-dev and ensure that `adapter.*.begin.ts` and 
`adapter.*.end.ts` fields do not show up unless you set debug logging for 
`org.apache.metron.enrichment.bolt`


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron turnoff_debugging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #915


commit ba241c9c9624e8b02a52952d8733414450bc7f15
Author: cstella 
Date:   2018-01-29T20:55:51Z

Only emit logging fields when debug is turned on.




> Only emit debugging timing fields in enrichment when debugging is turned on
> ---
>
> Key: METRON-1433
> URL: https://issues.apache.org/jira/browse/METRON-1433
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Right now we always emit performance debugging fields in the split/join 
> bolts.  We should only do that when debug logging is turned on for 
> org.apache.metron.enrichment.bolt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1433) Only emit debugging timing fields in enrichment when debugging is turned on

2018-01-29 Thread Casey Stella (JIRA)

Casey Stella created METRON-1433:


 Summary: Only emit debugging timing fields in enrichment when 
debugging is turned on
 Key: METRON-1433
 URL: https://issues.apache.org/jira/browse/METRON-1433
 Project: Metron
  Issue Type: Improvement
Reporter: Casey Stella


Right now we always emit performance debugging fields in the split/join bolts.  
We should only do that when debug logging is turned on for 
org.apache.metron.enrichment.bolt



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1397) JSONMap parser should support JSON Path expressions to split input into multiple messages


[ 
https://issues.apache.org/jira/browse/METRON-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343968#comment-16343968
 ] 

ASF GitHub Bot commented on METRON-1397:


GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/914

METRON-1397  Support for JSON Path and complex documents in JSONMapParser

It would be useful for implementors to have the ability to ingest more 
complex documents without having to stand up a NiFi node/cluster.

This PR adds support for splitting multiple messages from a single message 
by way of using JSON Path statements to select the List element in the document.

Since the parse() interface already returns a List the scope of 
this change is in the parser.

Example:

```json
{
"foo" :
[
{ "name" : "foo1", "value" : "bar", "number" : 1.0 },
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
]
}
 ```

with JSON Path statement "$.foo" will result in two messages emitted from 
the parser

```json
 { "name" : "foo1", "value" : "bar", "number" : 1.0 }
```

```json
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
```
Support in the configuration for a new configuration parameter: jsonpQuery 
has been added.
An example of the new configuration:

```json
{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"jsonMapQuery",
  "parserConfig": {"jsonpQuery":"$.foo"}
}
```

In order to integration test both with and without the query present, a new 
jsonMapQuery parser configuration has been added, with test data.


##Testing
- tests should run
- deployment should work


### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:


- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron jsonp-support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #914


commit 414611b9ea6eb06b79cba0cc86e63ea062335884
Author: Otto Fowler 
Date:   2018-01-29T16:58:16Z

JSON Path support for JSONMapParser
Tests, Integration Tests, Doc

commit ad03ac4d3e7c98fcf571f3f3906efbb915deb1f9
Author: Otto Fowler 
Date:   2018-01-29T17:13:10Z

checkstyle fixes

commit 7c76c5f5b7507ac55861ed44ad131700c1e6f124
Author: Otto Fowler 
Date:   2018-01-29T19:36:16Z

account for new config in rpm

fix handle empty




> JSONMap parser should support JSON Path expressions to split input into 
> multiple messages
> -
>
> Key: METRON-1397
> URL: https://issues.apache.org/jira/browse/METRON-1397
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> The parser interface takes byte[] and returns a list.  
> It is consistent with the

[jira] [Commented] (METRON-1432) JDK Install Fails on Ubuntu Development Environment


[ 
https://issues.apache.org/jira/browse/METRON-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343883#comment-16343883
 ] 

ASF GitHub Bot commented on METRON-1432:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/913
  
+1 by inspection


> JDK Install Fails on Ubuntu Development Environment 
> 
>
> Key: METRON-1432
> URL: https://issues.apache.org/jira/browse/METRON-1432
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When deploying Metron in the Ubuntu-based development environment, the JDK 
> fails to install.  The following exception is seen.
> {code:java}
> TASK [java_jdk : Check for java at "/usr/jdk64/jdk1.8.0_77"] 
> ***
> ok: [node1]
> TASK [java_jdk : Alternatives link for java] 
> ***
> skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/java', 
> u'link': u'/usr/bin/java', u'name': u'java'})
> skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/jar', 
> u'link': u'/usr/bin/jar', u'name': u'jar'})
> TASK [java_jdk : Install openjdk] 
> **
> failed: [node1] (item=[u'java-1.8.0-openjdk', u'java-1.8.0-openjdk-devel']) 
> => {"failed": true, "item": ["java-1.8.0-openjdk", 
> "java-1.8.0-openjdk-devel"], "module_stderr": "Shared connection to node1 
> closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File 
> \"/tmp/ansible_4ssOWN/ansible_module_yum.py\", line 25, in \r\n 
> import yum\r\nImportError: No module named yum\r\n", "msg": "MODULE FAILURE"}
> to retry, use: --limit 
> @/Users/nallen/tmp/metron-pr907/metron-deployment/development/ubuntu14/ansible/playbook.retry
> PLAY RECAP 
> *
> node1 : ok=42 changed=26 unreachable=0 failed=1
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1427) Add support for storm 1.1 and hdp 2.6


[ 
https://issues.apache.org/jira/browse/METRON-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343882#comment-16343882
 ] 

ASF GitHub Bot commented on METRON-1427:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/907
  
For 2, that's the same error @mmiklavc hit on centos.  It's very 
interesting that I'm not hitting it.


> Add support for storm 1.1 and hdp 2.6
> -
>
> Key: METRON-1427
> URL: https://issues.apache.org/jira/browse/METRON-1427
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1 becuase 
> of some classpath issues and not supporting the stack.  We should migrate 
> fulldev to run 2.6 (while still working with 2.5) and validate that we work 
> with storm 1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1427) Add support for storm 1.1 and hdp 2.6


[ 
https://issues.apache.org/jira/browse/METRON-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343872#comment-16343872
 ] 

ASF GitHub Bot commented on METRON-1427:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
For (2), the problem is that the HDFS configuration in Ambari has embedded 
'mustache handlebars' (see `{{major_stack_version}}`) that when returned from 
an API call confuses Ansible.  We need some way to strip that out or ignore it.

![screen shot 2018-01-29 at 2 31 44 
pm](https://user-images.githubusercontent.com/2475409/35530186-5072f3ae-0501-11e8-8bd9-93096da51f02.png)



> Add support for storm 1.1 and hdp 2.6
> -
>
> Key: METRON-1427
> URL: https://issues.apache.org/jira/browse/METRON-1427
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1 becuase 
> of some classpath issues and not supporting the stack.  We should migrate 
> fulldev to run 2.6 (while still working with 2.5) and validate that we work 
> with storm 1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1432) JDK Install Fails on Ubuntu Development Environment


[ 
https://issues.apache.org/jira/browse/METRON-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343760#comment-16343760
 ] 

ASF GitHub Bot commented on METRON-1432:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/913

METRON-1432 JDK Install Fails on Ubuntu Development Environment

The Ansible role used to install the JDK does not work correctly on Ubuntu. 
 This fixes the problem and ensures that the JDK can be installed on either 
Ubuntu or CentOS.

## Testing

1. Launch the Ubuntu development environment. 
* Run the Metron Service Check
* Ensure data is visible within the Alerts UI

1. Launch the CentOS development environment.
* Run the Metron Service Check
* Ensure data is visible within the Alerts UI

## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1432

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #913


commit bbc27bf6337067d51e2d6d7d03bcb19203e35f59
Author: Nick Allen 
Date:   2018-01-29T18:15:32Z

METRON-1432 JDK Install Fails on Ubuntu Development Environment




> JDK Install Fails on Ubuntu Development Environment 
> 
>
> Key: METRON-1432
> URL: https://issues.apache.org/jira/browse/METRON-1432
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When deploying Metron in the Ubuntu-based development environment, the JDK 
> fails to install.  The following exception is seen.
> {code:java}
> TASK [java_jdk : Check for java at "/usr/jdk64/jdk1.8.0_77"] 
> ***
> ok: [node1]
> TASK [java_jdk : Alternatives link for java] 
> ***
> skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/java', 
> u'link': u'/usr/bin/java', u'name': u'java'})
> skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/jar', 
> u'link': u'/usr/bin/jar', u'name': u'jar'})
> TASK [java_jdk : Install openjdk] 
> **
> failed: [node1] (item=[u'java-1.8.0-openjdk', u'java-1.8.0-openjdk-devel']) 
> => {"failed": true, "item": ["java-1.8.0-openjdk", 
> "java-1.8.0-openjdk-devel"], "module_stderr": "Shared connection to node1 
> closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File 
> \"/tmp/ansible_4ssOWN/ansible_module_yum.py\", line 25, in \r\n 
> import yum\r\nImportError: No module named yum\r\n", "msg": "MODULE FAILURE"}
> to retry, use: --limit 
> @/Users/nallen/tmp/metron-pr907/metron-deployment/development/ubuntu14/ansible/playbook.retry
> PLAY RECAP 
> *
> node1 : ok=42 changed=26 unreachable=0 failed=1
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1432) JDK Install Fails on Ubuntu Development Environment

2018-01-29 Thread Nick Allen (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-1432:
---
Summary: JDK Install Fails on Ubuntu Development Environment   (was: JDK 
Install Fails on Ubuntu )

> JDK Install Fails on Ubuntu Development Environment 
> 
>
> Key: METRON-1432
> URL: https://issues.apache.org/jira/browse/METRON-1432
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When deploying Metron in the Ubuntu-based development environment, the JDK 
> fails to install.  The following exception is seen.
> {code:java}
> TASK [java_jdk : Check for java at "/usr/jdk64/jdk1.8.0_77"] 
> ***
> ok: [node1]
> TASK [java_jdk : Alternatives link for java] 
> ***
> skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/java', 
> u'link': u'/usr/bin/java', u'name': u'java'})
> skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/jar', 
> u'link': u'/usr/bin/jar', u'name': u'jar'})
> TASK [java_jdk : Install openjdk] 
> **
> failed: [node1] (item=[u'java-1.8.0-openjdk', u'java-1.8.0-openjdk-devel']) 
> => {"failed": true, "item": ["java-1.8.0-openjdk", 
> "java-1.8.0-openjdk-devel"], "module_stderr": "Shared connection to node1 
> closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File 
> \"/tmp/ansible_4ssOWN/ansible_module_yum.py\", line 25, in \r\n 
> import yum\r\nImportError: No module named yum\r\n", "msg": "MODULE FAILURE"}
> to retry, use: --limit 
> @/Users/nallen/tmp/metron-pr907/metron-deployment/development/ubuntu14/ansible/playbook.retry
> PLAY RECAP 
> *
> node1 : ok=42 changed=26 unreachable=0 failed=1
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1432) JDK Install Fails on Ubuntu

2018-01-29 Thread Nick Allen (JIRA)

Nick Allen created METRON-1432:
--

 Summary: JDK Install Fails on Ubuntu 
 Key: METRON-1432
 URL: https://issues.apache.org/jira/browse/METRON-1432
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen


When deploying Metron in the Ubuntu-based development environment, the JDK 
fails to install.  The following exception is seen.
{code:java}
TASK [java_jdk : Check for java at "/usr/jdk64/jdk1.8.0_77"] ***
ok: [node1]

TASK [java_jdk : Alternatives link for java] ***
skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/java', 
u'link': u'/usr/bin/java', u'name': u'java'})
skipping: [node1] => (item={u'path': u'/usr/jdk64/jdk1.8.0_77/bin/jar', 
u'link': u'/usr/bin/jar', u'name': u'jar'})

TASK [java_jdk : Install openjdk] **
failed: [node1] (item=[u'java-1.8.0-openjdk', u'java-1.8.0-openjdk-devel']) => 
{"failed": true, "item": ["java-1.8.0-openjdk", "java-1.8.0-openjdk-devel"], 
"module_stderr": "Shared connection to node1 closed.\r\n", "module_stdout": 
"Traceback (most recent call last):\r\n File 
\"/tmp/ansible_4ssOWN/ansible_module_yum.py\", line 25, in \r\n import 
yum\r\nImportError: No module named yum\r\n", "msg": "MODULE FAILURE"}
to retry, use: --limit 
@/Users/nallen/tmp/metron-pr907/metron-deployment/development/ubuntu14/ansible/playbook.retry

PLAY RECAP *
node1 : ok=42 changed=26 unreachable=0 failed=1

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1427) Add support for storm 1.1 and hdp 2.6


[ 
https://issues.apache.org/jira/browse/METRON-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343732#comment-16343732
 ] 

ASF GitHub Bot commented on METRON-1427:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
Running this up on Ubuntu, I ran into two issues.

(1) JDK Install Failed on Ubuntu - I do not think this was caused by this 
PR.  This is something that I should have hit in #903, but the changes there 
never brought this bug to light.  Fortunately, it is an easy fix and I will 
open it as a separate PR.

(2) After the Ambari deployment completes, it begins to install the 
sensors.  It queries Ambari for information to configure the sensors and in one 
of those queries it hits this problem.  I am not sure exactly what the problem 
is yet.
```
TASK [ambari_gather_facts : Ask Ambari: hdfs_url] 
**
ok: [node1]

TASK [ambari_gather_facts : set_fact] 
**
fatal: [node1]: FAILED! =>{  
   "failed":true,
   "msg":"the field 'args' has an invalid value, which appears to include a 
variable that is undefined. The error was: {u'status': 200, u'content_type': 
u'text/plain', u'set_cookie': 
u'AMBARISESSIONID=5tlv0y7btgc24krm5ugdpdgj;Path=/;HttpOnly', u'expires': u'Thu, 
01 Jan 1970 00:00:00 GMT', u'vary': u'Accept-Encoding, User-Agent', u'user': 
u'admin', u'pragma': u'no-cache', u'x_frame_options': u'DENY', 
u'x_xss_protection': u'1; mode=block', u'url': 
u'http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED',
 u'changed': False, u'x_content_type_options': u'nosniff', u'content': u'{\\n  
\"href\" : 
\"http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED\",\\n
  \"items\" : [\\n{\\n  \"href\" : 
\"http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED\",\\n
  \"tag\" : \"TOPOLOGY_RESOLVED\",\\n  \"type\" : \"core-site\",\\n 
 \"version\" : 2,\\n  \"Config\" : {\\n\"cluster_name\" : 
\"metron_cluster\",\\n\"stack_id\" : \"HDP-2.6\"\\n  },\\n  
\"properties\" : {\\n\"fs.defaultFS\" : \"hdfs://node1:8020\",\\n   
 \"fs.trash.interval\" : \"360\",\\n
\"ha.failover-controller.active-standby-elector.zk.op.retries\" : \"120\",\\n   
 \"hadoop.custom-extensions.root\" : 
\"/hdp/ext/{{major_stack_version}}/hadoop\",\\n
\"hadoop.http.authentication.simple.anonymous.allowed\" : \"true\",\\n
\"hadoop.proxyuser.hbase.groups\" : \"*\",\\n
\"hadoop.proxyuser.hbase.hosts\" : \"*\",\\n
\"hadoop.security.auth_to_local\" : \"DEFAULT\",\\n
\"hadoop.security.authentication\" : \"simple\",\\n
\"hadoop.security.authorization\" : \"false\",\\n
\"hadoop.security.key.provider.path\" : \"\",\\n
\"io.compression.codecs\" : 
\"org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec\",\\n
\"io.file.buffer.size\" : \"131072\",\\n\"io.serializations\" : 
\"org.apache.hadoop.io.serializer.WritableSerialization\",\\n
\"ipc.client.connect.max.retries\" : \"50\",\\n
\"ipc.client.connection.maxidletime\" : \"3\",\\n
\"ipc.client.idlethreshold\" : \"8000\",\\n\"ipc.server.tcpnodelay\" : 
\"true\",\\n\"mapreduce.jobtracker.webinterface.trusted\" : 
\"false\",\\n\"net.topology.script.file.name\" : 
\"/etc/hadoop/conf/topology_script.py\"\\n  },\\n  
\"properties_attributes\" : {\\n\"final\" : {\\n  
\"fs.defaultFS\" : \"true\"\\n}\\n  }\\n}\\n  ]\\n}', 
u'connection': u'close', u'msg': u'OK (unknown bytes)', u'redirected': False, 
u'cache_control': u'no-store'}: 'major_stack_version' is undefined\n\nThe error 
appears to have been in 
'/Users/nallen/tmp/metron-pr907/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml':
 line 82, column 3, but may\nbe elsewhere in the file depending on the exact 
syntax problem.\n\nThe offending line appears to be:\n\n\n- set_fact:\n  ^ 
here\n"
}
```




> Add support for storm 1.1 and hdp 2.6
> -
>
> Key: METRON-1427
> URL: https://issues.apache.org/jira/browse/METRON-1427
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1 becuase 
> of some classpath issues and not supporting the stack.  We should migrate 
> fulldev to run 2.6 (while still working with 2.5) and validate that we work 
> with storm 1.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1419) Create a SolrDao

[
https://issues.apache.org/jira/browse/METRON-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343565#comment-16343565
]

ASF GitHub Bot commented on METRON-1419:

GitHub user merrimanr reopened a pull request:

https://github.com/apache/metron/pull/911

METRON-1419: Create a SolrDao

## Contributor Comments
This PR is an initial attempt at creating a SolrDao that implements the
IndexDao interface, is functionally equivalent to ElasticsearchDao and passes
all tests in SearchIntegrationTest and UpdateIntegrationTest.

A high level summary of the changes include:

- Upgraded the Solr client version to 6.6.0
- Updated the SolrComponent to work with Solr 6.6 and added a couple
convenience methods similar to ElasticsearchComponent
- Added a new SolrDao implementation with all IndexDao methods implemented
- Refactored the SearchIntegrationTest to work for both Solr and
Elasticsearch and added an Solr implementation (more detail below)
- Created an abstract UpdateIntegrationTest and added a Solr implementation
- Added Solr schemas for test data sets
- Added new tests to SearchIntegrationTest including filtering on fields
with different types, faceting on fields with different types, and faceting on
fields with missing types.
- Broke the IndexDao down in the SolrDao to smaller, easier to understand
classes. The ElasticsearchDao class has become very large so I attempted to
make the SolrDao more readable.

There were a couple areas where Elasticsearch and Solr behave slightly
different. I attempted to accommodated for that through the SolrDao
implementation, by adjusting existing tests, and by splitting out specific
tests:

- Column metadata is different between the 2 search engines so each
implementation has their own tests
- There are cases where the clients will return different types in search
results. I am handling this in SearchIntegrationTest by first converting the
types to strings and then comparing (other ideas?). For example, the ES client
returns an Integer for timestamp while Solr returns a Long.
- There are cases where ES throws an error under certain conditions while
Solr does not (and vice-versa). These were moved to either ES or Solr
SearchIntegrationTest implementations.
- There is no support in Solr for sorting group results so I am sorting
them client-side instead.

At this point the scope is limited to tests passing, meaning
metron-elasticsearch and metron-solr pass all tests. There are other PRs in
progress that are needed before automated testing with full dev can be done. I
am still actively working on manually testing in full dev and adding
documentation but this should get us started.

This PR is intended to be merged into the upgrade-solr feature branch but I
have it set to master temporarily so review is easier. We will need to merge
in master to the feature branch to get rid of the extra commits since this PR
is up to date with master.

I'm expecting a lengthy review and would request multiple reviewers and +1s.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```

- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these

[jira] [Commented] (METRON-1419) Create a SolrDao


[ 
https://issues.apache.org/jira/browse/METRON-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343564#comment-16343564
 ] 

ASF GitHub Bot commented on METRON-1419:


Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/911


> Create a SolrDao
> 
>
> Key: METRON-1419
> URL: https://issues.apache.org/jira/browse/METRON-1419
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Ryan Merriman
>Priority: Major
>
> Create an implementation of the IndexDao for Solr. This will involve 
> implementing the various IndexDao methods using the SolrJ library and also 
> providing a SolrSearchIntegrationTest that extends SearchIntegrationTest 
> (similar to ElasticsearchSearchIntegrationTest). An integration test similar 
> to ElasticsearchUpdateIntegrationTest should also be included.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-534) Metron Should have service to monitor HDFS for changes to files


 [ 
https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated METRON-534:
---
Priority: Major  (was: Minor)

> Metron Should have service to monitor HDFS for changes to files
> ---
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on 
> file changes. 
>  A service that utilized this and was configured with "rules" to process 
> notifications on files of interest ( configurations, rules etc ) in the 
> system would be able to send notifications to Zookeeper
>  This may allow for things that are better stored in HDFS to remain there, 
> and not force the system to put things in Zookeeper just for change 
> notifications.
> Issues:
>  * Inotify requires hdfs super user, what are the implications on a secure 
> cluster etc?
>  * performance / scalability and load
>  * Can it be deployed with yarn?
>  * Is Zookeeper actually the right place for all the configurations
> -
>  
> UPDATE: 1/29/18
> "
> In the end, what I’m thinking is this:
>  
> We have an ambari service that runs the notification -> zookeeper
> it reads the ‘registration area’ from zookeeper to get it’s state and what to 
> watch
> post 777 when parsers are installed and registered it is trivial to have my 
> installer also register the files to watch
>  
> the notifications service also has a notification from zookeeper for new 
> registrations.
>  
> On notify event, the ‘notification node’ has it’s content set to the event 
> details and time
> which the parser would pick up…. causing the reload
> "
>  
> POC: 
> [hdfs-inotify-zookeeper|https://github.com/ottobackwards/hdfs-inotify-zookeeper]
>  
> !inotify-ascii.png!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-534) Metron Should have service to monitor HDFS for changes to files


 [ 
https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated METRON-534:
---
Description: 
With the INotify library it is possible to receive notifications from HDFS on 
file changes. 
 A service that utilized this and was configured with "rules" to process 
notifications on files of interest ( configurations, rules etc ) in the system 
would be able to send notifications to Zookeeper
 This may allow for things that are better stored in HDFS to remain there, and 
not force the system to put things in Zookeeper just for change notifications.

Issues:
 * Inotify requires hdfs super user, what are the implications on a secure 
cluster etc?
 * performance / scalability and load
 * Can it be deployed with yarn?
 * Is Zookeeper actually the right place for all the configurations

-

 

UPDATE: 1/29/18

"
In the end, what I’m thinking is this:
 
We have an ambari service that runs the notification -> zookeeper
it reads the ‘registration area’ from zookeeper to get it’s state and what to 
watch
post 777 when parsers are installed and registered it is trivial to have my 
installer also register the files to watch
 
the notifications service also has a notification from zookeeper for new 
registrations.
 
On notify event, the ‘notification node’ has it’s content set to the event 
details and time
which the parser would pick up…. causing the reload
"

 

POC: 
[hdfs-inotify-zookeeper|https://github.com/ottobackwards/hdfs-inotify-zookeeper]

 

!inotify-ascii.png!

 

  was:
With the INotify library it is possible to receive notifications from HDFS on 
file changes. 
 A service that utilized this and was configured with "rules" to process 
notifications on files of interest ( configurations, rules etc ) in the system 
would be able to send notifications to Zookeeper
 This may allow for things that are better stored in HDFS to remain there, and 
not force the system to put things in Zookeeper just for change notifications.

Issues:
 * Inotify requires hdfs super user, what are the implications on a secure 
cluster etc?
 * performance / scalability and load
 * Can it be deployed with yarn?
 * Is Zookeeper actually the right place for all the configurations

-

 

!inotify-ascii.png!

 


> Metron Should have service to monitor HDFS for changes to files
> ---
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Minor
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on 
> file changes. 
>  A service that utilized this and was configured with "rules" to process 
> notifications on files of interest ( configurations, rules etc ) in the 
> system would be able to send notifications to Zookeeper
>  This may allow for things that are better stored in HDFS to remain there, 
> and not force the system to put things in Zookeeper just for change 
> notifications.
> Issues:
>  * Inotify requires hdfs super user, what are the implications on a secure 
> cluster etc?
>  * performance / scalability and load
>  * Can it be deployed with yarn?
>  * Is Zookeeper actually the right place for all the configurations
> -
>  
> UPDATE: 1/29/18
> "
> In the end, what I’m thinking is this:
>  
> We have an ambari service that runs the notification -> zookeeper
> it reads the ‘registration area’ from zookeeper to get it’s state and what to 
> watch
> post 777 when parsers are installed and registered it is trivial to have my 
> installer also register the files to watch
>  
> the notifications service also has a notification from zookeeper for new 
> registrations.
>  
> On notify event, the ‘notification node’ has it’s content set to the event 
> details and time
> which the parser would pick up…. causing the reload
> "
>  
> POC: 
> [hdfs-inotify-zookeeper|https://github.com/ottobackwards/hdfs-inotify-zookeeper]
>  
> !inotify-ascii.png!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-534) Metron Should have service to monitor HDFS for changes to files


 [ 
https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler updated METRON-534:
---
Description: 
With the INotify library it is possible to receive notifications from HDFS on 
file changes. 
 A service that utilized this and was configured with "rules" to process 
notifications on files of interest ( configurations, rules etc ) in the system 
would be able to send notifications to Zookeeper
 This may allow for things that are better stored in HDFS to remain there, and 
not force the system to put things in Zookeeper just for change notifications.

Issues:
 * Inotify requires hdfs super user, what are the implications on a secure 
cluster etc?
 * performance / scalability and load
 * Can it be deployed with yarn?
 * Is Zookeeper actually the right place for all the configurations

-

 

!inotify-ascii.png!

 

  was:
With the INotify library it is possible to receive notifications from HDFS on 
file changes. 
A service that utilized this and was configured with "rules" to process 
notifications on files of interest ( configurations, rules etc ) in the system 
would be able to send notifications to Zookeeper
This may allow for things that are better stored in HDFS to remain there, and 
not force the system to put things in Zookeeper just for change notifications. 

Issues:
* Inotify requires hdfs super user, what are the implications on a secure 
cluster etc?
* performance / scalability and load
* Can it be deployed with yarn?
* Is Zookeeper actually the right place for all the configurations


> Metron Should have service to monitor HDFS for changes to files
> ---
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Priority: Minor
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on 
> file changes. 
>  A service that utilized this and was configured with "rules" to process 
> notifications on files of interest ( configurations, rules etc ) in the 
> system would be able to send notifications to Zookeeper
>  This may allow for things that are better stored in HDFS to remain there, 
> and not force the system to put things in Zookeeper just for change 
> notifications.
> Issues:
>  * Inotify requires hdfs super user, what are the implications on a secure 
> cluster etc?
>  * performance / scalability and load
>  * Can it be deployed with yarn?
>  * Is Zookeeper actually the right place for all the configurations
> -
>  
> !inotify-ascii.png!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-534) Metron Should have service to monitor HDFS for changes to files


 [ 
https://issues.apache.org/jira/browse/METRON-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned METRON-534:
--

Assignee: Otto Fowler

> Metron Should have service to monitor HDFS for changes to files
> ---
>
> Key: METRON-534
> URL: https://issues.apache.org/jira/browse/METRON-534
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Minor
> Attachments: inotify-ascii.png
>
>
> With the INotify library it is possible to receive notifications from HDFS on 
> file changes. 
>  A service that utilized this and was configured with "rules" to process 
> notifications on files of interest ( configurations, rules etc ) in the 
> system would be able to send notifications to Zookeeper
>  This may allow for things that are better stored in HDFS to remain there, 
> and not force the system to put things in Zookeeper just for change 
> notifications.
> Issues:
>  * Inotify requires hdfs super user, what are the implications on a secure 
> cluster etc?
>  * performance / scalability and load
>  * Can it be deployed with yarn?
>  * Is Zookeeper actually the right place for all the configurations
> -
>  
> !inotify-ascii.png!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-534) Metron Should have service to monitor HDFS for changes to files