[jira] [Commented] (METRON-811) Enforce Maven Version in Top Level POM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989781#comment-15989781
 ] 

ASF GitHub Bot commented on METRON-811:
---

Github user zezutom closed the pull request at:

https://github.com/apache/incubator-metron/pull/557


> Enforce Maven Version in Top Level POM
> --
>
> Key: METRON-811
> URL: https://issues.apache.org/jira/browse/METRON-811
> Project: Metron
>  Issue Type: Improvement
>Reporter: David M. Lyle
>  Labels: newbie
> Fix For: 0.3.2
>
>
> METRON-793 revealed a latent issue with one of my computers. That computer 
> had been running Maven 3.2.3 without issue. Last night after a pull from 
> master, Metron would no longer build. ( 
> metron-storm-kafka couldn't find Guava)
> At first, I suspected it was a change to the Reactor plugin ordering, but 
> that wasn't the case. At this time, I haven't identified the cause, but 
> upgrading to Maven 3.3.9 allowed the build to complete. 
> If we add the enforcer plugin to the top level pom and enforce the Maven 
> version, we can give an actionable error message and avoid this kind of 
> weirdness. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException

[Anand Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989678#comment-15989678
 ] 

Anand Subramanian commented on METRON-897:
--

Hi [~nickwallen], I did the change on the nodes themselves, not through Ambari. 

> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start.  This 
> is in an environment with 1 master node and 2 data nodes all running on 3 
> separate hosts.  The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main" 
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: 
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block 
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error 
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited, 
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989676#comment-15989676
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/555
  
We should attach those logs to a jira about expanding the parser es 
integration tests


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989627#comment-15989627
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/555
  
The most recent time I noticed the issue was when I was testing #510.  In 
that case, I spun up full-dev and pushed [these 
logs](https://github.com/JonZeolla/incubator-metron/blob/master/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
 onto the bro kafka topic and noticed that only 7 of them were making it to ES, 
and it was spitting the error messages into the storm logs for the indexing 
worker.  If this fix works properly, all 10 of those logs should now get 
properly indexed.


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-811) Enforce Maven Version in Top Level POM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989544#comment-15989544
 ] 

ASF GitHub Bot commented on METRON-811:
---

GitHub user zezutom opened a pull request:

https://github.com/apache/incubator-metron/pull/557

METRON-811: Enforce Maven Version in Top Level POM

## Contributor Comments
Enforcing Maven version >= 3.3.1 and Java 8


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zezutom/incubator-metron METRON-811

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/557.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #557


commit 6d3309ac6683212ff5c608678bfd41ea287cc079
Author: zezutom 
Date:   2017-04-28T22:17:50Z

Enforcing Maven version >= 3.3.1 and Java 8




> Enforce Maven Version in Top Level POM
> --
>
> Key: METRON-811
> URL: https://issues.apache.org/jira/browse/METRON-811
> Project: Metron
>  Issue Type: Improvement
>Reporter: David M. Lyle
>  Labels: newbie
> Fix For: 0.3.2
>
>
> METRON-793 revealed a latent issue with one of my computers. That computer 
> had been running Maven 3.2.3 without issue. Last night after a pull from 
> master, Metron would no longer build. ( 
> metron-storm-kafka couldn't find Guava)
> At first, I suspected it was a change to the Reactor plugin ordering, but 
> that wasn't the case. At this time, I haven't identified the cause, but 
> upgrading to Maven 3.3.9 allowed the build to complete. 
> If we add the enforcer plugin to the top level pom and enforce the Maven 
> version, we can give an actionable error message and avoid this kind of 
> weirdness. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-811) Enforce Maven Version in Top Level POM

[Tomas Zezula (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989461#comment-15989461
 ] 

Tomas Zezula commented on METRON-811:
-

I've looked into this further and decided to go for the suggested enforcer 
plugin as a single point of control. I'd start with the following rules:
* Maven: a range of supported versions
* Java 8


> Enforce Maven Version in Top Level POM
> --
>
> Key: METRON-811
> URL: https://issues.apache.org/jira/browse/METRON-811
> Project: Metron
>  Issue Type: Improvement
>Reporter: David M. Lyle
>  Labels: newbie
> Fix For: 0.3.2
>
>
> METRON-793 revealed a latent issue with one of my computers. That computer 
> had been running Maven 3.2.3 without issue. Last night after a pull from 
> master, Metron would no longer build. ( 
> metron-storm-kafka couldn't find Guava)
> At first, I suspected it was a change to the Reactor plugin ordering, but 
> that wasn't the case. At this time, I haven't identified the cause, but 
> upgrading to Maven 3.3.9 allowed the build to complete. 
> If we add the enforcer plugin to the top level pom and enforce the Maven 
> version, we can give an actionable error message and avoid this kind of 
> weirdness. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-196) Deployment Fails Without Ansible 2.0.0.2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989425#comment-15989425
 ] 

ASF GitHub Bot commented on METRON-196:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/219
  
@2xyo Did #499 take care of your concerns here (as @JonZeolla  mentioned)?  
If so, please close this PR when you get a chance.  Otherwise, please describe 
why this PR might still be needed.  Thanks!


> Deployment Fails Without Ansible 2.0.0.2
> 
>
> Key: METRON-196
> URL: https://issues.apache.org/jira/browse/METRON-196
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: David M. Lyle
>Priority: Minor
>  Labels: 0.2.2BETA
> Fix For: 0.4
>
>
> The following error occurs when deploying Metron with versions other than 
> 2.0.0.2; particularly version 2.0.1.  The current work around is to ask users 
> to downgrade Ansible version per 
> https://cwiki.apache.org/confluence/display/METRON/Downgrade+Ansible.
> ASK [elasticsearch : Add Elasticsearch templates for topologies] 
> **
> failed: [node1] (item={u'sensor': u'bro', u'file': {'mappings': {'bro_doc': 
> {'_timestamp': {'enabled': True}, 'properties': 
> {'enrichments:geo:ip_dst_addr:location_point': {'type': 'geo_point'}, 
> 'timestamp': {'type': 'date', 'format': 'epoch_millis', 'template': 
> 'bro_index*'}}) => {"content": "", "content_length": "450", "content_type": 
> "application/json; charset=UTF-8", "failed": true, "item": {"file": 
> {"mappings": {"bro_doc": {"_timestamp": {"enabled": true}, "properties": 
> {"enrichments:geo:ip_dst_addr:location_point": {"type": "geo_point"}, 
> "timestamp": {"format": "epoch_millis", "type": "date", "template": 
> "bro_index*"}, "sensor": "bro"}, "msg": "Status code was not [200]: HTTP 
> Error 400: Bad Request", "redirected": false, "status": 400, "url": 
> "http://node1:9200/_template/template_bro"}
> failed: [node1] (item={u'sensor': u'yaf', u'file': {'mappings': {'yaf_doc': 
> {'_timestamp': {'enabled': True}, 'properties': {'uflags': {'type': 
> 'string'}, 'pkt': {'type': 'string'}, 'app': {'type': 'string'}, 'rtt': 
> {'type': 'string'}, 'tag': {'type': 'string'}, 'duration': {'type': 
> 'string'}, 'riflags': {'type': 'string'}, 'sip': {'type': 'string'}, 'proto': 
> {'type': 'string'}, 'rtag': {'type': 'string'}, 'oct': {'type': 'string'}, 
> 'risn': {'type': 'string'}, 'end-time': {'type': 'string'}, 'end-reason': 
> {'type': 'string'}, 'timestamp': {'type': 'date', 'format': 'epoch_millis'}, 
> 'dp': {'type': 'string'}, 'enrichments:geo:ip_dst_addr:location_point': 
> {'type': 'geo_point'}, 'roct': {'type': 'string'}, 'sp': {'type': 'string'}, 
> 'iflags': {'type': 'string'}, 'isn': {'type': 'string'}, 'ruflags': {'type': 
> 'string'}, 'rpkt': {'type': 'string'}, 'dip': {'type': 'string', 
> 'template': 'yaf_index*'}}) => {"content": "", "content_length": "450", 
> "content_type": "application/json; charset=UTF-8", "failed": true, "item": 
> {"file": {"mappings": {"yaf_doc": {"_timestamp": {"enabled": true}, 
> "properties": {"app": {"type": "string"}, "dip": {"type": "string"}, "dp": 
> {"type": "string"}, "duration": {"type": "string"}, "end-reason": {"type": 
> "string"}, "end-time": {"type": "string"}, 
> "enrichments:geo:ip_dst_addr:location_point": {"type": "geo_point"}, 
> "iflags": {"type": "string"}, "isn": {"type": "string"}, "oct": {"type": 
> "string"}, "pkt": {"type": "string"}, "proto": {"type": "string"}, "riflags": 
> {"type": "string"}, "risn": {"type": "string"}, "roct": {"type": "string"}, 
> "rpkt": {"type": "string"}, "rtag": {"type": "string"}, "rtt": {"type": 
> "string"}, "ruflags": {"type": "string"}, "sip": {"type": "string"}, "sp": 
> {"type": "string"}, "tag": {"type": "string"}, "timestamp": {"format": 
> "epoch_millis", "type": "date"}, "uflags": {"type": "string", "template": 
> "yaf_index*"}, "sensor": "yaf"}, "msg": "Status code was not [200]: HTTP 
> Error 400: Bad Request", "redirected": false, "status": 400, "url": 
> "http://node1:9200/_template/template_yaf"}
> failed: [node1] (item={u'sensor': u'snort', u'file': {'mappings': 
> {'snort_doc': {'_timestamp': {'enabled': True}, 'properties': 
> {'enrichments:geo:ip_dst_addr:location_point': {'type': 'geo_point'}, 
> 'timestamp': {'type': 'date', 'format': 'epoch_millis', 'template': 
> 'snort_index*'}}) => {"content": "", "content_length": "450", "content_type": 
> "application/json; charset=UTF-8", "failed": true, "item": {"file": 
> {"mappings": {"snort_doc": {"_timestamp": {"enabled": true}, "properties": 
> {"enrichments:geo:ip_dst_addr:location_point": {"type": "geo_point"}, 
> "timestamp": {"format": "epoch_millis", "type": "date", "template

[jira] [Commented] (METRON-836) Use Pycapa with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989419#comment-15989419
 ] 

ASF GitHub Bot commented on METRON-836:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/524


> Use Pycapa with Kerberos
> 
>
> Key: METRON-836
> URL: https://issues.apache.org/jira/browse/METRON-836
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> How can we use Pycapa with Kerberos?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989389#comment-15989389
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/555
  
How should we test this @JonZeolla ?  Can you layout some steps for us?


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-836) Use Pycapa with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989324#comment-15989324
 ] 

ASF GitHub Bot commented on METRON-836:
---

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/524
  
Looks good!  +1


> Use Pycapa with Kerberos
> 
>
> Key: METRON-836
> URL: https://issues.apache.org/jira/browse/METRON-836
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> How can we use Pycapa with Kerberos?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-836) Use Pycapa with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989299#comment-15989299
 ] 

ASF GitHub Bot commented on METRON-836:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/524
  
> One last minor documentation issue: this PR description includes steps to 
install Python 2.7 but nowhere is that mentioned in the README.

The latest README now mentions Python 2.7.

> The only issue above I feel is necessary to address would adding the 
extra Kafka ACL command for the group to the README

I updated the README and the PR description to set the ACL on the topic and 
on the group.

Yay!


> Use Pycapa with Kerberos
> 
>
> Key: METRON-836
> URL: https://issues.apache.org/jira/browse/METRON-836
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> How can we use Pycapa with Kerberos?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (METRON-865) Additional Mpack bug fixes and improvements, that affect Ambari database

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated METRON-865:
--
Description: 
Multiple bug fixes and recommended improvements were found in the course of 
implementing METRON-608 that are unrelated to METRON-609 (singlenode install).  
Almost all items relate to Elasticsearch.

About half the fixes did not impact the Ambari database, and were done in 
METRON-634 (PR#532).
This jira provides the work items for changes that do impact the Ambari 
database, and should therefore be done with an Mpack version bump and database 
upgrade script. 

h2. Affects Ambari database, needs db upgrade script:
* status_params.py, which redundantly defines pid_dir as a python variable, is 
unnecessary and unused by the ES portion of the Mpack.  It can be removed.
* pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than 
elastic-env.xml, as it is a parameter that must be provided to ES at 
launch-time, but is not something there's any reason for the admin to change in 
usual circumstances.
* conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in 
elastic-sysconfig.xml.  While it too is a parameter that must be provided to ES 
at launch-time, it is typically left to the installing admin where to put the 
config files.
* The configuration parameter names in elastic-site.xml should be improved in 
several instances to make the semantics more obvious to the human reader (who 
may not be real familiar with Elasticsearch configuration).  Mouse-over 
documentation will continue to provide the ES config parameter equivalents.  In 
particular, suggest:
{code}
cluster_name -> es_cluster_name  (to distinguish ES cluster from Stack cluster)
zen_discovery_ping_unicast_hosts -> es_cluster_hosts
network_host -> network_bindings  (these are in fact interface names, not host 
names)
{code}
* This item moved to METRON-905 : -There are at least two places in 
elasticsearch.master.yaml.j2 (zen_discovery_ping_unicast_hosts and 
network_host) where needed square brackets are either missing or included in 
the configuration string.  To be consistent with other usages, and less prone 
to human error, the square brackets should not be in the configuration string 
but rather should be provided in the template text.-
* "data_dir" apparently should be eliminated (from elastic-sysconfig) in 
preference for "path_data" (in elastic-site.xml).  The latter value ends up 
overriding the former anyway, but the existence of the former is confusing and 
unnecessary.
* All four configuration parameters in elastic-env.xml should be moved to 
elastic-site.xml, because they are all reasonable to set in a "site.xml" file 
and do end up in the .yml file that ES uses instead of "site.xml", and do NOT 
end up in environment variables.  The only parameters that end up in env vars 
are set in elastic-sysconfig, and the ES launch process in fact ignores the 
elastic-env.sh file that is templated in elastic-env.xml (which consists only 
of JAVA_HOME and PATH).  Therefore we could also eliminate elastic-env.sh and 
hence entirely remove elastic-env.xml, or we could choose to keep the small 
elastic-env.sh file and its template, just to remind people that it is 
necessary to have JAVA_HOME defined.
* In METRON/0.3.0/configuration/metron-env.xml and 
METRON/0.3.0/package/scripts/params/params_linux.py, the value 
"metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is 
appropriate to require it to be subordinate to "metron_apps_hdfs_dir".  Thus it 
can be removed from metron-env.xml and set to 
"\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py.  This also 
eliminates a really unacceptable use of "double format".



  was:
Multiple bug fixes and recommended improvements were found in the course of 
implementing METRON-608 that are unrelated to METRON-609 (singlenode install).  
Almost all items relate to Elasticsearch.

About half the fixes did not impact the Ambari database, and were done in 
METRON-634 (PR#532).
This jira provides the work items for changes that do impact the Ambari 
database, and should therefore be done with an Mpack version bump and database 
upgrade script. 

h2. Affects Ambari database, needs db upgrade script:
* status_params.py, which redundantly defines pid_dir as a python variable, is 
unnecessary and unused by the ES portion of the Mpack.  It can be removed.
* pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than 
elastic-env.xml, as it is a parameter that must be provided to ES at 
launch-time, but is not something there's any reason for the admin to change in 
usual circumstances.
* conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in 
elastic-sysconfig.xml.  While it too is a parameter that must be provided to ES 
at launch-time, it is typically left to the installing admin where to put the 
config files.
* The configura

[jira] [Created] (METRON-905) Fix square-bracket tolerance and default network interface bindings for ES

[Matt Foley (JIRA)]

Matt Foley created METRON-905:
-

 Summary: Fix square-bracket tolerance and default network 
interface bindings for ES
 Key: METRON-905
 URL: https://issues.apache.org/jira/browse/METRON-905
 Project: Metron
  Issue Type: Sub-task
Affects Versions: 0.3.1
Reporter: Matt Foley
Assignee: Matt Foley


Community agrees we should change Elasticsearch default binding to "all 
interfaces" (0.0.0.0).

In addition, taking opportunity to fix the problem with square brackets on 
these parameters:
* zen_discovery_ping_unicast_hosts 
* network_host

For all other parameters in ES and other Metron sub-components, the square 
brackets around lists are not needed in the parameter value, but are instead 
provided in the template text.  Making it so here, with the additional step 
that if the user provides them (due to old habit) the extra square brackets 
will be stripped before they cause a problem.  This should make the change less 
backward-incompatible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-836) Use Pycapa with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989157#comment-15989157
 ] 

ASF GitHub Bot commented on METRON-836:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/524
  
Yes, good feedback.  I had updated the README to mention Py 2.7, but I now 
just realized that I never pushed that out.  Will get that pushed out and 
updated.


> Use Pycapa with Kerberos
> 
>
> Key: METRON-836
> URL: https://issues.apache.org/jira/browse/METRON-836
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> How can we use Pycapa with Kerberos?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-819) Document kafka console producer parameter for sensors with kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989143#comment-15989143
 ] 

ASF GitHub Bot commented on METRON-819:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/507
  
I also remember (after fighting with this for a while this morning) that if 
you don't have a JAAS config file defined then it won't work.  Should this be 
part of the docs in this PR??

Without it, you just get this error.
```
[root@y136 ~]# kafka-console-consumer.sh  --zookeeper y113:2181 --topic bro 
--security-protocol SASL_PLAINTEXT
[2017-04-28 16:47:20,596] WARN Could not login: the client is being asked 
for a password, but the Zookeeper client code does not currently support 
obtaining a password from the user. Make sure that the client is configured to 
use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' 
and restart the client. If you still get this message after that, the TGT in 
the ticket cache has expired and must be manually refreshed. To do so, first 
determine if you are using a password or a keytab. If the former, run kinit in 
a Unix shell in the environment of the user who is running this Zookeeper 
client using the command 'kinit ' (where  is the name of the 
client's Kerberos principal). If the latter, do 'kinit -k -t  ' 
(where  is the name of the Kerberos principal, and  is the 
location of the keytab file). After manually refreshing your cache, restart 
this client. If you continue to see this message after manually refreshing your 
cache, ensure that your KDC host's clock is in sync with this host's clock. 
(org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-28 16:47:20,599] WARN SASL configuration failed: 
javax.security.auth.login.LoginException: No password provided Will continue 
connection to Zookeeper server without SASL authentication, if Zookeeper server 
allows it. (org.apache.zookeeper.ClientCnxn)
No brokers found in ZK.
```
After doing the following, then it works for me.

1. Define `~/.java.login.config` 

```
[root@y137 ~]# cat ~/.java.login.config
KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=false
  useKeyTab=true
  principal="yaf/y137...@example.com"
  keyTab="/etc/security/keytabs/yaf.service.keytab"
  renewTicket=true
  debug=true
  serviceName="kafka"
  storeKey=true;
};
```

2. Tell the JVM where to find your JAAS file.

```
[root@y137 ~]# cat 
/usr/lib/jvm/java-1.8.0-openjdk/jre/lib/security/java.security | grep login
# Class to instantiate as the javax.security.auth.login.Configuration
login.configuration.provider=sun.security.provider.ConfigFile
# Default login configuration file
#login.config.url.1=file:${user.home}/.java.login.config
login.config.url.1=file:${user.home}/.java.login.config
```


> Document kafka console producer parameter for sensors with kerberos
> ---
>
> Key: METRON-819
> URL: https://issues.apache.org/jira/browse/METRON-819
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Snort and Yaf use the Kafka console producer. These sensors need an 
> additional parameter to work with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-836) Use Pycapa with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989136#comment-15989136
 ] 

ASF GitHub Bot commented on METRON-836:
---

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/524
  
I was able to get this to work by performing one extra step.  The PR 
description did not include the Kafka ACL command to permission the topic, only 
the "pycapa" group.  After I did this, I was able to verify data with the 
pycapa --consumer command.  The topic ACL command is included in the Pycapa 
README so I think simply adding the ACL command for the group is all that's 
needed.

I did notice that running the pycapa --producer command causes a Kafka 
service ticket to be cached.  Since we're passing in the metron principal and 
keytab I would expect that ticket to be in the cache instead.  While it's 
unexpected it did not cause any errors.

One last minor documentation issue:  this PR description includes steps to 
install Python 2.7 but nowhere is that mentioned in the README.  I know it's 
only needed because full dev is centos 6 but it might be helpful to at least 
call out Python 2.7 as a prereq.  Maybe it's assumed the user knows that but it 
tripped me up when I was initially trying to spin this up and that wasn't 
included in the PR description.

The only issue above I feel is necessary to address would adding the extra 
Kafka ACL command for the group to the README.  Pending that change, +1.


> Use Pycapa with Kerberos
> 
>
> Key: METRON-836
> URL: https://issues.apache.org/jira/browse/METRON-836
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> How can we use Pycapa with Kerberos?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (METRON-904) service sensor-stub restart is broken

[Jon Zeolla (JIRA)]

Jon Zeolla created METRON-904:
-

 Summary: service sensor-stub restart is broken
 Key: METRON-904
 URL: https://issues.apache.org/jira/browse/METRON-904
 Project: Metron
  Issue Type: Bug
Reporter: Jon Zeolla


I'm still seeing an issue with `service sensor-stubs restart bro snort` - it 
looks like it's restarting all `sensor-stubs`, including `yaf`, even when I 
pass specific sensor stubs.  

```
[root@node1 ~]# service sensor-stubs restart bro snort
Stopping sensor-stubs...
..   bro: Stopped [510]
   yaf: Not running
.. snort: Stopped [32669]
Starting sensor-stubs...
   bro: Ok [20884]
   yaf: Ok [20889]
 snort: Ok [20912]
[root@node1 ~]# service sensor-stubs status
Checking sensor-stubs...
   bro: Running [20884]
   yaf: Running [20889]
 snort: Running [20912]
```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-896) Document Having Kerberos Issue Renewable Tickets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989110#comment-15989110
 ] 

ASF GitHub Bot commented on METRON-896:
---

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/553
  
Updated after merging in master, which included the README fixes as well as 
moving the Ambari instructions alongside the Manual instructions.


> Document Having Kerberos Issue Renewable Tickets
> 
>
> Key: METRON-896
> URL: https://issues.apache.org/jira/browse/METRON-896
> Project: Metron
>  Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> Apparently in some circumstances, a default kerberos install on CentOS7 will 
> not be configured to issue renewable keytabs.  This causes issues with 
> deploying topologies.
> Add documentation for both initial setup, as well as allowing a principal to 
> get renewable tickets if the KDC is already setup.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-903) Create a connections report in Zeppelin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989097#comment-15989097
 ] 

ASF GitHub Bot commented on METRON-903:
---

GitHub user justinleet opened a pull request:

https://github.com/apache/incubator-metron/pull/556

METRON-903: Create a connections report in Zeppelin

## Contributor Comments
The added Zeppelin dashboard is pretty straightforward.

For each of the sensors (Bro/Snort/Yaf) and for all three as a whole, 
groups everything by (ip_src_addr,ip_dst_addr) for a given range, counts it all 
up, and spits out tables.

The range is provided by two fields, start and end, which are dates that 
can be formatted by a user provided formatting string.  e.g.
```
2017-04-28 14:19:19 -> -MM-dd HH:mm:ss
2017-04-28 -> -MM-dd
```

## Test Plan
To test, spin up full-dev. Ensure data flows through for the sensors.  For 
Yaf, which is not enabled by default, it'll be necessary to start the 
sensor-stub
```
service sensor-stubs start yaf
```

It'll also be necessary to add Yaf to the list of sensors run in Ambari. To 
do so, stop Metron, edit "Metron Parsers" to include "yaf" (or if the other 
sensors already have data output, feel free to just make it "yaf" only).

Let data flow through for a little while (enough so that we can reasonable 
make range filters).

Once some data has gone through, we'll need to have an instance of 
Zeppelin.  Because of the size of the Vagrant instance, we'll want to shut down 
unneeded services.  Shutdown Metron, Kibana, Storm, Kafka, and HBase.

Install Zeppelin from "Actions - Add Service".  It'll prompt you to install 
Spark and Hive, do so.  Configuration is pretty trivial, all that's necessary 
is to set an arbitrary Hive database password.  Let this run.  The Hive service 
check likes to fail on our Vagrant, but it's benign (some impersonation 
configuration issue unrelated to actually running our queries).  Ignore it and 
accept the installation.

From Metron's "Service Actions", run the "Zeppelin Notebook Import", to 
load our notebooks into Zeppelin.  Use the quick links to navigate to the 
Zeppelin UI.

Go into the "Metron - Connection Report" notebook.

Each service (and the aggregate) will have something similar to the 
following:
https://cloud.githubusercontent.com/assets/5077341/25536613/764888ea-2c09-11e7-876f-0779cf955ee7.png";>

In my case, note the second row:
```
192.168.66.1192.168.66.121  439
```

We want to validate this range, so we can check ES and see what a count 
against the same range returns
```
curl -XPOST 'node1:9200/y*/_count?pretty' -H 'Content-Type: 
application/json' -d'
{
   "query": {
  "filtered": {
 "filter": {
"and": [
   {
  "bool": {
 "must": [
{
   "term": {
  "ip_src_addr": "192.168.66.1"
   }
},
{
   "term": {
  "ip_dst_addr": "192.168.66.121"
   }
}
 ]
  }
   },
   {
  "range": {
 "timestamp": {
"gte": 1493389159000,
"lt": 1493389859000
 }
  }
   }
]
 }
  }
   }
}
'
```
The result is:
```
{
  "count" : 439,
  "_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
  }
}
```
They're the same, so we're good.

Additionally, the dates and formatting can be changed and run as well.

This exercise can be repeated across the sensors.
## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989077#comment-15989077
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/555
  
Or maybe we need something new, specifically for these templates.


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989061#comment-15989061
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/555
  
We should look at extending the indexing integration tests maybe.
another idea for me is if the ES templates get bundled with the parsers, we 
can have new parser integration tests for that


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989043#comment-15989043
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/555
  
Good catch, @JonZeolla   I am trying to think of a way we could automate 
the testing of this thing.  Somehow grab a particular record that triggers the 
issue and make sure thats part of our test suite.  Don't have any ideas right 
now, but I need to noodle on that some more (and get some more coffee).


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs

[Jon Zeolla (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989024#comment-15989024
 ] 

Jon Zeolla commented on METRON-508:
---

We should also improve the tokenization to be more sane.  For instance, the 
addition of things like 
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pathhierarchy-tokenizer.html
 (forward or reverse, depending on the field) would be very helpful.

> Expand Elasticsearch templates to support the standard bro logs
> ---
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn, 
> HTTP, and DNS.  We should provide additional templates so that an 
> out-of-the-box bro install can send all of its logs into Metron and they will 
> get probably indexed in elasticsearch.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989017#comment-15989017
 ] 

ASF GitHub Bot commented on METRON-902:
---

GitHub user JonZeolla opened a pull request:

https://github.com/apache/incubator-metron/pull/555

METRON-902

## Contributor Comments
It appears that `bro_index.template` regressed a bit when it was migrated 
to the ambari mpack.  If you run this up in vagrant and let the bro sensor-stub 
run for a while you will see ES indexing issues for dns logs because answers is 
not always an IP address.  This fixes that, and I will try to get a look at the 
whole bro ES indexing template for other issues, as a part of METRON-348.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/incubator-metron METRON-902

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #555


commit c5955b37287e5e41acb6803de90d7f8705192bf1
Author: Jon Zeolla 
Date:   2017-04-25T13:47:17Z

METRON-883

commit ce055a0cab444ca92f4f80fa0f6f0adde8c20ab5
Author: Jon Zeolla 
Date:   2017-04-28T15:20:48Z

Merge branch 'master' of https://github.com/apache/incubator-metron

commit e00f11807c050c768e3906d754f16583d68ddbe1
Author: Jon Zeolla 
Date:   2017-04-28T15:30:23Z

METRON-902 ES improperly indexes Bro logs




> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/com

[jira] [Created] (METRON-903) Create a connections report in Zeppelin

[Justin Leet (JIRA)]

Justin Leet created METRON-903:
--

 Summary: Create a connections report in Zeppelin
 Key: METRON-903
 URL: https://issues.apache.org/jira/browse/METRON-903
 Project: Metron
  Issue Type: New Feature
Reporter: Justin Leet
Assignee: Justin Leet


User types in range into a search box
System generates connections report:
IP(A) -> IP(B) : # of times



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (METRON-902) ES improperly indexes Bro logs

[Jon Zeolla (JIRA)]

Jon Zeolla created METRON-902:
-

 Summary: ES improperly indexes Bro logs
 Key: METRON-902
 URL: https://issues.apache.org/jira/browse/METRON-902
 Project: Metron
  Issue Type: Bug
Reporter: Jon Zeolla
Assignee: Jon Zeolla


It appears that an old issue has been reintroduced into the ES template for 
indexing bro DNS logs.  It is possible that other issues have been reintroduced 
as well, as I have not yet reviewed the template holistically.

Initial fix:  
https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
Reintroduction:  
https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-835) Use Profiler with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988830#comment-15988830
 ] 

ASF GitHub Bot commented on METRON-835:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/554


> Use Profiler with Kerberos
> --
>
> Key: METRON-835
> URL: https://issues.apache.org/jira/browse/METRON-835
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> Define how the Profiler can be used with Kerberos. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-835) Use Profiler with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988807#comment-15988807
 ] 

ASF GitHub Bot commented on METRON-835:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/554
  
+1, great job, Mike.  Thank you very much for taking this up.


> Use Profiler with Kerberos
> --
>
> Key: METRON-835
> URL: https://issues.apache.org/jira/browse/METRON-835
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> Define how the Profiler can be used with Kerberos. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException

[Nick Allen (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988793#comment-15988793
 ] 

Nick Allen commented on METRON-897:
---

Anand - Did you make the change in Ambari and let it push out that change to 
master and data nodes?  Or did you make that change on the Elasticsearch master 
and data nodes?

If the fix for you was different, then that complicates matters.



> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start.  This 
> is in an environment with 1 master node and 2 data nodes all running on 3 
> separate hosts.  The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main" 
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: 
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block 
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error 
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited, 
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException

[Nick Allen (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988789#comment-15988789
 ] 

Nick Allen commented on METRON-897:
---

I think binding to all interfaces is a reasonable default.  We just want 
something that works out-of-the-box and binding to all interfaces is the most 
logical way to do that.

> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start.  This 
> is in an environment with 1 master node and 2 data nodes all running on 3 
> separate hosts.  The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main" 
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: 
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block 
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error 
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited, 
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-836) Use Pycapa with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988765#comment-15988765
 ] 

ASF GitHub Bot commented on METRON-836:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/524
  
Hey, the more reviewers the better.  No harm in that.


> Use Pycapa with Kerberos
> 
>
> Key: METRON-836
> URL: https://issues.apache.org/jira/browse/METRON-836
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> How can we use Pycapa with Kerberos?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-835) Use Profiler with Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988763#comment-15988763
 ] 

ASF GitHub Bot commented on METRON-835:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/554
  
+1 Very nice, Mike.  Thanks for cleaning that up.


> Use Profiler with Kerberos
> --
>
> Key: METRON-835
> URL: https://issues.apache.org/jira/browse/METRON-835
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> Define how the Profiler can be used with Kerberos. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException

[David M. Lyle (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988734#comment-15988734
 ] 

David M. Lyle commented on METRON-897:
--

Now that I look closely at it, I'd prefer not to default to eth0, with or 
without brackets. That seems brittle. How would people feel about binding to 
all interfaces by default. IIRC, that is achieved by setting network.bind_host 
to 0.

> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start.  This 
> is in an environment with 1 master node and 2 data nodes all running on 3 
> separate hosts.  The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main" 
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: 
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block 
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error 
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited, 
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-795) Install Metron REST with Ambari MPack

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988712#comment-15988712
 ] 

ASF GitHub Bot commented on METRON-795:
---

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/500
  
Matt,

METRON-795 is ready for review but it's substantial and I imagine it will
take some time to make it through the process.

Ryan

On Wed, Apr 26, 2017 at 7:33 PM, Matt Foley 
wrote:

> @merrimanr  , since METRON-859 / PR#535
> just went in, will this be going in too? Thanks.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> 
,
> or mute the thread
> 

> .
>



> Install Metron REST with Ambari MPack
> -
>
> Key: METRON-795
> URL: https://issues.apache.org/jira/browse/METRON-795
> Project: Metron
>  Issue Type: Improvement
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> The REST application should be included in the Ambari MPack.  This task 
> includes creating a RPM file for the metron-rest module and installing it via 
> Ambari MPack.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-896) Document Having Kerberos Issue Renewable Tickets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988611#comment-15988611
 ] 

ASF GitHub Bot commented on METRON-896:
---

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/553
  
Looks like https://github.com/apache/incubator-metron/pull/554 cleans up a 
lot of things. I'll wait for that to go in, then I'll adjust this PR and we'll 
revisit it.


> Document Having Kerberos Issue Renewable Tickets
> 
>
> Key: METRON-896
> URL: https://issues.apache.org/jira/browse/METRON-896
> Project: Metron
>  Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> Apparently in some circumstances, a default kerberos install on CentOS7 will 
> not be configured to issue renewable keytabs.  This causes issues with 
> deploying topologies.
> Add documentation for both initial setup, as well as allowing a principal to 
> get renewable tickets if the KDC is already setup.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException

[Anand Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988571#comment-15988571
 ] 

Anand Subramanian edited comment on METRON-897 at 4/28/17 10:08 AM:


I faced the same issue on an openstack deployment. In my case, adding the 
square brackets did not help on the ES master node, however adding the IP 
address of the node running ES master helped to bring up the service.

Adding square brackets worked on the data nodes though.


was (Author: anandsubbu):
I faced the same issue on an openstack deployment. In my case, adding the 
square brackets did not help, however adding the IP address of the node running 
ES master helped to bring up the service.

> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start.  This 
> is in an environment with 1 master node and 2 data nodes all running on 3 
> separate hosts.  The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main" 
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: 
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block 
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error 
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited, 
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elas

[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException

[Anand Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988571#comment-15988571
 ] 

Anand Subramanian commented on METRON-897:
--

I faced the same issue on an openstack deployment. In my case, adding the 
square brackets did not help, however adding the IP address of the node running 
ES master helped to bring up the service.

> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start.  This 
> is in an environment with 1 master node and 2 data nodes all running on 3 
> separate hosts.  The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main" 
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: 
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block 
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at 
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at 
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error 
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited, 
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)