[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1242 > specifically setting `topology.ackers.executors` equal to the number of inbound Kafka topic partitions And the number of Workers too. i.e. No. of Kafka partitions = No. of workers (`ra_indexing_workers`) = No. of Ackers (`ra_indexing_acker_executors`) In my test, I set this to 10 and saw no failed tuples in the kafkaSpout. My +1 holds. Thanks @mmiklavc ! ---
[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1242
> The last remaining bit before I merge this will be sharing some results
around the performance/regression testing performed by @anandsubbu.
I completed a set of performance comparison tests between the REST and
Transport clients. Here's the update.
### Note on the Testing Done
* The performance regression comparison tests are run on a multi-node
VM-based environment by:
a) Running a set of tests using the ES REST Client
b) Swap out the REST client with the Transport client, and re-run the same
set of tests.
### Observations
1) The indexing rate numbers with the REST client is similar to that of the
Transport client.
2) I had noticed storm rebalance error messages for the kafkaSpout for both
REST and Transport client runs, which required to adjust the values of
`request.timeout.ms` and `session.timeout.ms` in the ES flux file under
'kafkaProps'. The errors were resolved after making these changes.
3) For the REST client, I observed listener timeout exceptions with the
default settings. I had to increase max retry and socket timeout values in
global config as follows, in order to resolve the exceptions.
```
"es.client.settings" : {
"max.retry.timeout.millis" : "6",
"socket.timeout.millis" : "6"
}
```
4) For both cases, the kafkaSpout complete latency was seen to be on the
higher side (~30 secs)
5) Lastly, I noticed that there were a number of failed tuples seen in the
kafkaSpout of the RA Indexing topology. I am continuing to dig more into this,
but since this was seen for both the REST and Transport client tests, I am
suspecting it could be an anomaly with the test environment, or a different
issue common to both REST/Transport clients.
Based on the above, it is seen that there is no immediate regression in
performance b/w the REST and Transport client.
+1, great work @mmiklavc
---
[GitHub] metron pull request #1256: METRON-1861: REST fails to start when LDAP enable...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1256 METRON-1861: REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty ## Contributor Comments Set 'Active Spring profiles' to just `ldap`, when `metron_spring_profiles_active` is blank. ### Testing Steps 1. Set 'LDAP Enabled' to On and set 'Active Spring profiles' as blank 2. Restart REST service and validate that it starts up fine. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1861 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1256.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1256 commit ad781b6da1b03524ea63c318ebe68c47758c5b64 Author: Anand Subramanian Date: 2018-11-08T09:24:45Z Handle case when metron_spring_profiles_active is empty ---
[GitHub] metron issue #1246: METRON-1844: Allow for LDAP to be used for authenticatio...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1246 My +1 as well. I spun up a multi-node cluster and set this up using the demo LDAP service provided by Knox. I was able to authenticate using the demo users for all the Swagger, Alerts and Management UI interfaces. Nicely done, @justinleet ! ---
[GitHub] metron pull request #1246: METRON-1844: Allow for LDAP to be used for authen...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/1246#discussion_r229259363
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
---
@@ -267,6 +274,21 @@
if 'solr-config-env' in config['configurations']:
solr_principal_name = solr_principal_name.replace('_HOST',
hostname_lowercase)
+# LDAP
+metron_ldap_url =
config['configurations']['metron-security-env']['metron.ldap.url']
+metron_ldap_userdn =
config['configurations']['metron-security-env']['metron.ldap.bind.dn']
+metron_ldap_password =
config['configurations']['metron-security-env']['metron.ldap.bind.password']
+metron_ldap_user_pattern =
config['configurations']['metron-security-env']['metron.ldap.user.dnpattern']
+metron_ldap_user_password =
config['configurationsmetron_spring_profiles_active']['metron-security-env']['metron.ldap.user.password']
--- End diff --
Looks like a mispaste. `metron_spring_profiles_active` should be removed.
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen ! I have seen this issue previously as well.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen ! I have bumped into this issue previously
as well.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen ! I have bumped into this issue previously
as well.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen ! I have seen this issue previously as well.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen ! I have seen this issue previously as well.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen ! I have seen this issue previously as well.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
+1
Thanks for the fix @nickwallen !
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error (which I have also seen previously many
times!):
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` to see
that the ES mpack install goes through fine:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
ok: [node1]
```
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
Hey @nickwallen , I am seeing an error at the `ambari_gather_facts : Ask
Ambari: core_site_tag` step when trying to validate the fix.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` after
which I am seeing the below error:
```
TASK [ambari_gather_facts : Ask Ambari: namenode_host]
*
changed: [node1]
TASK [ambari_gather_facts : set_fact]
**
ok: [node1]
TASK [ambari_gather_facts : Ask Ambari: core_site_tag]
*
fatal: [node1]: FAILED! => {"changed": true, "cmd": "curl -s -u admin:admin
-X GET -H \"X-Requested-By: ambari\"
'http://node1:8080/api/v1/clusters/metron_cluster/hosts/node1/host_components/NAMENODE'
| python -c 'import sys, json; print
json.load(sys.stdin)[\"HostRoles\"][\"actual_configs\"][\"core-site\"][\"default\"]'",
"delta": "0:00:00.060271", "end": "2018-10-21 23:19:31.928568", "msg":
"non-zero return code", "rc": 1, "start": "2018-10-21 23:19:31.868297",
"stderr": "Traceback (most recent call last):\n File \"\", line 1, in
\nKeyError: 'core-site'", "stderr_lines": ["Traceback (most recent call
last):", " File \"\", line 1, in ", "KeyError: 'core-site'"],
"stdout": "", "stdout_lines": []}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
Am I missing something?
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
Hey @nickwallen , I am seeing an error at the `ambari_gather_facts : Ask
Ambari: core_site_tag` step when trying to validate the fix.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` after
which I am seeing the below error:
```
TASK [ambari_gather_facts : Ask Ambari: namenode_host]
*
changed: [node1]
TASK [ambari_gather_facts : set_fact]
**
ok: [node1]
TASK [ambari_gather_facts : Ask Ambari: core_site_tag]
*
fatal: [node1]: FAILED! => {"changed": true, "cmd": "curl -s -u admin:admin
-X GET -H \"X-Requested-By: ambari\"
'http://node1:8080/api/v1/clusters/metron_cluster/hosts/node1/host_components/NAMENODE'
| python -c 'import sys, json; print
json.load(sys.stdin)[\"HostRoles\"][\"actual_configs\"][\"core-site\"][\"default\"]'",
"delta": "0:00:00.060271", "end": "2018-10-21 23:19:31.928568", "msg":
"non-zero return code", "rc": 1, "start": "2018-10-21 23:19:31.868297",
"stderr": "Traceback (most recent call last):\n File \"\", line 1, in
\nKeyError: 'core-site'", "stderr_lines": ["Traceback (most recent call
last):", " File \"\", line 1, in ", "KeyError: 'core-site'"],
"stdout": "", "stdout_lines": []}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
Am I missing something?
---
[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1244
Hey @nickwallen , I am seeing an error at the `ambari_gather_facts : Ask
Ambari: core_site_tag` step when trying to validate the fix.
1. Re-created the problem first by:
a) Running `vagrant up`
b) Then attempting `vagrant provision` again
c) Noticed the following error:
```
TASK [ambari_master : Install Elasticsearch MPack on Ambari Host]
**
fatal: [node1]: FAILED! => {"changed": true, "cmd": "ambari-server
install-mpack --mpack=/tmp/elasticsearch_mpack-0.6.1.0.tar.gz", "delta":
"0:00:00.433435", "end": "2018-10-21 18:12:29.939320", "msg": "non-zero return
code", "rc": 255, "start": "2018-10-21 18:12:29.505885", "stderr": "",
"stderr_lines": [], "stdout": "Using python /usr/bin/python\nInstalling
management pack\nERROR: Management pack elasticsearch-ambari.mpack-5.6.2
already installed!\nERROR: Exiting with exit code -1. \nREASON: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "stdout_lines": ["Using
python /usr/bin/python", "Installing management pack", "ERROR: Management pack
elasticsearch-ambari.mpack-5.6.2 already installed!", "ERROR: Exiting with exit
code -1. ", "REASON: Management pack elasticsearch-ambari.mpack-5.6.2 already
installed!"]}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
2. Destroyed the VM, applied the fix and
a) Running `vagrant up`
b) Then ran `vagrant --ansible-skip-tags="build,sensors" provision` after
which I am seeing the below error:
```
TASK [ambari_gather_facts : Ask Ambari: namenode_host]
*
changed: [node1]
TASK [ambari_gather_facts : set_fact]
**
ok: [node1]
TASK [ambari_gather_facts : Ask Ambari: core_site_tag]
*
fatal: [node1]: FAILED! => {"changed": true, "cmd": "curl -s -u admin:admin
-X GET -H \"X-Requested-By: ambari\"
'http://node1:8080/api/v1/clusters/metron_cluster/hosts/node1/host_components/NAMENODE'
| python -c 'import sys, json; print
json.load(sys.stdin)[\"HostRoles\"][\"actual_configs\"][\"core-site\"][\"default\"]'",
"delta": "0:00:00.060271", "end": "2018-10-21 23:19:31.928568", "msg":
"non-zero return code", "rc": 1, "start": "2018-10-21 23:19:31.868297",
"stderr": "Traceback (most recent call last):\n File \"\", line 1, in
\nKeyError: 'core-site'", "stderr_lines": ["Traceback (most recent call
last):", " File \"\", line 1, in ", "KeyError: 'core-site'"],
"stdout": "", "stdout_lines": []}
to retry, use: --limit
@/Users/asubramanian/Desktop/Metron/anand-metron-fork/metron-latest/metron/metron-deployment/development/centos6/ansible/playbook.retry
```
Am I missing something?
---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Thanks @MohanDV ! ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Cool, thanks for your opinion @nickwallen . I will wait for @MohanDV 's thoughts as well before merging. ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Thanks much @nickwallen for giving it another spin! Hey @MohanDV , let me know if my below explanation makes sense. @nickwallen , would you have any thoughts on this? > @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. > > In my understanding, the following parameters are not directly related to the PCAP core service, but are functionally related to REST (used by the PCAP query panel UI): > > * pdml.script.path > * base.path > * base.interim.result.path > * final.output.path > * page.size > * yarn.queue > * finalizer.threadpool.size > > For this reason, I did not move these properties out of the rest_application.yml. > > > @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology > > With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both). The converse, which you noted, i.e. changing any REST config does not require restarting the PCAP topology. Here's a screenshot after making changes to REST config... > >  ---
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r222726135 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -119,8 +119,8 @@ metron_keytab_path = config['configurations']['metron-env']['metron_service_keytab'] # Pcap -metron_pcap_topology = 'pcap' -pcap_input_topic = 'pcap' +metron_pcap_topology = config['configurations']['metron-pcap-env']['spout_kafka_topic_pcap'] --- End diff -- Done, changed. ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @nickwallen suggested a great idea of making use of parser aggregation and thus starting the default parsers on full-dev as an aggregated parser topology. The latest commit makes use of PR #1215 which provides that ability. Now, on full-dev we start a single aggregated topology for the default configured sensors viz. "bro,snort,yaf". ## Testing Done - Spun up full-dev and noticed that a single topology named `bro__snort__yaf` is started - Verified Alerts UI has data from all three sensors:  ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. In my understanding, the following parameters are not directly related to the PCAP core service, but are functionally related to REST (used by the PCAP query panel UI): * pdml.script.path * base.path * base.interim.result.path * final.output.path * page.size * yarn.queue * finalizer.threadpool.size For this reason, I did not move these properties out of the rest_application.yml. > @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both). The converse, which you noted, i.e. changing any REST config does not require restarting the PCAP topology. Here's a screenshot after making changes to REST config...  ---
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Thanks again for the review, @nickwallen ! ---
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Re-run travis ---
[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...
GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up full-dev. 2. Stop the "Metron Parsers" service so that existing parser topologies are killed/stopped. 3. Go to Ambari -> Metron -> Configs -> Parsers 4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example) 5. Save changes and restart the Metron Parsers service. 6. Go to Storm UI, and verify that the aggregated topologies viz. "bro__snort" and "yaf" are started. ## Testing Done ### 1. Full Dev * Set the Metron Parsers value as "bro,snort.yaf" and restarted the service. * A single aggregated topology is seen to be started:  * Verified stop 'Metron Parsers' service to check that the aggregated topology is stopped properly * Verified restart 'Metron Parsers' service to check that the aggregated topology is restarted properly ### 2. Multi-node setup * Set the Metron Parsers value as: "bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort * The appropriate aggregated and single topologies are started:  * Changed the parser list as: bro,snort,yaf * Restarted the parsers service to validate that the three parser topologies are started individually.  ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1798 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1215.patch To close this pull requ
[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1215 ---
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Hi @nickwallen , thank you for the review. > @nickwallen One thing I notice right off is that we did not add any documentation for the parsers fields in Ambari. Would it make sense to add a brief description to the pop-up text describing how a user can define parsers? Sure, makes sense. I have also added a section to the Parser Chaining README as well with examples. Please have a look. > @nickwallen: When switching parser topologies, some of the original parser topologies can fail to be shut down properly. > I have found that the same issue occurs in master. Since that is the case, we could choose to tackle this as a separate ticket. It is your choice. I created [METRON-1802](https://issues.apache.org/jira/browse/METRON-1802) so this can be fixed outside of this PR. ---
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Please note: I did not find the Parser Aggregation functionality documented under the Parser Chaining README, save a mention under the parser chaining [use case](https://github.com/apache/metron/tree/master/use-cases/parser_chaining#aggregated-parsers-with-parser-chaining). Please let me know if it makes sense to add this to the main parser chaining README, and I will add it. ---
[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up full-dev. 2. Stop the "Metron Parsers" service so that existing parser topologies are killed/stopped. 3. Go to Ambari -> Metron -> Configs -> Parsers 4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example) 5. Save changes and restart the Metron Parsers service. 6. Go to Storm UI, and verify that the aggregated topologies viz. "bro__snort" and "yaf" are started. ## Testing Done ### 1. Full Dev * Set the Metron Parsers value as "bro,snort.yaf" and restarted the service. * A single aggregated topology is seen to be started:  * Verified stop 'Metron Parsers' service to check that the aggregated topology is stopped properly * Verified restart 'Metron Parsers' service to check that the aggregated topology is restarted properly ### 2. Multi-node setup * Set the Metron Parsers value as: "bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort * The appropriate aggregated and single topologies are started:  ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1798 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1215 commit 8aa7c77f9d99f987d2549f3f8eed3a72f09f8bb1 Author: Anand Subramanian Date: 2018-09-28T14:56:18Z Support parser aggregation ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Latest commit modifies full dev to start only the bro parser topology by default. The PCAP topology now gets the worker slot which was freed up. Note that on a regular deploy, all of bro, snort and yaf topologies will be deployed by default. I am also looking for the issue @nickwallen mentioned about the PCAP topology appearing stopped. Nothing so far. Will continue to monitor. ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 One thing I am not sure is, whether the topology name i.e. `metron_pcap_topology` should be fetched from the config or should be hard coded to 'pcap'. I see that for the enrichment, profiler, indexing etc., we have hardcoded the topology name and fetch the kafka topic from config. See [here](https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py#L329) for example. I would like opinions on whether we should follow the same convention here or should we fetch it from Ambari config. ---
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219736069 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -387,18 +388,32 @@ bolt_hdfs_rotation_policy_units = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units'] bolt_hdfs_rotation_policy_count = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count'] -# Pcap +# PCAP metron_pcap_topology = 'pcap' pcap_input_topic = 'pcap' --- End diff -- The latest commit fetches the topology and topic name from Ambari config. ---
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1207 METRON-1695: Expose pcap properties through Ambari ## Contributor Comments This change makes the PCAP config properties available via the Metron service -> Configs tab in Ambari. It also takes care of updating the `pcap.properties` file with relevant values depending on the deploy environment (single node vs. multi node vs. kerberized cluster). Here's a screenshot of the PCAP Config:  ### Testing Steps * Launch full dev * Metron PCAP service should come up successfully. * Go to Ambari -> Metron services -> Configs -> PCAP tab; change any config parameter; * Restart the required services and the config change take effect. ### Testing Done I ran the following different tests to validate the fix. A. Full Dev * Verified Metron PCAP service comes up successfully * Changed some of the PCAP config values from the Ambari UI, validated that this is reflected in the `pcap.properties` file upon service restart. - **Note**: I ran into [METRON-1786](https://issues.apache.org/jira/browse/METRON-1786) during testing. I was able to resolve the issue by applying the fix in PR #1206 B. Multi-node Cluster * Validated that the `kafka.zk` property in `pcap.properties` is updated with the correct Broker details * Changed a PCAP service-related property (E.g. `kafka_pcap_numpackets`) and validated that the `pcap.properties` file is updated. * Changed a PCAP REST-related property (E.g. `pcap_page_size`) and validated that the `/etc/default/metron` file is updated. C. Kerberized Cluster * Kerberized the multi-node cluster and setup pycapa * Ingested PCAP into the cluster * Ran queries through the PCAP panel and validated that the modified config settings are in effect (E.g. `pcap_page_size` set to 50). ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [NA] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/
[GitHub] metron issue #1206: METRON-1786: Pcap Topology Status Incorrect
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1206 +1 Verified the fix on full dev. The topology status is seen to be fine after the fix. No errors in ambari agent log. ---
[GitHub] metron issue #1201: METRON-1709 : Added the required scripts to start / stop...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1201 Hi @nickwallen , I would like to create a fresh pull request based on these changes. Could you go ahead and merge this please ? ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Hey @nickwallen ... Let me close this PR. Will create one afresh based on the latest changes from #1201 ---
[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1132 ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1134 @mmiklavc - in my setup, I have seen that non-kerb requires storm:hadoop permissions, while kerberized needs metron:hadoop for PCAP to write into HDFS. ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Sure @merrimanr , sounds good. @MohanDV has already begun work on METRON-1709. I will wait for him to complete and then submit this pull request so it would be a natural fit. ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1134 > metron:metron did not allow pcap topology to write as the Storm user. Yup @mmiklavc .. on a non-kerberized cluster, I faced the same issue. ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 This PR lays the foundation for exposing the properties to begin with. My thought was that it reduces one error prone manual step of hand-editing the `pcap.properties` file. I agree with your point about how users might be concerned about restarting all parsers after modifying PCAP config. I did not create this earlier under the feature branch since this was not related specifically to the PCAP query panel, it was a more generic change. I am fine to move this under the feature branch or I could wait until the fix for METRON-1709 (making the PCAP service its own) is available and then submit a fresh PR. Let me know which one works better. ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Yup, @mmiklavc I see what you are saying. Make sense to me. Let me know if the latest README update conveys the message. I also added a fix to prompt a service restart upon changes to the PCAP config settings. Thanks @MohanDV for the pointer! ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Btw, I created [METRON-1719](https://issues.apache.org/jira/browse/METRON-1719) to track PCAP sensor being an independent service entity. ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 > Please document the lifecycle for how pcap.properties is updated in the README. It should be clear to users that they will need to restart parsers for the changes to take effect after the initial install is performed. Can you confirm, is that the way the properties are/will be deployed with this PR @anandsubbu? Hi @mmiklavc - there is no change in behavior from earlier to now. Nothing changes from a usage perspective. Let me clarify more with an example for a multi-node deployment... **Earlier** * User deploys metron * `pcap.properties` file created with defaults under `$METRON_HOME/config` * User hand edits the properties file (sets ZK quorum and other parameters as required) * Starts the topology per the instructions [here](https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#starting-the-topology). **Now** * During deploy time, user is presented with a separate config tab in Ambari to configure/reconfigure PCAP properties. If user chooses to leave them untouched, the pcap.properties is initialized with appropriate defaults. ZK quorum is auto-populated as well. * `pcap.properties` file is created during metron-parsers startup step. * Starts the topology per the instructions [here](https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#starting-the-topology). The steps in the [README](https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend) still holds. Now, we can add a note in the README to indicate that the PCAP properties can be configured via the 'PCAP' tab in Ambari Metron config. But I noticed this was not explicitly mentioned for other components (e.g. [REST](https://github.com/apache/metron/tree/master/metron-interface/metron-rest#configuration), [elasticsearch](https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch#properties)). Let me know if you prefer to have this added. ---
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 @mmiklavc thanks for the review. I concur with your observations. PCAP topology does deserve its own place in Ambari and the Management UI. As far as the scope of this PR goes, this IMHO, would be a first cut towards it. This PR addresses the immediate need for exposing the parameters in `pcap.properties` via Ambari. It also addresses auto-populating the ZK quorum in `pcap.properties` thus simplifying the manual steps that would be otherwise required in a multi-node deployment. Would you be okay if we create a separate JIRA to cover the broader change? Let me know your thoughts. ---
[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1132#discussion_r205241710 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-pcap-env.xml --- @@ -0,0 +1,141 @@ + + + + + +pcap_topology_worker_childopts +PCAP Topology JVM Options + +PCAP Topology childopts + +true + + + +pcap_topology_workers +Number of PCAP Topology Workers +1 +Workers for PCAP Topology + + +spout_kafka_topic_pcap +PCAP Input Topic +pcap +PCAP Input Topic + + +hdfs_sync_every --- End diff -- The PCAP [config readme](https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#configuration) does not talk more about the units for this parameter (whether seconds, or minutes or something else). I looked through the code as well, but was not able to find more info. If someone could clarify about this, I can add the units so it is clearer. ---
[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/1132#discussion_r205237418
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py
---
@@ -54,6 +57,12 @@ def configure(self, env, upgrade_type=None,
config_dir=None):
commands.init_kafka_acls()
commands.set_acl_configured()
+File(format("{metron_config_path}/pcap.properties"),
--- End diff --
I had earlier place this under the [indexing_master.py
](https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py#L62)script,
but felt that it is more related to parser than with indexing. I am happy to
move this block to any other place that is deemed fit.
---
[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1132 METRON-1695: Expose pcap properties through Ambari ## Contributor Comments This change add a new jinja template for the pcap properties and adds a new tab in Ambari Metron config for the PCAP properties. Here are the screenshots: https://user-images.githubusercontent.com/20395490/43222746-a9e0026c-906e-11e8-9353-0ea34dbb4224.png;> https://user-images.githubusercontent.com/20395490/43222747-aa1ba704-906e-11e8-82e8-71d4a9731aba.png;> *Testing Done* Built mpack out of these changes, deployed a multi-node cluster and validated that the pcap.properties is being updated during Parser startup scripts per the screenshot below: https://user-images.githubusercontent.com/20395490/43222878-fe5fcc50-906e-11e8-80a7-45c73e173638.png;> Also validated that the pcap properties is updated with config changes (ZK quorum, kafka broker etc.) ``` [root@metron-12 config]# cat pcap.properties # Storm # topology.worker.childopts= topology.auto-credentials=[] topology.workers=1 # Kafka # spout.kafka.topic.pcap=pcap kafka.zk=metron-10:2181,metron-11:2181,metron-1:2181 hdfs.sync.every=1 hdfs.replication.factor=--1 kafka.security.protocol=PLAINTEXT # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST kafka.pcap.start=UNCOMMITTED_EARLIEST kafka.pcap.numPackets=1000 kafka.pcap.maxTimeMS=30 kafka.pcap.ts_scheme=FROM_KEY kafka.pcap.out=/apps/metron/pcap/input # Parallelism # kafka.pcap.ts_granularity=MICROSECONDS kafka.spout.parallelism=1 ``` *Verification Steps* * Spin up full dev (or a multi-node cluster) * Ensure that the PCAP tab in Ambari config allows for values to be entered. * Deploy cluster and validate that $METRON_HOME/config/pcap.properties file has the proper config values set. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1695 Alternatively you can review and apply these changes
[GitHub] metron issue #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1039 I ran a few performance tests comparing ver 1.1.0 and 1.2.1 on the enrichment topology using the [load_tool](https://github.com/apache/metron/blob/master/metron-contrib/metron-performance/README.md). The EPS numbers for 1.2.1 are seen to be on-par or better than 1.1.0. However, I noticed that the Avg CPU usage is higher on 1.2.1 (about 70%) as compared to 1.1.0 (about 40%). ---
[GitHub] metron issue #526: Metron-846: Add E2E tests for metron management ui
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/526 I am testing and updating this PR to get it to work on latest master. ---
[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably du...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/1004#discussion_r184889297
--- Diff: metron-interface/metron-alerts/protractor.conf.js ---
@@ -32,12 +33,13 @@ exports.config = {
'./e2e/alerts-list/tree-view/tree-view.e2e-spec.ts',
'./e2e/alerts-list/alert-filters/alert-filters.e2e-spec.ts',
'./e2e/alerts-list/alert-status/alerts-list-status.e2e-spec.ts',
-'./e2e/alert-details/alert-status/alert-details-status.e2e-spec.ts',
+//
'./e2e/alert-details/alert-status/alert-details-status.e2e-spec.ts', //This
will not work
'./e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts'
],
capabilities: {
'browserName': 'chrome',
'chromeOptions': {
+ args: ["--disable-gpu", "--window-size=1435,850" ],
--- End diff --
`-headless` can be included here to default to headless execution.
---
[GitHub] metron issue #1004: METRON-1489: Retrofit UI tests to run reliably during ni...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1004 @nickwallen , in order to get past the following error you are seeing.. ``` [13:44:09] E/launcher - All configured authentication methods failed [13:44:09] E/launcher - Error: All configured authentication methods failed at tryNextAuth (/Users/nallen/tmp/metron-pr1004/metron-interface/metron-alerts/node_modules/ssh2/lib/client.js:380:17) at SSH2Stream.onUSERAUTH_FAILURE (/Users/nallen/tmp/metron-pr1004/metron-interface/metron-alerts/node_modules/ssh2/lib/client.js:591:5) at SSH2Stream.emit (events.js:160:13) at parsePacket (/Users/nallen/tmp/metron-pr1004/metron-interface/metron-alerts/node_modules/ssh2-streams/lib/ssh.js:3929:10) at SSH2Stream._transform (/Users/nallen/tmp/metron-pr1004/metron-interface/metron-alerts/node_modules/ssh2-streams/lib/ssh.js:669:13) at SSH2Stream.Transform._read (_stream_transform.js:185:10) at SSH2Stream._read (/Users/nallen/tmp/metron-pr1004/metron-interface/metron-alerts/node_modules/ssh2-streams/lib/ssh.js:251:15) at SSH2Stream.Transform._write (_stream_transform.js:173:12) at doWrite (_stream_writable.js:407:12) at writeOrBuffer (_stream_writable.js:393:5) [13:44:09] E/launcher - Process exited with error code 199 ``` The error you are seeing is because of the ssh2 module which authenticates to the vagrant node to do cleanup tasks. This needs to be fixed. As a workaround, you can do the following in (metron-interface/metron-alerts/e2e/utils/clean_metron_update_table.ts) i.e. https://github.com/apache/metron/pull/1004/files#diff-2cc98b255ddf3a954ad22dfa2b5545fb ``` +resolve(); +//cleanupTable(resolve, reject); ``` i.e. you need to uncomment the `resolve()` function and comment out ` cleanupTable()`. ---
[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably du...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/1004#discussion_r184888777
--- Diff:
metron-interface/metron-alerts/e2e/utils/clean_metron_update_table.ts ---
@@ -0,0 +1,52 @@
+declare var Promise: any;
+var chalk = require('chalk');
+var Client = require('ssh2').Client;
+var errorMsg = '';
+
+export function cleanMetronUpdateTable() {
+ return new Promise(
+ function (resolve, reject) {
+// resolve();
+cleanupTable(resolve, reject);
--- End diff --
IMHO, it would be better to default to `resolve()` only owing to the issues
with Vagrant `ssh2` client.
---
[GitHub] metron pull request #981: METRON-1510: Update Metron website to include info...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/981 METRON-1510: Update Metron website to include info about github update subscription ## Contributor Comments Update to include links for the 'Issues' mailing list. Have also changed Archives links to point to lists.apache.org, which has a better viewing than the earlier one. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1510 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/981.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #981 commit e2889419a7fafe5b637fd6d49e829061c009b62c Author: Anand Subramanian <asubramanian@...> Date: 2018-04-05T04:59:37Z Added links to Issues. Also changed Archives to point to lists.apache.org which has better look and feel. ---
[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/943 My +1 still holds. Installed both mpacks on a 12-node cluster and was able to get bro indices to work successfully. Thanks @mmiklavc ! ---
