[GitHub] metron pull request #1227: METRON-1807: Auto populate the recommended values...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1227 METRON-1807: Auto populate the recommended values to some of the metron config parameters ## Contributor Comments Auto populate the recommended values to below metron config parameters Storm Rest Server Address Zeppelin Server Address Solr Zookeeper Urls ### Testing Done : Spun-up a multinode cluster and verified that the above property values got auto populated with recommended values , here are the screenshots for the same https://user-images.githubusercontent.com/12934693/46550498-e8500800-c8f2-11e8-8a02-ac35903fbdbb.png";> https://user-images.githubusercontent.com/12934693/46550514-f2720680-c8f2-11e8-8746-ed344221c4d7.png";> ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1807 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1227.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1227 commit 389552606146bedda5e670c6554919c3e75ce384 Author: Mohan Venkateshaiah Date: 2018-10-05T17:27:30Z METRON-1807: Auto populate the recommended values to some of the metron config parameters ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1207 > Cool, thanks for your opinion @nickwallen . I will wait for @MohanDV 's thoughts as well before merging. +1 Thanks @anandsubbu ---
[GitHub] metron issue #1205: METRON-1785 Automate deployment of packet capture for de...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1205 Deployed a full dev , I executed set of instructions as in Docs `vagrant up vagrant --ansible-tags="pcap" provision` #Stopped the Parser, Enrichment, Indexing, and Profiler topologies to free-up resources. `vagrant ssh sudo su - source /etc/default/metron yum -y install wireshark` I see that the pcap-replay and pycapa services not deployed .. `[root@node1 ~]# service pcap-replay start` `pcap-replay: unrecognized service` `[root@node1 ~]# service pycapa start` `pycapa: unrecognized service` ---
[GitHub] metron issue #1202: METRON-1782 Add Kafka Partition and Offset to Profiler D...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1202 +1 Verified on the full dev , below are the debug log from profiler `2018-09-28 10:17:36.465 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [DEBUG] Received message; topic=indexing, partition=0, offset=5490, kafkaTimestamp=1538129856340 ... 2018-09-28 10:17:43.893 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [DEBUG] Received message; topic=indexing, partition=0, offset=5491, kafkaTimestamp=1538129863880 . 2018-09-28 10:17:45.026 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [DEBUG] Received message; topic=indexing, partition=0, offset=5492, kafkaTimestamp=1538129864998 . 2018-09-28 10:17:45.859 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [DEBUG] Received message; topic=indexing, partition=0, offset=5493, kafkaTimestamp=1538129865850 ` ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1207 PCAP related properties needs to be taken out from the [rest_application](https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/config/rest_application.yml#L57). Also the dependency of PCAP on rest settings to be removes as Changing a REST Setting triggers Ambari to [Restart](https://hortonworks.jira.com/browse/BUG-111762) PCAP Topology . ---
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219501331 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -387,18 +388,32 @@ bolt_hdfs_rotation_policy_units = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units'] bolt_hdfs_rotation_policy_count = config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count'] -# Pcap +# PCAP metron_pcap_topology = 'pcap' pcap_input_topic = 'pcap' --- End diff -- pcap input topic is also exposed , so shouldn't we be getting the value from config tab ? ---
[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...
Github user MohanDV commented on a diff in the pull request:
https://github.com/apache/metron/pull/1207#discussion_r219503222
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
---
@@ -387,18 +388,32 @@
bolt_hdfs_rotation_policy_units =
config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_units']
bolt_hdfs_rotation_policy_count =
config['configurations']['metron-indexing-env']['bolt_hdfs_rotation_policy_count']
-# Pcap
+# PCAP
metron_pcap_topology = 'pcap'
pcap_input_topic = 'pcap'
--- End diff --
pcap input topic is also hardcoded in [status_param.py
](https://github.com/MohanDV/metron/blob/a46fda45b46a9f317e2ef7c3755ce6f072ed893b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py#L123)
we should get the value from config tab here too.
---
[GitHub] metron pull request #1206: METRON-1786: Pcap Topology Status Incorrect
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1206 METRON-1786: Pcap Topology Status Incorrect ## Contributor Comments Added missing params which caused the incorrect PCAP topology status in Ambari. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1786 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1206.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1206 commit a46fda45b46a9f317e2ef7c3755ce6f072ed893b Author: Mohan Venkateshaiah Date: 2018-09-20T17:49:48Z METRON-1786: Added missing params which caused the incorrect pcap topology status in Ambari. ---
[GitHub] metron issue #1201: METRON-1709 : Added the required scripts to start / stop...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1201 @nickwallen having separate panel for PCAP configs is been addressed in the pull request [1132](https://github.com/apache/metron/pull/1132). It will be a duplicate effort here. ---
[GitHub] metron issue #1201: METRON-1709 : Added the required scripts to start / stop...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1201 @nickwallen It's my bad I dint run this up in Full Dev, I verified my scripts on a 12 node cluster. I did the required changes to support the Full dev deploymwnt and updated the pull request. ---
[GitHub] metron pull request #1201: METRON-1709 : Added the required scripts to start...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1201 METRON-1709 : Added the required scripts to start / stop PCAP topology from Ambari ## Contributor Comments Created PCAP topology as Metron component, Created required scripts to create kafka topic, hdfs folder for pcap sequence files with required ACL's and added controls to start / stop the PCAP topology from Ambari. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1709 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1201 commit 048e250404427d1ffa43fb91e22642f34d3ecc4e Author: Mohan Venkateshaiah Date: 2018-09-18T11:14:24Z METRON-1709 : Created PCAP as a proper component for Metron, added the required scripts to start/stop PCAP topology from Ambari. ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 Closed as it is addressed in METRON-1738 ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 closing as it addressed in METRON-1738 ---
[GitHub] metron pull request #1134: METRON-1696: Create the HDFS directory for pcap s...
Github user MohanDV closed the pull request at: https://github.com/apache/metron/pull/1134 ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 I am fine wuth closing this PR as this is currently implemented in REST. When pcap gets its own start/stop script in Ambari we can move the code from REST to respective file. ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 Non Kerberized pcap topology runs under 'storm' which requires write under the 'hadoop' group so we should provide mode 0775 when the cluster is un secure and mode 0755 when the cluster is secure as the topology runs under 'metron' user. ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 @mmiklavc I see the 'storm' user as the owner of pcap topology in non-kerberized setup, where as the 'metron' as the owner in kerberized setup. IMHO The pcap should have 'metron' as owner of the topology, then having the hdfs permissions set to metron:hadoop will work on both the environment. ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 Thanks for pointing this out @merrimanr, Yes this will overwrite the the #1019 . If the feature branch is not getting to master sooner then we can keep this change and rebase the feature branch else we can discard this. @mmiklavc metron:metron should allow pcap topology to write to hdfs directory. I started the topology as metron user. ---
[GitHub] metron pull request #1141: Metron-1727: metron-rest service needs to be rest...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1141 Metron-1727: metron-rest service needs to be restarted after enabling x-pack for Elastic search. ## Contributor Comments metron-rest service needs to be restarted after enabling x-pack for Elastic search. If not the alerts will not be populated on the alerts UI. Documented the the required step under enabling X-pack section. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1727 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1141.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1141 commit 0744ad1e706fadf64f04a97db5fb7608dea99604 Author: Mohan Venkateshaiah Date: 2018-08-07T10:09:58Z METRON-1727 : metron-rest service needs to be restarted after enabling x-pack for Elastic search. ---
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/1134 @mmiklavc can you please floor this PR, I have create follow-on Jira [METRONâ-1709 ](https://issues.apache.org/jira/browse/METRON-1709). ---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1136#discussion_r206864051 --- Diff: metron-sensors/pycapa/README.md --- @@ -54,8 +54,8 @@ General notes on the installation of Pycapa. ``` export PREFIX=/usr -wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz -cd librdkafka-0.9.4/ +wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz +cd librdkafka-0.11.5/ --- End diff -- Yes it works for unsecured deployment as well. ---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1136#discussion_r206863799 --- Diff: metron-sensors/pycapa/README.md --- @@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab` * The service principal is `met...@example.com` +If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. +``` +yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi +``` + + 1. Use the non-binary wheel to install confluent-kafka-python. + +The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case: +``` +**NOTE:** The pre-built Linux wheels do NOT contain SASL Kerberos support. + If you need SASL Kerberos support you must install librdkafka and + its dependencies using the above repositories and then build + confluent-kafka from source. + ``` --- End diff -- done ---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1136#discussion_r206863111 --- Diff: metron-sensors/pycapa/README.md --- @@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab` * The service principal is `met...@example.com` +If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. +``` +yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi +``` + + 1. Use the non-binary wheel to install confluent-kafka-python. --- End diff -- Moved these instruction to the top as description . ---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
Github user MohanDV commented on a diff in the pull request: https://github.com/apache/metron/pull/1136#discussion_r206862766 --- Diff: metron-sensors/pycapa/README.md --- @@ -239,25 +239,47 @@ The probe can be used in a Kerberized environment. Follow these additional step * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab` * The service principal is `met...@example.com` +If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. --- End diff -- done ---
[GitHub] metron pull request #1136: METRON-1701: Update steps to run the pycapa on ke...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1136 METRON-1701: Update steps to run the pycapa on kerberized cluster. ## Contributor Comments Updated the steps to run Pycapa with Kerberos ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1701 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1136.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1136 commit 8f44e2732159bcf615f5883a9c48e7776954ac98 Author: Mohan Venkateshaiah Date: 2018-07-31T09:50:31Z Update steps to run the pycapa on kerberized cluster. ---
[GitHub] metron pull request #1134: METRON-1696: Create the HDFS directory for pcap s...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1134 METRON-1696: Create the HDFS directory for pcap sequence files and add required privileges to metron user ## Contributor Comments PCAP parser fails to write pacap sequence file to hdfs on kerberized cluster due to insufficient privileges to hdfs folder for 'metron' user. Create the HDFS directory for PCAP sequence files and add the required privileges to metron user. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1696 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1134 commit af92ad3b95c28b5ef3640fbe5637286eeff57025 Author: Mohan Venkateshaiah Date: 2018-07-26T11:46:38Z METRON-1696: Create the HDFS directory for pcap sequence files and add the required previlages to metron user ---
[GitHub] metron pull request #1086: METRON-1645: Check wether the Solr management pac...
GitHub user MohanDV opened a pull request:
https://github.com/apache/metron/pull/1086
METRON-1645: Check wether the Solr management pack is installed before
configuring the solr principal name.
## Contributor Comments
Check wether the Solr management pack is installed before configuring the
solr principal name.
Steps to reproduce:
1. Install Metron with Elastic Search as indexing server
2. Kerberize the cluster and restart services
3. All the Metron services including Metron client fails with below error
'resource_management.core.exceptions.Fail: Configuration parameter
'solr-config-env' was not found in configurations dictionary!'
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MohanDV/metron METRON-1645
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1086.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1086
commit 27f4026f4487399bb7ef12080738826cf4f2048b
Author: Mohan Venkateshaiah
Date: 2018-06-27T16:02:20Z
METRON-1645: Check wether the Solr management pack is installed before
configuring the solr principal name.
---
[GitHub] metron pull request #1052: METRON-1604 : Add power pc to OS family for the H...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1052 METRON-1604 : Add power pc to OS family for the HCP management pack repo info ## Contributor Comments Add power pc to OS family for the HCP management pack repo info ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1604 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1052.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1052 commit 7b503d7d2f9e08bf7f223aae05b036efeb18dc16 Author: Mohan Venkateshaiah Date: 2018-06-07T07:14:49Z METRON-1604 : Add power pc to OS family for the HCP management pack repo info ---
[GitHub] metron pull request #1035: METRON-1581: kill the profiler topology immediate...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/1035 METRON-1581: kill the profiler topology immediately before restarting the topology if it is in 'KILLED' state for longer time ## Contributor Comments Restarting profiler topology fails storm kill 'topology' command will leave the profiler topology in 'KILLED' state for longer time. ### Steps to reproduce: 1. Restart the profiler topology 2. The start topology fails with 'Topology with name `profiler` already exists on cluster' 3. execute 'storm list' on any node with supervisor which will return the profiler topology status as KILLED ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1581 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1035.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1035 commit a9d731e77037a2248b2802906f1ed5ea992f261a Author: Mohan Venkateshaiah Date: 2018-05-28T13:33:49Z jira:METRON-1581 kill the profiler topology immediately if it is in 'KILLED' state for long time. ---
[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/942 Thanks @cestella . Can you please merge this pull request. ---
[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...
Github user MohanDV commented on the issue: https://github.com/apache/metron/pull/942 If we've reached consensus can I get +1 for this ? ---
[GitHub] metron pull request #992: METRON-1522: Fix the typo errors at profiler debug...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/992 METRON-1522: Fix the typo errors at profiler debugger readme ## Contributor Comments Fix the typo errors at profiler debugger readme to keep the profile name consistent in flush profile section and reference to the profile across sections. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1522 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/992.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #992 commit b5e151af7602e5bde247dc1b6cd6592fcb467e1a Author: Mohan Venkateshaiah Date: 2018-04-12T11:05:39Z update the doc to fix typo errors. ---
[GitHub] metron pull request #968: METRON-1497: Rest endpoint '/api/v1/search/search'...
GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/968 METRON-1497: Rest endpoint '/api/v1/search/search' needs to handle null when elastic search response return null for getAggregations ## Contributor Comments Handle null value when elastic search response return null for getAgreegations method ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MohanDV/metron METRON-1497 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/968.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #968 commit f417510d232d0f1b4fde72ed89a53fab30d7dc69 Author: Mohan Venkateshaiah Date: 2018-03-19T12:49:29Z Handle null value when elastic search response return null for getAggregations method ---
