[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...

2018-11-15 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1242 > specifically setting `topology.ackers.executors` equal to the number of inbound Kafka topic partitions And the number of Workers too. i.e. No. of Kafka partitions = No. of work

[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...

2018-11-08 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1242 > The last remaining bit before I merge this will be sharing some results around the performance/regression testing performed by @anandsubbu. I completed a set of performance compari

[GitHub] metron pull request #1256: METRON-1861: REST fails to start when LDAP enable...

2018-11-08 Thread anandsubbu
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1256 METRON-1861: REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty ## Contributor Comments Set 'Active Spring profiles' to just `ldap`, when

[GitHub] metron issue #1246: METRON-1844: Allow for LDAP to be used for authenticatio...

2018-11-01 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1246 My +1 as well. I spun up a multi-node cluster and set this up using the demo LDAP service provided by Knox. I was able to authenticate using the demo users for all the Swagger, Alerts

[GitHub] metron pull request #1246: METRON-1844: Allow for LDAP to be used for authen...

2018-10-30 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1246#discussion_r229259363 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-22 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! I have seen this issue previously as well. 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting `vagrant

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! I have bumped into this issue previously as well. 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! I have bumped into this issue previously as well. 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! I have seen this issue previously as well. 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting `vagrant

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! I have seen this issue previously as well. 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting `vagrant

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! I have seen this issue previously as well. 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting `vagrant

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 +1 Thanks for the fix @nickwallen ! 1. Re-created the problem first by: a) Running `vagrant up` b) Then attempting `vagrant provision` again c) Noticed the following

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 Hey @nickwallen , I am seeing an error at the `ambari_gather_facts : Ask Ambari: core_site_tag` step when trying to validate the fix. 1. Re-created the problem first by: a) Running

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 Hey @nickwallen , I am seeing an error at the `ambari_gather_facts : Ask Ambari: core_site_tag` step when trying to validate the fix. 1. Re-created the problem first by: a) Running

[GitHub] metron issue #1244: METRON-1839 Install Elasticsearch MPack Step in Ansible ...

2018-10-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1244 Hey @nickwallen , I am seeing an error at the `ambari_gather_facts : Ask Ambari: core_site_tag` step when trying to validate the fix. 1. Re-created the problem first by: a) Running

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-10-05 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Thanks @MohanDV ! ---

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-10-04 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Cool, thanks for your opinion @nickwallen . I will wait for @MohanDV 's thoughts as well before merging. ---

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-10-04 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Thanks much @nickwallen for giving it another spin! Hey @MohanDV , let me know if my below explanation makes sense. @nickwallen , would you have any thoughts on this? > @Moha

[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...

2018-10-04 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r222726135 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-10-03 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @nickwallen suggested a great idea of making use of parser aggregation and thus starting the default parsers on full-dev as an aggregated parser topology. The latest commit makes use of PR #1215

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-10-03 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. In my understanding, the following parameters are not directly related

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

2018-10-03 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Thanks again for the review, @nickwallen ! ---

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

2018-10-03 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Re-run travis ---

[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...

2018-10-03 Thread anandsubbu
GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin

[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...

2018-10-03 Thread anandsubbu
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1215 ---

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

2018-10-03 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Hi @nickwallen , thank you for the review. > @nickwallen One thing I notice right off is that we did not add any documentation for the parsers fields in Ambari. Would it make se

[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation

2018-09-28 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Please note: I did not find the Parser Aggregation functionality documented under the Parser Chaining README, save a mention under the parser chaining [use case](https://github.com/apache/metron

[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...

2018-09-28 Thread anandsubbu
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-09-25 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 Latest commit modifies full dev to start only the bro parser topology by default. The PCAP topology now gets the worker slot which was freed up. Note that on a regular deploy, all of bro, snort

[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari

2018-09-24 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 One thing I am not sure is, whether the topology name i.e. `metron_pcap_topology` should be fetched from the config or should be hard coded to 'pcap'. I see that for the enrichment, profiler

[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...

2018-09-24 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1207#discussion_r219736069 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py

[GitHub] metron pull request #1207: METRON-1695: Expose pcap properties through Ambar...

2018-09-21 Thread anandsubbu
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1207 METRON-1695: Expose pcap properties through Ambari ## Contributor Comments This change makes the PCAP config properties available via the Metron service -> Configs tab in Ambari. It a

[GitHub] metron issue #1206: METRON-1786: Pcap Topology Status Incorrect

2018-09-21 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1206 +1 Verified the fix on full dev. The topology status is seen to be fine after the fix. No errors in ambari agent log. ---

[GitHub] metron issue #1201: METRON-1709 : Added the required scripts to start / stop...

2018-09-20 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1201 Hi @nickwallen , I would like to create a fresh pull request based on these changes. Could you go ahead and merge this please ? ---

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-09-20 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Hey @nickwallen ... Let me close this PR. Will create one afresh based on the latest changes from #1201 ---

[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...

2018-09-20 Thread anandsubbu
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1132 ---

[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...

2018-08-09 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1134 @mmiklavc - in my setup, I have seen that non-kerb requires storm:hadoop permissions, while kerberized needs metron:hadoop for PCAP to write into HDFS. ---

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-08-09 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Sure @merrimanr , sounds good. @MohanDV has already begun work on METRON-1709. I will wait for him to complete and then submit this pull request so it would be a natural fit. ---

[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...

2018-08-08 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1134 > metron:metron did not allow pcap topology to write as the Storm user. Yup @mmiklavc .. on a non-kerberized cluster, I faced the same issue. ---

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-08-08 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 This PR lays the foundation for exposing the properties to begin with. My thought was that it reduces one error prone manual step of hand-editing the `pcap.properties` file. I agree with your

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-08-02 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Yup, @mmiklavc I see what you are saying. Make sense to me. Let me know if the latest README update conveys the message. I also added a fix to prompt a service restart upon changes

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-08-01 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Btw, I created [METRON-1719](https://issues.apache.org/jira/browse/METRON-1719) to track PCAP sensor being an independent service entity. ---

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-08-01 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 > Please document the lifecycle for how pcap.properties is updated in the README. It should be clear to users that they will need to restart parsers for the changes to take effect af

[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari

2018-08-01 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 @mmiklavc thanks for the review. I concur with your observations. PCAP topology does deserve its own place in Ambari and the Management UI. As far as the scope of this PR goes

[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...

2018-07-25 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1132#discussion_r205241710 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-pcap-env.xml

[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...

2018-07-25 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1132#discussion_r205237418 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/parser_master.py

[GitHub] metron pull request #1132: METRON-1695: Expose pcap properties through Ambar...

2018-07-25 Thread anandsubbu
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1132 METRON-1695: Expose pcap properties through Ambari ## Contributor Comments This change add a new jinja template for the pcap properties and adds a new tab in Ambari Metron config

[GitHub] metron issue #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1

2018-06-08 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1039 I ran a few performance tests comparing ver 1.1.0 and 1.2.1 on the enrichment topology using the [load_tool](https://github.com/apache/metron/blob/master/metron-contrib/metron-performance

[GitHub] metron issue #526: Metron-846: Add E2E tests for metron management ui

2018-05-14 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/526 I am testing and updating this PR to get it to work on latest master. ---

[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably du...

2018-04-29 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1004#discussion_r184889297 --- Diff: metron-interface/metron-alerts/protractor.conf.js --- @@ -32,12 +33,13 @@ exports.config = { './e2e/alerts-list/tree-view/tree

[GitHub] metron issue #1004: METRON-1489: Retrofit UI tests to run reliably during ni...

2018-04-29 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1004 @nickwallen , in order to get past the following error you are seeing.. ``` [13:44:09] E/launcher - All configured authentication methods failed [13:44:09] E/launcher - Error: All

[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably du...

2018-04-29 Thread anandsubbu
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/1004#discussion_r184888777 --- Diff: metron-interface/metron-alerts/e2e/utils/clean_metron_update_table.ts --- @@ -0,0 +1,52 @@ +declare var Promise: any; +var chalk

[GitHub] metron pull request #981: METRON-1510: Update Metron website to include info...

2018-04-04 Thread anandsubbu
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/981 METRON-1510: Update Metron website to include info about github update subscription ## Contributor Comments Update to include links for the 'Issues' mailing list. Have also changed

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

2018-04-04 Thread anandsubbu
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/943 My +1 still holds. Installed both mpacks on a 12-node cluster and was able to get bro indices to work successfully. Thanks @mmiklavc ! ---