[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636948#comment-16636948 ] ASF GitHub Bot commented on METRON-1798: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1215 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636945#comment-16636945 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Thanks again for the review, @nickwallen ! > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636819#comment-16636819 ] ASF GitHub Bot commented on METRON-1798: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1215 +1 This is really good fix @anandsubbu . Thanks for the easily grokable docs and clean code. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636695#comment-16636695 ] ASF GitHub Bot commented on METRON-1798: GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up full-dev. 2. Stop the "Metron Parsers" service so that existing parser topologies are killed/stopped. 3. Go to Ambari -> Metron -> Configs -> Parsers 4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example) 5. Save changes and restart the Metron Parsers service. 6. Go to Storm UI, and verify that the aggregated topologies viz. "bro__snort" and "yaf" are started. ## Testing Done ### 1. Full Dev * Set the Metron Parsers value as "bro,snort.yaf" and restarted the service. * A single aggregated topology is seen to be started:  * Verified stop 'Metron Parsers' service to check that the aggregated topology is stopped properly * Verified restart 'Metron Parsers' service to check that the aggregated topology is restarted properly ### 2. Multi-node setup * Set the Metron Parsers value as: "bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort * The appropriate aggregated and single topologies are started:  * Changed the parser list as: bro,snort,yaf * Restarted the parsers service to validate that the three parser topologies are started individually.  ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1798 Alternatively you ca
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636692#comment-16636692 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1215 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636694#comment-16636694 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Re-run travis > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636668#comment-16636668 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Hi @nickwallen , thank you for the review. > @nickwallen One thing I notice right off is that we did not add any documentation for the parsers fields in Ambari. Would it make sense to add a brief description to the pop-up text describing how a user can define parsers? Sure, makes sense. I have also added a section to the Parser Chaining README as well with examples. Please have a look. > @nickwallen: When switching parser topologies, some of the original parser topologies can fail to be shut down properly. > I have found that the same issue occurs in master. Since that is the case, we could choose to tackle this as a separate ticket. It is your choice. I created [METRON-1802](https://issues.apache.org/jira/browse/METRON-1802) so this can be fixed outside of this PR. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634671#comment-16634671 ] ASF GitHub Bot commented on METRON-1798: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1215 > @nickwallen: When switching parser topologies, some of the original parser topologies can fail to be shut down properly. @anandsubbu I have found that the same issue occurs in master. Since that is the case, we could choose to tackle this as a separate ticket. It is your choice. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634336#comment-16634336 ] ASF GitHub Bot commented on METRON-1798: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1215 When switching parser topologies, some of the original parser topologies can fail to be shut down properly. Steps to Replicate: 1. Start with default parsers; [bro,snort] 1. Change Metron Parsers setting to use aggregation; ["bro","snort,yaf"] 1. The original "snort" topology is never shutdown. I would expect this to be shutdown. Parsers Running: bro, snort__yaf, snort Parsers Expected: bro, snort__yaf  It seems that the code needs to take into account the original field setting when shutting down the parsers, but then use the new setting when starting the parsers. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634056#comment-16634056 ] ASF GitHub Bot commented on METRON-1798: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1215 @anandsubbu This looks great. I am going to run it up now. One thing I notice right off is that we did not add any documentation for the parsers fields in Ambari. Would it make sense to add a brief description to the pop-up text describing how a user can define parsers? > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632153#comment-16632153 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Please note: I did not find the Parser Aggregation functionality documented under the Parser Chaining README, save a mention under the parser chaining [use case](https://github.com/apache/metron/tree/master/use-cases/parser_chaining#aggregated-parsers-with-parser-chaining). Please let me know if it makes sense to add this to the main parser chaining README, and I will add it. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631985#comment-16631985 ] ASF GitHub Bot commented on METRON-1798: GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up full-dev. 2. Stop the "Metron Parsers" service so that existing parser topologies are killed/stopped. 3. Go to Ambari -> Metron -> Configs -> Parsers 4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example) 5. Save changes and restart the Metron Parsers service. 6. Go to Storm UI, and verify that the aggregated topologies viz. "bro__snort" and "yaf" are started. ## Testing Done ### 1. Full Dev * Set the Metron Parsers value as "bro,snort.yaf" and restarted the service. * A single aggregated topology is seen to be started:  * Verified stop 'Metron Parsers' service to check that the aggregated topology is stopped properly * Verified restart 'Metron Parsers' service to check that the aggregated topology is restarted properly ### 2. Multi-node setup * Set the Metron Parsers value as: "bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort * The appropriate aggregated and single topologies are started:  ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1798 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1215 commit 8aa7c77f9d9
